Changeset View
Changeset View
Standalone View
Standalone View
DB/NewImageFinder.cpp
Show First 20 Lines • Show All 133 Lines • ▼ Show 20 Line(s) | |||||
134 | * | 134 | * | ||
135 | * DESCRIPTION OF KPHOTOALBUM IMAGE LOAD PROCESS | 135 | * DESCRIPTION OF KPHOTOALBUM IMAGE LOAD PROCESS | ||
136 | * ----------- -- ----------- ----- ---- ------- | 136 | * ----------- -- ----------- ----- ---- ------- | ||
137 | * | 137 | * | ||
138 | * KPhotoAlbum, when it loads an image, performs three processing steps: | 138 | * KPhotoAlbum, when it loads an image, performs three processing steps: | ||
139 | * | 139 | * | ||
140 | * 1) Compute the MD5 checksum | 140 | * 1) Compute the MD5 checksum | ||
141 | * | 141 | * | ||
142 | * 2) Extract the EXIF metadata | 142 | * 2) Extract the Exif metadata | ||
143 | * | 143 | * | ||
144 | * 3) Generate a thumbnail | 144 | * 3) Generate a thumbnail | ||
145 | * | 145 | * | ||
146 | * Previous to this round of performance tuning, the first two steps | 146 | * Previous to this round of performance tuning, the first two steps | ||
147 | * were performed in the first pass, and thumbnails were generated in | 147 | * were performed in the first pass, and thumbnails were generated in | ||
148 | * a separate pass. Assuming that the set of new images is large enough | 148 | * a separate pass. Assuming that the set of new images is large enough | ||
149 | * that they cannot all fit in RAM buffers, this results in the I/O | 149 | * that they cannot all fit in RAM buffers, this results in the I/O | ||
150 | * being performed twice. The rewrite results in I/O being performed once. | 150 | * being performed twice. The rewrite results in I/O being performed once. | ||
Show All 19 Lines | |||||
170 | * whether other candidate filenames are present, nor does it | 170 | * whether other candidate filenames are present, nor does it | ||
171 | * compute the MD5 checksum of any such files it does find. Rather, | 171 | * compute the MD5 checksum of any such files it does find. Rather, | ||
172 | * it only checks for files that are already in KPhotoAlbum, either | 172 | * it only checks for files that are already in KPhotoAlbum, either | ||
173 | * previously or as a result of the current load. Merely checking | 173 | * previously or as a result of the current load. Merely checking | ||
174 | * for the presence of another file is not cheap, and it's not | 174 | * for the presence of another file is not cheap, and it's not | ||
175 | * necessary; if an image will belong to a stack, we'll either know | 175 | * necessary; if an image will belong to a stack, we'll either know | ||
176 | * it now or when other images that can be stacked are loaded. | 176 | * it now or when other images that can be stacked are loaded. | ||
177 | * | 177 | * | ||
178 | * 4) The EXIF metadata extraction is now done only once; previously | 178 | * 4) The Exif metadata extraction is now done only once; previously | ||
179 | * it was performed several times at different stages of the loading | 179 | * it was performed several times at different stages of the loading | ||
180 | * process. | 180 | * process. | ||
181 | * | 181 | * | ||
182 | * 5) The thumbnail index is now written out incrementally rather than | 182 | * 5) The thumbnail index is now written out incrementally rather than | ||
183 | * the entire index (which can be many megabytes in a large image | 183 | * the entire index (which can be many megabytes in a large image | ||
184 | * database) being rewritten frequently. The index is fully rewritten | 184 | * database) being rewritten frequently. The index is fully rewritten | ||
185 | * prior to exit. | 185 | * prior to exit. | ||
186 | * | 186 | * | ||
▲ Show 20 Lines • Show All 108 Lines • ▼ Show 20 Line(s) | |||||
295 | * RAW - 1 scout 18% (12%) | 295 | * RAW - 1 scout 18% (12%) | ||
296 | * RAW - 2 scouts 25% (15%) | 296 | * RAW - 2 scouts 25% (15%) | ||
297 | * RAW - 3 scouts 25% (15%) | 297 | * RAW - 3 scouts 25% (15%) | ||
298 | * | 298 | * | ||
299 | * I also used kcachegrind to measure CPU consumption on smaller | 299 | * I also used kcachegrind to measure CPU consumption on smaller | ||
300 | * subsets of images (with and without thumbnail creation). In terms | 300 | * subsets of images (with and without thumbnail creation). In terms | ||
301 | * of user CPU consumption, thumbnail creation constitutes the large | 301 | * of user CPU consumption, thumbnail creation constitutes the large | ||
302 | * majority of CPU cycles for processing JPEG files, followed by MD5 | 302 | * majority of CPU cycles for processing JPEG files, followed by MD5 | ||
303 | * computation, with EXIF parsing lagging far behind. For RAW files, | 303 | * computation, with Exif parsing lagging far behind. For RAW files, | ||
304 | * MD5 computation consumes more cycles, likely in part due to the | 304 | * MD5 computation consumes more cycles, likely in part due to the | ||
305 | * larger size of RAW files but possibly also related to the smaller | 305 | * larger size of RAW files but possibly also related to the smaller | ||
306 | * filesize of embedded thumbnails (on the Canon 7D mkII, the embedded | 306 | * filesize of embedded thumbnails (on the Canon 7D mkII, the embedded | ||
307 | * thumbnail is full size but low quality). | 307 | * thumbnail is full size but low quality). | ||
308 | * | 308 | * | ||
309 | * With thumbnail generation: | 309 | * With thumbnail generation: | ||
310 | * ---- --------- ----------- | 310 | * ---- --------- ----------- | ||
311 | * | 311 | * | ||
312 | * RAW JPEG | 312 | * RAW JPEG | ||
313 | * | 313 | * | ||
314 | * Thumbnail generation 44% 82% | 314 | * Thumbnail generation 44% 82% | ||
315 | * libjpeg processing 43% 82% | 315 | * libjpeg processing 43% 82% | ||
316 | * MD5 computation 51% 13% | 316 | * MD5 computation 51% 13% | ||
317 | * Read EXIF 1% 1.0% | 317 | * Read Exif 1% 1.0% | ||
318 | * | 318 | * | ||
319 | * Without thumbnail generation: | 319 | * Without thumbnail generation: | ||
320 | * ------- --------- ----------- | 320 | * ------- --------- ----------- | ||
321 | * | 321 | * | ||
322 | * RAW JPEG | 322 | * RAW JPEG | ||
323 | * | 323 | * | ||
324 | * MD5 computation 92% 80% | 324 | * MD5 computation 92% 80% | ||
325 | * Read EXIF 4% 10% | 325 | * Read Exif 4% 10% | ||
326 | * | 326 | * | ||
327 | * | 327 | * | ||
328 | * CONCLUSIONS | 328 | * CONCLUSIONS | ||
329 | * ----------- | 329 | * ----------- | ||
330 | * | 330 | * | ||
331 | * For loading files from hard disk (likely the most common case), | 331 | * For loading files from hard disk (likely the most common case), | ||
332 | * there's no reason to consider any loading method other than using a | 332 | * there's no reason to consider any loading method other than using a | ||
333 | * single scout thread and computing thumbnails concurrently. Even | 333 | * single scout thread and computing thumbnails concurrently. Even | ||
▲ Show 20 Lines • Show All 194 Lines • ▼ Show 20 Line(s) | |||||
528 | 528 | | |||
529 | void NewImageFinder::loadExtraFile( const DB::FileName& newFileName, DB::MediaType type ) | 529 | void NewImageFinder::loadExtraFile( const DB::FileName& newFileName, DB::MediaType type ) | ||
530 | { | 530 | { | ||
531 | MD5 sum = Utilities::MD5Sum( newFileName ); | 531 | MD5 sum = Utilities::MD5Sum( newFileName ); | ||
532 | if ( handleIfImageHasBeenMoved(newFileName, sum) ) | 532 | if ( handleIfImageHasBeenMoved(newFileName, sum) ) | ||
533 | return; | 533 | return; | ||
534 | 534 | | |||
535 | // check to see if this is a new version of a previous image | 535 | // check to see if this is a new version of a previous image | ||
536 | // We'll get the EXIF data later, when we get the MD5 checksum. | 536 | // We'll get the Exif data later, when we get the MD5 checksum. | ||
537 | ImageInfoPtr info = ImageInfoPtr(new ImageInfo( newFileName, type, false, false )); | 537 | ImageInfoPtr info = ImageInfoPtr(new ImageInfo( newFileName, type, false, false )); | ||
538 | ImageInfoPtr originalInfo; | 538 | ImageInfoPtr originalInfo; | ||
539 | DB::FileName originalFileName; | 539 | DB::FileName originalFileName; | ||
540 | 540 | | |||
541 | if (Settings::SettingsData::instance()->detectModifiedFiles()) { | 541 | if (Settings::SettingsData::instance()->detectModifiedFiles()) { | ||
542 | // requires at least *something* in the modifiedFileComponent | 542 | // requires at least *something* in the modifiedFileComponent | ||
543 | if (m_modifiedFileCompString.length() >= 0 && | 543 | if (m_modifiedFileCompString.length() >= 0 && | ||
544 | newFileName.relative().contains(m_modifiedFileComponent)) { | 544 | newFileName.relative().contains(m_modifiedFileComponent)) { | ||
▲ Show 20 Lines • Show All 196 Lines • Show Last 20 Lines |