Changeset View
Changeset View
Standalone View
Standalone View
DB/NewImageFinder.cpp
| Show First 20 Lines • Show All 133 Lines • ▼ Show 20 Line(s) | |||||
| 134 | * | 134 | * | ||
| 135 | * DESCRIPTION OF KPHOTOALBUM IMAGE LOAD PROCESS | 135 | * DESCRIPTION OF KPHOTOALBUM IMAGE LOAD PROCESS | ||
| 136 | * ----------- -- ----------- ----- ---- ------- | 136 | * ----------- -- ----------- ----- ---- ------- | ||
| 137 | * | 137 | * | ||
| 138 | * KPhotoAlbum, when it loads an image, performs three processing steps: | 138 | * KPhotoAlbum, when it loads an image, performs three processing steps: | ||
| 139 | * | 139 | * | ||
| 140 | * 1) Compute the MD5 checksum | 140 | * 1) Compute the MD5 checksum | ||
| 141 | * | 141 | * | ||
| 142 | * 2) Extract the EXIF metadata | 142 | * 2) Extract the Exif metadata | ||
| 143 | * | 143 | * | ||
| 144 | * 3) Generate a thumbnail | 144 | * 3) Generate a thumbnail | ||
| 145 | * | 145 | * | ||
| 146 | * Previous to this round of performance tuning, the first two steps | 146 | * Previous to this round of performance tuning, the first two steps | ||
| 147 | * were performed in the first pass, and thumbnails were generated in | 147 | * were performed in the first pass, and thumbnails were generated in | ||
| 148 | * a separate pass. Assuming that the set of new images is large enough | 148 | * a separate pass. Assuming that the set of new images is large enough | ||
| 149 | * that they cannot all fit in RAM buffers, this results in the I/O | 149 | * that they cannot all fit in RAM buffers, this results in the I/O | ||
| 150 | * being performed twice. The rewrite results in I/O being performed once. | 150 | * being performed twice. The rewrite results in I/O being performed once. | ||
| Show All 19 Lines | |||||
| 170 | * whether other candidate filenames are present, nor does it | 170 | * whether other candidate filenames are present, nor does it | ||
| 171 | * compute the MD5 checksum of any such files it does find. Rather, | 171 | * compute the MD5 checksum of any such files it does find. Rather, | ||
| 172 | * it only checks for files that are already in KPhotoAlbum, either | 172 | * it only checks for files that are already in KPhotoAlbum, either | ||
| 173 | * previously or as a result of the current load. Merely checking | 173 | * previously or as a result of the current load. Merely checking | ||
| 174 | * for the presence of another file is not cheap, and it's not | 174 | * for the presence of another file is not cheap, and it's not | ||
| 175 | * necessary; if an image will belong to a stack, we'll either know | 175 | * necessary; if an image will belong to a stack, we'll either know | ||
| 176 | * it now or when other images that can be stacked are loaded. | 176 | * it now or when other images that can be stacked are loaded. | ||
| 177 | * | 177 | * | ||
| 178 | * 4) The EXIF metadata extraction is now done only once; previously | 178 | * 4) The Exif metadata extraction is now done only once; previously | ||
| 179 | * it was performed several times at different stages of the loading | 179 | * it was performed several times at different stages of the loading | ||
| 180 | * process. | 180 | * process. | ||
| 181 | * | 181 | * | ||
| 182 | * 5) The thumbnail index is now written out incrementally rather than | 182 | * 5) The thumbnail index is now written out incrementally rather than | ||
| 183 | * the entire index (which can be many megabytes in a large image | 183 | * the entire index (which can be many megabytes in a large image | ||
| 184 | * database) being rewritten frequently. The index is fully rewritten | 184 | * database) being rewritten frequently. The index is fully rewritten | ||
| 185 | * prior to exit. | 185 | * prior to exit. | ||
| 186 | * | 186 | * | ||
| ▲ Show 20 Lines • Show All 108 Lines • ▼ Show 20 Line(s) | |||||
| 295 | * RAW - 1 scout 18% (12%) | 295 | * RAW - 1 scout 18% (12%) | ||
| 296 | * RAW - 2 scouts 25% (15%) | 296 | * RAW - 2 scouts 25% (15%) | ||
| 297 | * RAW - 3 scouts 25% (15%) | 297 | * RAW - 3 scouts 25% (15%) | ||
| 298 | * | 298 | * | ||
| 299 | * I also used kcachegrind to measure CPU consumption on smaller | 299 | * I also used kcachegrind to measure CPU consumption on smaller | ||
| 300 | * subsets of images (with and without thumbnail creation). In terms | 300 | * subsets of images (with and without thumbnail creation). In terms | ||
| 301 | * of user CPU consumption, thumbnail creation constitutes the large | 301 | * of user CPU consumption, thumbnail creation constitutes the large | ||
| 302 | * majority of CPU cycles for processing JPEG files, followed by MD5 | 302 | * majority of CPU cycles for processing JPEG files, followed by MD5 | ||
| 303 | * computation, with EXIF parsing lagging far behind. For RAW files, | 303 | * computation, with Exif parsing lagging far behind. For RAW files, | ||
| 304 | * MD5 computation consumes more cycles, likely in part due to the | 304 | * MD5 computation consumes more cycles, likely in part due to the | ||
| 305 | * larger size of RAW files but possibly also related to the smaller | 305 | * larger size of RAW files but possibly also related to the smaller | ||
| 306 | * filesize of embedded thumbnails (on the Canon 7D mkII, the embedded | 306 | * filesize of embedded thumbnails (on the Canon 7D mkII, the embedded | ||
| 307 | * thumbnail is full size but low quality). | 307 | * thumbnail is full size but low quality). | ||
| 308 | * | 308 | * | ||
| 309 | * With thumbnail generation: | 309 | * With thumbnail generation: | ||
| 310 | * ---- --------- ----------- | 310 | * ---- --------- ----------- | ||
| 311 | * | 311 | * | ||
| 312 | * RAW JPEG | 312 | * RAW JPEG | ||
| 313 | * | 313 | * | ||
| 314 | * Thumbnail generation 44% 82% | 314 | * Thumbnail generation 44% 82% | ||
| 315 | * libjpeg processing 43% 82% | 315 | * libjpeg processing 43% 82% | ||
| 316 | * MD5 computation 51% 13% | 316 | * MD5 computation 51% 13% | ||
| 317 | * Read EXIF 1% 1.0% | 317 | * Read Exif 1% 1.0% | ||
| 318 | * | 318 | * | ||
| 319 | * Without thumbnail generation: | 319 | * Without thumbnail generation: | ||
| 320 | * ------- --------- ----------- | 320 | * ------- --------- ----------- | ||
| 321 | * | 321 | * | ||
| 322 | * RAW JPEG | 322 | * RAW JPEG | ||
| 323 | * | 323 | * | ||
| 324 | * MD5 computation 92% 80% | 324 | * MD5 computation 92% 80% | ||
| 325 | * Read EXIF 4% 10% | 325 | * Read Exif 4% 10% | ||
| 326 | * | 326 | * | ||
| 327 | * | 327 | * | ||
| 328 | * CONCLUSIONS | 328 | * CONCLUSIONS | ||
| 329 | * ----------- | 329 | * ----------- | ||
| 330 | * | 330 | * | ||
| 331 | * For loading files from hard disk (likely the most common case), | 331 | * For loading files from hard disk (likely the most common case), | ||
| 332 | * there's no reason to consider any loading method other than using a | 332 | * there's no reason to consider any loading method other than using a | ||
| 333 | * single scout thread and computing thumbnails concurrently. Even | 333 | * single scout thread and computing thumbnails concurrently. Even | ||
| ▲ Show 20 Lines • Show All 211 Lines • ▼ Show 20 Line(s) | |||||
| 545 | 545 | | |||
| 546 | ImageInfoPtr NewImageFinder::loadExtraFile( const DB::FileName& newFileName, DB::MediaType type ) | 546 | ImageInfoPtr NewImageFinder::loadExtraFile( const DB::FileName& newFileName, DB::MediaType type ) | ||
| 547 | { | 547 | { | ||
| 548 | MD5 sum = Utilities::MD5Sum( newFileName ); | 548 | MD5 sum = Utilities::MD5Sum( newFileName ); | ||
| 549 | if ( handleIfImageHasBeenMoved(newFileName, sum) ) | 549 | if ( handleIfImageHasBeenMoved(newFileName, sum) ) | ||
| 550 | return DB::ImageInfoPtr(); | 550 | return DB::ImageInfoPtr(); | ||
| 551 | 551 | | |||
| 552 | // check to see if this is a new version of a previous image | 552 | // check to see if this is a new version of a previous image | ||
| 553 | // We'll get the EXIF data later, when we get the MD5 checksum. | 553 | // We'll get the Exif data later, when we get the MD5 checksum. | ||
| 554 | ImageInfoPtr info = ImageInfoPtr(new ImageInfo( newFileName, type, false, false )); | 554 | ImageInfoPtr info = ImageInfoPtr(new ImageInfo( newFileName, type, false, false )); | ||
| 555 | ImageInfoPtr originalInfo; | 555 | ImageInfoPtr originalInfo; | ||
| 556 | DB::FileName originalFileName; | 556 | DB::FileName originalFileName; | ||
| 557 | 557 | | |||
| 558 | if (Settings::SettingsData::instance()->detectModifiedFiles()) { | 558 | if (Settings::SettingsData::instance()->detectModifiedFiles()) { | ||
| 559 | // requires at least *something* in the modifiedFileComponent | 559 | // requires at least *something* in the modifiedFileComponent | ||
| 560 | if (m_modifiedFileCompString.length() >= 0 && | 560 | if (m_modifiedFileCompString.length() >= 0 && | ||
| 561 | newFileName.relative().contains(m_modifiedFileComponent)) { | 561 | newFileName.relative().contains(m_modifiedFileComponent)) { | ||
| ▲ Show 20 Lines • Show All 192 Lines • Show Last 20 Lines | |||||