Change Details

We were using libheif... 1.4 or the like, and since then the library has updated quite a bit. Let's go over the new features. Color Profiles ============== Heif and av1f both have icc, restricted icc and nclx. Restricted ICC -------------- Restricted ICC profiles are a thing from jp2: http://wiki.opf-labs.org/display/TR/Handling+of+ICC+profiles https://www.bitsgalore.org/2013/07/01/icc-profiles-and-resolution-jp2-update-2011-d-lib-paper As far as I can gather, lcms supports it: https://sourceforge.net/p/lcms/mailman/lcms-user/thread/AANLkTinrH1DyqPGeYcmwBI3LMxLPaBJJIwKbMsiX-3mg%40mail.gmail.com/#msg25370985 [] **we should hunt down jpg2000 guy and ask him if he knows anything.** NCLX ---- nclx is an ancient quicktime color space description that roughly corresponds to a matrix shaper profile: https://poynton.ca/notes/misc/sde-nclc-vui-nclx.html GIMP converts this to a icc profile, but I need to double check it does elle stone's corrections: https://gitlab.gnome.org/GNOME/gimp/-/blob/master/plug-ins/common/file-heif.c#L528 Elle Stone's quantization corrections: https://ninedegreesbelow.com/photography/well-behaved-profiles-quest.html#hexadecimal-quantization C++ bindings ------------ The c++ bindings haven't been updated for a while and misses the ability to get the color profile. https://github.com/strukturag/libheif/commits/master/libheif/heif_cxx.h [] **We need c++ bindings first as everything in the current plugin uses these.** [] **NCLX will proly need a nclx_to_icc kind of function in the lcms color engine.** Color Spaces ============ Heif supports YCrCb/YUV and Monochrome, and in theory RGB(A). https://github.com/strukturag/libheif/issues/62 Av1f only supports YCrCb/YUV and Monochrome with seperate alpha channels. Libheif takes care of the conversion to yuv if we pass it rgb data, but the conversion algorithm has the usual rounding problems. The problem is that the algorithm is also spec-defined so we can't try to make a better conversion here. The result is that crisp line art will have very strange looking anti-aliasing. [] **Let's figure out what the current state of this is.** [] **We also should try to make sure that end-users are made aware of this limitation as it makes 'lossless' not truly lossless** [] **We probably should support monochrome. Right now we only support 8bit rgb** Bit-depth --------- The library right now supports 8, 10 and 12 bit. Krita and GIMP both only support 8 and 16 bits. GIMP solves this by just scaling up the values for 10 and 12 bit. I am unsure if this is correct, as I cannot tell if 10 and 12 bit heif/av1f images are always understood to be HDR images, especially as most of the time that also requires HDR metadata (maxfall/maxcll). [] **Figure out if 10 and 12 bit libheif images are always understood to be HDR** [] **We really need a maxfall/maxcll function, see T10627 .** Av1f support ============ Av1f support is possible if compiled with libaom. There's also rav1e support for 1.8.0, which is a rust library. The reason this is interesting is because libaom is a slow (unoptimized reference) encoder. Outside of the RGB thing, most features seem to be the exact same between the two formats. We will probably want to implement this support in the same way we have several fileformats use qimageio. Mimetypes can be found here: https://github.com/strukturag/libheif/blob/master/libheif/heif.h#L294 [] **Figure out just how sad sysadmin will be if we try to compile libheif with rav1e.** [] **We don't seem to need extra cpp bindings, as what is necessary is to change the encoder for libheif depending on the mimetype, and there's already bindings for those** Animation ========= libheif has support for multiple images in one file, in two different ways. One is an image sequence, the other is tiled images, which seems to be for panorama style images. The latter can only be read. These images are in a sort of tree structure, with the image sequence images being top level, and the tiled images being sub-images of the top level ones. Right now we only load the primary image. We could in theory load and save animation sequences using libheif. This would be great, because if av1f and heif are going to get the appropriate support on social media, then this would be a much newbie-friendlier way to handle small animation export than our ffmpeg situation. Given that we currently are seeing a couple of refactorings in animation, I would like to delay this until we get the blessing of @emmetoneill and @eoinoneill [] **Wait until refactors are merged before we plan how to tackle this.** [] **Figure out if it is desirable or at all necessary to handle tiled image reading** Depth map ========= heif files support a depth image, which is a grayscale image representing a z-buffer, and is used for adding 3d effects to images. Within image manipulation, the z-buffer is often used as a mask on a blur filter to create a depth-of-field effect. Exr files from blender can have the z-buffer as a render pass. We could in theory support this by loading depth images as grayscale layers, and allowing the user to select a layer as a depth image, which will then get converted to grayscale and removed/hidden on the layerstack before we save the rest of the image. [] **This needs some overview with artists, but it seems like a workflow that makes sense...** Thumbnails ========== We can save thumbnails with libheif. We don't do anything with thumbnail images ourselves, but we do of course have all the features to make thumbnail saving easy. I am currently not sure if libheif just generates a thumbnail for us. [] **Check if libheif generates thumbnail images for us.** Image transformations ================= So, because mobile phones do not want to transform the actual pixel data of an image for performance reasons, some images contain internal image transformations. Libheif supports loading an applying image transformations. I am only bringing this up because of https://bugs.kde.org/show_bug.cgi?id=425832 which is about a similar thing, but then for tiff. [] **Figure out what we want to do with these** Overlay images ============ Libheif supports read overlay images... I have no idea what these are [] **Figure out what these are.**

We were using libheif... 1.4 or the like, and since then the library has updated quite a bit. Let's go over the new features. Color Profiles ============== Heif and av1f both have icc, restricted icc and nclx. Restricted ICC -------------- Restricted ICC profiles are a thing from jp2: http://wiki.opf-labs.org/display/TR/Handling+of+ICC+profiles https://www.bitsgalore.org/2013/07/01/icc-profiles-and-resolution-jp2-update-2011-d-lib-paper As far as I can gather, lcms supports it: https://sourceforge.net/p/lcms/mailman/lcms-user/thread/AANLkTinrH1DyqPGeYcmwBI3LMxLPaBJJIwKbMsiX-3mg%40mail.gmail.com/#msg25370985 [] **we should hunt down jpg2000 guy and ask him if he knows anything.** NCLX ---- nclx is an ancient quicktime color space description that roughly corresponds to a matrix shaper profile: https://poynton.ca/notes/misc/sde-nclc-vui-nclx.html GIMP converts this to a icc profile, but I need to double check it does elle stone's corrections: https://gitlab.gnome.org/GNOME/gimp/-/blob/master/plug-ins/common/file-heif.c#L528 Elle Stone's quantization corrections: https://ninedegreesbelow.com/photography/well-behaved-profiles-quest.html#hexadecimal-quantization C++ bindings ------------ The c++ bindings haven't been updated for a while and misses the ability to get the color profile. https://github.com/strukturag/libheif/commits/master/libheif/heif_cxx.h [] **We need c++ bindings first as everything in the current plugin uses these.** [] **NCLX will proly need a nclx_to_icc kind of function in the lcms color engine.** Color Spaces ============ Heif supports YCrCb/YUV and Monochrome, and in theory RGB(A). https://github.com/strukturag/libheif/issues/62 Av1f only supports YCrCb/YUV and Monochrome with seperate alpha channels. Libheif takes care of the conversion to yuv if we pass it rgb data, but the conversion algorithm has the usual rounding problems. The problem is that the algorithm is also spec-defined so we can't try to make a better conversion here. The result is that crisp line art will have very strange looking anti-aliasing. [] **Let's figure out what the current state of this is.** [] **We also should try to make sure that end-users are made aware of this limitation as it makes 'lossless' not truly lossless** [] **We probably should support monochrome. Right now we only support 8bit rgb** Bit-depth --------- The library right now supports 8, 10 and 12 bit. Krita and GIMP both only support 8 and 16 bits. GIMP solves this by just scaling up the values for 10 and 12 bit. I am unsure if this is correct, as I cannot tell if 10 and 12 bit heif/av1f images are always understood to be HDR images, especially as most of the time that also requires HDR metadata (maxfall/maxcll). [] **Figure out if 10 and 12 bit libheif images are always understood to be HDR** [] **We really need a maxfall/maxcll function, see T10627 .** Av1f support ============ Av1f support is possible if compiled with libaom. There's also rav1e support for 1.8.0, which is a rust library. The reason this is interesting is because libaom is a slow (unoptimized reference) encoder. Outside of the RGB thing, most features seem to be the exact same between the two formats. We will probably want to implement this support in the same way we have several fileformats use qimageio. Mimetypes can be found here: https://github.com/strukturag/libheif/blob/master/libheif/heif.h#L294 [] **Figure out just how sad sysadmin will be if we try to compile libheif with rav1e.** [] **We don't seem to need extra cpp bindings, as what is necessary is to change the encoder for libheif depending on the mimetype, and there's already bindings for those** Animation ========= libheif has support for multiple images in one file, in two different ways. One is an image sequence, the other is tiled images, which seems to be for panorama style images. The latter can only be read. These images are in a sort of tree structure, with the image sequence images being top level, and the tiled images being sub-images of the top level ones. Right now we only load the primary image. We could in theory load and save animation sequences using libheif. This would be great, because if av1f and heif are going to get the appropriate support on social media, then this would be a much newbie-friendlier way to handle small animation export than our ffmpeg situation. Given that we currently are seeing a couple of refactorings in animation, I would like to delay this until we get the blessing of @emmetoneill and @eoinoneill [] **Wait until refactors are merged before we plan how to tackle this.** [] **Also see: https://github.com/strukturag/libheif/issues/57 and https://github.com/strukturag/libheif/issues/129, seems we can load and save sequences but not necessarily timing data** [] **Figure out if it is desirable or at all necessary to handle tiled image reading** Depth map ========= heif files support a depth image, which is a grayscale image representing a z-buffer, and is used for adding 3d effects to images. Within image manipulation, the z-buffer is often used as a mask on a blur filter to create a depth-of-field effect. Exr files from blender can have the z-buffer as a render pass. We could in theory support this by loading depth images as grayscale layers, and allowing the user to select a layer as a depth image, which will then get converted to grayscale and removed/hidden on the layerstack before we save the rest of the image. [] **This needs some overview with artists, but it seems like a workflow that makes sense...** Thumbnails ========== We can save thumbnails with libheif. We don't do anything with thumbnail images ourselves, but we do of course have all the features to make thumbnail saving easy. I am currently not sure if libheif just generates a thumbnail for us. [] **Check if libheif generates thumbnail images for us.** Image transformations ================= So, because mobile phones do not want to transform the actual pixel data of an image for performance reasons, some images contain internal image transformations. Libheif supports loading an applying image transformations. I am only bringing this up because of https://bugs.kde.org/show_bug.cgi?id=425832 which is about a similar thing, but then for tiff. [] **Figure out what we want to do with these** Overlay images ============ Libheif supports read overlay images... I have no idea what these are [] **Figure out what these are.**

We were using libheif... 1.4 or the like, and since then the library has updated quite a bit. Let's go over the new features. Color Profiles ============== Heif and av1f both have icc, restricted icc and nclx. Restricted ICC -------------- Restricted ICC profiles are a thing from jp2: http://wiki.opf-labs.org/display/TR/Handling+of+ICC+profiles https://www.bitsgalore.org/2013/07/01/icc-profiles-and-resolution-jp2-update-2011-d-lib-paper As far as I can gather, lcms supports it: https://sourceforge.net/p/lcms/mailman/lcms-user/thread/AANLkTinrH1DyqPGeYcmwBI3LMxLPaBJJIwKbMsiX-3mg%40mail.gmail.com/#msg25370985 [] **we should hunt down jpg2000 guy and ask him if he knows anything.** NCLX ---- nclx is an ancient quicktime color space description that roughly corresponds to a matrix shaper profile: https://poynton.ca/notes/misc/sde-nclc-vui-nclx.html GIMP converts this to a icc profile, but I need to double check it does elle stone's corrections: https://gitlab.gnome.org/GNOME/gimp/-/blob/master/plug-ins/common/file-heif.c#L528 Elle Stone's quantization corrections: https://ninedegreesbelow.com/photography/well-behaved-profiles-quest.html#hexadecimal-quantization C++ bindings ------------ The c++ bindings haven't been updated for a while and misses the ability to get the color profile. https://github.com/strukturag/libheif/commits/master/libheif/heif_cxx.h [] **We need c++ bindings first as everything in the current plugin uses these.** [] **NCLX will proly need a nclx_to_icc kind of function in the lcms color engine.** Color Spaces ============ Heif supports YCrCb/YUV and Monochrome, and in theory RGB(A). https://github.com/strukturag/libheif/issues/62 Av1f only supports YCrCb/YUV and Monochrome with seperate alpha channels. Libheif takes care of the conversion to yuv if we pass it rgb data, but the conversion algorithm has the usual rounding problems. The problem is that the algorithm is also spec-defined so we can't try to make a better conversion here. The result is that crisp line art will have very strange looking anti-aliasing. [] **Let's figure out what the current state of this is.** [] **We also should try to make sure that end-users are made aware of this limitation as it makes 'lossless' not truly lossless** [] **We probably should support monochrome. Right now we only support 8bit rgb** Bit-depth --------- The library right now supports 8, 10 and 12 bit. Krita and GIMP both only support 8 and 16 bits. GIMP solves this by just scaling up the values for 10 and 12 bit. I am unsure if this is correct, as I cannot tell if 10 and 12 bit heif/av1f images are always understood to be HDR images, especially as most of the time that also requires HDR metadata (maxfall/maxcll). [] **Figure out if 10 and 12 bit libheif images are always understood to be HDR** [] **We really need a maxfall/maxcll function, see T10627 .** Av1f support ============ Av1f support is possible if compiled with libaom. There's also rav1e support for 1.8.0, which is a rust library. The reason this is interesting is because libaom is a slow (unoptimized reference) encoder. Outside of the RGB thing, most features seem to be the exact same between the two formats. We will probably want to implement this support in the same way we have several fileformats use qimageio. Mimetypes can be found here: https://github.com/strukturag/libheif/blob/master/libheif/heif.h#L294 [] **Figure out just how sad sysadmin will be if we try to compile libheif with rav1e.** [] **We don't seem to need extra cpp bindings, as what is necessary is to change the encoder for libheif depending on the mimetype, and there's already bindings for those** Animation ========= libheif has support for multiple images in one file, in two different ways. One is an image sequence, the other is tiled images, which seems to be for panorama style images. The latter can only be read. These images are in a sort of tree structure, with the image sequence images being top level, and the tiled images being sub-images of the top level ones. Right now we only load the primary image. We could in theory load and save animation sequences using libheif. This would be great, because if av1f and heif are going to get the appropriate support on social media, then this would be a much newbie-friendlier way to handle small animation export than our ffmpeg situation. Given that we currently are seeing a couple of refactorings in animation, I would like to delay this until we get the blessing of @emmetoneill and @eoinoneill [] **Wait until refactors are merged before we plan how to tackle this.** [] **Also see: https://github.com/strukturag/libheif/issues/57 and https://github.com/strukturag/libheif/issues/129, seems we can load and save sequences but not necessarily timing data** [] **Figure out if it is desirable or at all necessary to handle tiled image reading** Depth map ========= heif files support a depth image, which is a grayscale image representing a z-buffer, and is used for adding 3d effects to images. Within image manipulation, the z-buffer is often used as a mask on a blur filter to create a depth-of-field effect. Exr files from blender can have the z-buffer as a render pass. We could in theory support this by loading depth images as grayscale layers, and allowing the user to select a layer as a depth image, which will then get converted to grayscale and removed/hidden on the layerstack before we save the rest of the image. [] **This needs some overview with artists, but it seems like a workflow that makes sense...** Thumbnails ========== We can save thumbnails with libheif. We don't do anything with thumbnail images ourselves, but we do of course have all the features to make thumbnail saving easy. I am currently not sure if libheif just generates a thumbnail for us. [] **Check if libheif generates thumbnail images for us.** Image transformations ================= So, because mobile phones do not want to transform the actual pixel data of an image for performance reasons, some images contain internal image transformations. Libheif supports loading an applying image transformations. I am only bringing this up because of https://bugs.kde.org/show_bug.cgi?id=425832 which is about a similar thing, but then for tiff. [] **Figure out what we want to do with these** Overlay images ============ Libheif supports read overlay images... I have no idea what these are [] **Figure out what these are.**