Refine behaviour for copying modified images to external applications
ClosedPublic

Authored by rkflx on May 21 2018, 9:13 PM.

Details

Summary

In b94861f27a29 copying images for pasting into other applications was
added for cases where only adding the URL to the clipboard was not
enough, e.g. for GIMP. This was done by adding a raster based
representation to the mime data.

However, for modified (e.g. cropped) images, the pasted result would
depend on whether the external application used the original URL
containing the unmodified image, or the updated raster data. This was
confusing for the user, and for example for Dolphin pasting the modified
image was not possible at all.

This patch modifies the mime handling so that pasting the modified image

  • works in Dolphin
  • does not require extra steps in LibreOffice
  • is disabled for applications only supporting handling of the unmodified URL (i.e. users have to save the image first)
Test Plan

Press Ctrl+C in Gwenview and paste to the target for every combination of:

  • Source: JPG, PNG, NEF, SVG
  • Regular and Cropped copy for every source file format
  • Target: Dolphin, GIMP, LibreOffice, Inkscape, Firefox, Kate

Copying and pasting non-image files and multiple images should work as before and produce no warnings.

Diff Detail

Repository
R260 Gwenview
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.
rkflx requested review of this revision.May 21 2018, 9:13 PM
rkflx created this revision.
rkflx added a comment.May 21 2018, 9:14 PM

BTW: For pasting into Firefox I used https://codepen.io/netsi1964/pen/IoJbg.

(For dropping there is https://codepen.io/esausilva/pen/RKVoQw, but this seems to only work with URLs.)

Works good for all images and destination apps I found. :)
There's only one issue with non image selections (see inline comment).

app/fileopscontextmanageritem.cpp
104–105

This fails if the selected item is not an image (e.g. folder, archive) and gives a console warning Can't load full image: loading has already failed.

123

Is this necessary for raster images? setImageData() already creates a couple of mime types in the clipboard.
You can list them with e.g. xclip -o -selection clipboard -t TARGETS and extract any specific to check and compare.

rkflx planned changes to this revision.May 22 2018, 7:40 AM
rkflx added inline comments.
app/fileopscontextmanageritem.cpp
104–105

Thanks for finding ;) Will fix.

123

Hm, the whole point of the patch was to set the correct mimetype, so pasting a cropped image.jpg would result in Dolphin pasting a JPG. Without this, the filename would be image.jpg, but a PNG would be pasted, which cannot be opened subsequently.

However, I just noticed that pasting a cropped NEF will set the correct mimetype (due to my patch), but the pasting itself will fail in the end. Maybe Dolphin can only paste Qt's image formats after all?

We might have to go back to doc->image() only, but I don't know yet how to set the correct filename extension for this. I don't think we can assume PNG in every case? Or perhaps we need to fix Dolphin's paste dialog to automatically set the correct extension?

rkflx edited the test plan for this revision. (Show Details)May 22 2018, 7:41 AM
rkflx edited the test plan for this revision. (Show Details)
@rkflx wrote:

However, I just noticed that pasting a cropped NEF will set the correct mimetype (due to my patch), but the pasting itself will fail in the end. Maybe Dolphin can only paste Qt's image formats after all?

I suspect Gwenview only loads the JPG preview embedded in the raw files but the mimetype is identified by the original file.

I don't think we can assume PNG in every case?

It seems application/x-qt-image is always a PNG image. At least I did not see anything other so far.

muhlenpfordt added a comment.EditedMay 22 2018, 8:51 AM

It seems application/x-qt-image is always a PNG image. At least I did not see anything other so far.

I'm wrong - it's the format that doc->image() returns.
Using mimeData->setImageData(doc->rawData()) works and copies e.g. a JPG image to application/x-qt-image.
Edit: To be clear - 'works' means the JPG data is copied, but unlike setting a QImage the image is not converted and all other mimetypes are empty.

@rkflx wrote:

However, I just noticed that pasting a cropped NEF will set the correct mimetype (due to my patch), but the pasting itself will fail in the end. Maybe Dolphin can only paste Qt's image formats after all?

I suspect Gwenview only loads the JPG preview embedded in the raw files but the mimetype is identified by the original file.

But then renaming to JPG should work, however nothing gets pasted in the first place. This also happens for TGA, which should not have a JPG preview embedded. And indeed, after removing doc->image() Dolphin will not even initiate the paste dialog.

I don't think we can assume PNG in every case?

It seems application/x-qt-image is always a PNG image. At least I did not see anything other so far.

it's the format that doc->image() returns.
Using mimeData->setImageData(doc->rawData()) works and copies e.g. a JPG image to application/x-qt-image.

That's strange, because for me Dolphin (and xclip too) presents a list of multiple different mimetypes, and upon selecting one it will convert the format correctly (cross-checked by examining the pasted file with Okteta).

Maybe we can do something like this:

  • If Qt supports our format, directly set the correct format and filename extension.
  • If not, don't set the format, and set the extension to PNG (assuming it will always be the first item in the list).
  • Possibly extend Dolphin in the future to add the correct extension when changing the format selection.
rkflx updated this revision to Diff 34633.May 22 2018, 11:38 AM
rkflx edited the test plan for this revision. (Show Details)
  • fix warning by only loading proper images
  • fix NEF case by only setting the mimetype if supported
  • force PNG filename extension otherwise
  • even more comments
rkflx marked 4 inline comments as done.May 22 2018, 11:47 AM

Works perfect for the common file formats.
For a couple of image formats (e.g. TGA, BMP, PBM, PNG) doc->rawData() returns an empty array (modified or not). If the file was modified this results in Dolphin not saving for image/x-tga, etc. and you have to select another mimetype to create the file. PNG seems to be a special case and works anyway.
I'm not sure if this is a bug or feature of Document?
Any idea how Dolpin selects the preferred mimetype - maybe we could just mark this without setting the data?

rkflx added a comment.May 23 2018, 9:17 PM

Hm, this patch is getting really complicated…

For a couple of image formats (e.g. TGA, BMP, PBM, PNG) doc->rawData() returns an empty array (modified or not).

Yeah, we only need doc->rawData() for the SVG case, for the rest merely the setData(mimeType, …) part seems important. I don't think there is a way to set a preferred mimetype in KIO/Dolphin yet.

If the file was modified this results in Dolphin not saving for image/x-tga, etc. and you have to select another mimetype to create the file.

Here it gets interesting: I cannot reproduce your claim for BMP, but I can see it for TGA and PBM. I don't think the empty data is the cause, but some X11 clipboard weirdness. The reason is that Qt's and our mimetypes are image/x-tga and image/x-portable-bitmap, while somehow the clipboard announces image/tga and image/pbm, which then leads to failure. You can see this in the combobox, where there is a long description for the former, and the latter only has image/… (relevant code), or by comparing qDebug() << mimeData->formats() and xclip -o -selection clipboard -t TARGETS.

Qt reads the formats from the clipboard here, then KIO tries to read the data here. As the clipboard has image/tga, but we lookup data for image/x-tga, we don't get something useful back.

So far I could not find the place where the other formats get added to the clipboard, so I'm kinda stuck. Probably somewhere after here.

On Wayland only BMP seem to work at all for me, so somehow this is broken too? Or do we need a workaround like this?

Unfortunately more questions than answers at this point…


Anyway, I guess we have two options:

  1. Set the proper image data:
QByteArray bArray;
QBuffer buffer(&bArray);
buffer.open(QIODevice::WriteOnly);
doc->image().save(&buffer, QFileInfo(url.fileName()).suffix().toLatin1());

mimeData->setData(mimeType, bArray);

This means actually performing the conversion to the target format by ourselves, while currently this conversion seems to take place only on demand (no idea where, though?).

Also, this is more of a workaround, as normally setImageData(doc->image()) should be enough.

  1. Only go with PNG for now:
  • Revert setData(mimeType, …).
  • Possibly later add a patch to KIO to determine the preferred format from the suggested filename extension (or set PNG if there is none), and then adapt Gwenview accordingly. TGA would still be broken, of course.

I'm leaning towards option #2, as it's less of a hack which would probably bite us later on Wayland. Thoughts?

Any tips for debugging image/tga further, i.e. where is this coming from, and how does mapping from application/x-qt-image to the various formats work?

rkflx added a comment.EditedMay 24 2018, 7:55 AM

Any tips for debugging image/tga further, i.e. where is this coming from, and how does mapping from application/x-qt-image to the various formats work?

That function looks related…

…and then we end up in here and over there – might be a bug in Qt, after all (unless some specification says otherwise, e.g. the clipboard should not contain XDG mimetypes on Windows?).

I wonder why in imageWriteMimeFormats() Qt uses QImageWriter::supportedImageFormats() and not QImageWriter::supportedMimeTypes(). Seems like in https://bugreports.qt.io/browse/QTBUG-28177 the concept of mimetypes was introduced, but still in a couple of places the custom file extension "mimetypes" are used (e.g. here).

Eventually this decision will break showing the correct description in KIO, unless there is a way to map from extension back to mimetype.

Note that in Qt 5.12 there will be a mapping for the opposite direction (see https://bugreports.qt.io/browse/QTBUG-49714) (still the wrong direction, and for images only, of course).

If the file was modified this results in Dolphin not saving for image/x-tga, etc. and you have to select another mimetype to create the file.

Here it gets interesting: I cannot reproduce your claim for BMP, but I can see it for TGA and PBM.

Right, for BMP the correct mimetype image/bmp is returned. It's confusing since pasting to Dolphin also fails for bigger images sometimes (BMP/TGA 2812x2370 fails but e.g. scaled to 2400x2022 works).

The reason is that Qt's and our mimetypes are image/x-tga and image/x-portable-bitmap, while somehow the clipboard announces image/tga and image/pbm, which then leads to failure.

I think the image/x-... should not be used if no content is attached to it, only the mimetypes provided by Qt.
Just converting e.g. image/x-tga to image/tga and attaching an empty QByteArray via setData() seems to work but it's not a nice solution... ;)

  1. Only go with PNG for now: I'm leaning towards option #2, as it's less of a hack which would probably bite us later on Wayland. Thoughts?

Seems to be the best option for now. With setting the provided filename suffix to .png this should work ok.

Any tips for debugging image/tga further, i.e. where is this coming from, and how does mapping from application/x-qt-image to the various formats work?

It looks like the pasting application (or Qt here) has to provide a callback function which converts the image data on request. If we want to offer additional formats/mimetypes we have to extend this callback by ourself.
It's for Gnome but I think this is the normal X11 clipboard: https://developer.gnome.org/gtk3/stable/gtk3-Clipboards.html

rkflx added a comment.May 24 2018, 8:31 PM

It's for Gnome but I think this is the normal X11 clipboard: https://developer.gnome.org/gtk3/stable/gtk3-Clipboards.html

Thanks, that gave a nice introduction.

In the meantime, I also found where image/tga comes from (see edits above).


Summary on what we've learned:

  • Unsetting the URL for modified images makes sense, as then the clipboard only contains the correct modified raster data.
  • KIO should be modified to infer the default mimetype to use from the filename extension (or provide a method to set it explicitly).
  • The way Qt works results in the clipboard's TARGETS atom announcing "formats", but not "mimetypes".
rkflx updated this revision to Diff 34838.May 24 2018, 8:31 PM
  • Only default to PNG for now.
muhlenpfordt accepted this revision.May 25 2018, 7:10 AM

Looks flawless to me and I can't spot any more issues. :)
I think the behaviour is totally comprehensible and it's not worth fiddling with the original image formats.

This revision is now accepted and ready to land.May 25 2018, 7:10 AM

Looks flawless to me and I can't spot any more issues. :)

Thanks for finding the issues in the previous iterations ;)

I think the behaviour is totally comprehensible and it's not worth fiddling with the original image formats.

You mean KIO should not be adapted to react to the passed filename extension, and users should manually switch from PNG to JPG when they are pasting from a JPG? Or was this about setData()?

This revision was automatically updated to reflect the committed changes.

You mean KIO should not be adapted to react to the passed filename extension, and users should manually switch from PNG to JPG when they are pasting from a JPG?

Sure, if any of the libraries offers a better way to handle this, it will be an improvement but for the current conditions it's a good solution.