KFileItem: Improve isSlow to not block when a network mount is unresponsive, make SkipMimeTypeFromContent skip only on slow fs
AbandonedPublic

Authored by meven on Jan 4 2020, 12:20 AM.

Details

Reviewers
ngraham
broulik
dfaure
Group Reviewers
Frameworks
Summary

Alternative to D23198
BUG: 401579

Diff Detail

Repository
R241 KIO
Branch
arcpatch-D26407
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 22208
Build 22226: arc lint + arc unit
There are a very large number of changes, so older changes are hidden. Show Older Changes
broulik added inline comments.Jan 4 2020, 12:07 PM
src/core/kfileitem.cpp
785

when a network mount is unresponsive

If I do a QFileInfo on an unresponsive mount it will still block. You already freeze before you even know it's a network mount.

dfaure requested changes to this revision.Jan 4 2020, 5:22 PM
dfaure added inline comments.
src/core/kfileitem.cpp
49–50

I agree.

789

You're right, the issue on the FreeBSD jail is fstab, not mnttab. Ignore what I said.

I still don't really like the assumption "no mountpoint found for a given path => we're on android" in a comment. I bet there are other corner cases where this can happen. If you really want to find out you're on android, surely there's Q_OS_ANDROID. Or until we find out what those other corner cases are, the code can stay, but the comment should say "for instance" or "maybe", not "can only mean".

This revision now requires changes to proceed.Jan 4 2020, 5:22 PM
meven added inline comments.Jan 4 2020, 5:42 PM
src/core/kfileitem.cpp
785

Good point, indeed the current solution does not work.
We need an alternative to KMountPoint::findByPath to check the path without calling QFileInfo.

789

I did not mean to avoid android, just surfaced the comment in KMountPoint::currentMountPoints regarding its limitation to make it clear why we need an else block here in the first place.

meven updated this revision to Diff 73260.Jan 11 2020, 10:48 AM
meven marked an inline comment as done.

And KMountPoint::List::findByPathDirectly to check mount point without resolving symlinks

meven updated this revision to Diff 73261.Jan 11 2020, 10:49 AM

remove a commo in comment, fix @since version

meven retitled this revision from KFileItem: Improve isSlow to not block when a network mount is unresponsive, make SkipMimeTypeFromContent skip only on network fs to KFileItem: Improve isSlow to not block when a network mount is unresponsive, make SkipMimeTypeFromContent skip only on slow fs.Jan 11 2020, 10:50 AM
meven updated this revision to Diff 73282.Jan 11 2020, 4:05 PM
meven marked 2 inline comments as done.

Move static variables to KFileItemPrivate::getMountPoints

dfaure requested changes to this revision.Jan 12 2020, 10:27 AM
dfaure added inline comments.
src/core/kfileitem.cpp
771

You forgot to update lastMountPointUpdate here.

And to avoid calling currentDateTime() twice (this is a relatively slow method), make sure to put the result into a local var used in both places.

789

OK, but I still read "can only mean" like a possibly incorrect assumption.

I suggest s/can only mean/for instance/

1248

Did you mean || ?

Otherwise this changes the behaviour also on fast local paths (when SkipMimeTypeFromContent is set).

Then again, D19887 (which introduced this if) was apparently about network mounts.

src/core/kmountpoint.h
76

I don't really like the name, it's non-obvious until reading the documentation.
Maybe it should be findByPath(path, KMountPoint::DontResolveSymlinks) ?
(with a 2-args findByPath overload and a KF6 TODO to merge the two)

This revision now requires changes to proceed.Jan 12 2020, 10:27 AM
meven updated this revision to Diff 73524.Jan 14 2020, 2:22 PM
meven marked 4 inline comments as done.

Address review: Make KMountPoint::Ptr KMountPoint::List::findByPath non-blocking, checking recursively intermediate symlinks

meven added a comment.Jan 14 2020, 2:23 PM

I am thinking about adding some api to check status of nfs, sftp or smb mounts by checking their hostname network reachability, to complement isSlow(), and perhaps then running a stat of the mount with a timeout.
This would be kept in a cache with a short duration.
We still need this because we will lack icons for network files and folders as long as we skip mimetype checking solely on fs type.
And this would allow to make this check default and remove m_bSkipMimeTypeFromContent at this point.

src/core/kfileitem.cpp
789

I meant this and this what KMountPoint does, better be precise here.

1248

I meant to change the behavior : this isSlow check would be triggered only when m_bSkipMimeTypeFromContent is set or I will create regression in Dolphin for instance where icons for folders in network mount would disappear.

src/core/kmountpoint.h
76

Nice suggestion !

dfaure requested changes to this revision.Jan 16 2020, 8:45 AM
dfaure added inline comments.
src/core/kfileitem.cpp
789

I see. Sorry, I hadn't realized that there is a Q_OS_ANDROID check in kmountpoint itself.

src/core/kmountpoint.cpp
437

Compared to canonicalPath, this only works if the very last component is a symlink.

If I make /home/dfaure a symlink to /opt/dfaure (because I have more disk space there), then findByPath("/home/dfaure/Documents/foo.odt") used to return the mountpoint for /opt (which was correct) while now it will return the mount point for /home.
isSymlink() will say false because foo.odt isn't a symlink.

I don't see a perfect solution to this. The best we can do IMHO is make things work for local symlinked directories and for remote mounts, but we can't fully avoid slow calls in the case of symlinks-to-remote-mounts.

So I would just call canonicalPath here, once we find out that the orig path isn't a slow mount. Yes it might point to a slow mount, but IMHO that's the user asking for (performance) trouble :)

This revision now requires changes to proceed.Jan 16 2020, 8:45 AM
meven updated this revision to Diff 73687.Jan 16 2020, 10:10 AM
meven marked an inline comment as done.

Check parent dir transitive symlinks, aka manual canonicalPath

meven updated this revision to Diff 73690.Jan 16 2020, 10:22 AM

Fix poping .length() > 0 ordering

dfaure requested changes to this revision.Jan 16 2020, 11:14 PM
dfaure added inline comments.
src/core/kmountpoint.cpp
438

Hmm, what if the symlink *is* the very last component, like your previous iteration tried to handle?

445

I don't want to be a pain, but this is still wrong....

If /home/dfaure is a symlink to /opt/dfaure, and then /opt/dfaure/tmp is a symlink to /tmp, then the canonical path (and therefore the mount point) for /home/dfaure/tmp is in fact /tmp.

But this is going to call findByPath(/opt/dfaure) (the symlink target of the first symlink found in the path), and stop there, assuming that everything at that target is part of the same mountpoint, which isn't necessarily the case.

I guess this should be findByPath(symLinkTarget() + remainder_of_path)

One possible objection is a case like /a/b/c/d/e/f/g where g is a symlink to h (in the same directory), because then both levels of recursion will stat a, b, c, d, e, f. But maybe this is unavoidable. I don't know how clever canonicalPath() implementations are to optimize such things, while still allowing for /a/b/c/d/e to be a symlink to something totally different, like /x/y, where /x itself might be a symlink (!!).
OK so maybe the recursion and redoing the stat's for all levels is correct.

This revision now requires changes to proceed.Jan 16 2020, 11:14 PM
meven updated this revision to Diff 73819.Jan 18 2020, 8:28 AM
meven marked 3 inline comments as done.

Make the recursive canonical work

meven added inline comments.Jan 18 2020, 8:28 AM
src/core/kmountpoint.cpp
438

I made the incorrect assumption, I had already checked it

445

On the contrary, thanks this is again a great review.
Sorry for missing cases.

OK so maybe the recursion and redoing the stat's for all levels is correct.

I think so, that's what canonicalFilePath does anyway.

dfaure added inline comments.Jan 18 2020, 11:31 AM
src/core/kmountpoint.cpp
443

Why reuse and assign, compared to just const QFileInfo fileinfo(parentPath)?

(Same for parentPath -- I prefer C++ over C)

447

Wouldn't .mid() be faster than split+join? Just wondering. I see why you need to split, just wondering about join..

449

Doesn't this reverse the order?

meven updated this revision to Diff 73831.Jan 18 2020, 1:58 PM
meven marked 3 inline comments as done.

Use mid/lastIndexOf instead of split/join, use const assign in loop

meven added inline comments.Jan 18 2020, 1:59 PM
src/core/kmountpoint.cpp
443

I expected reviewers to tell to have declaration outside of loops...

447

I am pretty sure a mid operation would be faster.
It was just that algorithmic-wise it was clearer and faster to put together for me.

meven marked an inline comment as done.Jan 18 2020, 2:08 PM
dfaure added inline comments.Jan 18 2020, 6:00 PM
src/core/kmountpoint.cpp
436

is the word "for" missing in this sentence? (as in "check A for B?")

Even then I fail to parse it...

439

Ctor syntax is simpler than assignment syntax (which is in fact a copy ctor). And mid(0,N) is left(N).

const QFileInfo fileInfo(path.left(cursor));
src/core/kmountpoint.h
62

Really? that's not what all callers might want.

e.g. for "free disk space" calculation we want the real final mountpoint for that path.

If you need something else, it should be a different method, or indeed a flag

65

where's this param? leftover docu?

meven updated this revision to Diff 73836.Jan 18 2020, 6:18 PM
meven marked 3 inline comments as done.

Fix comment issues and improve syntax used

meven planned changes to this revision.Jan 18 2020, 6:24 PM

Really? that's not what all callers might want.
e.g. for \"free disk space\" calculation we want the real final mountpoint for that path.
If you need something else, it should be a different method, or indeed a flag

Good point, I need to take care of this.
Although the other use case "return the final mountpoint" may be blocking.
I am back at square "add a function or enum".

meven updated this revision to Diff 75216.Feb 8 2020, 9:05 AM

Add a enum parameter to findByPath to choose its behavior regarding returning first slow path

meven added a comment.Feb 8 2020, 9:15 AM

We might want to use Solid instead since it is capable of sending signals when mnttab is updated, rather than like here having a mount point list updated every 5 seconds.
With this code we might have issues with potential run condition : when there is a mount within the 5 seconds cache refresh window and this function is called, it will return incorrect data by default mount point for / .
If this filesystem becomes unresponsive right away, this will cause a freeze because the slowness state would be incorrect.

And a great thing I learned more about at Fosdem, there is a new upcoming Linux API io_uring https://lwn.net/Articles/810414/ that will allow us to make asynchronous statx calls with timeout (linux 5.6+), making dealing with unresponsive file system that much easier.

In the meantime this works.

dfaure requested changes to this revision.Feb 8 2020, 9:18 AM

The approach makes sense to me.

src/core/kmountpoint.cpp
420

You should rename this variable, it's no longer the result of canonicalFilePath() (also called "realpath" in glibc).

src/core/kmountpoint.h
45

Missing @since

68

This is BIC, you changed the signature of an existing exported method.

You need to overload it, instead.
i.e. add a method with two args, and a comment like

// TODO KF6 merge with the above method using RealMountPoint as default value

(and a @since flag of course)

This revision now requires changes to proceed.Feb 8 2020, 9:18 AM
cfeck added a subscriber: cfeck.Mar 19 2020, 5:01 PM

Are more changes planned?

meven planned changes to this revision.Mar 24 2020, 4:06 PM
meven added inline comments.May 16 2020, 4:25 PM
src/core/kmountpoint.cpp
444

Need to pass the flag here.