Tentative patch to reduce I/O overhead of plasmashell when copying files
AbandonedPublic

Authored by jtamate on Jan 12 2018, 9:44 AM.

Details

Reviewers
dfaure
Group Reviewers
Frameworks
Summary

kservice uses kdirwatch to be notified when one of the directories containing it's source of information changes.
But also the method TimestampChecker().checkTimestamps(allResourceDirs) is called once every 1500 miliseconds.
When copying files from any directory to another one, this check introduces a big use of io/wait.

Test Plan

Before, when copying 20Gb files, the cpu spent in io/wait, as seen by ksysguard, was around 80%.
After, when copying 20Gb files, the cpu spent in io/wait, as seen by ksysguard, is around 50%

Diff Detail

Repository
R309 KService
Lint
Lint Skipped
Unit
Unit Tests Skipped
jtamate created this revision.Jan 12 2018, 9:44 AM
Restricted Application added a project: Frameworks. ยท View Herald TranscriptJan 12 2018, 9:44 AM
jtamate requested review of this revision.Jan 12 2018, 9:44 AM

Does this patch then also affects the speed of the copy operation?

Does this patch then also affects the speed of the copy operation?

Not that I've noticed. Only the cpu in io/wait.

dfaure requested changes to this revision.Jan 14 2018, 12:00 PM

This goes against the whole redesign of ksycoca that I did some time ago. It's supposed to check timestamps on dirs, to rebuild the cache on demand, much like many other caches out there. Did you run the unittests in kservice after this change? I strongly doubt they pass.

I don't understand why a copy triggered by the user (so, I assume, not related to ~/.local) would trigger an unusual effort on the part of ksycoca? How can this be related?
It's just the I/O of the copying happening at the same time as the I/O from ksycoca when using the K menu or something?

This revision now requires changes to proceed.Jan 14 2018, 12:00 PM

Yes, you are right, it doesn't pass the unittests with this tentative patch.

Some local changes should have triggered the heavy load, because now I'm not able to reproduce them.

But there is a problem somewhere (I'll investigate further) because today, everytime I change directory in dolphin (even in /d/....), or change tabs in konsole, or change file to edit in kate or save a file in kate, a messagebox is shown or a notification is shown, there is a check for the timestamps, and this check has to check all the directories in the QMap:

checking file timestamps for QMap(("/etc/xdg/menus", 1515667084956)("/home/jtorres/.config/menus", 1515507849227)("/home/jtorres/.local/share/applications", 1515507849219)("/home/jtorres/.local/share/kservices5", 0)("/home/jtorres/.local/share/kservicetypes5", 0)("/home/jtorres/.local/share/mime", 1512993366020)("/usr/local/share/applications", 1515667254872)("/usr/local/share/kservices5", 0)("/usr/local/share/kservicetypes5", 0)("/usr/local/share/mime", 0)("/usr/share/applications", 1515667274588)("/usr/share/kservices5", 1515667109372)("/usr/share/kservicetypes5", 1515667086640)("/usr/share/mime", 1515667195828))

And the directories watched by KDirWatch m_fileWatcher are:
kf5.kcoreaddons.kdirwatch: Entries watched:
kf5.kcoreaddons.kdirwatch: [ Entry for "/", dir, using INotify inotify_wd=6, has 0 clients , nonexistent subentries: 0x20e45f0 "/.directory" ]
kf5.kcoreaddons.kdirwatch: dependent entries:
kf5.kcoreaddons.kdirwatch: 0x20e45f0 "/.directory" NonExistent
kf5.kcoreaddons.kdirwatch: [ Entry for "/.directory", file, non-existent, using INotify inotify_wd=-1, has 2 clients ]
kf5.kcoreaddons.kdirwatch: by "KDirWatch-3" ( 1 times) ""
kf5.kcoreaddons.kdirwatch: by "KDirWatch-5" ( 1 times) ""
kf5.kcoreaddons.kdirwatch: [ Entry for "/etc/xdg/taskmanagerrulesrc", file, using INotify inotify_wd=9, has 1 clients ]
kf5.kcoreaddons.kdirwatch: by "KDirWatch-8" ( 1 times) ""
kf5.kcoreaddons.kdirwatch: [ Entry for "/home/jtorres/.cache/ksycoca5_es_EE55KafscqW408BVSKZuHAbUvuI=", file, using INotify inotify_wd=1, has 1 clients ]
kf5.kcoreaddons.kdirwatch: by "KDirWatch-1" ( 2 times) ""
kf5.kcoreaddons.kdirwatch: [ Entry for "/home/jtorres/.config", dir, using INotify inotify_wd=8, has 0 clients , nonexistent subentries: 0x43c53f0 "/home/jtorres/.config/taskmanagerrulesrc" 0x446ba20 "/home/jtorres/.config/klaunchrc" ]
kf5.kcoreaddons.kdirwatch: dependent entries:
kf5.kcoreaddons.kdirwatch: 0x43c53f0 "/home/jtorres/.config/taskmanagerrulesrc" NonExistent
kf5.kcoreaddons.kdirwatch: 0x446ba20 "/home/jtorres/.config/klaunchrc" NonExistent
kf5.kcoreaddons.kdirwatch: [ Entry for "/home/jtorres/.config/klaunchrc", file, non-existent, using INotify inotify_wd=-1, has 1 clients ]
kf5.kcoreaddons.kdirwatch: by "KDirWatch-9" ( 1 times) ""
kf5.kcoreaddons.kdirwatch: [ Entry for "/home/jtorres/.config/ksmserverrc", file, using INotify inotify_wd=7, has 1 clients ]
kf5.kcoreaddons.kdirwatch: by "KDirWatch-7" ( 1 times) ""
kf5.kcoreaddons.kdirwatch: [ Entry for "/home/jtorres/.config/plasma-org.kde.plasma.desktop-appletsrc", file, using INotify inotify_wd=5, has 1 clients ]
kf5.kcoreaddons.kdirwatch: by "KDirWatch-2" ( 1 times) ""
kf5.kcoreaddons.kdirwatch: [ Entry for "/home/jtorres/.config/plasmarc", file, using INotify inotify_wd=2, has 1 clients ]
kf5.kcoreaddons.kdirwatch: by "KDirWatch-2" ( 60 times) ""
kf5.kcoreaddons.kdirwatch: [ Entry for "/home/jtorres/.config/taskmanagerrulesrc", file, non-existent, using INotify inotify_wd=-1, has 1 clients ]
kf5.kcoreaddons.kdirwatch: by "KDirWatch-8" ( 1 times) ""
kf5.kcoreaddons.kdirwatch: [ Entry for "/usr/share/icons/breeze/index.theme", file, using INotify inotify_wd=4, has 1 clients ]
kf5.kcoreaddons.kdirwatch: by "KDirWatch-2" ( 1 times) ""
kf5.kcoreaddons.kdirwatch: [ Entry for "/usr/share/plasma/desktoptheme/default/metadata.desktop", file, using INotify inotify_wd=3, has 1 clients ]
kf5.kcoreaddons.kdirwatch: by "KDirWatch-2" ( 1 times) ""
kf5.kcoreaddons.kdirwatch: [ Entry for "/usr/share/solid/actions", dir, using INotify inotify_wd=10, has 1 clients ]
kf5.kcoreaddons.kdirwatch: by "KDirWatch-10" ( 1 times) ""

jtamate abandoned this revision.Jan 15 2018, 10:12 AM

Just to let you know that I know how to reproduce the high I/O of plasmashell and continuous calls to Kservice.
You can reproduce it just having opened the vivaldi web browser. (I never though it could be a problem).
If it goes to a webpage that still uses flash, the effect is even bigger.

.

Just to let you know that I know how to reproduce the high I/O of plasmashell and continuous calls to Kservice.
You can reproduce it just having opened the vivaldi web browser. (I never though it could be a problem).
If it goes to a webpage that still uses flash, the effect is even bigger.

.

Do any of the active or recently-landed patches address this?

! In D9840#203803, @ngraham wrote:
Do any of the active or recently-landed patches address this?

No, but I have one in https://phabricator.kde.org/D10342

Can you strace vivaldi to see what it's doing? It must be doing something wrong if it's triggering so much I/O.

Can you strace vivaldi to see what it's doing? It must be doing something wrong if it's triggering so much I/O.

This is related to https://phabricator.kde.org/D10342

I've straced vivaldi and main process is continuously doing futex and gettid, and the active child the same + write + sendto. Every tab is a process, just like chromium.

mwolff added a subscriber: mwolff.Feb 14 2018, 9:37 AM

I guess this can be abandoned now that the real culprit was found, no?