poboiko (Igor Poboiko)
User

Projects

User does not belong to any projects.

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Wednesday

  • Clear sailing ahead.

User Details

User Since
Feb 14 2017, 10:36 AM (113 w, 5 d)
Availability
Available

Recent Activity

Sun, Apr 7

poboiko added a comment to T9805: Overhaul Baloo database scheme.

I didn't realize LMDB does not modify pages; and if we change one, it creates a new one, copies the data from the old page, modifies it and marks old one as "dirty".
In that case we'll most likely end up modifying a single page in both cases.

Sun, Apr 7, 6:30 PM · Baloo

Fri, Apr 5

poboiko added a comment to T9805: Overhaul Baloo database scheme.

Just because you hide the RMW, it does not mean it does not happen. For LMDB, duplicate keys are just a plain array, see e.g. MDB_APPENDDUP in http://www.lmdb.tech/doc/group__mdb.html#ga4fa8573d9236d54687c61827ebf8cac0.

The data entries are sorted by LMDB as soon as you do a mdb_put, and this is of course a RMW cycle.

I thought the structure behind MDB_DUPSORT is a bit more clever that just a plain array, i.e. it's still a search tree.
And insertion still should cost much less compared to what is done now (fetch whole list + insert + put it back).
Of course it still might do some RMW work (tree rebalancing, for example), my point is that it should be more efficient.

Fri, Apr 5, 4:47 PM · Baloo

Thu, Apr 4

poboiko added a comment to T9805: Overhaul Baloo database scheme.

I'd like to revive this discussion.

Thu, Apr 4, 10:04 AM · Baloo

Mar 20 2019

poboiko committed R293:e58804bf9eb8: React to config updates inside indexer (authored by poboiko).
React to config updates inside indexer
Mar 20 2019, 9:02 AM
poboiko closed D15983: React to config updates inside indexer.
Mar 20 2019, 9:01 AM · Baloo, Frameworks

Mar 19 2019

poboiko added a comment to D15983: React to config updates inside indexer.

Can you rebase this on master again? Sorry for the radio silence. :(

Mar 19 2019, 10:58 AM · Baloo, Frameworks
poboiko added a comment to D15983: React to config updates inside indexer.

Ping?

Mar 19 2019, 8:47 AM · Baloo, Frameworks
poboiko added reviewers for D15983: React to config updates inside indexer: bruns, ngraham.
Mar 19 2019, 8:47 AM · Baloo, Frameworks

Mar 17 2019

poboiko accepted D17162: Harmonize handling of underscore in query parser.
Mar 17 2019, 4:34 PM · Baloo, Frameworks

Feb 23 2019

poboiko added a comment to D18664: Baloo engine: treat every non-success code as a failure.

I've looked through the patch (quite large indeed), apart from the single note I think it's good to go.

Feb 23 2019, 2:46 PM · Baloo, Frameworks

Feb 15 2019

poboiko committed R293:5e1add922ab2: [baloo/KInotify] Notify if folder was moved from unwatched place (authored by poboiko).
[baloo/KInotify] Notify if folder was moved from unwatched place
Feb 15 2019, 4:04 PM
poboiko closed D18698: [baloo/KInotify] Notify if folder was moved from unwatched place.
Feb 15 2019, 3:42 PM · Baloo, Frameworks
poboiko updated the diff for D18698: [baloo/KInotify] Notify if folder was moved from unwatched place.

Forgot to define fname inside EventMoveTo

Feb 15 2019, 3:41 PM · Baloo, Frameworks
poboiko updated the diff for D18698: [baloo/KInotify] Notify if folder was moved from unwatched place.

Updated comment, removed duplicated QFile::decodeName

Feb 15 2019, 11:52 AM · Baloo, Frameworks

Feb 12 2019

poboiko added inline comments to D18698: [baloo/KInotify] Notify if folder was moved from unwatched place.
Feb 12 2019, 3:58 PM · Baloo, Frameworks
poboiko added inline comments to D18698: [baloo/KInotify] Notify if folder was moved from unwatched place.
Feb 12 2019, 11:12 AM · Baloo, Frameworks

Feb 6 2019

poboiko added a comment to D18698: [baloo/KInotify] Notify if folder was moved from unwatched place.

Something like that? I've decided not to emit created signal from inside the function, just to have a bit less branching in the code (and documented this behavior, since it might be a bit confusing)

Feb 6 2019, 10:33 PM · Baloo, Frameworks
poboiko updated the diff for D18698: [baloo/KInotify] Notify if folder was moved from unwatched place.

Added recursive iteration over all contents for Create event as well

Feb 6 2019, 10:24 PM · Baloo, Frameworks

Feb 5 2019

poboiko updated the diff for D18698: [baloo/KInotify] Notify if folder was moved from unwatched place.

Cosmetics

Feb 5 2019, 4:59 PM · Baloo, Frameworks
poboiko updated the diff for D18698: [baloo/KInotify] Notify if folder was moved from unwatched place.

Added code to work with first entry that pops from FilteredDirIterator (that is the directory itself)
Test still works; but we should emit created() signal for it as well.

Feb 5 2019, 11:25 AM · Baloo, Frameworks
poboiko added a comment to D18698: [baloo/KInotify] Notify if folder was moved from unwatched place.

I am not sure if I understood your description correctly, but I am quite sure this race condition does not exist - the files/folders inside the moved folder are not created/moved one by one, but the containing folder ist just "renamed" - it is unlinked from the old parent and linked into the new one, atomically.

Sure. That concern corresponded only to the second note - if moving from another device, system has to do actual copy/move.

Feb 5 2019, 9:36 AM · Baloo, Frameworks

Feb 4 2019

poboiko updated the diff for D18698: [baloo/KInotify] Notify if folder was moved from unwatched place.

Explained the race condition in summary, expanded test to check if watches were installed correctly.

Feb 4 2019, 11:43 AM · Baloo, Frameworks

Feb 3 2019

poboiko requested review of D18698: [baloo/KInotify] Notify if folder was moved from unwatched place.
Feb 3 2019, 3:44 PM · Baloo, Frameworks
poboiko abandoned D15637: Make DBusMenu work correctly with dynamically generated menus.
Feb 3 2019, 1:20 PM · Plasma
poboiko closed D18688: Check Exiv2::ValueType::typeId before converting it to rational.
Feb 3 2019, 9:10 AM · Baloo, Frameworks
poboiko committed R286:6e449d44bb5d: Check Exiv2::ValueType::typeId before converting it to rational (authored by poboiko).
Check Exiv2::ValueType::typeId before converting it to rational
Feb 3 2019, 9:10 AM

Feb 2 2019

poboiko added a comment to D18664: Baloo engine: treat every non-success code as a failure.

Nice! I like it, it's definitely much better than Q_ASSERT_X macros that are just silently ignored in non-debug builds.

Feb 2 2019, 10:21 PM · Baloo, Frameworks
poboiko requested review of D18688: Check Exiv2::ValueType::typeId before converting it to rational.
Feb 2 2019, 9:48 PM · Baloo, Frameworks

Dec 18 2018

poboiko added a comment to D15960: Don't check if file is directory based on mime-type.

@poboiko You broke the build, please fix it.

Dec 18 2018, 8:44 PM · Baloo, Frameworks
poboiko committed R293:c7416a41ddef: Fix mistakes introduced in a632a72a (authored by poboiko).
Fix mistakes introduced in a632a72a
Dec 18 2018, 8:42 PM
poboiko committed R293:a632a72a354e: Don't check if file is directory based on mime-type (authored by poboiko).
Don't check if file is directory based on mime-type
Dec 18 2018, 2:11 PM
poboiko added a comment to D15960: Don't check if file is directory based on mime-type.

Sorry, fell through the cracks - give a ping next time something is blocked for no apparent reason ...

Dec 18 2018, 2:07 PM · Baloo, Frameworks
poboiko closed D15960: Don't check if file is directory based on mime-type.
Dec 18 2018, 2:06 PM · Baloo, Frameworks

Nov 21 2018

poboiko added a comment to D16878: Resolve symlinks in exclude folders.

I believe can do something better here.
I think if we stick to canonical paths everywhere, and resolve symlinks ASAP (but still follow them), that might solve all the problems.

Nov 21 2018, 3:28 PM · Baloo, Frameworks

Nov 15 2018

poboiko added a comment to D16878: Resolve symlinks in exclude folders.

IMHO we should just disallow specifying symlinks in both include/excludeFolders. The user can just use exludeFolders = /storage/stuff if he wants to exclude it.

Nov 15 2018, 9:40 AM · Baloo, Frameworks
poboiko added a comment to D16876: [balooctl] Add possibility to create a copy of the index without freelist.

You are replicating mdb_copy -c here.
[...]

Nov 15 2018, 9:30 AM · Baloo, Frameworks
poboiko added a comment to D16498: [KFileMetaData] Add extractor for DSC conforming (Encapsulated) Postscript.

It seems like you've pushed something that was not intended to be pushed (XML extractor parts)

Nov 15 2018, 9:23 AM · Baloo, Frameworks

Nov 14 2018

poboiko accepted D16498: [KFileMetaData] Add extractor for DSC conforming (Encapsulated) Postscript.

Apart from trivial comment, this looks fine. I've tested it on my setup (with bunch of (e)ps files), and randomly chosen files seems to be indexed nicely. It also reduced the size of the index by almost 50MB, because those are not indexed as plaintext anymore :)
Yet I would also vote for replacing it (eventually) with a full-featured extractor based on libspectre
(I'm not a security specialist in any way, but that CVE doesn't look too harmful, and from my point of view it's not worth to abandon full support of (E)PS because of it)

Nov 14 2018, 3:50 PM · Baloo, Frameworks
poboiko requested review of D16878: Resolve symlinks in exclude folders.
Nov 14 2018, 3:14 PM · Baloo, Frameworks
poboiko requested review of D16876: [balooctl] Add possibility to create a copy of the index without freelist.
Nov 14 2018, 2:10 PM · Baloo, Frameworks
poboiko updated the diff for D15983: React to config updates inside indexer.

It's a bad idea to removeRecursively starting from root of the tree (documentid 0).
If user has indexed /home/username folder, there is also an index entry for /home (that's how IdTreeDB works).
However, /home should not be indexed, according to checks (because it's not in includeFolders, while /home/username is)
This will lead to removeRecursively("/home") call, which will wipe index for /home/username as well.

Nov 14 2018, 1:47 PM · Baloo, Frameworks
poboiko updated the diff for D15983: React to config updates inside indexer.

Rebase on master

Nov 14 2018, 1:11 PM · Baloo, Frameworks

Oct 30 2018

poboiko accepted D15826: [Balooshow] Avoid out-of-bounds access when accessing corrupt db data.

Yep, fine by me

Oct 30 2018, 12:26 PM · Baloo, Frameworks
poboiko added a comment to D16523: [Extractor] Replace homegrown IO handler with QDataStream, catch HUP.

That's nice! I'll test it a little.

Oct 30 2018, 8:13 AM · Baloo, Frameworks
poboiko added a comment to T9595: [KAddressbook] Use KPeople model for contact list.

Sorry, I've postponed this one for some time :(

Oct 30 2018, 7:31 AM · KDE PIM
poboiko updated the summary of D15960: Don't check if file is directory based on mime-type.
Oct 30 2018, 7:29 AM · Baloo, Frameworks

Oct 20 2018

poboiko added a comment to T9805: Overhaul Baloo database scheme.

BTW. Is there any reason why i.e. inside PostingDB Baloo stores key -> encoded list of values, instead of using MDB_DUPSORT | MDB_DUPFIXED (which allow to store multiple values for keys)?

Oct 20 2018, 7:57 PM · Baloo
poboiko abandoned D15959: Wait for the extraction process to finish before scheduling.

Dropped in favor of D16265: [Scheduler] Use flag to track when a runner is going idle, which handles this problem better.

Oct 20 2018, 1:08 PM · Baloo, Frameworks
poboiko added a comment to D16265: [Scheduler] Use flag to track when a runner is going idle.

I like it, it's better than D15959: Wait for the extraction process to finish before scheduling.
And it seems to be working, as far as I can see :)

Oct 20 2018, 1:07 PM · Baloo, Frameworks
poboiko added inline comments to D16266: [Extractor] Make extractor crash resilient.
Oct 20 2018, 9:16 AM · Baloo, Frameworks

Oct 17 2018

poboiko added inline comments to D16266: [Extractor] Make extractor crash resilient.
Oct 17 2018, 1:16 PM · Baloo, Frameworks
poboiko added inline comments to D16266: [Extractor] Make extractor crash resilient.
Oct 17 2018, 10:01 AM · Baloo, Frameworks

Oct 16 2018

poboiko added a comment to T9873: Investigate memory issues in Baloo.

Here is some test data for my (rather simple) setup.

Oct 16 2018, 6:54 PM · Baloo
poboiko committed R286:5eee9ac75b7d: Don't crash on invalid exiv2 data (authored by poboiko).
Don't crash on invalid exiv2 data
Oct 16 2018, 1:56 PM
poboiko closed D16165: Don't crash on invalid exiv2 data.
Oct 16 2018, 1:56 PM · Baloo, Frameworks

Oct 15 2018

poboiko created T9873: Investigate memory issues in Baloo.
Oct 15 2018, 4:44 PM · Baloo

Oct 14 2018

poboiko added a comment to D16165: Don't crash on invalid exiv2 data.

Unfortunately 375131 is quite vague, they all speak about "baloo" hanging, but as "baloo" is not a single process, it is impossible to tell for sure which process crashes or hangs.

Apparent hangs of baloo_file are easy to produce, create a file 'baloo_file_extractor' with the following content:

#! /bin/bash
echo "S foobar"

make it executable, and then (after killing the current baloo_file), run PATH=./<path_with_fake_extractor> <path_to>/baloo_file. When you run baloo_monitor, you will see it says "Indexing foobar", but will make no further progress as the fake extractor has exited (equivalent to a crash here).

When you run baloo_file from build/bin, make sure you have the relevant baloo_file_extractor in the PATH first.

Oct 14 2018, 4:59 PM · Baloo, Frameworks

Oct 13 2018

poboiko added a comment to D16165: Don't crash on invalid exiv2 data.

These three should be CCBUG: (not test file, not able to confirm fixed), while 375131 should be BUG: (confirmed and fixed).

Oct 13 2018, 5:52 PM · Baloo, Frameworks
poboiko added a comment to D16165: Don't crash on invalid exiv2 data.

I guess bugs
352856
353848
361259
will also be fixed by this?

Oct 13 2018, 8:56 AM · Baloo, Frameworks
poboiko updated the diff for D16165: Don't crash on invalid exiv2 data.

Replaced size() by count(), which is more appropriate here

Oct 13 2018, 8:54 AM · Baloo, Frameworks
poboiko updated the summary of D16165: Don't crash on invalid exiv2 data.
Oct 13 2018, 8:51 AM · Baloo, Frameworks

Oct 12 2018

poboiko requested review of D16165: Don't crash on invalid exiv2 data.
Oct 12 2018, 7:12 PM · Baloo, Frameworks
poboiko added a comment to D15985: [balooctl] Print current state & indexing file when monitor starts.

That's weird. I caught it only once and thought that it is because I trivially forgot the endl. But now I cannot reproduce it anymore.
Seems like it should be fine, since we do emit finished signal, which will write the 'OK' part.
Sorry for the noise.

Oct 12 2018, 2:46 PM
poboiko added a comment to D15985: [balooctl] Print current state & indexing file when monitor starts.

No, definitely not, see D15995.

Oct 12 2018, 1:13 PM
poboiko added a comment to D15985: [balooctl] Print current state & indexing file when monitor starts.

Woops! startedIndexingFile does not print a newline. I guess I can just add m_out << endl...

Oct 12 2018, 8:10 AM
poboiko updated the diff for D15983: React to config updates inside indexer.

Address raised issues

Oct 12 2018, 8:05 AM · Baloo, Frameworks
poboiko added a comment to D15826: [Balooshow] Avoid out-of-bounds access when accessing corrupt db data.

I'm not entirely sure how translating system works, but won't it pop up as 4 identical lines in i.e. Lokalize, causing frustration to our translators?
It would also mean that if we would want to change the message, we would have to do it in 4 different lines.

Oct 12 2018, 7:55 AM · Baloo, Frameworks
poboiko added a comment to D15939: Perform checks for unindexed files and stale index entries on startup.

https://phabricator.kde.org/D11529 was already up for review, implemented the index cleaner and checked for removeable volumes before removing index entries. Exporting the storagedevices object was required: https://phabricator.kde.org/D15047. The alternative, implementing a path lookup function is here: https://phabricator.kde.org/D15843.

Oct 12 2018, 7:40 AM · Baloo, Frameworks
poboiko added a comment to D15959: Wait for the extraction process to finish before scheduling.

This works for me:

 void FileIndexScheduler::scheduleIndexing()
 {
-    if (m_threadPool.activeThreadCount() || m_indexerState == Suspended) {
+    if (m_indexerState == Suspended) {
         return;
     }
Oct 12 2018, 7:34 AM · Baloo, Frameworks

Oct 9 2018

poboiko added a comment to D15939: Perform checks for unindexed files and stale index entries on startup.

Does this run at startup? If so, this will erase the entries of files on a removable volume not already mounted.

Whoa, thanks for the notice. Not cool, forgot about it.
Probably need to add some checks in IndexCleaner, about the device.

Oct 9 2018, 6:33 AM · Baloo, Frameworks

Oct 8 2018

poboiko added a comment to D15939: Perform checks for unindexed files and stale index entries on startup.

Hah, looks like you pushed a commit for it last year!

I'm not wedded to symlink support, but if we're not going to do it, we should close the bug with some good reasons.

Oct 8 2018, 10:33 PM · Baloo, Frameworks
poboiko committed R293:f8897a2511c4: Perform checks for unindexed files and stale index entries on startup (authored by poboiko).
Perform checks for unindexed files and stale index entries on startup
Oct 8 2018, 10:26 PM
poboiko closed D15939: Perform checks for unindexed files and stale index entries on startup.
Oct 8 2018, 10:26 PM · Baloo, Frameworks
poboiko committed R293:a9696978322c: [balooctl] Print current state & indexing file when monitor starts (authored by poboiko).
[balooctl] Print current state & indexing file when monitor starts
Oct 8 2018, 10:16 PM
poboiko closed D15985: [balooctl] Print current state & indexing file when monitor starts.
Oct 8 2018, 10:16 PM
poboiko committed R293:3fb45683fd27: [balooctl] Monitor also for state changes (authored by poboiko).
[balooctl] Monitor also for state changes
Oct 8 2018, 10:12 PM
poboiko closed D15943: [balooctl] Monitor also for state changes.
Oct 8 2018, 10:12 PM · Baloo, Frameworks
poboiko committed R293:7e5c005e6a3b: [balooctl] Fix "index" command with already indexed, but moved file (authored by poboiko).
[balooctl] Fix "index" command with already indexed, but moved file
Oct 8 2018, 10:11 PM
poboiko closed D15944: [balooctl] Fix "index" command with already indexed, but moved file.
Oct 8 2018, 10:11 PM · Baloo, Frameworks
poboiko added inline comments to D15826: [Balooshow] Avoid out-of-bounds access when accessing corrupt db data.
Oct 8 2018, 9:40 PM · Baloo, Frameworks
poboiko added a comment to D11828: Simplify orPostingIterator and make it faster.

Looks fine to me. But do we really need to optimize it? I mean, I didn't see it running more than ~20 ms, and with this patch for small queries it runs like ~16 ms. Worst case is when user types something in KRunner, but again, the lag is negligible there.

Oct 8 2018, 7:20 PM · Baloo, Frameworks
poboiko added a comment to D15826: [Balooshow] Avoid out-of-bounds access when accessing corrupt db data.

Maybe we should print also a suggestion to user, something like (maybe rephrase it better)

WARNING:
Looks like your index is corrupted.
We suggest you to run `balooctl disable && balooctl disable` to wipe it and rebuild from scratch

so they won't have to google what to do (there's still not much can be done in that case)

Oct 8 2018, 7:09 PM · Baloo, Frameworks

Oct 7 2018

poboiko added a comment to D15826: [Balooshow] Avoid out-of-bounds access when accessing corrupt db data.

I never experienced such corruption, though, but sanity check shouldn't hurt.

Oct 7 2018, 9:22 AM · Baloo, Frameworks
poboiko added a comment to T9805: Overhaul Baloo database scheme.

I would like to comment on that and add my 2 cents:

Oct 7 2018, 9:04 AM · Baloo

Oct 6 2018

poboiko added a comment to D15939: Perform checks for unindexed files and stale index entries on startup.

I've found the relevant bug: https://bugs.kde.org/show_bug.cgi?id=333678

Oct 6 2018, 8:22 PM · Baloo, Frameworks
poboiko requested review of D15985: [balooctl] Print current state & indexing file when monitor starts.
Oct 6 2018, 5:46 PM
poboiko added a comment to D15959: Wait for the extraction process to finish before scheduling.

That doesn't looks like it works all the time: sometimes I still have this issue.
Apparently, it still can happen that even when the process is finished, signal is emitted, but the thread is not finished yet.

Oct 6 2018, 4:38 PM · Baloo, Frameworks
poboiko requested review of D15983: React to config updates inside indexer.
Oct 6 2018, 4:27 PM · Baloo, Frameworks
poboiko added a comment to D15939: Perform checks for unindexed files and stale index entries on startup.

Works great, thanks for all these Baloo patches! They are very much appreciated. Code looks good. As with the others, please commit for Frameworks 5.52.

While you're cranking away at these, might I suggest a few others that are sorely needed:

I can look at these, shouldn't be too hard.

Oct 6 2018, 10:51 AM · Baloo, Frameworks
poboiko updated the summary of D15943: [balooctl] Monitor also for state changes.
Oct 6 2018, 10:16 AM · Baloo, Frameworks
poboiko added a comment to D15943: [balooctl] Monitor also for state changes.

Works for me and the code looks sane. Looks like this fixes https://bugs.kde.org/show_bug.cgi?id=364858, so can you add BUG: 364858 and FIXED-IN: 5.52 to the summary section and also make the changes that @bruns requested? Thanks!

Weird, bug 364858 should be fine as it is - user asks for a way to display files Baloo is currently processing, but that's exactly what monitor does (even without this patch).

Oct 6 2018, 10:14 AM · Baloo, Frameworks
poboiko updated the summary of D15944: [balooctl] Fix "index" command with already indexed, but moved file.
Oct 6 2018, 10:08 AM · Baloo, Frameworks

Oct 5 2018

poboiko requested review of D15960: Don't check if file is directory based on mime-type.
Oct 5 2018, 8:23 AM · Baloo, Frameworks
poboiko requested review of D15959: Wait for the extraction process to finish before scheduling.
Oct 5 2018, 7:57 AM · Baloo, Frameworks
poboiko updated the summary of D15943: [balooctl] Monitor also for state changes.
Oct 5 2018, 7:25 AM · Baloo, Frameworks

Oct 4 2018

poboiko updated the diff for D15939: Perform checks for unindexed files and stale index entries on startup.

Fixed typos

Oct 4 2018, 7:31 PM · Baloo, Frameworks
poboiko requested review of D15944: [balooctl] Fix "index" command with already indexed, but moved file.
Oct 4 2018, 1:10 PM · Baloo, Frameworks
poboiko requested review of D15943: [balooctl] Monitor also for state changes.
Oct 4 2018, 1:02 PM · Baloo, Frameworks
poboiko committed R293:ade5188979c4: Fix invalid char constant (authored by poboiko).
Fix invalid char constant
Oct 4 2018, 12:58 PM
poboiko closed D15941: Fix invalid char constant.
Oct 4 2018, 12:57 PM · Baloo, Frameworks