To be honest, this is something I wanted to contribute. I've had moments as I'd code that my computer would lock up and it was because baloo was indexing pictures (not even the source code). I'm not sure but it's not explicit that Baloo runs only when the computer is idle. I'd want to extend this to communicate:

• How much time / progress is made for a full index
• What progress is made during a reindexing (after the above when adding new files)
• A want for it to run explicitly when the computer is idle
• Or to the above, keeping it running under a particular niceness (that's a Linux-y thing but I'm sure there's equivalents on other systems)

jackyalcine: thank you very much for answer.
I am sorry, I am not programmer, I am not understand of all your questions.
I send you any screenshots. In Stacer app monitor application you can see up applications who use most processor.
Baloo file extractor use sometimes 1 GB RAM

In T13298#233444, @jackyalcine wrote:

To be honest, this is something I wanted to contribute. I've had moments as I'd code that my computer would lock up and it was because baloo was indexing pictures (not even the source code). I'm not sure but it's not explicit that Baloo runs only when the computer is idle. I'd want to extend this to communicate:

• How much time / progress is made for a full index

I think this is difficult to know in advance, you can only tell the progress if it already scanned the drive. What it can tell you already is the number of files already processed.

• What progress is made during a reindexing (after the above when adding new files)

This may be possible more easily if Baloo is able to collect the number of changed files fast enough. But usually, this counter should be very low anyways as it is already using a file system watcher to look for changes and immediately queues it for reindex. Such a progress meter could thus be mostly useless.

• A want for it to run explicitly when the computer is idle

Ah, well... Actually, it already does this by setting itself to low IO priority and idle CPU scheduler. But it could be improved by watching the current load (this needs to use an inverse extrapolation of the three loadavg values to get an accurate "just-now" value). Such a calculated load value could be used to put Baloo on pause immediately for maybe 10s or 60s. But care should be taken that we can still make progress. It could also watch for the sys time used by IO calls which would be most accurate, as this is the value that affects perceived interactivity the most. The btrfs tool "bees" uses such an approach and it works exceptionally well, you may want to look at its load estimation algorithm.

• Or to the above, keeping it running under a particular niceness (that's a Linux-y thing but I'm sure there's equivalents on other systems)

That's what it already does, tho it doesn't use nice (this is actually not very useful with most modern kernels due to session auto-grouping). It may be better to adapt it for the recently introduced kernel feature to fork children directly into a cgroup, then adjust this cgroup. There's also PSI (pressure stall information) for cgroups which can be used as a resources pressure indicator.

All in all, a lot of investigation has already been done, and most of your ideas have already been implemented almost to perfection when looking at the source code. But a few things stand out:

1. Baloo cannot work correctly with filesystems that use unstable device numbers, and it is also build around a wrong assumption about device and inode numbers: POSIX does not claim these to be stable across reboots or even umount/mount. NFS as a virtual filesystem (it has no block device associated with it) is problematic here as its device number changes with every mount. This confused Baloo: It reindexes the same files all over again after each remount because the device id changed, but it doesn't purge the files associated to the previous device id from the database because it cannot know if those files are only temporarily unavailable. So the database only grows with every reboot and easily surpasses your RAM size (which is not too bad but read below). The same problem applies to btrfs and ZFS which do not use physical block devices but map their pool of partitions through virtual block devices: The device ids are not stable across reboots. Similar problems may exist for file systems with potentially unstable inode numbers (probably FAT file systems or maybe also NTFS). So all in all, Baloo is currently only compatible with ext and xfs, maybe a few others which are less common.

2. After having outlined (1), let's see what could be fixed: Baloo should migrate to using its own device id database. That could be easily coded by adding another table to the database which stores a mapping of device id to filesystem uuid. The nice aspect of this approach is that we could just map the currently known device ids at table creation time and all the database entries stay compatible. Newly added devices would then allocate a free device number from this table and remember the uuid, reappearing device would match the uuid and use the device id mapped in the table. This will fix all file systems with unstable device ids (according fstat call). But it will not fix file systems with unstable inode numbers. Also, there's a problem with modern filesystems using 64 bit inode numbers: Baloo only has 32bit (device id) + 32bit (inode id) for file mapping, it currently just discards the upper 32bit if they exist. This is okay most of the times but creates a new set of problems.

3. While the Baloo database is using lmdb and that database system is very powerful, it is not tuned optimally when used as a background service. Actually, I don't believe lmdb was designed to work as a background system but all it's other properties are very well designed. I already submitted patches to tune some of the parameters, i.e. disable read-ahead as it is mostly useless for Baloo and how it uses the database. This reduces file system cache thrashing. And actually, that is what you want to care most about: Baloo should be able to tell the system early that it no longer needs some data in cache.

4. When the Baloo database grows very large, this may become problematic for the kernel. While lmdb actually doesn't really use memory (it's memory mapped and works more like the swap file), it still maintains a good amount of page table space. Especially in combination with components that do a lot of memory allocations (like btrfs), this introduces a vast amount of micro latencies and puts the memory manager under pressure. We will probably also see high memory fragmentation due to a lot of small allocations needed from lmdb. Memory compaction, when needed and running, will be observed as severely affecting interactive response of the system. And it may even lead to early out of memory situations for some hardware that needs contiguous memory allocations (mostly GPU applications). So it's important to keep the memory footprint of lmdb as low as possible. But that's not really possible as lmdb needs to map the whole database all the time. But it may be possible to start with a small database file and dynamically grow it when running out of space. But lmdb is not designed to do this while the database is opened, and Baloo probably needs some redesign to do this as far as I understood the source code.

So, all in all, that's a lot more complex than I maybe thought, and most of the points you are suggesting are already implemented. The devil is in the details, like "nice" doing not exactly that what you think it's doing on modern kernels, or memory latency issues, or unstable ids... Actually, when getting Baloo under control (correct filesystem, reduced database size, much RAM to reduce early latency from memory fragmentation) you'd probably not even notice it is running. Memory consumption itself isn't really the problem, I tried it, it's not solved by reducing the memory used by Baloo. The problems come from its interaction with cache, readahead and memory fragmentation. It would probably be easier to solve this with a database system that locks a fixed memory block into memory and only operates within the window (by shifting the mmap around in the file). There are database with very similar design to lmdb that could do that but they have different problems, and it's a lot of effort before even reaching a point where you could compare both implementations and test against each other. So we probably should make the best out of lmdb.

See also: T11859

hurikhan77 : thank you very much for post.
Yes, I use dual boot with Windows 10.