Create Task
Maniphest T7069

resource database uses a lot more space than it actually requires.
Closed, ResolvedPublic
Actions

Description

A full sync with 40k mails results in a 1.6 gb file, although only ~110mb are occupied.

This is not overly problematic as this space will be reused as the actual required size grows, but it's not pretty either.

It is not clear so far why we end up with such a large file. The size seems to imply that at some point we actually use 1.6 gb, but end up freeing much of it again. The only thing we are rewriting though are some indexes, so no idea why this happens.

Background info:
https://symas.com/understanding-lmdb-database-file-sizes-and-memory-utilization/

Related Objects

cmollekopf created this task.Sep 26 2017, 10:40 AM
bash-4.3$ ls -lisah  ~/.local/share/sink/storage/\{7c69545a-2169-4667-a2a9-e8ac3d507434\}/
total 1.6G
662342 4.0K drwxr-xr-x  3 developer developer 4.0K Sep 25 10:04 .
662341 4.0K drwxr-xr-x 17 developer developer 4.0K Sep 25 10:10 ..
662343 4.0K drwxr-xr-x  3 developer developer 4.0K Sep 25 10:04 data
662360 1.6G -rw-r--r--  1 developer developer 1.6G Sep 25 10:10 data.mdb
662359 4.0K -rw-r--r--  1 developer developer 8.0K Sep 25 10:10 lock.mdb
bash-4.3$ sinksh stat {7c69545a-2169-4667-a2a9-e8ac3d507434}
        Database: __flagtable
        Size [kb]: 4
        Database: addressbook.index.parent
        Size [kb]: 4
        Database: addressbook.main
        Size [kb]: 4
        Database: contact.index.uid
        Size [kb]: 4
        Database: contact.main
        Size [kb]: 4
        Database: default
        Size [kb]: 4
        Database: event.index.uid
        Size [kb]: 4
        Database: event.main
        Size [kb]: 4
        Database: folder.index.name
        Size [kb]: 4
        Database: folder.index.parent
        Size [kb]: 4
        Database: folder.main
        Size [kb]: 20
        Database: folderuids
        Size [kb]: 4
        Database: mail.index.date
        Size [kb]: 4776
        Database: mail.index.draft
        Size [kb]: 4
        Database: mail.index.folder
        Size [kb]: 4
        Database: mail.index.folder.sort.date
        Size [kb]: 8976
        Database: mail.index.messageId
        Size [kb]: 8068
        Database: mail.index.messageIdthreadId
        Size [kb]: 8152
        Database: mail.index.parentMessageId
        Size [kb]: 3656
        Database: mail.index.threadIdmessageId
        Size [kb]: 5452
        Database: mail.main
        Size [kb]: 60664
        Database: mailuids
        Size [kb]: 3412
        Database: revisionType
        Size [kb]: 1716
        Database: revisions
        Size [kb]: 4864
        Database: uids
        Size [kb]: 4
        Disk usage [kb]: 1'606'676
Total [kb]: 109'812

1606676/109812= 15 times more than what we should have?

Can also be compacted

mdb_copy -c /home/developer/.local/share/sink/storage/{7c69545a-2169-4667-a2a9-e8ac3d507434} foo
bash-4.3$ ls -sh foo/
total 116M
116M data.mdb
bash-4.3$

We're not creating superfluos revisions (we have about 48k).

Maybe the problem is that normally we would fit many more items on a page and for some reason pages are underused.
cmollekopf closed this task as Resolved.Feb 22 2020, 9:31 PM
cmollekopf claimed this task.

The reason was that we had a concurrent read-only transaction so we ended up accumulating a lot of free pages. Fixed in 0dc8aa249d063a3d6eaa248950c57ed5a1709524