Differential D13902

Use Key API in indexes
ClosedPublic
Actions

Authored by rnicole on Jul 5 2018, 1:44 PM.

Details

Reviewers

cmollekopf

Commits

R9:620c4f551d3d: Use Key API in indexes

Summary

Only in TypeIndex, not in Index (since we might want to store something other than identifiers as values)
We might want to do the same in the SynchronizerStore for localId ↔ remoteId indexes

Depends on D13735

Some quick benchmarks (against develop and D13735):

bench8 KBDownload

Diff Detail

Repository

R9 Sink

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

rnicole requested review of this revision.Jul 5 2018, 1:44 PM

rnicole created this revision.

Looks good to me.

This revision is now accepted and ready to land.Jul 6 2018, 10:14 AM

Increment database version

Some disk statistics:

disk8 KBDownload

Account 1: a lot of mails, but mostly text (occasional big attachments)
Account 2: a good number of mails, but mostly HTML heavy messages, with images, etc.

Results of the Dummy resource write benchmark:

writebench12 KBDownload

Some interesting bits:

Run benchmarks

Before	After
Current Rss usage [kb]: 40700	Current Rss usage [kb]: 38484
Peak Rss usage [kb]: 40700	Peak Rss usage [kb]: 38484
Rss growth [kb]: 15920	Rss growth [kb]: 13660
Rss growth per entity [byte]: 3260	Rss growth per entity [byte]: 2797
Rss without db [kb]: 29736	Rss without db [kb]: 29504
Percentage peak rss error: 0	Percentage peak rss error: 0
On disk [kb]: 10788	On disk [kb]: 8804
Buffer size total [kb]: 898	Buffer size total [kb]: 898
Write amplification: 12.0075	Write amplification: 9.79923

Test Disk Usage:

Before	After
Free pages: 412	Free pages: 312
Total pages: 760	Total pages: 607
Used size: 1425408	Used size: 1208320
Calculated key + value size: 856932	Calculated key + value size: 702866
Calculated total db sizes: 970752	Calculated total db sizes: 950272
Main store on disk: 3112960	Main store on disk: 2486272
Total on disk: 3293184	Total on disk: 2666496
Used size amplification: 1.66339	Used size amplification: 1.71913
Write amplification: 3.63268	Write amplification: 3.53733

The benchmark matches the expectations I'd say

I'd say so too! But I'm still not sure why the disk statistics are so underwhelming :-/

In D13902#289950, @rnicole wrote:

I'd say so too! But I'm still not sure why the disk statistics are so underwhelming :-/

Yeah, those seem wrong to be honest. The only way those numbers make any sense if the uid size is completely dwarfed by the rest of the data, but that has not been our experience so far.

rnicole added a dependent revision: D14099: Use Key API in DataStoreQuery.Jul 13 2018, 4:11 PM

Closed by commit R9:620c4f551d3d: Use Key API in indexes (authored by rnicole, committed by cmollekopf). · Explain WhyJul 27 2018, 2:15 PM

This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

			Path	Packages
M			common/definitions.cpp (2 lines)
M			common/index.h (3 lines)
M			common/index.cpp (12 lines)
M			common/storage/entitystore.cpp (17 lines)
M			common/typeindex.h (17 lines)
M			common/typeindex.cpp (66 lines)

Diff	ID	Base	Description	Created	Lint	Unit
Base			Base
Diff 1	37186			Jul 5 2018, 12:20 PM	★	★
Diff 2	37241			Jul 6 2018, 11:07 AM	★	★
Diff 3	38607	d1838e5	R9:620c4f551d3de830a516475ad02965695cb25945	Jul 27 2018, 11:47 AM	★	★

Status	Author	Revision
Closed	rnicole	D14404 Use Key API in SinkSH
Closed	rnicole	D14289 Use Key API in ResultSet
Closed	rnicole	D14099 Use Key API in DataStoreQuery
Closed	rnicole	D13902 Use Key API in indexes
Closed	rnicole	D13735 New Key API in storage layer