Implement Overlap queries
ClosedPublic

Authored by rnicole on Jun 11 2018, 9:20 AM.

Details

Summary

Notes:

  • Introduces the concept of queries on multiple properties (which meant changing query's internals a bit)
  • Dates are stored as well as the "reference" in the index to allow quick filtering without fetching the whole entity
  • Buckets are weeks starting on Monday (guaranteed by the use of the Julian calendar)
  • Some size improvements are definitely possible (dates are padded numbers again, not using integer databases, Julian calendar starts at a very old date, etc.)
Test Plan

Tested in querytest

Diff Detail

Repository
R9 Sink
Lint
Lint Skipped
Unit
Unit Tests Skipped
rnicole requested review of this revision.Jun 11 2018, 9:20 AM
rnicole created this revision.

Some general notes first; Good job, I like the approach =) Using the buckets makes the whole index smaller, which is great. The tradeoff is probably the added complexity of storing start and end date directly in the value and filtering accordingly.
The alternative would have been to just store the samples as key, and then relying on a post filtering step that reads the relevant dates from the entities directly. I don't know what would perform better either (depends on the dataset I suppose), so I'm fine with this approach as well (If you did make any estimations I'd be interested).

cmollekopf added inline comments.Jun 18 2018, 8:48 AM
tests/querytest.cpp
1620

You'll have to test the live query case (a running live query, to which events are added) as well. So far we never use the indexes in the incremental update case, but always filter directly on the properties (the assumption is that this is cheaper for small updates). I think your code handles the case though with the datastorequery changes.

cmollekopf accepted this revision.Jun 18 2018, 8:49 AM

Looks overall good to me. I'll apply the patch locally, and will merge it if not problems surface.

This revision is now accepted and ready to land.Jun 18 2018, 8:49 AM
rnicole planned changes to this revision.EditedJun 18 2018, 9:03 AM

Wait before merging it, storing the dates alongside the reference in the index is a "bad optimization" (changed my mind over the WE):

It was to optimize memory when fetching with a period not aligned with an entire week

So in the general case it's better, but since we use it for a calendar view, we will most likely only fetch events in a whole week (ie: week view and month view), so it should be better to remove them

rnicole updated this revision to Diff 36269.Jun 18 2018, 10:34 AM
  • Test live queries
  • Remove dates from the index (worst general case, but we're generally not the general case)
This revision is now accepted and ready to land.Jun 18 2018, 10:34 AM
This revision was automatically updated to reflect the committed changes.
This comment was removed by rnicole.