[Akonadi] Foreign payload
Open, Needs TriagePublic

Description

Our handling of locally-stored data is extremely ineffective. The data are almost always not in the cache (because local resources have very short cache expiration timeouts), so fetching such items is rather expensive:

  1. client -> server
  2. server -> resource
  3. resource -> server
  4. server->client ...
  5. cachecleaner

Where step 3 involves unnecessarily copying the already-locally-available data into our cache. This does not even make use of COW on FS that supports it because we actually pass around the raw data. It generates unnecessary amount of IO and SQL load.

Foreign payloads would basically allow clients to access the data from the resource's local storage directly without needing to temporarily duplicate the data on disk.

There are two possible approaches:

  1. The resource stores the absolute filepath to local data in PLD:BODY and sets a special flag that is understood by client libraries to interpret it differently (this is different from the "external" flag in PartTable).
    • Pros: very fast, simple
    • Cons: resource needs to maintain the path up-to-date, only works for data that are stored per-file in already encoded format (that is would not work for mbox resource for instance)
  1. The resource stores sets a special flag for the Item that the client libraries will understand. The libraries that talk to the resource via DBus to request actual data and/or absolute filepath to the data (or file descriptor of the file/mmaped page)
    • Pros: works even for mbox resources, allows pre-processing of the data by the resource
    • Cons: slower and less efficient than 1)
  1. ???

Related Objects

dvratil created this task.Aug 27 2015, 7:13 PM
dvratil updated the task description. (Show Details)
dvratil raised the priority of this task from to Needs Triage.
dvratil added a project: KDE PIM.
dvratil moved this task to Backlog on the KDE PIM board.
dvratil added a subscriber: dvratil.
dvratil renamed this task from [Akonadi] Payload PassThrough to [Akonadi] Foreign payload.Mar 26 2017, 2:55 PM
dvratil claimed this task.
dvratil updated the task description. (Show Details)
dkurz added a subscriber: dkurz.Jun 18 2018, 8:21 AM

The first proposed approach is just an External Payload without the need to cache the local file, but referencing it directly, right? So in other words, can External Payloads be viewed as Foreign Payloads + caching of the actual payload in a file?

Also, in https://community.kde.org/KDE_PIM/Akonadi/Architecture you wrote "There are also Foreign Payloads, but right now they are not actually used by anyone." Does that mean that Foreing Payloads are already implemented, but unused so far? If so, which approach was implemented. If not, I'll update the c.k.o Architecture article.

dvratil added a comment.EditedJun 18 2018, 8:37 AM

Yes, Foreign Payloads are implemented following the first proposal. On the database level, the external flag was replaced with storage enumerator, which can be Internal (data are stored in the DB), External (data are stored in file_db_data, DB holds only the filename), and Foreign (data are stored as files in resource's storage (maildir), DB holds absolute path to the file). This is fully opaque to the server, on the client side the Item API has been extended (Item::setPayloadPath()) and the ProtocolHelper was extended to handle this special case by setting the storage flag appropriately.

The code is there and works, but it isn't actually used by any resource (no resource sets payload via the Item::setPayloadPath() API). The reason is that the asynchronous nature of Akonadi led to data loss when moving Items between different resources. This should no longer be an issue nowadays thanks to T639 (Notification Payloads) but needs to be tested properly first.