Change Details

Our handling of locally-stored data is extremely ineffective. The data are almost always not in the cache (because local resources have very short cache expiration timeouts), so fetching such items is rather expensive: 1) client -> server 2) server -> resource 3) resource -> server 4) server->client ... 5) cachecleaner Where step 3 involves unnecessarily copying the already-locally-available data into our cache. This does not even make use of COW on FS that supports it because we actually pass around the raw data. It generates unnecessary amount of IO and SQL load. Payload PassThrough would basically allow clients to access the data from the resource's local storage directly without needing to temporarily duplicate the data on disk. There are two possible approaches: 1) The resource stores the absolute filepath to local data in PLD:BODY and sets a special flag that is understood by client libraries to interpret it differently (this is different from the "external" flag in PartTable). * Pros: very fast, simple * Cons: resource needs to maintain the path up-to-date, only works for data that are stored per-file in already encoded format (that is would not work for mbox resource for instance) 2) The resource stores sets a special flag for the Item that the client libraries will understand. The libraries that talk to the resource via DBus to request actual data and/or absolute filepath to the data (or file descriptor of the file/mmaped page) * Pros: works even for mbox resources, allows pre-processing of the data by the resource * Cons: slower and less efficient than 1) 3) ???

Our handling of locally-stored data is extremely ineffective. The data are almost always not in the cache (because local resources have very short cache expiration timeouts), so fetching such items is rather expensive: 1) client -> server 2) server -> resource 3) resource -> server 4) server->client ... 5) cachecleaner Where step 3 involves unnecessarily copying the already-locally-available data into our cache. This does not even make use of COW on FS that supports it because we actually pass around the raw data. It generates unnecessary amount of IO and SQL load. Foreign payloads would basically allow clients to access the data from the resource's local storage directly without needing to temporarily duplicate the data on disk. There are two possible approaches: 1) The resource stores the absolute filepath to local data in PLD:BODY and sets a special flag that is understood by client libraries to interpret it differently (this is different from the "external" flag in PartTable). * Pros: very fast, simple * Cons: resource needs to maintain the path up-to-date, only works for data that are stored per-file in already encoded format (that is would not work for mbox resource for instance) 2) The resource stores sets a special flag for the Item that the client libraries will understand. The libraries that talk to the resource via DBus to request actual data and/or absolute filepath to the data (or file descriptor of the file/mmaped page) * Pros: works even for mbox resources, allows pre-processing of the data by the resource * Cons: slower and less efficient than 1) 3) ???

Our handling of locally-stored data is extremely ineffective. The data are almost always not in the cache (because local resources have very short cache expiration timeouts), so fetching such items is rather expensive: 1) client -> server 2) server -> resource 3) resource -> server 4) server->client ... 5) cachecleaner Where step 3 involves unnecessarily copying the already-locally-available data into our cache. This does not even make use of COW on FS that supports it because we actually pass around the raw data. It generates unnecessary amount of IO and SQL load. Payload PassThroughForeign payloads would basically allow clients to access the data from the resource's local storage directly without needing to temporarily duplicate the data on disk. There are two possible approaches: 1) The resource stores the absolute filepath to local data in PLD:BODY and sets a special flag that is understood by client libraries to interpret it differently (this is different from the "external" flag in PartTable). * Pros: very fast, simple * Cons: resource needs to maintain the path up-to-date, only works for data that are stored per-file in already encoded format (that is would not work for mbox resource for instance) 2) The resource stores sets a special flag for the Item that the client libraries will understand. The libraries that talk to the resource via DBus to request actual data and/or absolute filepath to the data (or file descriptor of the file/mmaped page) * Pros: works even for mbox resources, allows pre-processing of the data by the resource * Cons: slower and less efficient than 1) 3) ???