The KIO protocol can only send a maximum of 0xFFFFFF bytes of data.
TransferJob has an additional limit of 14 MiB.
BUG: 375765
elvisangelaccio | |
dfaure |
The KIO protocol can only send a maximum of 0xFFFFFF bytes of data.
TransferJob has an additional limit of 14 MiB.
BUG: 375765
Downloaded a 132MiB file, no more "slave died unexpectedly".
Size and md5sum match.
No Linters Available |
No Unit Test Coverage |
Buildable 2933 | |
Build 2951: arc lint + arc unit |
Thanks a lot for working on this.
What about a fixed chunk size? TransferJob seems to recommend 1MB at most:
/** * Request for data. * Please note, that you shouldn't put too large chunks * of data in it as this requires copies within the frame * work, so you should rather split the data you want * to pass here in reasonable chunks (about 1MB maximum) * * @param job the job that emitted this signal * @param data buffer to fill with data to send to the * slave. An empty buffer indicates end of data. (EOD) */ void dataReq(KIO::Job *job, QByteArray &data);
We definitely need to improve the documentation of this stuff. Adding @dfaure for more insights
The bigger, the better (=faster).
The 0xFFFFFF limit in KIO itself is due to the protocol only having 24 bits for size AFAIK - KLocalSocket can handle more than that at once without copying.
I'm not sure where TransferJob is used actually. It worked fine with more than 14MiB as well.
The only reason to use a smaller chunk size is to work better over TCP, but who uses KIO over TCP nowadays?
Yes to chunking.
Yes to changing the documentation, the 1MB must be very old in there, if 14MB works then let's say so.
As to the implementation, would it be possible to use QByteArray::fromRawData to avoid copying? After all it's all "synchronous" here, i.e. the lifetime of the chunk will always be less than the lifetime of the overall data.
src/kio_gdrive.cpp | ||
---|---|---|
645 | Sure. I didn't believe the documentation as that means a QByteArray can never exceed 2GiB in size. |
Please push to the stable branch (1.2). Thanks!
src/kio_gdrive.cpp | ||
---|---|---|
642 | data() also here |
! In D15448#330352, @ngraham wrote:
How is this looking now that D15426 has gone in?
Those diffs are fully independant - this fixes the bug that larger file transfers (over 16MiB) simply failed at the end while D15426 improves performance of all transfers using the KIO AccessManager massively by removing the 100% CPU bottleneck.