[sftp] bump pending request count from 1 to 128
ClosedPublic

Authored by sitter on Sep 12 2018, 3:12 PM.

Details

Summary

with the previous value we basically did sync reading which means that
network and cyrptographic overhead head a huge impact on throughput.
meanwhile the perfect way to use asyncness is to schedule a whole bunch of
requests before starting to read.
previously this was documented as auto-adjusting, which it never was,
there's also little to be gained from adjusting this value on the fly.
more requests in most scenarios will simply mean a larger RAM footprint as
more data potentially sits in libssh waiting to be read. with 128 requests
that'd be ~8mb (assuming the file transferred is that large)

this improves read performance with libssh 0.8 by up to 20 times for large
files. read performance with libssh 0.6 is 2 to 3 times better.
the faster the connection the higher the gain of course.

128 gives somewhat competitive performance results compared to openssh's
ssh implementations while not having too large a footprint.

in raw numbers: a local link read was averaging around 10mb/s on both
libssh versions. with 128 requests this goes up to 200. competitive is
between 200 and 300 it seems (obviously all specific to my system)

CHANGELOG: sftp file reading is now up to 20 times faster

Diff Detail

Repository
R320 KIO Extras
Branch
multirequest
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 2763
Build 2781: arc lint + arc unit
sitter requested review of this revision.Sep 12 2018, 3:12 PM
sitter created this revision.

Oh. FTR: I tied to properly thread this with a thread constantly queuing new requests (which should technically perform better with a smaller queue) but it made a relatively small (10% maybe) difference while featuring more complicated code. I think this is in part because libssh isn't thread safe within the same session, so I had to guard the begin_read and the async_read calls which meant that most of the expensive code (i.e. the reading) was still serialized anyway. Something to explore in detail in the future I expect.

broulik accepted this revision.Sep 18 2018, 11:21 AM
This revision is now accepted and ready to land.Sep 18 2018, 11:21 AM
This revision was automatically updated to reflect the committed changes.