[sftp] bump pending request count from 1 to 128
ClosedPublic

Authored by sitter on Sep 12 2018, 3:12 PM.

Details

Summary

with the previous value we basically did sync reading which means that
network and cyrptographic overhead head a huge impact on throughput.
meanwhile the perfect way to use asyncness is to schedule a whole bunch of
requests before starting to read.
previously this was documented as auto-adjusting, which it never was,
there's also little to be gained from adjusting this value on the fly.
more requests in most scenarios will simply mean a larger RAM footprint as
more data potentially sits in libssh waiting to be read. with 128 requests
that'd be ~8mb (assuming the file transferred is that large)

this improves read performance with libssh 0.8 by up to 20 times for large
files. read performance with libssh 0.6 is 2 to 3 times better.
the faster the connection the higher the gain of course.

128 gives somewhat competitive performance results compared to openssh's
ssh implementations while not having too large a footprint.

in raw numbers: a local link read was averaging around 10mb/s on both
libssh versions. with 128 requests this goes up to 200. competitive is
between 200 and 300 it seems (obviously all specific to my system)

CHANGELOG: sftp file reading is now up to 20 times faster

Diff Detail

Repository
R320 KIO Extras
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.
sitter requested review of this revision.Sep 12 2018, 3:12 PM
sitter created this revision.

Oh. FTR: I tied to properly thread this with a thread constantly queuing new requests (which should technically perform better with a smaller queue) but it made a relatively small (10% maybe) difference while featuring more complicated code. I think this is in part because libssh isn't thread safe within the same session, so I had to guard the begin_read and the async_read calls which meant that most of the expensive code (i.e. the reading) was still serialized anyway. Something to explore in detail in the future I expect.

broulik accepted this revision.Sep 18 2018, 11:21 AM
This revision is now accepted and ready to land.Sep 18 2018, 11:21 AM
This revision was automatically updated to reflect the committed changes.