smb: make sure we use balanced arguments on qmax

previously there was opportunity for failure here when the deduced literal
type was different from the actual off_t type and qmax is very picky
about explicit input types. so, instead cast to off_t, as any int will
accommodate 0 anyway so the cast has no downsides.

BUG: 419702