Remove some extra code for calculating scratchpad bounds
(cherry picked from commit 2dcef0215e81eb2c6a24971a459aa26c3ac3eec8)