Fix clang-tidy warning

This code is intentionally trying get the sizeof a pointer value so
use ptrdiff_t directly to avoid confusion.