Not that I’m an expert on these things but don’t we have a limit on smallness, the Planck length, but not necessarily any limit on how far apart two objects can be? Things moving away via the expansion of spacetime would be able to continue unbounded, but shrinking not so - not without the laws of physics becoming meaningless.

You can still say that spacetime expanding is the same as us shrinking but then you’re just into semantics and sure, you can call things whatever you like.