Your orientation doesn’t matter if you always take the direction as if you were in the drivers seat

Correct. And you would refer to that as something specific because if you just said left people might think that you are talking about your personal left. So you come up with short hand to mean the left when facing the front of the ship, and right when facing the front of the ship. The words that were agreed upon for this purpose? Port and starboard.