Most things in our regular day-to-day lives would leave residue, even water (maybe deionized wouldn’t?) just a matter of how much and how noticeable. About the soap, well if you smell it on your hands after wash, then it’s still there and you will rub it onto stuff. At least it’s just soap tho…
I don’t know if you can reasonably remove all contaminants off your skin, unless you plan to like glove up or destroy the skin by washing waayyyyy too often. But you know, good hygiene will keep things you touch nicer for longer so keep at it.
And it’s not like the houses (not apartment buildings) these days dont look all the same. Unless it’s some rich people building their own and even those usually end up looking pretty unimaginative.