There are a lot of Matrix clients to choose from; they’re not all electron web apps and there’s even a terminal app available. There could easily be a Matrix client that behaves the way you want, but I couldn’t tell you if one exists. You’d have to try a few and see if they have settings that work well for you.

I guess I’m a little curious about why you wouldn’t want persistent chat history. Major IRC networks log all chats anyway, so it’s only you who’s missing out on having that chat history available if you are ever offline.