A disproportionate share of Confederate soldiers also came from slave-owning households, compared to the general population - something like a third - but non-slave owning households still provided the majority of recruits. But yeah, it wasn’t a tiny minority, it was widespread and normalized. It was a ‘realistic’ thing to aspire to, to be a slaver shithead, like climbing from the working to the middle class. Then you can sneer at all the other poors and their lack of slaves.