Yeah, you can’t apeal to “society has decided X” as an argument, if you also hold that half the people in society don’t count because you disagree with them. That’s actually the opposite of what ‘society’ means in that context.