Your number leaves out all of the people whose pay rates are above minimum wage, but are still poverty wages. There is quite a large gap between minimum and poverty, and not in the direction that benefits the working class.

Furthermore, raising the minimum wage leads to people in that gap also getting raises. People can and do benefit tremendously from the minimum wage being raised, even if they have never personally worked at minimum wage. As such, the minimum wage is relevant to far, far more workers than are actually getting it.