an AI resume screener had been trained on CVs of employees already at the firm, giving people extra marks if they listed “baseball” or “basketball” – hobbies that were linked to more successful staff, often men. Those who mentioned “softball” – typically women – were downgraded.
Marginalised groups often “fall through the cracks, because they have different hobbies, they went to different schools”
They’re not looking for the exceptional, out there exceptions - they’re looking for statistical pattern which have predicted current success. You may as well say that BMI is a useless metric for long term health complications. They both explicitly misestimate anomalous outliers because they are not designed to identify or classify anomalous outliers.