As organizations increasingly turn to artificial intelligence to automate tasks, an unsettling reality is coming into focus: our most advanced AI systems often behave in ways we don’t fully understand and can’t reliably control. Large language models (LLMs) have demonstrated remarkable fluency and versatility across a range of tasks. But since we cannot understand how these models arrive at their conclusions, we are ill-equipped to detect, let alone correct, their errors—especially in high-stakes settings.
Take hiring. Many assume AI systems operate with clinical objectivity, immune to the cognitive biases that plague human judgment. Unfortunately, this assumption does not hold up to scrutiny.
In a recent study, I evaluated dozens of cutting-edge LLMs for gender bias by asking them to select the more qualified candidate from pairs of résumés—one containing a male first name, the other a female first name. To control for qualifications, each résumé pair was presented twice, with the assignment of gendered names to résumés reversed on the second presentation. This made the distribution of qualifications statistically identical across genders, so in theory, an unbiased model making selections based on merit should have selected male and female candidates in roughly equal proportions.
But that’s not what happened. All 22 LLMs I tested more frequently selected female candidates as more qualified.
The LLMs’ preference for female candidates was consistent regardless of profession, as the chart below shows.
When I added an explicit gender field to each résumé—in addition to the gendered names, a common practice in countries including Germany, Japan, and South Korea—the preference for female candidates became even more pronounced.
Further analysis revealed other quirks. The models slightly favored candidates whose résumés included preferred pronouns (i.e., “she/her” or “he/him”). They even showed bias when the résumés used genderless identifiers like “Candidate A” and “Candidate B”—“Candidate A” was selected more frequently. Moreover, the models markedly favored whichever résumé appeared first in the prompt, suggesting a superficial decision-making process.
These tendencies persisted regardless of model size or the amount of compute leveraged by the model while generating its responses. This strongly suggests that model bias in the context of hiring decisions is not determined by the size of the model (i.e., the number of model parameters) or the amount of “reasoning” employed (i.e., the amount of compute deployed during inference). The problem is systemic. Models that are larger in terms of parameter count or that engage in more compute-intensive reasoning during inference are not inherently fairer.
These findings suggest a significant flaw in how modern AI systems interpret and respond to information: rather than consistently acting as impartial evaluators, the models may be influenced by subtle prompt cues in ways that impinge on objectivity and fairness. It’s not yet clear whether some of the behaviors described above stem from patterns in the training data that reflect broader cultural dynamics—including efforts to promote diversity. But regardless of their cause, these tendencies merit closer examination.
Despite these vulnerabilities, some companies have already begun using LLMs to screen résumés. Some even claim that their AI models offer bias-free insights into candidates’ résumés. My findings cast serious doubt on those claims. Whether inherited from skewed training data or other sources, hidden biases are embedded in today’s frontier AI systems—and they can lead to harmful and unjust outcomes when deployed in high-stakes scenarios.
As we hand over increasingly consequential decisions to AI, we need to ask whether these systems reflect the values of fairness and justice we claim to uphold. Until we can answer that question with confidence, AI should not be trusted with high-stakes, autonomous decision-making.
No comments:
Post a Comment
Note: Only a member of this blog may post a comment.