Amazon's Secret AI Hiring Tool Reportedly 'Penalized' Resumes With the Word 'Women's'

Image for article titled Amazon's Secret AI Hiring Tool Reportedly 'Penalized' Resumes With the Word 'Women's'
Photo: Getty

For years, a team at Amazon reportedly worked on software that vetted the resumes of job applicants in an effort to surface the most likely hires. It gradually became clear that no matter how hard engineers tried to fix it, the recruitment engine found a way to discriminate against women, Reuters reports.


On Wednesday, the outlet cited five sources familiar with the automated resume review program that began in 2014. According to those sources, a team that consisted of around a dozen engineers was tasked with building a program that would utilize machine learning to review a decade’s worth of resumes submitted to Amazon and its subsequent hiring practices. The goal was to teach an AI how to identify the most likely hires to streamline the list of potential recruits that would have to be subsequently vetted by human recruiters. From Reuters:

In effect, Amazon’s system taught itself that male candidates were preferable. It penalized resumes that included the word “women’s,” as in “women’s chess club captain.” And it downgraded graduates of two all-women’s colleges, according to people familiar with the matter. They did not specify the names of the schools.

Amazon edited the programs to make them neutral to these particular terms. But that was no guarantee that the machines would not devise other ways of sorting candidates that could prove discriminatory, the people said.

Gizmodo reached out to Amazon for comment on the report and a spokesperson sent us the following statement: “This was never used by Amazon recruiters to evaluate candidates.”

The algorithm’s gender discrimination issues became apparent about a year into the project’s lifecycle and it was eventually abandoned last year, the report said. It appears one of the primary issues was the dataset that Amazon had to work with. Most of the resumes submitted to the company over the previous decade came from men, and the tech sector has been controlled by men from its earliest days.

Another issue cited in the report was the algorithm’s preference for language that was often used by male applicants. Common words and phrases like a proficiency in a certain programming language would be ignored and verbs like “executed” and “captured” were given more weight.

After 500 iterations that were each trained to understand 50,000 unique terms, the team just couldn’t get the tool to stop reverting to discriminatory practices, Reuters reported. As time went on, the models often spiraled into recommending unqualified applicants at random.


The team’s effort highlights the limitations of algorithms as well as the difficulty of automating practices in a changing world. More women are joining the tech sector and all of the major tech giants have diversity initiatives in some form or another. But change has been painfully slow. Machines simply do what we tell them to do. If a machine is learning from example and we can only provide a sexist example, we’ll get sexist results.

According to Reuters, a new team has been assembled at Amazon’s Edinburgh engineering hub to take another crack at the “holy grail” of hiring.




Lars Vargas is still suspicious of 2021's motives

The AI analyzed a historical database of resumes and learned based on hiring results. Correct me if my impression of this is wrong, but I’m going to guess that gender and perhaps other forms of discrimination were present in the hiring process controlled by imperfect and biased humans. This discrimination may have been intentional or accidental, but there’s no doubting it’s there.

So really, the AI just learned what the humans did and copied that in its own approach. I suspect the AI would have behaved quite differently with say an even 50/50 mix of men and women hired from the same pool of resume, and would even discriminate against men if it were 80% women hired from that pool.

The AI is not inherently biased. It learns bias from humans. Heck, it did it’s job. The only way to “fix” this is to feed it data and results that reflect a different reality. But at that point everything is skewed and new biases are introduced, tainting the outcome in different ways. We can’t win as long as humans act like humans.