AI Hiring Tools Are Already Discriminating Against You

Given that almost 65% of the Fortune 500 companies rely on Workday products and services^[1], there is a narrow policy window for Canadian regulators to establish federal algorithmic transparency and accountability requirements for companies using AI hiring tools.

Workday Lawsuit

In June, a California judge allowed a discrimination lawsuit to proceed against Workday, the provider of cloud-based AI hiring systems^[2]. Workday is alleged to have created AI hiring systems that unlawfully discriminate against applicants based on race, age, and disability status. One plaintiff, a 40-year-old African American man, alleged that he was rejected from over 100 jobs from companies using Workday’s hiring platform, often mere minutes after applying and sometimes in the middle of the night.

Gender Bias in AI Hiring Tools

AI ethicists and researchers have spoken at great lengths about the risks of AI hiring tools in perpetuating historical biases through unrepresentative data or biased learning algorithms that perceive patterns that are inaccurate or non-existent.

In 2018, Amazon’s AI hiring system was found to have downgraded resumes containing female-coded or “female-adjacent” words or phrases (e.g., “women’s chess club captain”), rating them lower in comparison to resumes from male applicants. This was not surprising given that the system was trained on resumes previously submitted to Amazon – the majority of which were from male applicants. Amazon attempted to fix the system using de-biasing techniques, including tweaking the learning algorithm to correlate female-centric words or phrases in a more neutral manner. The solution partially worked, but Amazon could not guarantee that the system would not hallucinate and generate discriminatory outputs in other ways and decided to scrap the system.

How many companies have taken Amazon’s lead, and how many have been savvy enough in masking the flaws of their systems?

Algorithmic Monoculture

Most recently, Stanford University released a landmark study analyzing AI hiring tools built by the game-based recruitment company Pymetrics^[3] and found evidence of “systemic rejections at scale” against Black and Asian applicants, along with the rise of “algorithmic monoculture” – a phenomenon where different companies use the same set of AI hiring algorithms from a small number of third-party vendors^[4]. Algorithmic monoculture entrenches systemic discrimination and increases the likelihood of excluding historically marginalized groups from employment opportunities. An applicant might expect to be evaluated independently for each job they apply to but find they are being systemically rejected because a single discriminatory AI hiring algorithm is used by employers for screening across multiple positions^[5].

The study analyzed 4 million applications that were submitted to around 1,700 white-collar job postings across 150 employers (spanning professional services, finance, technology and manufacturing)^[9] and found that Black and Asian applicants were recommended for jobs by Pymetrics’ AI hiring models at significantly lower rates in comparison to White applicants. To be precise, 26% of Black candidates and 15% of Asian candidates applied to jobs where Pymetrics’ AI model discriminated against their racial group at the screening stage^[6]. The study found that if a model recommended Black and Asian applicants at the same rate as it was recommending White applicants, 40,000 more applications from these minority groups would have advanced to the next stage of hiring^[10].

Playing Games

Pymetrics builds online assessment games that measure an applicant’s cognitive abilities in areas like risk taking, planning, processing speed, trust and altruism. A company directs applicants to play these games and uses the results to assess suitability for a particular job. Specifically, Pymetrics builds a machine learning (ML) hiring model for a specific job that generates a score for the applicant based on their performance on the assessment games. That score will direct employers to either reject an applicant or recommend that they move forward in the hiring process. The results of an applicant’s assessment games are stored by Pymetrics for 330 days and used whenever an applicant applies to other jobs that are also managed using Pymetrics hiring tools.

Pymetrics notes that an applicant’s demographic information like race, gender or disability status is not collected when applicants play assessment games. However, ML models can still discriminate through proxy variables – which are indirect measures of sensitive characteristics and attributes like race. For example, postal codes can be correlated with the level of affluence of a particular neighbourhood and, by extension, can act as a proxy variable for the neighbourhood’s racial composition.

Critically, the ML model is trained on game performance data from current employees of the company who are in the same role. This means the ML hiring model learns patterns about what “good” game performance looks like (and by extension, an applicant’s suitability for a specific job) based on how existing employees played the assessment games, including the kinds of cognitive abilities and skills they exhibited. The employer determines which employees performed “well” on the games, and these employees serve as positive examples used to train the ML model. In this way, the employer can influence what the model learns, but risks embedding implicit biases that can penalize applicants of minority demographic groups. Applicants showing different patterns in their gameplay – e.g., in terms of reaction time, tactical decision-making or risk taking – could be penalized by the hiring model as “underperforming” because they don’t align with the performance of demographic groups that are typically favoured and/or hired by the company.

Canadian Context

How can Canadian regulators ensure transparency when many AI hiring systems, including the datasets used to train them, are proprietary in nature and protected from public review? Currently, Canada does not have horizontal federal legislation outlining how AI systems are to be managed. Ontario introduced a new requirement under its Employment Standards Act that requires employers to publicly disclose whether AI hiring tools are being used to screen applicants for job postings^[7]. Quebec’s “Law 25” outlines provisions for employers using AI systems (including for hiring), including the requirement to inform applicants that AI was used, provide applicants with recourse to have a human review a decision made by an AI system, and disclose the type of personal information that was used to influence the outcome generated by an AI system^[8]. At the federal level, existing legislation like the Personal Information Protection and Electronic Documents Act (PIPEDA) outlines minimum requirements for the collection, use, and disclosure of personal data for commercial activities, but does not have specific AI-related provisions. Canada is therefore relying on a patchwork of federal and provincial laws, liability regimes and non-binding guidelines to manage biases from AI-mediated hiring. However, this does not go far enough in protecting Canadians and increases liability risks for companies using AI-powered screening tools.

Every Job Application Sounds the Same

Both the Workday lawsuit and the Stanford study exemplify a shift in labour market dynamics in which the recruitment process has become ironically less efficient, but more homogenized. AI hiring systems are screening resumes written with AI tools, while AI-powered platforms and agents are conducting interviews with applicants who are providing responses generated by AI chatbots. It has become difficult to distinguish between the signal and the noise within the labour market. Applicants are starting to sound the same while being evaluated by algorithms with the same underlying pattern recognition mechanisms. The liminal space in which human discernment and judgement can be used to evaluate applications is narrowing while systemic biases are being further entrenched.