How Women Leaders Are Exposing AI’s Bias Problem

“The coded gaze of AI systems can perpetuate bias at a scale never seen before.” – Dr. Joy Buolamwini^[1]

Key Takeaways

• AI systems don’t just reflect biases—they amplify them across millions of decisions
• Women leaders in tech are exposing systematic bias in AI systems
• Bias multiplication happens through speed, scale, and false assumptions of objectivity
• Solutions require diverse perspectives and systematic accountability

The Promise vs. Reality

For decades, Silicon Valley has sold us a seductive narrative: technology would be the great equalizer, and algorithms would free us from human bias. While critical voices have always challenged this techno-utopian vision, they’ve been consistently drowned out by economic interests and favorable media coverage.

Now, as technology reshapes our world at an unprecedented pace, these overlooked warnings are proving all too true.

Recent developments tell a sobering story: 

• OpenAI’s dismantling of its nonprofit arm—originally designed as an ethical safeguard
• An exodus of high-level employees raising red flags about the industry’s direction
• Mounting evidence that AI systems aren’t eliminating bias—they’re multiplying it

Amid this rapid shift, a powerful group of women leaders has stepped up. Not just to sound the alarm, but to push for change. These researchers and advocates aren’t just studying the problem; they’re exposing AI’s systemic biases and fighting back.

The Women Leading the Change

It was 2016, and I was wandering the perfectly chaotic aisles of Green Apple Bookstore in San Francisco. That’s when “Weapons of Math Destruction” caught my eye. As someone who finds joy in mathematical patterns and has always been fascinated by the rapid rise of algorithmic systems, I couldn’t resist picking it up. The deeper I read, the more it echoed what I was already noticing in my own workplace and in emerging technologies. I knew I needed to understand more.

Fast forward seven years. During a conversation about ethical AI, someone mentioned Dr. Joy Buolamwini’s research. It felt like uncovering another piece of a puzzle I didn’t even realize I was solving. These moments led me down a path of exploring systemic bias through the lens of brilliant women pioneers who are actively reshaping how we think about AI.

Dr. Joy Buolamwini’s work at MIT’s Media Lab uncovered something that made me actually yelp out loud: facial recognition systems failed to identify darker-skinned women 34% of the time, while achieving 99% accuracy for lighter-skinned men^[2]. Let that sink in. That’s not a small gap – that’s a so-called “neutral” system working almost flawlessly for some while completely failing others.

Her documentary “Coded Bias”^[3] and recent book “Unmasking AI” peel back layers of systemic bias that run far deeper than facial recognition. What makes Dr. Joy’s work truly stand out is how she blends artistry with technical expertise—using poetry and creativity to make these urgent insights accessible to everyone.

This story isn’t just about one voice. A growing coalition of women researchers and leaders is shaping this conversation, each bringing their own powerful insights to the fight for ethical AI:

• Dr. Timnit Gebru^[4], who was famously pushed out of Google for raising concerns about bias in large language models
• Dr. Margaret Mitchell^[5], who co-authored groundbreaking work on model cards to bring more transparency to AI
• Cathy O’Neil, whose book “Weapons of Math Destruction”^[6] exposed how algorithms perpetuate inequality (I was truly heartbroken reading this)
• Safiya Noble, whose work “Algorithms of Oppression”^[7] uncovered racial bias baked into search engines

These women aren’t just publishing papers or writing books – they’re leading a movement. They’re forcing us to confront uncomfortable truths about the technology we’re building, exposing how AI’s biases ripple through society, causing real-world harm at a scale we’re only beginning to grasp.

But more importantly? They’re showing us that a different path is possible.

When Small Biases Create Big Problems

Here’s what’s becoming painfully clear: AI isn’t just inheriting our biases—it’s amplifying them at scale.

Case Study – Amazon’s AI Recruitment Tool

Think about this: as recently as 2015—not some distant past—Amazon’s engineers ran into a problem they couldn’t solve. Their AI-powered hiring tool was supposed to streamline recruitment. Instead, it started discriminating against women.

And this wasn’t some scrappy startup figuring things out as they go. This was Amazon. One of the most technologically advanced companies in the world, with virtually unlimited resources.

According to Reuters, Amazon built the system in 2014 to help screen resumes. But by 2015, they discovered a serious issue: the AI, trained on 10 years of the company’s hiring data, had taught itself to penalize resumes that included words like “women’s” and automatically downgraded graduates from women’s colleges.

Amazon’s engineers tried to fix it. They tweaked the models, adjusted the inputs. But they couldn’t guarantee that the system wouldn’t just find new, more subtle ways to discriminate.

Let that sink in: some of the world’s top engineers could not remove bias from an AI system. Eventually, Amazon shut down the project in early 2017. ^[8]

Real-World Impact Examples

Not all biased systems are caught before deployment. And when they make it into the real world? The consequences are massive.

💰 Financial Services

• In 2019, the New York Department of Financial Services launched an investigation after Apple Card’s algorithm offered women dramatically lower credit limits than men with identical financial profiles—including cases where women were given lower limits than their husbands despite having higher credit scores ^[9]. 

🏥 Healthcare

• A 2019 study published in Science found that healthcare algorithms used by major U.S. hospitals systematically underestimated Black patients’ health needs, affecting millions. The issue? The system used healthcare costs as a proxy for health needs. But because Black patients historically received less care and had lower medical spending, they had to be significantly sicker than white patients before being flagged for additional treatment. ^[10]

💼 Employment

• Research published in 2021 showed Facebook’s ad delivery system was showing different job ads to men and women regardless of qualifications, keeping higher-paying job ads from reaching women—all without advertisers requesting this discrimination. ^[11]

⚖️ Criminal Justice

• ProPublica’s “Machine Bias” investigation exposed how criminal risk assessment algorithms used in sentencing showed significant racial bias, leading to harsher sentences for people of color.^[12]

As Dr. Joy Buolamwini writes in “Unmasking AI,” “The coded gaze of AI systems can perpetuate bias at a scale never seen before.”

THIS, my friends, is the real problem. We’re not talking about one biased decision anymore. We’re talking about bias being multiplied across millions of decisions, all while hiding behind the illusion of technological objectivity.

These aren’t hypothetical risks or test cases. These are real systems affecting real people right now—determining who gets loans, who gets healthcare, who gets jobs, and even who goes to prison.

The Amazon case shows how bias can creep in even when people are actively trying to prevent it. But these other examples? They show us what happens when biased AI isn’t caught in time—when it moves beyond testing and starts shaping people’s lives.

Understanding the Feedback Loop

Dr. Timnit Gebru calls it the “data cascade effect,” and once you see it, you can’t unsee it.

How Bias Breeds Bias

Here’s how it works: AI systems learn from historical data – data that already reflects society’s existing biases. But instead of just mirroring those biases, AI amplify them. Then, these biased decisions create new training data, reinforcing and intensifying the cycle.

Take Amazon’s AI recruitment tool. It wasn’t just passively reflecting past hiring trends—it was actively penalizing resumes that included the word “women’s” and downgrading graduates from women’s colleges. If it had been deployed at scale, each biased hiring decision would have generated new “successful hire” data, further cementing discrimination into future hiring models.

The Amplification Factors

Once bias gets encoded into AI, three critical factors supercharge its impact:

1. 🚀 Speed and Scale: These systems make millions of decisions daily, each one potentially carrying embedded bias.
2. ⚖️ False Objectivity: “The computer said no” carries an illusion of fairness that masks underlying discrimination.
3. 🔄 Cross-System Effects: Biased outputs from one system become training data for another, compounding harm over time.

The Cascade Effect in Action

A single biased decision doesn’t just stay in one system—it ripples outward, creating destructive feedback loops:

• Biased lending algorithms → lower credit scores
• Lower credit scores → higher insurance rates
• Higher insurance rates → fewer housing opportunities
• Limited housing → restricted school access
• Restricted schools → fewer job prospects
• Fewer job prospects → limited financial mobility → reinforcing biased lending models

And the cycle repeats.

Cathy O’Neil calls this “destructive feedback loops”—systems that don’t just reflect inequality but actively reinforce it, trapping entire communities in cycles of algorithmic discrimination.Each loop of the cycle amplifies the initial bias, creating what O’Neil calls “destructive feedback loops” that can trap entire communities in cycles of algorithmic discrimination.

Breaking the Cycle

These researchers aren’t just identifying problems – they’re showing us where and how to intervene. Here’s what their work teaches us about creating real change:

🔍 Question Your Data (Inspired by Dr. Timnit Gebru’s work)

• Who is represented in the training data—and who isn’t?
• What biases and limitations exist in the data?
• How does historical context shape potential harm?

⚖️ Build in Accountability (Following Dr. Joy Buolamwini’s framework)

• Establish transparent audit processes for AI systems.
• Test outputs across diverse demographic groups to catch disparities.
• Track real-world outcomes to spot hidden compounding effects.

🌍 Diversify Your Teams (As advocated by Dr. Margaret Mitchell)

• Bring in diverse perspectives throughout AI development.
• Ensure inclusive decision-making at every stage.
• Value lived experience alongside technical expertise—it’s just as crucial.

🛑 Enable Oversight (Based on Cathy O’Neil’s recommendations)

• Make decision criteria explicit so AI’s reasoning isn’t a black box.
• Create clear audit trails for accountability.
• Build in human review for critical, high-impact decisions.

AI isn’t creating bias in a vacuum – it’s taking our societal biases and amplifying them at scale. Understanding how this happens also shows us exactly where we can step in and disrupt the cycle.

It’s not just about spotting the problem anymore – it’s about identifying the leverage points where our actions can make the biggest impact.

Your Role in the Equation

Understanding AI’s multiplier effect isn’t just some academic exercise. It’s a roadmap for change. When we question our data, build in real accountability, and bring new voices to the table, we shape what technology actually does in the world.

I’m in awe of these women who stood up and said ‘This isn’t good enough.’ They didn’t just point out the problems. They mapped out a way forward.

AI bias isn’t some inevitable “tech tax” we have to pay. It’s a choice. Every time we build a system, deploy an algorithm, or train a model, we’ make decisions that ripple outward, shaping lives in ways we may never see.

This isn’t just someone else’s problem. Whether you’re building AI, using it in your business, or simply navigating a world shaped by algorithms, you have power here.

📖 Read these researchers’ work.
🤔Question your systems.
💡 Support the people fighting for algorithmic justice.

And please, PLEASE amplify the diverse voices working to make tech more equitable.

We may never fully eliminate bias (we’re human, after all), but we can refuse to let technology become a megaphone for inequality.

As Dr. Buolamwini said, “Who codes matters, who designs matters, who creates matters.”

And I’ll add this:
Who speaks up matters.
Who takes action matters.
Who demands better matters.

Let’s make damn sure our AI systems amplify our best intentions, not our biases.

References & Resources

Ali, M., et al. (2021). “Ad Delivery Algorithms: The Hidden Arbiters of Political Messaging.” arXiv. ↩ The original research paper (2021): https://arxiv.org/abs/2104.14558

Angwin, J., et al. (2016). “Machine Bias.” ProPublica. ↩ https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing

Buolamwini, J. (2023). Unmasking AI: My Mission to Protect What Is Human in a World of Machines. Random House. https://www.penguinrandomhouse.com/books/670356/unmasking-ai-by-dr-joy-buolamwini/

Buolamwini, J., & Gebru, T. (2018). Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. Proceedings of Machine Learning Research, 81, 77-91. [Key facial recognition bias findings] https://proceedings.mlr.press/v81/buolamwini18a.html

Dastin, J. (2018). “Amazon scraps secret AI recruiting tool that showed bias against women.” Reuters. ↩ https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08G

Gebru, T., et al. (2020). On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 610-623. [Language model bias research] https://dl.acm.org/doi/10.1145/3442188.3445922

Mitchell, M., et al. (2019). Model Cards for Model Reporting. Proceedings of the Conference on Fairness, Accountability, and Transparency, 220-229. [Model transparency framework] https://dl.acm.org/doi/10.1145/3287560.3287596

New York State Department of Financial Services. (2021). “Investigation of Apple Card Program.” ↩ https://www.dfs.ny.gov/reports_and_publications/press_releases/pr202103231

Noble, S. U. (2018). Algorithms of Oppression: How Search Engines Reinforce Racism. NYU Press. https://nyupress.org/9781479837243/algorithms-of-oppression/

O’Neil, C. (2016). Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy. Crown. https://www.penguinrandomhouse.com/books/241363/weapons-of-math-destruction-by-cathy-oneil/

Obermeyer, Z., et al. (2019). “Dissecting racial bias in an algorithm used to manage the health of populations.” Science. ↩ https://www.science.org/doi/10.1126/science.aax2342

Raji, I. D., & Buolamwini, J. (2019). Actionable Auditing: Investigating the Impact of Publicly Naming Biased Performance Results of Commercial AI Products. AIES ’19: Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, 429-435. [Commercial AI audit findings] https://dl.acm.org/doi/10.1145/3306618.3314244

Related: Gender Shades project