Here’s a study that should make every educator using AI tools sit up straight: researchers at Stanford just found that large language models give Black students more praise and less constructive criticism than White students — on identical essays.
Same words. Same arguments. Same quality. Different feedback, depending on who the AI thinks wrote it.
The paper, Marked Pedagogies, by Mei Tan, Lena Phalen, and Dorottya Demszky from Stanford’s Graduate School of Education, was nominated for best paper at the International Learning Analytics and Knowledge Conference. It should be required reading for anyone selling AI tutoring tools to schools.
What the Researchers Did
The team took 600 identical argumentative essays written by 8th graders and submitted them to four major language models: GPT-4o, GPT-3.5-turbo, Llama-3.3 70B, and Llama-3.1 8B. Each essay was tested with different student profiles attached: “The student is Black,” “The student is White,” “female,” “has a learning disability,” “English language learner,” and so on.
Every essay was tested roughly 13 times across each configuration. The models didn’t see different writing — they saw the same writing with different names attached.
What They Found
The results were consistent across all four models:
- Black students received more praise (“powerful,” “great perspective”), more encouragement about leadership and cultural identity, and less constructive criticism about structure, evidence, and argumentation. “Black-marked” words increased by 180% compared to baseline feedback.
- White students received more direct, critical feedback on argument structure, evidence quality, and clarity — the kind of feedback that actually improves writing.
- Hispanic and English language learners got more feedback on grammar and spelling, with less attention to higher-order thinking.
- Female students received more affectionate language (“I love your confidence!”).
- Students described as low-achieving got upbeat pep talks instead of substantive critique.
As lead author Mei Tan put it: “They are picking up on the biases that humans exhibit.” The models learned these patterns from training data that reflects how humans treat different groups — and they’re reproducing them at scale.
Why “Nice” Bias Is Still Bias
Here’s the uncomfortable part: this looks like kindness. More encouragement, more affirmation, more cultural validation — what’s wrong with that?
Plenty, actually. It’s the algorithmic version of what education researchers call the “soft bigotry of low expectations.” When you withhold constructive criticism from a group, you’re not helping them — you’re implicitly saying their work doesn’t need improvement. You’re lowering the bar and calling it support.
A student who gets “Great perspective!” instead of “Your evidence doesn’t support your third argument — try finding a stronger source” is being shortchanged. The praise feels good in the moment. The skill gap compounds over years.
This is the same pattern researchers have documented in human teachers for decades. The difference is scale: an AI tutor deployed across thousands of schools can reproduce this bias simultaneously for millions of students, every day, without anyone noticing.
The Paper’s Own Controversial Footnote
The study has already sparked debate on X, and not just for the findings. A footnote in the paper states: “We do not assume all forms of discrimination are bad. Positive discrimination in favor of Black students may be considered morally justified.”
That’s an honest academic position — but it’s also a loaded one. The researchers are noting that not all bias is harmful, and some may even be intentional correction. Fair enough in a philosophy seminar. In a deployed EdTech product making real-time decisions about real children? That’s a different conversation.
The question isn’t whether some differential treatment could be beneficial. It’s whether AI systems should be making that call without teachers, parents, or policymakers knowing about it.
The NZ Angle
New Zealand schools are increasingly adopting AI tools — from Google’s free Gemini Pro for students to dedicated AI tutoring platforms. But our education system, with its commitment to Māori and Pasifika learner success, has a particular vulnerability here.
If AI tools are giving Māori or Pasifika students softer feedback because training data reflects patterns of how educators interact with these groups, we could be systematically undermining the very achievement goals the curriculum is designed to address. The bias might look supportive. It might even align with cultural responsiveness frameworks. But if it’s withholding the critical feedback that drives improvement, it’s a trap.
NZ’s AI education policies are still embryonic. This study makes the case for mandatory bias auditing before AI tools are deployed in classrooms — not after.
What Should Happen Next
The researchers’ own recommendation is straightforward: “Maybe a takeaway is that we shouldn’t leave the pedagogy to the large language model. Humans should be in control.”
Specifically:
- Mandatory bias audits for any AI tool used in schools — test with diverse student profiles before deployment, not after complaints
- Teacher oversight — AI feedback should be a starting point, not a final verdict
- Explicit pedagogical guardrails in prompts — tell the model to focus on argument quality, not student identity
- Transparency — EdTech companies must disclose how their tools handle demographic information
🔍 THE BOTTOM LINE
AI didn’t invent biased feedback — it learned it from us. But it can now deliver it at a scale no single teacher ever could. The soft bigotry of low expectations has gone algorithmic, and if we don’t build guardrails fast, we’ll be praising students straight into the achievement gap. The kindest thing an AI tutor can do is tell you the truth about your work.
