Can an AI system be accurate but still biased?

Yes. Accuracy measures how often a system is right overall; it says nothing about whether it is wrong more often for particular groups. Fairness requires separate tests: demographic parity, equalised odds, and individual fairness.

What is the four-fifths rule?

The four-fifths rule compares the selection rate for one group against the selection rate for the highest-selected group, expressed as a ratio. A ratio below 80% is the long-standing threshold that triggers regulatory scrutiny for disparate impact.

Does NYC Local Law 144 require an independent bias audit?

Yes. Local Law 144 requires employers using an automated employment decision tool on NYC candidates to have an independent bias audit conducted within the prior year, publicly post a summary, and give candidates at least 10 business days' notice. It has been DCWP-enforced since 5 July 2023, with penalties of $500 to $1,500 per violation per day. Responsible AI Studio produces the audit document and evidence structure — it is not the independent auditor the law requires.

What fairness tests does an AI bias audit include?

Demographic parity, equalised odds, and individual fairness, alongside a disparate-impact ratio calculated against the four-fifths rule.

AI Governance · Fairness

How to Run an AI Bias Audit

The fairness tests that matter, the four-fifths rule, and what NYC Local Law 144 actually requires.

Last reviewed: 2026-07-24

An AI system can pass every accuracy test and still discriminate. Accuracy tells you how often it's right; it says nothing about who it's wrong for, more often.

The fairness tests that matter

Demographic parityAre outcomes similar across groups?
Equalised oddsAre error rates similar across groups?
Individual fairnessAre similar people treated similarly, case by case?

NYC Local Law 144

LL144 requires an independent bias audit for automated employment decision tools used on NYC candidates — conducted within the prior year, results posted publicly, at least 10 business days' notice to candidates. DCWP-enforced since 5 July 2023; penalties $500- $1,500 per violation, per day. Responsible AI Studio produces the audit document and evidence structure — we are not the independent auditor the law requires. A qualified reviewer signs it.

Get the free 34-item bias-audit checklist — no cost, just your email.

Email me the free checklist

Or go straight to the full audit — $39, one-time.

Frequently asked

Can an AI system be accurate but still biased?: Yes. Accuracy measures how often a system is right overall; it says nothing about whether it is wrong more often for particular groups. Fairness requires separate tests: demographic parity, equalised odds, and individual fairness.
What is the four-fifths rule?: The four-fifths rule compares the selection rate for one group against the selection rate for the highest-selected group, expressed as a ratio. A ratio below 80% is the long-standing threshold that triggers regulatory scrutiny for disparate impact.
Does NYC Local Law 144 require an independent bias audit?: Yes. Local Law 144 requires employers using an automated employment decision tool on NYC candidates to have an independent bias audit conducted within the prior year, publicly post a summary, and give candidates at least 10 business days' notice. It has been DCWP-enforced since 5 July 2023, with penalties of $500 to $1,500 per violation per day. Responsible AI Studio produces the audit document and evidence structure — it is not the independent auditor the law requires.
What fairness tests does an AI bias audit include?: Demographic parity, equalised odds, and individual fairness, alongside a disparate-impact ratio calculated against the four-fifths rule.

This page and the linked tool produce first-draft, AI-generated documents — not legal advice. Qualified review is required before you rely on any output.

Sources: Regulation (EU) 2024/1689 (EU AI Act), Article 10 · NIST AI Risk Management Framework · NYC Local Law 144 (2021), NYC DCWP · Responsible AI Studio .

← All AI governance resources