When the Human in the Loop is the Problem

The safeguard nobody audits!

   


QUESTION 

We have been transitioning to AI use in several departments. As a board member with governance responsibilities, I am interested in the human-in-the-loop responsibility, which you discussed in Will AI Reduce Fair Lending Violations? We discussed your April 1st article, AI Replace Me. It was a sequel to your article, Will AI Replace Me? 

In your article, you discussed the future role of humans with regard to AI. You said that we should have a "human-governed" approach to AI. The matrix you provided on the roles and the corresponding human oversight requirement for AI was eye-opening. Accordingly, we are developing a human-in-the-loop policy. 

In our last board meeting, someone made an interesting point. The presumption is that the human in the loop acts lawfully and applies the proper regulatory guidelines. But what if the human disregards or deliberately violates the rules and guidelines? What then? We'd like your view on what our policy should be. 

What if the human in the loop deliberately violates the laws, rules and regulations? 

RESPONSE 

This is a challenging feature of the human-in-the-loop requirement. We have a human-in-the-loop (HITL) policy, which includes conditions, procedures, and suggestions. If you want a copy, please click here.

I will provide some basics. I also provide two matrices outlining key factors. 

A human in the loop protects people from AI only if that human acts in good faith. Almost every AI governance framework treats the reviewer as the fix: the person who catches the model's errors, bias, and hallucinations before they reach a real decision. Very few ask what happens when the reviewer is the source of the harm. 

I will look at how a reviewer can misuse AI output, where the law currently places responsibility, and how to design oversight that still works when one person in the chain does not. 

That gap matters more each year. Federal and state regulators across the US are writing human review into law as the core protection for people affected by automated decisions. But if the review step can be captured, coerced, or corrupted, the law's main safeguard becomes a liability shield instead: an AI output with a human signature looks more legitimate, not less. 

Five Ways toward HITL obstruction 

Most oversight design assumes the HITL reviewer's failure mode is fatigue or over-trust, not intent. A reviewer acting unlawfully exploits exactly that assumption. The patterns below are distinct, and each defeats a different control. 

Pattern	What it looks like	Why standard oversight misses it
Laundering	A decision already made for an unlawful reason (bias, retaliation, a bribe) is run through the AI until it produces a supporting output, which becomes the official rationale.	The record shows a model recommendation and a human sign-off, which is what auditors look for.
Selective override	The reviewer accepts the model when it agrees with their preference and overrides it when it doesn't, targeting particular people or groups.	Overrides are expected - each one looks like legitimate judgment in isolation.
Cherry-picking	The reviewer regenerates or reruns outputs, or swaps prompts and inputs, until a favorable answer appears, and records only that one.	Logs often capture the final output, not the attempts before it.
Tampering	The reviewer edits the AI output or the input data, then presents the result as the system's own finding.	Downstream readers assume the text or score came from the model unaltered.
Rubber-stamping by design	A manager directs reviewers to approve outputs without real scrutiny, or sets quotas that make scrutiny impossible, so the organization can claim human review occurred.	The human step exists on paper, so the system appears compliant.

The common thread is that the AI's apparent objectivity becomes cover for obstruction. A biased manager acting alone leaves a trail of personal judgment. The same manager acting through an AI system can point to the model, while pointing regulators to the human review.

What the law currently expects of the human 

The law is converging on one idea: human review must be meaningful, not ceremonial. That standard targets lazy reviewing, but it is also the strongest hook for catching malicious review. 

Federal law

No federal AI statute exists, so existing law does the work. Title VII, the ADA, and the ADEA apply to employment decisions, whatever tool informs them, and ECOA and the FCRA already require adverse action notices in credit and background-check decisions. Federal policy is currently pushing the other way on new rules. Unfortunately, Executive Order 14365 set up a Justice Department task force to challenge state AI laws, and it has intervened against Colorado's law. 

Colorado

In May 2026, Colorado repealed its original AI Act and replaced it with SB 26-189, which applies to consequential decisions from January 1, 2027. After an adverse decision, a consumer can request meaningful human review, and the reviewer must be trained, able to override the system, and must not simply defer to its output. The Attorney General's October 2026 interim rules would go further, requiring independence from improper management steering. That is one of the first legal texts to treat pressure on the reviewer, not just the reviewer's diligence, as a compliance risk. Enforcement timing is tied to pending federal litigation (see, xAI v. Weiser). 

Illinois

Since January 1, 2026, HB 3773 makes it a civil rights violation for an employer to use AI that has a discriminatory effect and requires notice when AI is used in employment decisions. Because the test is the effect, routing a biased outcome through a human sign-off does not cure it. 

New York City

Local Law 144, in force since 2023, requires bias audits of automated hiring and promotion tools and notice to candidates. A December 2025 New York State Comptroller audit found significant enforcement gaps. The audits also test the tool, not the people reviewing its output, which is exactly where a bad actor sits. 

General law still applies

None of these regimes replace existing law. A reviewer who uses AI output to discriminate, defraud, or falsify records is exposed under federal and state anti-discrimination statutes, fraud and records-falsification laws, and professional rules, exactly as if no AI were involved. In fact, Colorado's new law says so expressly for discrimination claims and voids contract terms that try to indemnify a party for its own discriminatory use of the technology. 

Who answers for it 

When an HITL reviewer misuses AI output, liability rarely stops with the reviewer. Responsibility tends to spread across four parties, and each argues for why it should sit elsewhere. 

Party	Likely exposure	Their usual defense	Why it may fail
The individual reviewer	Personal liability for fraud, falsification, discrimination, or misconduct; professional discipline; dismissal.	"I followed the AI" or "I was told to approve it."	Meaningful-review standards assume the reviewer had authority and a duty to disagree; following orders rarely excuses knowing wrongdoing.
The employer or deployer	Vicarious liability for acts in the course of employment; direct liability for inadequate oversight design, training, or logging.	"A rogue employee acted outside their role."	If the system let one person act unchecked, or management set quotas that made real review impossible, the failure is organizational.
Management	Direct liability where they directed or pressured rubber-stamping.	"We had a human review policy."	A policy that is not resourced or that punishes overrides is evidence of the problem, not a defense.
The AI developer	Limited, unless the tool was built or marketed in a way that invited misuse, such as hiding regeneration history or allowing silent edits.	"The tool worked as designed; the customer misused it."	Under Colorado's new law, developers must document intended uses and how to monitor and review the system; poor documentation weakens this defense.

Admittedly, the practical takeaway for organizations is uncomfortable. Having an HITL reviewer does not move liability onto that human. If anything, a documented review step that was easy to subvert can show a court that the organization understood the risk and did not control it. 

Designing oversight that survives a bad actor 

The goal is not to remove trust in reviewers but to make misuse visible and costly. Controls that already exist in finance and audit translate well, because those fields have long assumed that one insider might be dishonest. In my view, there are at least seven ways to implement such controls. 

1.   Log everything, not just the final answer

Record every prompt, input, regeneration, and edit, with timestamps and user IDs. This is the single most effective control against cherry-picking and tampering. It also supports the three-year record retention Colorado now requires. 

2.   Make edits visible

Any change to an AI output should show as a tracked difference from the original, so readers can see what the model said and what the human changed. 

3.   Require reasons for overrides and acceptances

A short written rationale for each decision creates a record you can compare across cases. A reviewer who overrides the model only for one group will show up in the pattern. 

4.   Separate duties for high-stakes decisions

The person who requests a decision should not be the one who reviews it. For the most consequential cases, require a second, independent reviewer. 

5.   Audit the reviewers, not just the model

Track override rates, approval speed, and outcomes by reviewer and by affected group. Unusually fast approvals suggest rubber-stamping; skewed overrides suggest bias. 

6.   Protect reviewers from pressure

Set realistic workloads, never penalize justified overrides, and give reviewers a confidential route to report being steered. Colorado's interim rules point in exactly this direction. 

7.   Give affected people a way in

Notice, explanation, and a right to independent reconsideration let the person harmed act as the final check, as Colorado's SB 26-189 now requires. 

Licensing Requirement

In my view, specific licensing and training should be required, with licensure and compliance in the Nationwide Multistate Licensing System & Registry (NMLS). NMLS itself does not issue, deny, or revoke licenses. The state and federal regulatory agencies retain final authority. NMLS would designate a category for the human in the loop. 

Licensing under the NMLS, with its permanent, unique ID requirements, would allow all banking departments and federal agencies to monitor performance and administrative history.

No control is perfect against a determined insider with broad access. But the combination of complete logs, visible edits, and pattern auditing, plus licensing and training requirements, turns invisible abuse into detectable abuse, which is the realistic standard. 

Conclusion: oversight needs its own oversight 

Human-in-the-loop is a control, and every control can fail or be defeated. The debate so far has focused on reviewers who trust AI too much. The harder case is the HITL reviewer who trusts it exactly as much as is convenient, and uses its apparent neutrality to dress up a decision they had already made. 

Regulators are starting to notice. The shift from "a human was involved" to "a trained, empowered, and independent human meaningfully reviewed this" is the right direction. 

Organizations that build for that standard now, by logging fully, auditing reviewers, and protecting them from pressure, will be ready when the human in the loop goes wrong. 

------------------ 

This article, When the Human in the Loop is the Problem, published on November 5, 2026, is authored by Jonathan Foxx, PhD, MBA, the Chairman & Managing Director of Lenders Compliance Group, founded in 2006, the first and only full-service mortgage risk management firm in the United States, specializing exclusively in residential mortgage compliance.