When the Human in the Loop is the Problem

The safeguard nobody audits!

   


QUESTION 

We have been transitioning to AI use in several departments. As a board member with governance responsibilities, I am interested in the human-in-the-loop responsibility, which you discussed in Will AI Reduce Fair Lending Violations? We discussed your April 1st article, AI Replace Me. It was a sequel to your article, Will AI Replace Me? 

In your article, you discussed the future role of humans with regard to AI. You said that we should have a "human-governed" approach to AI. The matrix you provided on the roles and the corresponding human oversight requirement for AI was eye-opening. Accordingly, we are developing a human-in-the-loop policy. 

In our last board meeting, someone made an interesting point. The presumption is that the human in the loop acts lawfully and applies the proper regulatory guidelines. But what if the human disregards or deliberately violates the rules and guidelines? What then? We'd like your view on what our policy should be. 

What if the human in the loop deliberately violates the laws, rules and regulations? 

RESPONSE 

This is a challenging feature of the human-in-the-loop requirement. We have a human-in-the-loop (HITL) policy, which includes conditions, procedures, and suggestions. If you want a copy, please click here.

I will provide some basics. I also provide two matrices outlining key factors. 

A human in the loop protects people from AI only if that human acts in good faith. Almost every AI governance framework treats the reviewer as the fix: the person who catches the model's errors, bias, and hallucinations before they reach a real decision. Very few ask what happens when the reviewer is the source of the harm. 

I will look at how a reviewer can misuse AI output, where the law currently places responsibility, and how to design oversight that still works when one person in the chain does not. 

That gap matters more each year. Federal and state regulators across the US are writing human review into law as the core protection for people affected by automated decisions. But if the review step can be captured, coerced, or corrupted, the law's main safeguard becomes a liability shield instead: an AI output with a human signature looks more legitimate, not less. 

Five Ways toward HITL obstruction 

Most oversight design assumes the HITL reviewer's failure mode is fatigue or over-trust, not intent. A reviewer acting unlawfully exploits exactly that assumption. The patterns below are distinct, and each defeats a different control. 

Pattern	What it looks like	Why standard oversight misses it
Laundering	A decision already made for an unlawful reason (bias, retaliation, a bribe) is run through the AI until it produces a supporting output, which becomes the official rationale.	The record shows a model recommendation and a human sign-off, which is what auditors look for.
Selective override	The reviewer accepts the model when it agrees with their preference and overrides it when it doesn't, targeting particular people or groups.	Overrides are expected - each one looks like legitimate judgment in isolation.
Cherry-picking	The reviewer regenerates or reruns outputs, or swaps prompts and inputs, until a favorable answer appears, and records only that one.	Logs often capture the final output, not the attempts before it.
Tampering	The reviewer edits the AI output or the input data, then presents the result as the system's own finding.	Downstream readers assume the text or score came from the model unaltered.
Rubber-stamping by design	A manager directs reviewers to approve outputs without real scrutiny, or sets quotas that make scrutiny impossible, so the organization can claim human review occurred.	The human step exists on paper, so the system appears compliant.

The common thread is that the AI's apparent objectivity becomes cover for obstruction. A biased manager acting alone leaves a trail of personal judgment. The same manager acting through an AI system can point to the model, while pointing regulators to the human review.