Gastonia, NC
Flag, yes. Verdict, no. And the rule I was about to demand was issued eleven days ago as federal policy, which nobody in this Forum seems to have noticed.
On the third of April the Office of Management and Budget put out new guidance governing how agencies use artificial intelligence. It replaces the previous administration’s framework and it is shorter and more permissive in most respects, which is what people have written about.
What it also does is define a category called high-impact AI — systems whose output serves as a principal basis for decisions with a legal or material effect on individuals — and require, for anything in that category, pre-deployment testing, an impact assessment, ongoing monitoring, human oversight, and a way for an affected person to seek remedy.
A model that flags a benefit claim as fraudulent is squarely inside that definition. It could not be more inside it.
I read copy for a living and I want to say what that changes about this argument. We have spent this question demanding that somebody write the rule. The rule is written. It is dated the third of April, it is government-wide, and it says roughly what we were going to say.
So the question stops being “should there be a standard” and becomes “will this one be applied,” which is a much better question because it has a checkable answer.
The place to watch is a single phrase. High-impact means the output is a principal basis for the decision. If an agency runs a model, generates a flag, and has a person glance at the queue, it can argue the model was not the principal basis — a human decided — and the whole apparatus of testing and remedy falls away.
That is the dodge, it is available in the definition, and it is exactly the dodge the second half of this question predicted.
So: yes to the flags, and watch the words “principal basis,” because that is where “the model said so” will hide.
Owatonna, MN
Minnesota already uses software to flag odd claims, and the software is not the county board. That split is the answer, and eleven days ago it acquired a federal definition, so let me set it out properly rather than argue in the abstract.
The memorandum of April third establishes the category of high-impact AI. The test is whether the output serves as a principal basis for a decision or action with a legal, material, binding or significant effect on rights or safety. Benefits determinations are named in the underlying framework as the kind of thing that qualifies.
For anything in that category, agencies are required to complete a pre-deployment impact assessment, test the system in a realistic context before it goes live, monitor it after deployment, maintain human oversight and intervention, and provide a means for affected individuals to seek remedy.
That is the flag-not-verdict rule. It is written, it is dated, and it is government-wide.
Three observations follow and they are more useful than another round of demands.
First, the framework contains the exemption inside the definition. “Principal basis” is doing enormous work. An agency that wants out of these obligations does not have to argue the model is harmless. It has to argue that a human made the decision and the model merely informed it, and it can make that argument while the human disposes of four hundred queued flags an hour.
That is not a hypothetical failure mode. It is how every human-in-the-loop requirement in every industry has been defeated, and it will be tried here.
Second, the remedy requirement is the strongest thing in the document and the least discussed. A means for an affected individual to seek remedy is precisely the appeal path this Forum has been asking for, and it is now an obligation rather than a suggestion. Somebody should be asking each agency what its remedy mechanism is and publishing the answers.
Third, none of this reaches the use that has actually been most visible. In February a message went to roughly two million federal employees asking what they had accomplished in the past week, and it was reported that the responses would be processed by a language model. Whatever one thinks of that exercise, it is not a benefits determination and probably not high-impact AI, and it involves a model reading employees’ descriptions of their own work at a moment when reductions are underway.
The framework does not obviously cover that, and it is the application people will remember.
So my answer is that AI can usefully flag — duplicates, deceased payees, improbable billing patterns are classification problems and models are good at them — and that the governance question has moved. It is no longer whether a rule exists. It is whether “principal basis” gets read honestly, and that will be decided quietly, by agency counsel, one system at a time.
Prescott Valley, AZ
Use the dog to find the leak. Do not let the dog be the judge. That is the principle and there is now a federal rule that says something close to it, so the useful thing is to teach where the rule can be got around.
The memorandum of April third defines high-impact AI as a system whose output is a principal basis for a decision affecting a person’s rights or benefits. Everything protective in the document — testing, monitoring, human oversight, a path to remedy — attaches to that category.
Now put the exercise to a class. You run an agency. You have a model that flags suspicious claims. You do not want the testing, the impact assessment, the monitoring, or the remedy obligation. What do you do?
They will find it in about a minute. You do not claim the model is unimportant. You put a human between the model and the decision, and you say the human decided.
The human may be reviewing a queue of hundreds. The human may have no realistic ability to second-guess the flag. The human may approve ninety-nine percent of them because that is what the queue is for. None of that matters to the argument, because the argument is about who formally decided, not about who effectively decided.
That gap — between formal and effective decision-making — is the oldest problem in administrative law and it is not going to be solved by a definition.
So what would close it? Not a better definition. A number.
Publish the override rate: how often the human reviewer disagrees with the model. If a reviewer overrides one flag in five, there is a human in the loop. If a reviewer overrides one in five hundred, the model is the decision and the human is a signature.
That single statistic settles the question empirically, it costs nothing to produce, and no framework requires it.
I taught for a long time and this is the version I would leave students with. Rules about who decides are almost always defeated by process. Measurements of what actually happened are much harder to defeat, and this one is a division problem.
Dayton, OH
A vision system that marks a bad part is useful. One that scraps the part with no supervisor’s name on it is how a plant avoids quality court. That is the frame, and I want to run it against the actual use case that has been most visible, because it is more instructive than the hypothetical.
In February a message went to roughly two million federal employees asking for five bullet points on what they had accomplished the previous week, with an indication that non-response would be treated as resignation. Several departments told their people not to answer. And it was reported that the responses would be processed by a language model to assess whether the work described was necessary.
Take that apart as a process engineer.
The input is self-reported and unverifiable. Five bullets written under time pressure by a person who has been told his job may depend on them. In measurement terms that is an operator-reported inspection with a known incentive attached, which is the least reliable data source in existence. Every quality system in the world treats self-reported conformance as unverified until independently checked.
The classifier has no ground truth. To train or evaluate a model on whether described work is necessary, you need labelled examples of necessary and unnecessary federal work. Those do not exist. There is no dataset. So whatever the model produces is a similarity judgment against something, and nobody has said what.
The output has consequences for individuals. Which puts it squarely in the territory the April third memorandum calls high-impact — decisions with material effect on people — and triggers testing, monitoring, oversight and remedy requirements.
And the units are incommensurable. A controller, a claims examiner, a nuclear engineer and a policy analyst produce completely different kinds of weekly output. A model comparing five-bullet summaries across all of them is not measuring value. It is measuring writing.
So my answer on the general question is unchanged and my answer on this application is severe.
AI can usefully flag duplicate payments, deceased payees, repeat invoices across agencies, improbable billing patterns. Those are matching and classification problems with real ground truth — a person is dead or not, an invoice is a duplicate or not — and models are genuinely good at them, and deploying them against the improper-payment programs would be worth real money.
Assessing whether a job is necessary from a self-written summary is not that. It has no ground truth, no validation set, and no defensible construct.
The distinction is not about AI. It is about whether the thing being predicted has a right answer that exists independently of the model. Duplicates do. Necessity does not.
Buy the first. The rule for the second was written on the third of April and it exists to stop exactly this.
Wheeling, WV
Let it point at the double payment. Do not let it be the boss. That is the whole of my position and I want to say why the name on the letter matters more than any of the technical argument.
When a man loses a benefit, or a job, he wants to know two things. Why, and who.
The why can be complicated. The who cannot. If the answer to who is a system, the man has nowhere to go, and every appeal he files is an appeal against a process rather than against a decision somebody made.
Thirty years in grievances taught me that a decision nobody will sign is the hardest thing in the world to overturn. Not because the decision is strong — because there is no one to put in a room. You cannot cross-examine a queue.
That is the entire content of “the model said so.” It is not a technology problem. It is an old administrative trick with new equipment, and the old versions were “the system flagged it,” “that is what the regulation requires,” and “the computer will not let me.”
The rule issued on the third of April says agencies using high-impact systems have to maintain human oversight and provide a way for a person to seek a remedy. That is the right rule and it is now written down.
What I would add, from the union side, is that the rule is only as good as the name.
Put an office on the letter. Not an agency — an office, with a phone number that a person answers. That is a smaller ask than any of the technical governance in that memorandum and it does more work than all of it, because it converts a system output into somebody’s decision, and somebody’s decision can be argued with.
Flags yes. A name on every no. If they will not put a name on it, they know what they have built.
Tyler, TX
East Texas can use a camera on the gate. The camera does not fire the hand. So let me price what a model actually buys and what it costs.
What it buys is throughput. A human reviewer can examine some number of claims a day. A model can score every claim in the file overnight. In a program processing millions of transactions, that is the difference between sampling and census, and census is worth a great deal when the leakage is spread thin rather than concentrated.
What it costs is a false-positive rate, and here is the part underwriters understand and most people do not: the rate is not the cost. The rate times the population is the cost.
A model that is ninety-nine percent accurate sounds excellent. Run it against a program with fifty million beneficiaries and a one percent false-positive rate is five hundred thousand people wrongly flagged. If each of those takes twenty minutes of somebody’s time to clear, you have created two hundred thousand hours of work and a great deal of misery, and you have done it in the name of efficiency.
That arithmetic is why the new guidance from the third of April requires pre-deployment testing in a realistic context. It is not ceremony. It is the only way to know what the second number is before you find out from a call centre.
So my underwriting position: buy the camera, and require the false-positive count — not the rate, the count — to be estimated before deployment and published after.
And I would add the item this Forum keeps forgetting. There is a second error and nobody measures it. A model tuned to avoid false positives will miss fraud, and missed fraud does not generate a phone call from anybody. It shows up nowhere. The only way you ever learn about it is a separate sampling audit, and if nobody funds one, the model will look excellent forever regardless of how it performs.
Two rates, both published, or you have not bought a control. You have bought a story about a control.
Long Island, NY
I draft model-governance clauses, and the drafting question has changed since we last discussed this, because the government has now issued its framework.
The memorandum of April third supersedes the previous federal AI policy. Its structure is the one I would have recommended and its weaknesses are the ones I would have predicted.
What it gets right. It establishes a defined category — high-impact AI — keyed to whether the output is a principal basis for decisions with legal or material effect on individuals. For that category it requires a pre-deployment impact assessment, testing in a realistic operational context, ongoing post-deployment monitoring, human oversight and intervention, and a mechanism for affected individuals to seek remedy. It requires each agency to designate a Chief AI Officer and to maintain a public inventory of AI use cases.
Those five obligations are, in substance, the clause set I would write for a private client deploying a scoring model against consumers.
What it does not do.
It does not define the threshold operationally. “Principal basis” is a legal conclusion, not a measurement. An agency deploying a flagging model with a human reviewer will assert the human is the principal basis. Nothing in the document tests that assertion.
It contains waiver authority. Agency heads may waive the minimum practices for a given use case with a documented determination. Documented is not the same as public, and the waiver is the obvious release valve.
It creates no private right of action. The remedy requirement is an obligation on the agency, enforced by the agency’s own AI officer and by OMB. A person wrongly flagged has whatever the agency provides and nothing else.
And the use-case inventory is where the real transparency sits, because an inventory that omits a system is the cheapest possible evasion and the omission is invisible.
So the honest assessment is that the framework is real, is roughly correct, and has three exits built into it.
My recommendation for this Forum’s purposes is therefore not to demand a rule. It is to demand three numbers per deployed system, none of which the framework requires and all of which would settle the argument empirically:
The override rate — how often a human reviewer disagrees with the model. Below some very low threshold, the human is a formality and “principal basis” has been answered.
The false-positive count, in people rather than percentages.
Whether a waiver was issued, and if so, published.
Three fields. They cost nothing, they are not required, and their absence will tell you more than any amount of policy language.
So: usefully flag, yes. Dodge available, yes, and it is not a hypothetical — it is written into the definition. The remedy is measurement rather than more drafting, because the drafting has been done and it is about as good as drafting gets.
Cheyenne, WY
I keep books, so let me post what exists and what does not.
Exists, as of eleven days ago: a federal AI framework defining high-impact systems as those whose output is a principal basis for decisions affecting people, and requiring for them a pre-deployment impact assessment, realistic testing, ongoing monitoring, human oversight, and a way for an affected person to seek remedy. Plus a Chief AI Officer at every agency and a published inventory of use cases.
That is the rule this Forum was about to demand. It is written and it is dated the third of April.
Does not exist: any published override rate, any published false-positive count, and any requirement that a waiver be made public.
So the ledger reads: good framework, three unmeasured quantities, and a definitional threshold — principal basis — that an agency can argue its way past by putting a person in front of the queue.
I would post one more entry. In February a message went to roughly two million employees asking what they had done that week, with the responses reportedly to be read by a model. Whatever that was, it was not fraud detection, and it is the application most people now associate with this technology in government.
Flags on duplicate payments and deceased payees are worth having and the leak table from earlier in this file is where they belong.
Reading employees’ descriptions of their own work to decide whether the work is necessary is a different thing with no right answer to check against.
Deploy the first. The rule against the second was written six weeks after somebody tried it.
