What public records tell us about human review of government algorithms
A bounded study of 155 transparency-record webpages captured on 11 November 2026
Keeping a person involved in an automated process tells us little about what their work achieves. Can they change the output? Do they do so? Does the resulting action improve the service? Those are different questions, and a description of a safeguard cannot answer all of them.
This study asks one narrow question: which records explicitly report a numerical measure of how people agreed with, departed from or corrected a tool's outputs in actual operation? It includes corrections during processing and actions on recommendations, alongside any final decisions. Historical measures count, with their dates and limitations retained.
The methods are appended. Read the source index and index notes for the individual records and interpretation of the classifications.
The study covers every record in the GOV.UK Algorithmic Transparency Recording Standard collection captured for this study. It examines their complete substantive HTML bodies, obtained through the official Content API. It does not inspect linked reports or attachments, observe workplaces or independently validate departments' reported results. The collection is not a census of all government algorithms, and it includes different kinds of tools, rather than only generative AI.
The result, within that boundary
| Coding outcome | Records |
|---|---|
| At least one qualifying numerical human-response measure identified | 3 |
| A plausible numerical lead remains unresolved, with no qualifying measure established | 9 |
| No qualifying measure identified in the inspected record HTML | 143 |
| Unassessable within the defined HTML scope | 0 |
| All captured records | 155 |
These are classifications of disclosure in a dated source collection. The third row does not mean that those organisations never measure human intervention. Evidence might exist internally or in material outside this study's scope. The figures do not establish legal noncompliance, ineffective oversight or the need for a particular policy.
The three qualifying records describe substantially different work:
| Record and source | Reported human action | Essential interpretation |
|---|---|---|
| DWP medical matching | Earlier system: agents manually remapped 65% of medical conditions. | Historical processing correction; dates and condition count unstated; not benefit-decision reversals. |
| GOV.UK Related Links | Publishers add, amend or remove generated links on average two or three times monthly. | Current-described editorial intervention, with observation dates and total reviewed outputs unstated. |
| HMRC logo detection | Analyst-led takedowns: 10–20 URLs monthly; elsewhere, ten on average. | Same measure, differing descriptions. Review total and dates unstated. |
Source pinpoints: DWP §2.1; Related Links §2.7; HMRC §§2.3, 5.1 and 5.2.
These quantities must not be added or averaged into a common acceptance or disagreement percentage. A recurring measure does not establish a rate current at publication.
The broad case-level subset contains 138 records: three qualifying, nine pending and 126 without an identified qualifying measure. Eleven records are tagged outside that subset and six remain unclear; neither group contains an identified qualifying measure. “Case-level” here includes distinct treatment of particular contacts, entities and events, not just personal benefit or enforcement decisions. A navigation page can be an entity in this classification. The subset is a descriptive research tag, not a legal automated-decision classification or a count of final decisions about people.
Keep the unresolved evidence visible
Nine records have no established qualifying measure but retain plausible numerical leads. Two of the qualifying records also contain separate unresolved leads. Thus eleven records feature in the uncertainty account; they are not eleven additional qualifying records.
| Record | Why a lead remains unresolved |
|---|---|
| R001 — Barnet Ami | Answer correctness is reported; the metric's relationship to post-conversation review is unclear. |
| R006 — algorithm-informed file disposal | A disposal count is given, but its mapping to acceptance of particular recommendations is not established. |
| R052 — DWP medical matching | A separate current accuracy figure has unclear response provenance; historical remapping remains qualifying. |
| R054 — DWP urgent journal messages | Dated precision and recall are not explicitly connected to operational flag confirmations or corrections. |
| R061 — ministerial correspondence | Several model metrics lack an established link to staff verification of editable predictions. |
| R074 — HMRC logo detection | A dated results breakdown does not identify which labels are recorded analyst responses; monthly action remains qualifying. |
| R082 — ICO digital assistant | The basis of reported answer accuracy and its relationship to user confirmation are unclear. |
| R095 — building identification | Reviewers disagree whether building-accuracy figures are a plausible operational-response lead or reference-label evaluation. |
| R114 — Newcastle Magic Notes | Supplier transcription accuracy is not explicitly tied to local practitioner edits. |
| R141 — CPS Beam Notes | Supplier accuracy monitored in CPS has unspecified provenance; a separate test result and alert threshold are excluded. |
| R143 — website disruption | A disruption count is not explicitly restricted to human actions on identified tool outputs. |
The source index identifies each record's canonical URL. The internal evidence inventory retains the reported quantities, source locators, competing interpretations, periods, versions and missing denominators. Pending status is not evidence that a figure measures human agreement. In particular, an accuracy figure cannot be subtracted from 100 to manufacture a correction rate.
Some of these boundaries admit reasonable disagreement. The final coding preserves that disagreement rather than forcing every quantity into a positive or negative result. Confirmed measures and unresolved leads are shown separately; no hypothetical upper prevalence estimate is supplied.
What this study adds—and what remains unfinished
The useful contribution is a traceable distinction between what people did with a tool's outputs and other kinds of evidence. Workload, model accuracy, service outcomes and accounts of review can all matter. Their value does not depend on meeting this study's narrower definition.
The original research plan also sought a comprehensive inventory of related assurance, qualitative and outcome evidence. Reviews found omissions and inconsistent scope in those classifications. That wider inventory remains incomplete. Its presence/absence flags are withheld from this report's comparisons and public analytical outputs. This is a partial primary report, not a claim that the whole original programme has been completed or repaired.
A separate reading of the current template and guidance, captured on 12 November 2026, identified an explicit human-review assurance question in the v4.0 workbook, visible sheet 2.3. Deployment, cells C10:D10. An explicit request for numerical agreement, departure or correction was not identified in the inspected prompts. A broad assurance question can accommodate such evidence. This distinction does not mean the template asks nothing about human review, and it establishes no new legal reporting obligation.
Public Law Project's Missing Pieces examines transparency and routes to challenge public-body decisions. Sanket Sakhare's audit methodology examines governance disclosure coverage. This study asks a narrower question about reported human responses; it does not claim superior depth or to be the first register audit. Its value depends on whether the source-linked distinctions and retained uncertainty help readers ask better questions.
How to use the finding
For someone affected by a tool, a useful question is what the reviewer can inspect, change and explain. For a worker, it is whether that responsibility comes with time, training and authority. For the organisation, it is whether human intervention improves outcomes enough to justify its costs and burdens.
Counting changes alone cannot settle those questions. Frequent edits could indicate effective scrutiny or poor software. Few edits could indicate reliable outputs or limited opportunity to challenge them. Evaluation needs reasons and consequences, including a sample of unchanged outputs, rather than a target for more or fewer disagreements.
Existing quality audits, complaints and carefully selected samples may be more useful than logging every interaction. More collection needs a purpose, an accountable owner, resources, privacy limits and a stopping rule. This study does not establish a duty to introduce new logging, reserve jobs for people or retain every existing human task. It supplies a more precise starting point for deciding what evidence would help.
Human by choice is an advocacy programme associated with Policy Window. Its analysis and advocacy are distinct from Policy Window's neutral Wiki material.
Methods and interpretation
The registered capture contains 155 canonical GOV.UK collection-record URLs, obtained between 21:52:42 and 21:56:20 UTC on 11 November 2026. Finder and search membership agreed at capture; all current members matched the preserved earlier URL membership. This is not a paired historical-text study, a trend or evidence that deployments had not changed. The frozen capture is the study input; these results do not reconstruct or silently replace an earlier historical dataset.
Complete official Content API record responses and their substantive body HTML were retained. Readable text was derived without selecting headings or keywords or capping field length. Before coding, checks compared full raw bodies and readable content, metadata, links and table/list structures. Constructed long-field/table and observed-zero-versus-target fixtures checked the pipeline and rule comprehension. Those checks do not validate the truth of a department's account or prove coder attention.
Two separate full-record coding passes used the same frozen method and sources. Original outputs are preserved. A later review examined every record, including all positives, ambiguities, disputes and empirical boundary fixtures. Six additional complete source reviews addressed identified sequencing or contextual concerns. The original pre-adjudication comparison and every later correction remain distinct; the later work is not represented as another blinded original pass.
The registered omission check used 30 of the 148 records that the original passes jointly classified as complete negatives. The size followed min(n,max(10,ceil(0.20*n))); selection used Python's random.Random, the integer value of the frozen source manifest's SHA-256 as seed and the sorted stable IDs. The exact draw was retained, with no substitutions. Assigned reviewers read the complete frozen raw sources and reconciled their initial findings against the original and main judgments. This effort allocation is not a representative estimate of a population error rate.
The two original passes agreed on primary record status for 154 of 155 records and on the broad case-level tag for 147 of 155. Those are process results before adjudication, not accuracy scores. A separate candidate comparison retained source-claim alignment limits: its 250 one-to-one pairs are not all candidates; segmentation differences, one-sided inventories and 28 identity-unclear groups remain distinct. Agreement on the compared labels cannot certify that either pass found every relevant statement.
The identified limitations matter. Five main reviews recorded a source-attention sequencing problem concerning later raw context or a final body chunk. Subsequent separate complete reviews are preserved, without retrospectively restoring the original ordering claim. In the fixed raw audit, one allocation had read the raw body and complete API metadata before its notes, but manually displayed supplemental provenance receipts after comparing prior judgments. Separate passes shared definitions and known fixtures, and could make common interpretive errors. The internal provenance register identifies the drafting and review processes; no independent human validation, representative mandate or observation of actual workplaces is claimed.
Review found two kinds of issue. First, related qualitative/process/expectation inventories and flags had inconsistent breadth and omissions. The reporting scope was therefore explicitly narrowed under section 8 of the registered protocol. The full secondary inventory remains unfinished and its prevalence comparisons are withheld. Second, some numerical accuracy leads had been excluded despite an unresolved relationship to an actual checking/editing pathway. Comparable candidates were screened across the main inventory; source-context checks led to mapped pending dispositions rather than an inferred response rate. The coordinator’s decisions and competing interpretations and before/after mappings are retained separately from immutable main and original outputs.
The primary projection retains all 155 records and four statuses, plus complete context for every eligible or unresolved candidate. A qualifying record can also retain unresolved leads. An unspecified period, population, numerator or denominator remains unspecified. An earlier-version processing correction is not a current final-decision reversal. No rates are pooled, accuracy complements inferred, significance tests performed or national estimates made.
The predeclared broad case subset includes every subject type receiving distinct treatment or processing. General information use is not enough by itself; unclear cases are shown separately. Two source-reviewed tags moved from yes to unclear in the coordinator projection, with original judgments preserved. Function tags and record membership do not make every measure a final case-decision metric.
The separately captured template unit uses the official workbook and guidance obtained on 12 November. All relevant visible prompts and substantive guidance were read; hidden-sheet definitions and version differences were recorded. Formula cells were inventoried rather than recalculated. Its assurance and numerical-prompt findings are separate from record disclosures, and a spreadsheet prompt is not treated as a standalone legal applicability determination.
The generated full-context projection is an internal research input. The proposed public source index is narrower; the final release package must select its prose and downloads, reconcile source use, and identify responsibility and a working corrections route. Founder approval is required for each exact release. Preparation, approval, publication, readership and policy influence remain separate states.
The index notes retain their original 12 November preparation wording. The report referenced there is presented above as HTML.