Comparative quality
Does Policy Window actually match or beat the references it is benchmarked against? For 10subjects, three independent judge agents fetched the live Policy Window article, the live Wikipedia article, and (best-effort) a published review, scoring each 0–5 on six dimensions under a refute-by-defaultmandate to hunt Policy Window's weaknesses.
Headline: Policy Window strongest on 7/10 subjects (3 where it is not); mean overall 4.37 vs Wikipedia 3.65 (0–5). Machine-readable: /wiki/comparative-quality.json.
Where Policy Window falls short (8)
- Citation mix is skewed toward AI-safety/x-risk academic literature (RAND bioweapon red-team, extinction-risk surveys, sleeper-agents) used for the 'does it work' evidence section; this under-weights the legislative/political record that is the core of this subject and reads as somewhat advocacy-adjacent relative to Wikipedia's news-and-record-heavy sourcing of the actual events.
- Does not capture the public-opinion / polling dimension (AIPI, YouGov, Chamber of Commerce, David Binder) that Wikipedia presents with figures and bias caveats.
- Drops the named supporter/opponent coalition almost entirely. Wikipedia names ~25 supporters (Bengio, Hinton, Musk, Hendrycks, Stuart Russell, 113 AI-company employees, 120+ Hollywood signers) and ~20 opponents (Andrew Ng, Fei-Fei Li, Yann LeCun, 9 U.S. Congress members incl. Pelosi, a16z, Meta, OpenAI, CCIA, TechNet); the CACM piece also names both sides. PW reduces the whole 'created its own weather system' fight to 'Anthropic & Meta objected on different grounds' — a serious neutrality/breadth gap.
- Far thinner support/opposition coverage: PW names only 'Anthropic + Meta', whereas Wikipedia names Hinton, Bengio, Musk, SAG-AFTRA and 120+ entertainment figures on one side and Andrew Ng, Fei-Fei Li, Pelosi, Lofgren, Khanna, Andreessen Horowitz, Y Combinator on the other — PW loses the political/coalition story.
- Imprecise on the critical-harm definition: PW states critical harm = 'mass casualties or economic damage exceeding $500M' generally. Per the statute, the $500M floor attaches specifically to the cyberattack-on-critical-infrastructure and autonomous-crime categories; the CBRN-weapon category is mass-casualties with no dollar threshold. Wikipedia states all four categories precisely; PW flattens them.
Where Policy Window falls short (8)
- BREADTH GAP (largest): PW omits the conceptual/technical core that Wikipedia treats at length — intelligence explosion, recursive self-improvement, instrumental convergence, the orthogonality thesis, corrigibility, and the alignment problem generally. It reads as a regulation tracker, not an encyclopedia entry on the subject.
- Citation volume far lower: PW has ~22 distinct citations vs Wikipedia's ~190+; on raw breadth-of-evidence PW cannot match the encyclopedia.
- Conversely, on neutrality PW is solid but its framing is implicitly governance-advocacy ('gap signal' for 20 silent regimes, 'no impact evidence that governance reduces risk') — it adjudicates a policy gap rather than neutrally surveying the underlying scientific debate the way the review does.
- Far less current on the most recent EMPIRICAL x-risk evidence than Wikipedia: PW's empirics stop at 2024 bio-uplift null results (RAND/OpenAI), whereas Wikipedia covers the Dec-2024 Apollo Research o1 deception findings (sandbagging/self-exfiltration in 0.3-10% of scenarios) and the Anthropic/Claude 'alignment faking' result (12%->78%) — directly relevant deceptive-alignment evidence PW name-checks as a risk category but never substantiates with these studies.
- Heavier reliance on arXiv preprints for several load-bearing claims (Shevlane, Anderljung, Schuett, Grace, Hubinger, Scholefield) — defensible and well-chosen, but the Hadshar review and Wikipedia both anchor more on the established Bostrom/Russell/Ord canon PW barely cites.
Where Policy Window falls short (8)
- CONFIRMED ERROR/STALENESS: PW lists US Executive Order 14110 as 'governs/high (§4.5 content authentication, watermarking)' and never acknowledges it was RESCINDED by President Trump on 2025-01-20. Wikipedia maintains a current, dedicated EO 14110 article documenting the revocation. PW's stale status is especially damning because its own 'last verified' date is 2026-05-24 — 16 months after the rescission — and it does list the replacement EO 14179 (as merely 'silent') without ever stating the rescission relationship.
- Citation-verifiability risk: PW leans on bleeding-edge preprints (e.g. Cuevas & Horta Ribeiro 2025 arXiv:2602.02754; Meding & Sorge 2024 arXiv:2412.09961) at least one of which could not be independently confirmed, whereas Wikipedia and the review cite established, checkable sources.
- Detection technology is treated only in passing (a single fault-line sentence citing Harris 2024 on detector limits), whereas Wikipedia devotes a whole subsection to audio/video forensics, blinking artifacts, biometric and blockchain verification.
- Far fewer citations in absolute terms: ~16 sources vs Wikipedia's ~240+ and the MethodsX review's ~140+. PW's are higher-quality (peer-reviewed/primary-law), but the sheer breadth of claim-by-claim backing is lower, and several whole subtopics (techniques, history) carry essentially no citations because they are barely covered.
- Far narrower breadth than Wikipedia: PW gives little-to-no coverage of deepfake HISTORY/origins (Wikipedia traces 1997 Video Rewrite onward), the underlying TECHNIQUES (GANs, autoencoders, latent space — Wikipedia explains these), or benign APPLICATIONS (film VFX, art, satire). PW is a law map, not an encyclopedia of the phenomenon.
Follow-up re-run (2026-06-22): did fixing the losses close the gap?
After the baseline above recorded 3 losses, their specific recorded weaknesses were addressed and the same adversarial methodology was re-run on those 3 subjects. Outcome: 1 flipped to a Policy Window win (deepfakes); the other 2 improved but still lose. On all three, the recorded factual/accuracy weaknesses are resolved (verbatim-verified) — the residual losses are encyclopedia breadth + citation volume, axes the governance-map scope does not target, not errors.
PW 4.06 → 4.33 · Wikipedia 4.11 → 4.28 · weaknesses resolved 2/8 (1 partial)
FLIPPED to a PW win. The confirmed EO-14110 staleness error is fixed (the cell + prose now state the 20 Jan 2025 rescission by EO 14148, EO 14179 silent) and the alarmism/evidence meta-critique is foregrounded; PW now leads on source quality, currency and neutrality. Residual losses are breadth/citation-volume (no history/techniques section) — inherent to the governance-map scope.
PW 4.19 → 4.28 · Wikipedia 4.33 → 4.5 · weaknesses resolved 2/8 (2 partial)
Still loses, but the factual gap is closed: a 'mechanisms and evidence' section now grounds the debate (specification-gaming vs goal-misgeneralization vs power-seeking; Apollo scheming + Anthropic alignment-faking) with the honest no-public-power-seeking-example caveat — judged MORE precise than Wikipedia on the mechanism distinctions. PW leads claim-accuracy + currency; the loss is the conceptual-canon breadth (intelligence explosion, instrumental convergence, the Bostrom/Russell/Ord canon) PW does not aim to be.
PW 3.75 → 3.89 · Wikipedia 3.97 → 4.56 · weaknesses resolved 2/8 (5 partial)
Still loses (the gap widened as the judge panel rated Wikipedia's record-completeness higher this run), but the two most damaging accuracy defects are fixed: the central-mechanism distortion (now correctly a safety-protocol + self-cert bill with 2026 third-party audits, not a pre-deployment testing mandate) and the critical-harm threshold precision, plus a real named coalition. Residual loss is encyclopedic record-completeness (vote counts, public-opinion polling, full coalition) — Wikipedia's strength, not a PW error.
Same self-administered adversarial methodology as the baseline: 3 refute-by-default judge agents per subject fetched the LIVE post-fix Policy Window article and the LIVE Wikipedia article, scored each 0-5 on the 6 dimensions, and verified (verbatim) whether each recorded baseline weakness is now resolved. Not an independent study; absolute scores shift across judge panels, so the durable signals are the weakness resolution and the within-run direction.
Method & honest limits
self-administered adversarial head-to-head with controls: per subject, 3 independent judge agents fetched the LIVE Policy Window article, the LIVE Wikipedia article, and (best-effort) a published review/survey, then scored each 0-5 on 6 dimensions under a refute-by-default mandate to hunt Policy Window weaknesses. Means across judges. Two scopes of comparison are reported and are NOT the same: the per-subject outcome label ('PW strongest' / 'PW not strongest') is computed against the strongest of ALL sources (Wikipedia AND the published review), whereas the headline 'X/10' tally and mean-vs-Wikipedia figure are PW-vs-Wikipedia only. In this snapshot the two coincide (every win also beats the review; every loss is to Wikipedia), but a future snapshot where PW beats Wikipedia yet trails the review could diverge. NOT an independent third-party study: the judges are Policy Window-run LLM agents, so treat as a self-assessment with adversarial controls, not external validation.
The recurring weakness pattern: Policy Window is strongest on primary-law citation density, currency, and governance depth, but Wikipedia is broader on conceptual/historical scope and named contestation (most visibly on catastrophic-risk, which reads as a governance tracker rather than a full encyclopedia treatment of the subject). The follow-up re-run above closed the recorded factual gaps on the three losing subjects (one flipped to a win); the remaining conceptual/historical breadth gap is treated as an intentional scope boundary of a governance-research engine, not a defect to erase.