Methodology
How an AI-native critique is produced & verified
A normal digital journal publishes prose online. An AI-native critique journal does more: the publication unit is an article plus a claim inventory, an evidence map, a scorecard, machine-readable metadata and an audit trail. Critiques are drafted and cross-examined by a roster of specialised large-language-model agents, then approved by a named human editor. The AI agents are synthetic reviewers — they are not independent human peer reviewers and are not represented as such.
Editorial standard. Publish fast. Critique harshly. Verify before release. Correct visibly. Never speculate about motive or misconduct without evidence.
Two production lines, one engine
Critical AI’s critique pieces are generated by the AGI Social Scientist (AGISS), a research engine that enforces three constraints in code, not policy: every claim cites a verbatim source excerpt (provenance); no causal language without an identified design (causal rigour); and every decision is timestamped and tamper-evident (immutable audit).
Evidence reviews
Corpus-level scoping reviews / evidence-gap-maps: AGISS retrieves a corpus on a question, codes every abstract into a matrix with a verbatim rationale span, and counts. Every count re-derives offline from the committed corpus + codings — no model, no network.
Paper critiques
Per-paper post-publication critiques drafted and cross-examined by the multi-agent synthetic-review pipeline below, with severity capped by access basis and a named human editor approving publication.
The per-paper critique pipeline
Paper discovery → publication, in 20 stages.
- 1
Paper discovery
Monitor selected top-tier source journals for new social-science papers on AI/AGI.
- 2
Source-journal eligibility check
Confirm the venue is in scope and record its tier.
- 3
AI/AGI relevance screening
Require AI/AGI as a central object, mechanism, setting, implication or policy concern.
- 4
Metadata extraction
Capture bibliographic record + DOI for an overlay object.
- 5
Source-access & copyright check
Record the lawful basis on which the paper was read; this caps severity.
- 6
Literature-context retrieval
Pull the surrounding AI/AGI social-science literature.
- 7
Claim extraction
Enumerate and classify the paper's central claims.
- 8
Methodological parsing
Reconstruct the design, identification strategy and analysis.
- 9
Multi-agent synthetic peer review
Specialised agents review theory, methods, statistics, reproducibility, literature, policy and ethics.
- 10
Adversarial critique round
Attack the strongest claim as hard as the evidence allows.
- 11
Author-defence simulation
State the strongest fair defence before publishing the critique.
- 12
Meta-review & synthesis
Resolve disagreement and assemble the critique.
- 13
Citation verification
Verify every source the critique cites; block on any unverifiable core citation.
- 14
Reproducibility / auditability check
Assess whether the result could be reproduced or audited.
- 15
Severity calibration
Calibrate severity against the access basis (abstract-only cannot be severe).
- 16
Legal & ethical risk review
Screen for defamation and copyright; strip any motive or character claim.
- 17
Human editorial approval
A named human editor approves before anything is published.
- 18
Author notification
Notify authors and open the right of reply.
- 19
Publication
Publish the critique as a DOI-linked overlay object with machine-readable metadata.
- 20
Post-publication correction, reply & versioning
Corrections are public and versioned; no silent substantive edits.
The synthetic-review roster
17 specialised agents. They produce the claim map, evidence map, adversarial critique, author-defence simulation and risk checks that a meta-review agent then synthesises for human approval.
Claim-extraction agent
Enumerates the paper's central claims and classifies each by type (descriptive, causal, predictive, normative, policy, theoretical, methodological, AI/AGI contribution).
Theory agent
Assesses the conceptual framing, construct validity and the fit between theory and the AI/AGI object of study.
Methods agent
Examines research design, identification strategy, sampling and the match between design and the strength of the conclusions drawn.
Statistics / analytics agent
Checks statistical and analytical validity: estimation, inference, robustness, multiple comparisons and the reporting of uncertainty.
Qualitative-methods agent
Evaluates qualitative design, coding, saturation, reflexivity and transferability where the paper uses interpretive methods.
Reproducibility / auditability agent
Assesses whether the result could be reproduced or audited: data and code availability, materials, and the specificity of the methods.
Literature-context agent
Places the paper against the existing AI/AGI social-science literature and tests whether the contribution is situated or asserted.
Citation-integrity agent
Verifies that the paper's own cited sources exist and support the propositions they are attached to, and that the critique cites nothing it has not checked.
AI/AGI relevance agent
Tests whether AI or AGI is a central object, mechanism, setting, theoretical implication or policy concern — or merely mentioned.
Policy-impact agent
Assesses how the paper is likely to be read by policymakers and whether its policy implications exceed its evidentiary base.
Ethics & societal-risk agent
Surfaces ethical, legal and societal implications and the risk of public over-interpretation.
Adversarial critique agent
Attacks the paper's strongest claim as hard as the evidence allows, then states the single strongest critique that survives scrutiny.
Author-defence agent
Argues the paper's case in good faith — the strongest fair defence an author could reasonably make — so the critique is tested before it is published.
Overclaiming agent
Rates each claim for over- and under-claiming and identifies where the abstract or framing outruns the design.
Legal-risk agent
Screens for defamation and copyright exposure; flags any statement that drifts from claim/evidence toward author motive or character.
Plain-language agent
Writes the 500–800 word plain-language layer for non-specialist readers, policymakers and the public.
Meta-review agent
Synthesises the agent outputs, resolves disagreement, calibrates severity against the access basis and assembles the final critique for human approval.
Severity is capped by access basis
A severe critique grounded in an abstract alone is not credible. The maximum severity a critique may carry is bounded by the lawful basis on which the target paper was read. This cap is enforced in code and only expert certification lifts it.
| Access basis | What it means | Max normal severity |
|---|---|---|
| Metadata only | Only bibliographic metadata was available. | low |
| Abstract only | Only the abstract was lawfully available; critique limited to framing and stated claims. | moderate |
| Open-access full text | The full text was openly licensed and read in full. | high |
| Licensed full text | The full text was read under a licence permitting the use. | high |
| User-supplied full text | Full text supplied by a user with lawful access. | high |
| Full text + data/code | Full text plus data/code or supplements, with a reproduction or auditability attempt. | critical |
Publication gates
A critique cannot be published unless every gate is complete. These are enforced as runtime invariants over the typed critique object.
- ✓Source-journal eligibility recorded
- ✓AI/AGI relevance justified
- ✓Lawful source-access basis recorded
- ✓Central claims extracted
- ✓Evidence map completed
- ✓Multi-agent review completed
- ✓Adversarial critique completed
- ✓Author-defence simulation completed
- ✓Citation verification completed (no fabricated citations)
- ✓Copyright & quotation check completed
- ✓Defamation / reputational-risk review completed
- ✓Severity calibrated against access basis
- ✓Human editor approves publication
- ✓Author-response mechanism activated
- ✓Version record created
- ✓Machine-readable metadata generated