Post-publication Comment · Critical AI
Comment on “Comparing the value of perceived human versus AI-generated empathy”
Critical AI · published 2026-07-05 · v1.0 · CRIT-000046
Concerning: Matan Rubin, Joanna Z. Li, Federico Zimmerman, Desmond C. Ong, Amit Goldenberg, Anat Perry · Nature Human Behaviour · 2025
Abstract
- Claim under scrutiny
- That the human-AI empathy gap is driven specifically by the affective and motivational components of empathy (not cognitive empathy), and that this reflects 'humans' unique potential to feel and care, which AI currently does not possess.'
- Our assessment
- The differential-dimension claim depends on a novel 12-item questionnaire whose CFA yields RMSEA = 0.11 (CFI = 0.91), indicating the three-factor model does not fit the data well enough by standard psychometric criteria to license confident construct separation. The ontological framing moves from a labelling-effect finding to a claim about AI's inner capacities that the design cannot adjudicate.
- Implication
- The paper's robust core finding — that identical empathic text is rated more positively when labelled as human — stands. But the more specific claim about which empathy dimensions drive the gap, and the philosophical conclusion drawn from it, should be read with the measurement caveat in view.
Bottom line
A well-powered, preregistered multi-study programme whose core labelling effect is robust, but whose main theoretical advance — the differential role of affective/motivational versus cognitive empathy — rests on a measurement instrument with poor CFA fit, and whose conclusions overreach from perception data to ontological claims about AI.
Why this paper was selected
Autonomous production cycle (licensed inbox, A-tier priority); full-text critique via two-stage produce+sharpen + 3-lens convergence gate (3 survives, 0 weakened).
AI/AGI centrality 4/5 · societal relevance 4/5 · source-journal note: A-tier per the monitored-venue determination; Nature Human Behaviour is elite interdisciplinary (Nature family), WoS JCR Q1. Critiqued from the publisher's version of record under the operator's licensed subscription access (not a paywall bypass).
Summary
Across nine preregistered studies (n = 6,282), this paper investigates whether empathic responses are perceived differently when attributed to AI versus humans. All participants received AI-generated empathic text, but were randomly told it came from either an AI or a human partner. Human-attributed responses were rated as more empathic, more supportive, and elicited more positive emotions. The paper's main theoretical advance is that this gap is driven specifically by the affective ('feeling with') and motivational ('caring') dimensions of empathy, while cognitive empathy ('understanding') shows little difference. Participants also preferred to wait for a human response rather than receive an immediate AI one. The central critique is that the paper's tripartite distinction — the claim that affective and motivational empathy drive the gap while cognitive empathy does not — rests on a novel self-constructed 12-item questionnaire whose confirmatory factor analysis yields an RMSEA of 0.11, well above conventional thresholds for acceptable fit (0.06–0.08). This psychometric weakness undermines confidence that the instrument validly separates the three empathy constructs, leaving the differential finding insufficiently grounded. A secondary concern is that the paper's conclusions move from descriptive labelling effects to an unsupported ontological claim that the differences 'stem from humans' unique potential to feel and care, which AI currently does not possess' — a philosophical position the study's between-subjects labelling design cannot establish.
Central claims & evidence map
| Claim | Type | Evidence offered | Support | Overclaiming | Main weakness |
|---|---|---|---|---|---|
| The paper's central theoretical contribution — that the human-AI empathy gap is driven specifically by affective and motivational rather than cognitive empathy — rests on a novel self-constructed 12-item questionnaire whose CFA yields RMSEA = 0.11, well above conventional thresholds for acceptable fit. | Methodological | showed an acceptable fit over a single-factor model (root mean square error of approximation (RMSEA), 0.11 | Moderate | Moderate | The three-factor measurement model does not fit the data well enough to support confident separation of the three empathy constructs, undermining the differential findings. |
| The paper's conclusions move from a descriptive labelling effect to the ontological claim that these differences 'stem from humans' unique potential to feel and care, which AI currently does not possess,' which the study's between-subjects labelling design cannot establish. | Causal | unique potential to feel and care, which AI currently does not possess | Weak | Major | An inferential leap from self-report ratings under a label manipulation to claims about AI's inner states, which the design cannot adjudicate. |
| Across the nine studies, attention-check failure rates are remarkably high (23–29%), yet no analysis examines whether exclusion rates differed by experimental condition or whether excluded participants differed systematically from retained ones. | Descriptive | failed attention checks (n = 263) or reported technical issues (n = 10), | Moderate | Minor | High exclusion rates with no analysis of differential attrition by condition could bias estimated labelling effects. |
Point-by-point
Each objection as a scholarly disputation — the authors’ claim, our assessment, the strongest reply they could give, our response, and what would change our mind.
- Point 1Moderate overreachMethodological
The authors claim: The paper's central theoretical contribution — that the human-AI empathy gap is driven specifically by affective and motivational rather than cognitive empathy — rests on a novel self-constructed 12-item questionnaire whose CFA yields RMSEA = 0.11, well above conventional thresholds for acceptable fit.
However — our assessment
The paper describes the three-factor CFA as showing 'acceptable fit' with RMSEA = 0.11 and CFI = 0.91. By standard criteria (Hu & Bentler, 1999: RMSEA at most 0.08 for adequate, at most 0.06 for good; CFI at least 0.95 for good fit), the model fit is poor. The CFI of 0.91 is at the floor of lenient acceptability and below the 0.95 threshold. The paper's main advance over prior work is the differential-dimension claim, which depends on the instrument cleanly separating cognitive, affective, and motivational empathy. With poor discriminant validity evidence, observed differences across subscales may reflect measurement artifact rather than genuine construct separation.
Anticipated reply
The authors could point to Study 3's experimental prompt-manipulation as convergent validity: when the AI was prompted to emphasise a specific empathy component, participants rated that component highest, confirming discriminant sensitivity. Additionally, the CFI of 0.91 meets some lenient thresholds, and the three-factor model fits better than a single-factor alternative.
Our response
Study 3's prompt-manipulation is a construct-response correspondence test — it shows the items react to prompts designed to foreground each dimension, but it does not resolve whether the items reliably separate the latent constructs in the primary between-subjects design where the same holistic prompt is used. The CFA fit indices (RMSEA 0.11, CFI 0.91) are the direct test of whether the measurement model adequately captures the tripartite structure, and they fall outside standard bounds. Relative fit over a single-factor model shows only that three factors are better than one, not that the fit is adequate in absolute terms.
What would change our mind
An independent psychometric validation of the 12-item questionnaire — CFA with RMSEA below 0.08 and CFI above 0.95 in an independent sample — would retire this objection and confirm the instrument's construct separation.
- Point 2Major overreachCausal
The authors claim: The paper's conclusions move from a descriptive labelling effect to the ontological claim that these differences 'stem from humans' unique potential to feel and care, which AI currently does not possess,' which the study's between-subjects labelling design cannot establish.
However — our assessment
The experimental design manipulates labels on identical AI-generated text. All observed rating differences therefore reflect how humans respond to labels, not anything about AI's actual emotional capacities. The conclusion that differences 'stem from' AI's inability to feel conflates what the data show (a perception/labelling effect) with what the authors believe philosophically (an ontological claim about AI phenomenology). The data are equally consistent with in-group favouritism, social-desirability effects, or mere prejudice against AI sources — none of which require the premise that AI 'does not possess' the capacity to feel.
Anticipated reply
The authors could argue that current LLMs are widely agreed to lack subjective experience (they cite Refs 48–54), and that their framing reflects the scientific consensus rather than an empirical claim their design must support.
Our response
Whether AI truly feels is an open philosophical question on which reasonable experts disagree. Even granting the authors' position, the study's data cannot be adduced as evidence for it — the data show a labelling effect on human perception, not a window into AI phenomenology. The sentence 'these differences stem from humans' unique potential to feel and care' asserts a causal origin for the observed gap that the design was not built to test. The hedge 'currently' partially mitigates but does not eliminate the gap between what was measured and what is concluded.
What would change our mind
A study that manipulates participants' beliefs about whether AI can genuinely feel (rather than just the source label) and shows this belief mediates the empathy-gap effect would provide the missing link between perception and the ontological claim.
- Point 3Minor overreachDescriptive
The authors claim: Across the nine studies, attention-check failure rates are remarkably high (23–29%), yet no analysis examines whether exclusion rates differed by experimental condition or whether excluded participants differed systematically from retained ones.
However — our assessment
Study 1a loses 263 of 998 participants (~26%) to attention checks. Similar rates appear across studies (1b: ~23%, 1c: ~25%, 1d: ~28%, 2a: ~29%, 2b: ~27%, 3: ~23%). If AI-condition participants were more likely to disengage and fail attention checks — a plausible differential-attrition mechanism given the nature of the manipulation — the retained sample would be differentially motivated across conditions, potentially inflating the labelling effect. The paper transparently reports exclusion counts but does not analyse whether attrition was condition-dependent.
Anticipated reply
The authors preregistered the attention-check exclusion criteria, and all studies use random assignment, so baseline characteristics should be balanced. High exclusion rates in online studies are common and do not by themselves indicate bias.
Our response
Random assignment balances baseline characteristics, but post-randomisation differential attrition can reintroduce systematic differences. The concern is not that exclusions are high per se, but that they may be condition-dependent — something a simple condition-by-attrition chi-square test could check. Its absence leaves the question open.
What would change our mind
Reporting that attention-check failure rates did not differ significantly between human-label and AI-label conditions across the studies would directly address this concern.
Scorecard
Sub-scores are 0–5 editorial judgements on fixed scales (higher is better, except methodological risk and overclaiming where higher is worse). They are contestable and open to a severity challenge from authors.
Strongest critique
The paper's central theoretical contribution — that the human-AI empathy gap is driven specifically by affective and motivational rather than cognitive empathy — rests on a novel self-constructed 12-item questionnaire whose confirmatory factor analysis yields an RMSEA of 0.11, well above conventional thresholds for acceptable model fit (at most 0.08 for adequate, at most 0.06 for good). The CFI of 0.91 is at the floor of lenient acceptability and below the 0.95 standard. This poor fit raises serious doubts about whether the instrument validly separates the three empathy constructs — without that separation, the differential findings that constitute the paper's main advance over prior human-preference research are insufficiently grounded.
Strongest fair defence
The paper is methodologically ambitious and admirably transparent. Nine preregistered studies with a combined n of 6,282 provide substantial statistical power. The authors systematically address alternative explanations (halo effects via studies 2a/2b, model dependence via study 1c with Llama 3.1, single-exchange limitation via study 1d with iterative exchanges). They share data and analysis code on OSF and document deviations from preregistration. The limitations section is forthright about the short-interaction scope, the absence of actual human-response comparisons, and the online-sample constraint. Study 3's prompt manipulation provides convergent validity for the three-construct distinction, partially offsetting the CFA concern. The basic descriptive finding — that labelling identical empathic responses as human versus AI shifts ratings — is robust and well-replicated. Effect sizes are reported throughout with confidence intervals.
Conclusion
This is a well-powered, preregistered, multi-study investigation of an important and timely question. Its core descriptive finding — that labelling identical empathic responses as human versus AI shifts self-reported empathy ratings — is robust and well-replicated internally. However, the paper's main theoretical advance (the differential role of affective/motivational versus cognitive empathy) depends on a measurement instrument with poor CFA fit, and the conclusions section makes an unsupported inferential leap from perceptual labelling effects to claims about AI's capacity to feel and care. These are bounded overclaims on an otherwise sound body of work.
Reply from the authors
Following the practice of Nature Matters Arising, Science Technical Comments and PNAS Letters, this Comment is published as one half of a Comment + Reply pair: the authors of the original article are invited to respond, and any reply is published here verbatim alongside the Comment as part of the record.
Reply: not yet invited. No reply has been received for publication.
The authors have a right of reply and no veto. A reply may request a factual correction, a methodological rebuttal, a clarification, a data/code update, or a severity challenge, and is published unedited. See the right-of-reply policy.
Source-grounding attestation
- ✓Verbatim source spans present in the critique — 3/3 provenance spans re-derived in the critique prose
- ✓Passes the publication validator — no errors
- ✓Zero fabricated citations — 0 fabricated
- ✓Severity within the access-basis cap — severity "moderate" ≤ cap "high" for licensed_access
Every verbatim span the critique relies on is re-derived in the prose in-app; span-in-source is re-verifiable offline (the abstract is re-fetched, not stored, per the no-reproduce policy).
Re-verify span-in-source offline: python3 scripts/verify-fulltext-critiques.py
Independent faithfulness review
A refute-by-default adversarial panel (two independent reviewers — an overreach lens and a mischaracterization lens — that fetched the real source) tried to prove this critique misread the paper. This is an AI adversarial review recorded with its reasoning, not a deterministic check.
All three verbatim spans independently verified as exact substrings. The critique correctly credits the paper's extensive methodological strengths (nine preregistered studies, n = 6,282, OSF data sharing, systematic alternative-explanation testing). Severity calibrated to moderate — appropriate for bounded overclaims on an otherwise sound study. The critique targets claims and inference, never authors' motives.
Version & correction history
| Version | Date | Change |
|---|---|---|
| v1.0 | 2026-07-05 |
No silent substantive corrections — every change is versioned and visible.
How to cite this Comment
Critical AI. Comment on “Comparing the value of perceived human versus AI-generated empathy” (Matan Rubin et al., Nature Human Behaviour, 2025). Critical AI; 2026. https://policywindow.org/critique/c/perceived-human-vs-ai-empathy
A registered DOI will replace the URL once minted; until then the canonical URL is the persistent identifier. Highwire/Dublin-Core citation tags and a schema.org Review record are embedded in this page for Google Scholar and reference managers.
Verify this Comment. Its checkable facts (target DOI, access-basis severity cap, zero fabricated citations) are served — as the app’s self-report — at /critique/api/critiques/perceived-human-vs-ai-empathy/verify; to confirm them independently of this site, re-derive the same checks (and resolve the target DOI) with npx tsx scripts/verify-critical-ai.ts --critique perceived-human-vs-ai-empathy --live.
Content fingerprint cf07547c294b2306 (v1.0) — this Comment’s substantive content is content-addressed; a silent post-publication edit would change it.