Comment on “Multimodal large language models can make context-sensitive hate speech evaluations aligned with human judgement”

The headline claim is a deployment claim: the Discussion asserts MLLMs can provide context-sensitive automated screening at scale. Yet every piece of evidence for context sensitivity comes from a forced-choice design in which two posts are compared and lexical content is effectively neutralized across the pair, so that author identity can register at the margin. The Methods reveal the design was selected precisely because the realistic one-at-a-time format would not exhibit the phenomenon: the author was 'concerned that presenting posts one at a time could induce heuristic use, such that slurs are used to make a decision, irrespective of contextual factors,' and states that if models 'always chose or ignored profiles with specific features, it would not be possible to observe contextual sensitivity.' When the author ran the ecologically valid single-profile replication, that is exactly what happened — some models 'nearly always selected posts containing certain slurs,' i.e., near-ceiling lexical flagging irrespective of author identity. This is not a minor external-validity caveat: near-ceiling flagging of the reclaimed slur regardless of the author's race is precisely the racialized false-positive failure mode (silencing Black users who use in-group reclaimed terms) that the paper claims context sensitivity would help avoid. A capacity that manifests only when the measurement instrument suppresses the lexical heuristic, and disappears under the task structure actual moderation systems face, cannot support the inference from 'context sensitivity was measurable in a conjoint' to 'MLLMs can make context-sensitive moderation decisions.' The forced-choice conjoint is a legitimate tool for estimating relative feature weights — the flaw is the load-bearing generalization from format-bound relative weights to a deployable capability, against the paper's own contrary evidence in the deployment-like condition.

• “Turning to the methodology, the forced-choice design differs from the one-at-a-time decisions typically made by moderators, so the single-task design may be closer to the way content is moderated in practice.”
• “Indeed, I replicated the AI experiments using a single-profile design and found that some models nearly always selected posts containing certain slurs (Supplementary Fig. 6).”

The skeptic's argument turns on attributing to the paper a claim it does not make and on overstating what the single-profile replication shows. First, the headline is a capability claim about what larger models 'can' do under controlled measurement, not an autonomous-deployment claim: the paper's explicit deployment recommendation is the opposite of what the skeptic attacks — 'As things stand, I recommend that MLLMs are used to assist human moderators rather than to make decisions autonomously.' Second, the forced-choice format is not a mere lab instrument alien to practice. The outcome variable throughout is triage — the model must 'select the post that should be prioritized for manual review' — and the paper argues that 'content moderation necessitates triage and relative judgements,' proposing that 'Pairwise comparisons could prove to be a useful way for automated systems to prioritize the most challenging material for human evaluation.' There is thus a concrete deployment architecture in which the capability applies in exactly the format where it was measured; 'format-bound' does not entail 'non-deployable.' Third, the skeptic's claim that sensitivity 'largely vanishes' in the single-task design overreads the paper's own report: the rank order 'was less apparent,' 'some' models almost always selected 'certain' slurs (racial and homophobic epithets) — not all models, not all slurs — and crucially, 'Comparable identity-based discrepancies are also present in the single-task experiments (Supplementary Fig. 6),' showing author identity continued to register outside the conjoint. Fourth, the paper's interpretation of the format difference is methodologically standard, not circular: strong lexical weights 'mask underlying contextual sensitivities' in single-post evaluation. Masking by a stronger co-present signal is not evidence that the weaker signal is an artifact of the instrument — decomposing co-varying influences is precisely what conjoint designs are for, and the paper transparently prices the cost as 'a trade-off between ecological validity and internal validity.' Finally, the near-ceiling lexical flagging the skeptic treats as a buried refutation is one of the paper's two headline findings, stated in the abstract ('pervasive demographic and lexical biases remain') and driving its human-in-the-loop recommendation. A narrow concession is warranted: the paper does not demonstrate that reclaimed-slur discrimination survives at practically useful magnitude in single-post classification with current off-the-shelf models, and the Discussion sentence about 'automated screening at scale' reads stronger than that evidence if detached from the adjacent qualifications. But the flaw's stronger conclusion — that context sensitivity is an artifact of elicitation — is contradicted by the persistence of identity effects in the single-task replication and by the paper's own carefully bounded conclusions.

• “As things stand, I recommend that MLLMs are used to assist human moderators rather than to make decisions autonomously and stress the need to establish clear oversight mechanisms for accountability and democratic deliberation about the values these systems should reflect.”
• “the model is instructed to select the post that should be prioritized for manual review on the basis of a hate speech policy.”
• “While it sacrifices some external validity, the comparative forced-choice framework is not wholly unrealistic, as content moderation necessitates triage and rela- tive judgements105,106. Pairwise comparisons could prove to be a useful way for automated systems to prioritize the most challenging material for human evaluation.”
• “In this case, the rank order was less apparent because most models showed a similar propen- sity to select posts containing any identity-based slur (except sexism, which was also chosen less often), and some almost always selected posts with racial and homophobic epithets (Supplementary Fig. 6).”
• “Comparable identity-based discrepancies are also present in the single-task experiments (Supplementary Fig. 6).”
• “In this case, however, the tendency to give more weight to certain terms when evaluating an individual post masks underlying con- textual sensitivities (Supplementary Discussion 2).”
• “Ultimately, the forced-choice approach entails a trade-off between ecological validity and internal validity but provides a scale-free approach to measuring the relative impact of different factors on content moderation decisions.”
• “However, pervasive demographic and lexical biases remain, particularly among smaller models.”

The rebuttal wins real ground but concedes the load-bearing points, so the flaw survives in narrowed form. What the rebuttal defeats: my word 'artifact' overreached. The identity effects are real properties of the models' weightings, the paper transparently prices the design as 'a trade-off between ecological validity and internal validity', and the triage framing gives the pairwise format one coherent deployment reading. I withdraw the implication that the measured sensitivity is illusory. What survives: (1) The proponent concedes both pillars — that 'the paper does not demonstrate that reclaimed-slur discrimination survives at practically useful magnitude in single-post classification' and that the screening-at-scale sentence 'reads stronger than that evidence.' But that sentence is not detachable rhetoric; it is the paper's summary evidence claim: 'This study demonstrates that MLLMs can facilitate more sophisticated automated screening at scale, providing context-sensitive decisions' ... 'that align closely with human judgements.' 'Demonstrates' asserts the evidence shows this, and in the format the author concedes is 'closer to the way content is moderated in practice', the evidence shows the opposite. The human-in-the-loop recommendation qualifies deployment posture; it does not retract the demonstrative claim my flaw targeted. (2) The rebuttal's strongest empirical counter cites the wrong phenomenon. 'Comparable identity-based discrepancies are also present in the single-task experiments' sits in the paragraph on identity main effects that the paper glosses as 'intersectional disparities in hate speech evaluations' — what persists in the realistic format is the bias (flagging Black users' posts), while the paper's own Discussion says the protective sensitivity behaviorally disappears there: 'This probably explains why the models made less distinction between authors when evaluating racist language, since such posts were almost always selected, irrespective of the author’s identity. This tendency is particularly evident in the robustness check using a single-profile conjoint design, where some models selected certain slurs in almost every instance (Supplementary Fig. 6).' In the deployment-like condition, the harmful identity effect survives and the beneficial one does not — that sharpens my flaw rather than defeating it. (3) 'Masking' concedes behavioral absence: a masked sensitivity produces no different moderation decisions; it relocates the capability from output to latent weight, which cannot support a claim about evaluations the model 'can make.' (4) The triage rescue is, in the paper's own words, hypothetical — 'Pairwise comparisons could prove to be a useful way for automated systems to prioritize the most challenging material for human evaluation' — and untested; real triage queues compare lexically heterogeneous content, where the documented lexical over-weighting ('the models put excessive weight on these terms') would dominate exactly as it did in the single-task check. Even the severity rank order, a headline alignment result, degraded there: 'In this case, the rank order was less apparent'. Restated surviving flaw: the headline capability is format-bound; in the paper's own deployment-like replication the protective sensitivity vanishes behaviorally while the demographic biases persist, and the Discussion's 'demonstrates... automated screening at scale' claim outruns this evidence — as the proponent concedes.

The benchmark against humans is entirely distributional: model AMCEs and marginal means are plotted beside human ones and judged similar by inspection. No decision-level agreement between models and humans is ever computed — and it cannot be, because humans evaluated 15 pairs 'randomly sampled from the corpus' while each model evaluated an independently drawn 30,000 pairs, so the design precludes measuring whether a model and a human ever make the same call on the same pair. Two decision processes can share coefficient rank order while disagreeing on a large fraction of individual adjudications, which is what moderation alignment actually requires. Even at the only level measured, the paper's own numbers contradict 'closely aligned': on the single largest driver of decisions, the racism AMCE is 0.53 for humans versus 0.70 for both GPT-4o and Gemini 2.5 Flash — confidence intervals nowhere near overlapping, an effect roughly a third larger, which the author himself glosses as models putting 'excessive weight on these factors relative to human moderators.' On the dimension where alignment matters most — racialized context — the flagship model diverges qualitatively: GPT-4o flagged slur-free posts by Black men at a higher rate than anonymous users where 'human decisions did not vary systematically when slurs were absent,' penalized Black men but not white men for sexist language where humans showed no author effect, and showed no statistically significant author sensitivity for anti-Black racism where humans did ('there is less evidence of context sensitivity among the MLLMs'). 'Closely aligned' therefore rests on a discretionary reading that privileges rank-order agreement on slur severity and the direction of reclaimed-slur effects while discounting CI-rejected magnitude gaps and sign divergences the paper itself documents. Absent any defined alignment metric, the identical evidence would support the opposite headline — that MLLMs systematically diverge from human judgement — making the paper's central benchmarking inference unfalsifiable as stated.

• “The results demonstrate that larger, more advanced models can make context-sensitive evaluations that are closely aligned with human judgement.”
• “the AMCE for racism is 0.53 (95% confidence interval, (0.52, 0.55)) for human participants and 0.70 (0.69, 0.71) for GPT-4o and for Gemini 2.5 Flash. This suggests that models may put excessive weight on these factors relative to human moderators.”

The claim that alignment is 'never operationalized or tested' mistakes a deliberate choice of estimand for the absence of one. Alignment is operationalized as correspondence between identical causal quantities — AMCEs and differences in marginal means — estimated for humans and models over the same randomized 210,000-post attribute space, with interval estimates on every comparison: 'The error bars indicate 95% confidence intervals: the MLLM results use bootstrap confidence intervals, and the human experiment uses participant-level clustered standard errors.' This is not eyeballing; it is comparison of estimated causal parameters, and it is the design's stated purpose: 'similar experiments can be fielded to both humans and machines, allowing the results to be benchmarked against human judgement.' Because AMCEs are population-level estimands over the same stimulus universe and randomization, human-model comparability does not require item-level pairing; decision-level percent agreement is a different quantity — confounded by ties the design deliberately treats as random noise — and the paper explains why it targets factor weights instead: 'Unlike evaluations that focus on differences in predictive performance, the conjoint approach provides insight into the factors that contribute to decisions.' Second, the unfalsifiability charge fails on the paper's own record: the very evidence the skeptic cites (racism AMCE 0.53 vs 0.70, glossed as 'excessive weight') was produced by this framework, reported prominently, and folded into the conclusions — 'At the same time, the results demonstrate that MLLMs perpetuate known biases and introduce new risks,' and 'even the most sophisticated models showed some evidence of bias, making testing and validation essential.' A measure that generates and reports adverse results, which the author then integrates as headline caveats, is not unfalsifiable. Third, 'closely aligned' is a scoped claim about the largest models on enumerated dimensions, each independently documented: an identical severity ordering ('A consistent rank order emerged among the terms for both humans and most MLLMs. In general, anti-Black racism had the highest likelihood of selection, followed by homophobia, reclaimed anti-Black slurs, reverse racism and finally sexism'); directionally matching reclaimed-slur context effects ('Human participants were, on average, less likely to choose posts using racism or reclaimed slurs when the user was depicted as Black and were more likely to select them when the author was white. Larger models made similar judgements with respect to the reclaimed slur'); and the summary that frontier models 'judged racism and reclaimed racism differently depending on the author's race.' On the dimension that motivates the entire study — reclaimed-slur false positives — the paper states the agreement precisely: 'both humans and MLLMs can be sensitive to the context of reclaimed slur usage, helping avoid a common source of false positives.' Fourth, the author applies the alignment concept consistently rather than opportunistically: he also declines to score model-human convergence on anti-white language as misalignment precisely because 'the models reflect judgements made by human participants' — alignment means correspondence to measured human judgement, applied in both flattering and unflattering directions. A narrow concession: the paper reports no decision-level concordance statistic on shared pairs and pre-specifies no quantitative threshold that 'closely' must clear; the summary adjective rests on a qualitative synthesis of the estimated effects. That is a real limitation worth pressing. But it supports 'the alignment claim could be more stringently quantified,' not the skeptic's conclusion that the claim was never operationalized, never tested, or immune to refutation — the paper's own CI-rejected divergences, transparently reported and incorporated into its qualified conclusions, demonstrate the contrary.

• “The error bars indicate 95% confidence intervals: the MLLM results use bootstrap confidence intervals, and the human experiment uses participant-level clustered standard errors.”
• “the capacity to prompt generative AI means that similar experiments can be fielded to both humans and machines, allowing the results to be benchmarked against human judgement55– 57.”
• “Unlike evaluations that focus on differences in predictive performance, the conjoint approach provides insight into the factors that contribute to decisions, providing some interpretability into the output of ‘black box’ models.”
• “At the same time, the results dem- onstrate that MLLMs perpetuate known biases and introduce new risks.”
• “Nonetheless, even the most sophisticated models showed some evidence of bias, making testing and validation essential, regardless of the chosen model.”
• “A consistent rank order emerged among the terms for both humans and most MLLMs. In general, anti-Black racism had the highest likelihood of selection, followed by homophobia, reclaimed anti-Black slurs, reverse racism and finally sexism.”
• “Human participants were, on average, less likely to choose posts using racism or reclaimed slurs when the user was depicted as Black and were more likely to select them when the author was white. Larger models made similar judgements with respect to the reclaimed slur (third column).”
• “The largest frontier models exhibited the closest performance to human raters across several categories: GPT-4o and Gemini 2.5 Flash assigned the highest weights to the terms people found most likely to violate the policy, and they judged racism and reclaimed racism differently depending on the author’s race.”
• “This demonstrates that both humans and MLLMs can be sensitive to the context of reclaimed slur usage, helping avoid a common source of false positives.”
• “Importantly, this does not represent misalignment between MLLMs and humans, insofar as the models reflect judgements made by human participants.”

I concede the rebuttal's two strongest points, but they trim the flaw's rhetorical extremes without touching its core, which the proponent explicitly concedes — so the flaw survives in restated form. Honest concessions first: (1) 'never operationalized or tested' was overbroad at the comparison level. Estimating identical causal quantities for humans and models over the same randomized attribute space, with interval estimates throughout, is a genuine operationalization of comparison, and it is more than eyeballing. (2) Strict unfalsifiability of the framework fails: the framework detected and the author reported adverse results ('models may put excessive weight on these factors relative to human moderators'). But the flaw's target was never the measurement framework — it was the verdict. What survives, largely by the proponent's own concession: (1) The rebuttal concedes there is 'no decision-level concordance statistic on shared pairs,' no 'quantitative threshold that closely must clear,' and that 'the summary adjective rests on a qualitative synthesis.' That is precisely the flaw: comparison was operationalized; closeness never was. The title and abstract claim is 'closely aligned with human judgement', and no criterion anywhere in the paper converts the estimated effects into that adjective — nor is any human-model equivalence or difference formally tested. (2) The decision-level gap is confirmed, not closed. Humans each evaluated '15 pairs of posts randomly sampled from the corpus.' while 'The same set of posts was used in all AI experiments to rule out any variation due to differences in sample composition' — the shared set spans AI experiments only, so no human and model ever adjudicated the same pair. The proponent's defense that decision agreement is 'a different quantity' relabels the gap: for moderation, 'aligned with human judgement' connotes making similar calls, and factor-weight similarity places no bound on decision-level disagreement. This is unrebutted. (3) The scoped unfalsifiability claim — 'as stated' — stands. Within this paper, the framework returned a CI-rejected divergence on the single largest driver ('the AMCE for racism is 0.53 (95% confidence interval, (0.52, 0.55)) for human participants and 0.70 (0.69, 0.71) for GPT-4o and for Gemini 2.5 Flash.'), a sign divergence on sexism ('In contrast to the racialized slurs, there is no evidence that human judgement of sexist language varied systematically according to the apparent gender of the user.' — while GPT-4o penalized Black men), an anti-Black divergence on slur-free posts ('human decisions did not vary systematically when slurs were absent.' versus GPT-4o flagging Black men's posts), and a null where humans showed sensitivity ('With regard to anti-Black racism, there is less evidence of context sensitivity among the MLLMs.', with 'no statistically significant difference for the larger GPT-4o.') — and the headline adjective was unaffected. Transparent reporting of divergences is a virtue, but it demonstrates that no divergence magnitude observed in this study was permitted to defeat 'closely.' (4) The rebuttal's consistency point (anti-white convergence 'does not represent misalignment') illustrates the elasticity: alignment is credited on direction-matching without any magnitude standard, while CI-rejected magnitude gaps are absorbed as caveats. Restated surviving flaw: not that nothing was measured, but that the load-bearing verdict — 'closely aligned' — has no operationalization, no decision-level evidence, and is contradicted where alignment matters most by the paper's own interval estimates.