SOURCE-LINKED INTELLIGENCE

Fallacy Benchmarks Measure Scheme Recognition, Not Fallacy Detection

arXiv · AI, language, vision and robotics · article · Sep 16, 2026 · UTC

Fallacy-detection benchmarks pair fallacy classes with a single "valid" or "none" class that takes everything data collection did not label as a fallacy. This construction is misleading: a classifier can learn cues that do well on this class without learning to tell a fallacy from a correct argument. We show that the low false-positive rates benchmarks report are an artifact of how the class is built, not evidence of detection ability. The most informative negative for a fallacy is a correct argument using the same argumentation scheme, and such arguments are at most a few percent of the valid

Read original source ↗ Open in workspace

recordType

paper

region

Global