The Class That Matters Most Is the One There's Least Of

In the corpora that argument-mining systems are trained and measured on, disagreement is rare. The overwhelming majority of annotated relations are support: this reason backs that claim. Attack relations — this objection undercuts that reason — are a small minority, in some datasets under a tenth of the labelled links.

That single fact produces the field's most stubborn failure mode. A system that simply predicts support every time is right most of the time, and any model learning from that distribution absorbs the same bias. It becomes agreeable.

Which is exactly backwards for the reason you would deploy one. Nobody revisits a decision six months later wondering what everyone agreed about. They want the objection — and the objection is the class the model is least confident about and least likely to predict.

Why Imbalance Is Worse Here Than Elsewhere

Class imbalance is an ordinary machine-learning problem with ordinary machine-learning remedies — reweight the loss, oversample the rare class, report macro-averaged scores rather than accuracy. All of that is standard practice, and all of it helps somewhat.

What makes argument mining harder is that the imbalance is not a sampling artefact you can correct by collecting more data. It reflects how people write. In a persuasive essay the author is building a case, so most sentences support the thesis by design. Scaling the corpus scales the imbalance with it.

This is why the field reports macro-F1 and separate per-class scores rather than raw accuracy. A model that never predicts attack can post a respectable-looking overall number while being useless for the thing you wanted it for. Reading only the headline figure hides the failure completely.

Tone Is Not Just Unhelpful — It's Actively Misleading

The intuitive approach is to look at the language. Negative words, hedging, adversarial phrasing — surely those signal an attack.

They do not, and the counterexamples are not edge cases. Consider a claim that a project is risky. The sentence timeline pressure adds to the risk is phrased negatively and supports that claim — it supplies another reason the project is risky. Now consider long-term savings will offset the initial investment, attached to a claim that the costs are prohibitive. Positive words throughout, and it attacks the claim.

Whether something supports or attacks is a property of the relation, not of the sentence. The same sentence attached to a different parent flips its label. This is why sentiment analysis, which is very good at what it does, is the wrong tool: it reads the sentence, and the answer is not in the sentence.

The Objection to a Step That Was Never Written Down

The third reason is the deepest, and it goes back to Toulmin. Many real objections do not dispute the facts at all. They dispute the <em>inference</em> — the unstated principle connecting the evidence to the conclusion.

Someone cites a study showing a 13% productivity improvement and concludes the policy should be adopted. The objection is that the study population was nothing like this organisation. No fact has been challenged. What has been challenged is the warrant: that a result there transfers to here. That principle appears nowhere in the transcript, because nobody ever said it.

As the first post in this series covered, warrant reconstruction was Toulmin's central insight and remains the field's least-solved problem. A system asked to classify a relation must sometimes reconstruct a proposition that is not in its input, and then decide whether the objection targets that proposition, the evidence, or the conclusion.

But Surely a Bigger Model Fixes This?

This is the reasonable expectation, and it has been partially wrong for long enough to be interesting.

Scale helps with knowledge and fluency, and it has genuinely improved argument mining — that story is the next post in this series. But none of the three problems above is a knowledge problem. Imbalance is a property of how people write. Context dependence is a property of the task definition. The missing warrant is a property of the text.

A larger model reading the same transcript still finds no sentence stating the principle it needs, still sees mostly agreement in whatever it was tuned on, and still has to decide which of three plausible targets an objection was aimed at. Capability rose; the structure of the problem did not move.

What Actually Helps

• Score the classes separately. A single headline number lets a model that never predicts disagreement look competent. Per-class F1 makes the failure visible immediately.
• Ask about the parent, never the topic. The only question that yields a correct label is what this contribution does to the thing directly above it.
• Make the reasoning explicit before the verdict. Forcing a written justification of the relation before committing to support or attack surfaces the error while it can still be caught.
• Keep a human in the loop for the rare class. Disagreement is where models are weakest and where the value is highest — precisely the place to spend review attention.

The Useful Class Is the Rare One

Everything about the data pushes a system toward agreement. The distribution rewards it, the phrasing misleads toward it, and the crucial connecting step is frequently absent from the text entirely.

And the one thing you actually needed was the objection nobody answered. That is why this is the hardest problem in argument mining, and why it is worth the effort: the rare class is the useful class.

The argument mining series

Five posts on how machines learned to read arguments — where the field came from, why its hard problems are hard, and how we apply it.