Two Experts, One Paragraph, Two Different Answers

Give two trained annotators the same paragraph of a student essay and ask a simple question: which sentence is the claim, and which sentences are the reasons for it? They will often disagree. Not about the topic, not about whether the argument is any good — about where it is.

This is the fact that shaped the first decade of computational argument mining, and it is easy to underrate. Everything downstream depends on it: you cannot build a benchmark for a task that people cannot perform consistently, and you cannot claim a model beats human performance on a judgement humans do not share.

Try it yourself on the last email thread you argued in. Marking the claim is usually easy. Marking which sentence each objection was aimed at is where you will start hesitating — and that hesitation is the whole story of this decade.

Why the Theory Sat Unused for Twenty Years

By the early 1990s the theoretical apparatus was complete. Toulmin had given the field its anatomy, Walton its catalogue of reasoning patterns, Freeman the distinction between attacking a conclusion and attacking the inference to it. As we covered in the first post in this series, an argument had been redescribed as a labelled graph.

A labelled graph is something a computer can be scored on — in principle. In practice you need something to score it against. Without annotated data, two research groups claiming different results have no way to settle it, and a field where results cannot be compared is not a field. It is a collection of demos.

So the decade that turned argument mining into a discipline was not a decade of algorithms. It was a decade of annotation guidelines.

The Three Corpora That Defined the Task

Different research groups annotated different genres, and the genre choice mattered more than anyone initially expected.

• Persuasive essays (Stab & Gurevych). Student argumentative essays, annotated for claims, premises and the support/attack relations between them. Structured writing by people who were explicitly trying to argue clearly — about as favourable as text gets.
• Argumentative microtexts (Peldszus & Stede). Short controlled texts built specifically to study argument structure, each one a compact argument with a deliberate shape. Small, clean, and designed so that structural questions have answers.
• Public rulemaking comments (Park & Cardie). Comments submitted by the public to US regulatory consultations. Real, unedited, frequently rambling — much closer to the text organisations actually have.

That progression from clean to messy is the useful thing to notice. Performance drops as you move down the list, and it drops most on relations. The essays were the friendly case, and even there the relation numbers were modest.

Inter-Annotator Agreement Is the Real Ceiling

Annotation projects report how often independent annotators agree, and in argument mining that number is not a footnote. It is the ceiling.

If two trained humans following the same guidelines agree only moderately about which premise attaches to which claim, then a model scored against one of them cannot meaningfully be described as beating the other. The measurement itself becomes unstable before the model does.

Agreement is consistently higher on components than on relations — people mostly concur about which sentence is a claim, and diverge about what it is responding to. That single asymmetry predicted everything the benchmarks would later show.

The Guidelines Are the Theory, Made Operational

An annotation guideline sounds like clerical work. It is closer to a specification. Every ambiguity the theory left open has to be resolved before annotators can start, and each resolution is a real commitment.

Is a rhetorical question a claim? Does a restatement in the conclusion count as a new component or the same one? When a sentence supports two claims at once, does it attach to both, or to the nearer, or to the more general? If someone concedes a point before rebutting it, is the concession an attack on their own position?

None of these have obvious answers, and each corpus answered them slightly differently. That is why models transfer poorly between corpora: they are partly learning a genre, and partly learning one team's answers to those questions.

But Couldn't You Just Annotate More Data?

It is the natural response, and for component detection it broadly worked. More annotated spans produced better span detection. That part of the field improved steadily and predictably.

For relations it did not, and the reason is structural rather than a matter of volume. Relations depend on context outside the sentence — which is expensive to annotate consistently. Disagreement relations are rare, so scaling the corpus scales the imbalance along with it. And the connecting step is frequently unstated, so the annotator is not marking something visible in the text; they are marking their reconstruction of it.

More data multiplies all three problems rather than solving them. That argument is the subject of the next post.

What the Decade Left Behind

Two things, both still load-bearing.

The first is the task definition itself. The four-part breakdown that every argument-mining paper now opens with — find the components, label them, link them, classify the links — is a product of these annotation projects, not of the philosophy. Toulmin never proposed a pipeline.

The second is the benchmark set. When a modern system reports a number, it is usually reporting it on essays or microtexts annotated in this decade. That is worth remembering when comparing figures across papers: the same metric name on two different corpora is not the same measurement, and a great many published comparisons quietly ignore that.

What This Means Outside the Lab

• If humans can't agree where the argument is, no tool will hand you certainty. Treat extracted structure as a draft to correct, not a verdict to accept.
• Genre matters more than volume. A system tuned on tidy essays meets a very different problem in a meeting transcript full of interruptions and half-finished thoughts.
• The disagreement about targets is the real signal. When two people think an objection was aimed at different things, that gap is usually where the decision is actually stuck.
• Compare benchmark numbers only within a corpus. The same metric on a different dataset is a different measurement, however similar the label looks.

Where the Argument Is

The decade of annotation is the least glamorous part of this story and the part everything else rests on. No model, however large, escapes the fact that its scoreboard was built by people who sometimes disagreed with each other about where an argument was.

Which is a useful thing to carry into your own meetings. If finding the argument is hard for two trained annotators with a written guideline, the fact that your team keeps relitigating a decision is not a discipline problem. It is the same problem, without the guideline.

The argument mining series

Five posts on how machines learned to read arguments — where the field came from, why its hard problems are hard, and how we apply it.