Plot.plot({
...chartBase(250),
marginLeft: narrow ? 16 : 174,
marginRight: 24,
x: {label: "Rare topics preserved (out of 7)", domain: [0, 7], grid: true, ticks: [0, 1, 2, 3, 4, 5, 6, 7], tickFormat: d => d},
y: {domain: ["Corpus summarization", "Record-level labeling"], label: null, axis: narrow ? null : "left", padding: 0.5},
marks: [
Plot.ruleY(conditionRange, {y: "condition", x1: d => Math.round(d.minimum * 7 / 100), x2: d => Math.round(d.maximum * 7 / 100), stroke: d => d.condition === "Record-level labeling" ? C.accent : C.neutral, strokeWidth: 8}),
Plot.dot(conditionRange, {y: "condition", x: d => Math.round(d.minimum * 7 / 100), fill: C.panel, stroke: C.ink, r: 5}),
Plot.dot(conditionRange, {y: "condition", x: d => Math.round(d.maximum * 7 / 100), fill: C.panel, stroke: C.ink, r: 5}),
...(narrow ? [Plot.text(conditionRange, {y: "condition", x: 0, text: "condition", textAnchor: "start", dy: -24, fill: C.ink, fontWeight: 600})] : []),
Plot.ruleX([0], {stroke: C.ink})
]
})What LLM Summaries Miss in Customer Feedback
How summaries of customer feedback can lose rare issues, even when you ask for them.
A summary can capture the main themes in thousands of customer comments and still leave out a small group whose experience matters. In our comparison, even prompts that explicitly asked for rare issues did not reliably preserve them.
Imagine using a summary of rail passengers’ feedback to decide what deserves a closer look. It covers delays, service, and ticketing. But does its silence about travelling with children mean nobody raised the issue, or that the issue disappeared during summarization? The summary alone cannot tell you.
We tested this problem on public text with human annotations. Across 15 summaries of 2,095 posts about Deutsche Bahn, only 1 to 3 of 7 rare topics were clearly preserved. Labeling each post individually retained at least one correctly labeled example for 5 of 7 topics with one model and all 7 with another.
That is what we call the average trap: a readable account of the main themes can be mistaken for a complete account of the evidence. The question is how to keep smaller issues visible while still producing a useful overview.
Scroll through the steps below to see the result before the details.
Start with the evidence. 2,095 posts about Deutsche Bahn, tagged by human annotators. Each dot is one tag. The blue ones are easy to miss already: together they are 33 of 1,069.
Group the tags by topic. A few large themes dominate, led by the train journey itself and other irregularities. Seven topics have ten or fewer posts each: the long tail.
Summarize, and the tail fades. A summary describes what is common. Across 15 summaries, only 1 to 3 of the 7 rare topics were clearly covered. The numbers show in how many summaries each one was clear.
Size is not the whole story. Toilets and travelling with children both had seven posts. Toilets were clear in 13 of 15 summaries, travelling with children in one.
Label first, then count. Labeling each post on its own keeps every dot addressable. One labeling run kept a correct example of all 7 rare topics, the other of 5. The overview can then be written from counts that still include the tail.
Two ways to turn comments into an overview
The main benchmark was the test_syn portion of GermEval 2017: 2,095 German social and web posts about Deutsche Bahn. Human annotators had assigned topic categories to the posts, giving us a reference against which to check the models. After excluding the broad catch-all category Allgemein, 18 categories remained.
We called a category rare if it appeared in ten or fewer documents. Seven met that definition, ranging from QR codes in one document to accessibility in nine. These small categories form the long tail: issues that occur infrequently but may still matter to the people affected.
We compared two approaches. Corpus summarization asks the model to turn the collection into prose, either in one request or by summarizing chunks and then combining those summaries. Record-level labeling applies a known set of categories to each document separately and then aggregates the results.
The distinction is easiest to see with luggage, which appeared in two human-annotated documents. A summary might discuss luggage explicitly, fold it into a broad statement about comfort, or leave it out. A labeling system can attach the luggage category to the individual posts, allowing a reader to count and inspect them later. It can also get those labels wrong, so both approaches need checking.
For summaries, an LLM judge compared the final text with the human category inventory and classified each topic as clearly covered, vaguely covered, or absent. Positive verdicts required a supporting quotation. An independent audit checked those quotations and reproduced the aggregate results from the saved outputs, although some category matches remained open to interpretation.
For labeling, a rare category counted as preserved only if the model assigned it to at least one document that also carried that category in the human annotations. Emitting a label on an unrelated document did not count. This measures whether a topic survives somewhere in the output; it does not measure whether every relevant document was found or every assigned label was correct.
How many rare topics survived?
The 15 GermEval summary variants clearly covered between one and three of the seven rare categories. The two labeling runs retained five and seven respectively: five with GPT-5.4 mini and seven with DeepSeek v4 flash.
Summaries preserved 1–3 rare topics; individual labels preserved 5–7. Observed ranges across 15 summary variants and two labeling runs. A topic survives in the labeling result only if at least one assigned document matches the human annotation.
Source: GermEval summary coverage scores and labeling evaluations against human annotations. Ranges describe the tested configurations, not statistical confidence intervals.
The summary variants included different models, prompts, output lengths, input orders, and chunk sizes. None reached the rare-topic coverage of either labeling run. This is a comparison of the configurations we tested, rather than evidence that every possible summary would behave the same way.
A second benchmark showed the same direction. M-ABSA contained 538 short restaurant-review sentences and three rare categories. Its summaries clearly preserved between zero and two of those categories; all three labeling models retained at least one correct example of all three.
The main result is therefore about topic coverage, not overall model accuracy. Finding one correct luggage example keeps that category visible. It says nothing by itself about how many other luggage references the model missed or how often it applied the label incorrectly.
What disappeared, and what became vague?
A single coverage score hides an important distinction. A topic can be absent from the summary, or it can survive only as a vague reference that is hard to recognize as a separate issue. The table separates those outcomes for each rare GermEval category.
| Rare topic | Source documents | Clear | Vague | Absent |
|---|---|---|---|---|
| QR codes | 1 | 1* | 0 | 14 |
| Luggage | 2 | 3 | 7 | 5 |
| Catering | 3 | 0 | 1 | 14 |
| Design | 4 | 0 | 4 | 11 |
| Travelling with children | 7 | 1 | 12 | 2 |
| Toilets | 7 | 13 | 1 | 1 |
| Accessibility | 9 | 11 | 2 | 2 |
Source documents counts human-annotated posts. Clear, Vague, and Absent count summaries and add up to 15 in each row. Topic names are translated from the German categories. The asterisk marks a borderline judgment discussed below.
Travelling with children and toilets each appeared in seven documents, yet their outcomes differed sharply. Toilets received clear treatment in 13 summaries; travelling with children did so in only one. Frequency alone therefore does not explain which topics survived. Some issues may be easier to name distinctly, while others may be absorbed into broader themes, but this experiment does not isolate those causes.
Catering and design never received clear treatment. Every rare category was absent from at least one summary. By contrast, every category represented by 24 or more documents appeared at least vaguely in all 15 summaries. The summaries could capture the major themes while still providing an incomplete view of the smaller ones.
The QR-code example
The most extreme case was the category represented by a single document. Fourteen summaries omitted QR codes entirely. The fifteenth received credit for clear coverage because it mentioned “QR/PDF tickets” within a broader discussion of digital services. The audit flagged that verdict as borderline: the phrase named the subject, but did not clearly develop it as a distinct issue.
We retain the original score in the table and headline results. A stricter judgment would count no clear QR-code coverage at all. This illustrates why the difference between a passing mention and meaningful coverage needs to be visible to the reader.
The DeepSeek labeling run correctly labeled the source document; the other labeling model missed it. Individual labeling did not eliminate errors, but it left a result that could be checked against that specific post. A prose summary can also be audited, as we did here, but that requires returning to the source material to find out what is missing.
Can a better prompt or more context fix it?
The experiment tested more than a short request for the main themes. Summary prompts ranged from a basic management overview to a detailed inventory and an explicit request to preserve rare, minority, and contradictory views. Output targets were roughly 300 or 1,200 words; some responses exceeded those targets.
For larger workflows, we tested summarizing chunks and then combining their summaries, often called map-reduce. The variations included chunks of 50, 100, and 200 documents, as well as shuffled document order. We also tested putting all 2,095 documents into a single context, so there was no intermediate summary in which information could disappear.
The chart below focuses on the long-output prompt comparisons. Within this subset, no result clearly preserved more than two of the seven rare categories. The three-category maximum reported earlier came from another configuration in the broader experiment.
Explicit requests for rare issues did not consistently improve coverage. Long-output prompt comparisons. “Rare-aware” asks for rare, minority, and contradictory views. GPT-5.1 received the full corpus in one request; its detailed-inventory variant was not tested here.
Source: long-output GermEval prompt comparisons. Each dot is one tested configuration; dots are offset slightly to keep overlapping results visible.
Compared with the basic prompt, the explicit rare-topic request recovered one additional category for DeepSeek and full-context GPT-5.1, but lost one for GPT-5.4 mini. With one run per configuration, these small differences are not a reliable ranking of prompt strategies. They do show that asking for rare issues was insufficient to produce a complete inventory in these tests.
The full-context result matters because it rules out one explanation for those particular omissions: nothing had been discarded by an earlier chunk summary. In a map-reduce workflow, that is an additional risk: a final summary that only sees intermediate summaries cannot reliably recover an issue they omitted. When all documents are available, omissions can still occur. These results establish that behavior; they do not establish the model’s internal reason for selecting some topics over others.
What the comparison can and cannot tell us
The findings support a practical concern about completeness, with several limits on how far to take the numbers.
First, this was a comparison on two public benchmarks with known categories. It does not test whether a topic-discovery process would find every rare issue, or establish expected coverage for another industry or dataset. Labeling can only preserve a category if the schema provides somewhere for it to go.
Second, each summarization configuration was run once. The observed gap is worth investigating, but these ranges are not confidence intervals. More repeated runs would be needed to estimate how stable each configuration’s score is.
Third, the outputs and their scoring differ. A summary gets credit for clearly discussing a category; labeling gets credit for at least one correct document assignment. This provides a useful comparison of whether topics remain visible, but 100% category survival is not 100% labeling accuracy. Rare labels can still have low precision, meaning many documents assigned to a category do not belong there.
Finally, an LLM judged the summaries. Supporting quotations and an independent audit make those judgments inspectable, but they do not remove ambiguity. The QR-code example shows that even a quoted phrase can support more than one reasonable interpretation.
Keep the evidence before writing the overview
A summary is useful when the task is to get oriented quickly. When the task involves counting issues, investigating small groups, or tracing a finding back to the people who raised it, the underlying records need to remain part of the analysis.
A practical workflow is to define or discover the categories, label each document, retain the source references, and check the resulting assignments. Counts and filters can then identify both common themes and smaller groups for closer inspection. The written overview comes after that evidence is available.
01 Define the categories Give distinct issues a place in the schema. Apply to each document
02 Label and retain evidence Keep each assignment linked to its source. Check assignments
03 Count and inspect Examine smaller groups as well as major themes. Select findings
04 Write the overview Explain the findings with counts and examples.
For the rail-feedback example, this means a reader can open the posts labeled as travelling with children and check what they say. If that topic is missing from the final overview, its evidence is still available. Checking a sample of the original posts also helps reveal issues the labeling step missed.
The goal is an overview whose omissions can be investigated. Keeping document-level evidence makes that possible without asking a few paragraphs to carry the entire dataset.