Reading Customer Opinions at Scale
Why customer text is worth analyzing, how the methods evolved from keywords to language models, and what each approach can and cannot do.
Star ratings and survey scores tell you how customers feel on average. The text they write tells you what happened, in their own words. This article explains why that text is worth analyzing, where it adds value, and how the ways of reading it at scale have changed.
A four-star review of a heating repair can contain praise for a friendly technician, a complaint about a late arrival, and a surprise on the invoice. The rating compresses all three into one number. The text keeps them apart.
That difference is the starting point for everything on this site. We test methods for turning large volumes of customer text into evidence that can be counted, checked, and traced back to what people actually wrote. Before the individual experiments, it helps to see the landscape: what text adds, where it is used, which families of methods exist, and why none of them removes the need to validate the result.
Why read what customers write
Structured measures ask customers to answer questions someone chose in advance. Text works the other way round. Customers decide what to mention, and they describe it in their own terms. That makes it possible to find issues nobody thought to ask about.
Text analysis sits between qualitative and quantitative research. It keeps some of the depth of reading individual accounts while reaching the scale of a survey. A widely cited review in the Journal of Marketing describes a simple sequence that still holds: prepare the text, analyze it, turn the result into measures, and validate those measures before using them.
Text also has limits. People who write reviews choose to do so, and they choose what to write about. Text shows what customers say, not necessarily what most customers experience.
What text has revealed in research
Research has shown repeatedly that text carries information the usual numbers miss.
- Market structure without a survey. Mining car and health forums produced maps of which products consumers compare. They matched brand-switching data closely, and the same approach surfaced drug side effects that a major health reference did not list (Marketing Science, 2012).
- Signals that financial data misses. In peer-to-peer lending, the text of loan applications helped predict default beyond the financial information.
- Ratings hide more than text. Across many review platforms, star ratings are polarized by who chooses to write, while the review text is less extreme. An average rating can conceal what the text still says.
Where the analysis creates value
Text analysis is rarely the end product. Its value comes from feeding familiar marketing and customer decisions with evidence that numbers alone do not provide.
| Use | The question it answers | What text adds |
|---|---|---|
| Consumer and brand perception | Where do customers place us next to competitors, and on which attributes? | Attributes in customers’ own words, and which brands they mention together |
| Satisfaction and experience tracking | Why did satisfaction drop this quarter, and in which customer group? | The reasons behind a score, coded the same way in every wave |
| Marketing-mix modelling | Did the campaign change how customers talk about the brand? | Time series of what customers mention, such as price, service failures, or brand attributes, that can help explain shifts beyond spend |
| Product and service development | Which needs come up that our roadmap does not cover? | Needs customers rarely state in surveys but describe in reviews and forums |
| Early warning and risk | What small problem is growing before it shows in the scores? | Rare issues that appear in text before they move an average |
| Competitive and market monitoring | What do competitors’ customers complain about that we could do better? | Comparable topic profiles across brands, regions, or providers |
In each case the analysis does not replace the existing measure. It explains it, adds early signals, or covers what the measure never asked about.
One review, five ways to read it
There are many ways to read text at scale, and they see different things. The example below follows one short review through five common approaches. Switch between them to compare what each one picks up and what it misses.
★★★★★ Booked a boiler repair for Tuesday. They came a day late and didn't call. The technician was friendly and fixed it quickly, but the bill was €80 more than quoted.
Illustrative review written for this article. Each panel shows the typical behaviour of the method, not the output of a measured run.
The example is deliberately small, but the pattern is general. Methods that rely on fixed words are transparent and cheap but brittle. Methods that learn from examples read context but need those examples. Methods that discover structure need interpretation. Language models combine several of these strengths and add new costs and risks.
How the approaches compare
| Approach | What you define in advance | Main strength | Main limit | Typical use |
|---|---|---|---|---|
| Keyword and Boolean queries | The words to search for | Transparent, fast, cheap | Misses paraphrase and unexpected issues | Monitoring known terms in social listening |
| Dictionaries such as LIWC | Validated word lists per concept | Interpretable, reproducible | Weak on context, negation, and domain language | Measuring well-defined constructs |
| Supervised classifiers | Categories plus labeled examples | Often the most accurate for a fixed task | Needs training data; categories are fixed | High-volume, stable classification |
| Topic models (LDA, BERTopic) | The number of topics, roughly | Discovers themes without a list | Topics need naming and can be unstable | Exploring what a collection is about |
| Language models | Instructions and, optionally, a topic list | Handles paraphrase; plain-language labels; little or no training data | Cost, variation between runs, needs validation | Discovery and labeling with evidence |
Two points from the research behind this table are worth keeping in mind. Trained models usually beat word lists on accuracy, and in sentiment analysis the difference can be large enough to change the conclusions drawn. Topic models, on the other hand, discover themes but do not name them: the statistically best model is not always the one people find easiest to interpret.
What language models change, and what they do not
Large language models changed three practical things.
First, categories no longer need training data. A model can be given a list of topics and asked to apply them, or asked to propose topics from the text itself. Prompt-based approaches such as TopicGPT generate topic descriptions and then assign documents to them.
Second, outputs can be read directly. Instead of a list of words, a model can return a named topic, a sentiment, and the phrase that supports the label. That evidence makes individual assignments easy to inspect.
Third, annotation became cheap enough to test at scale. Language models have proved competitive with crowd workers on several annotation tasks, and marketing research is testing where they can support parts of human analysis.
Other things did not change. Customers who write reviews are still a self-selected group, and associations in text are still associations, not causes. Some problems are new:
- Results can vary. Small changes in wording, or simply running the same input again, can change a model’s output.
- Models change or disappear. Hosted models can be updated or retired, which makes exact replication difficult.
- Benchmarks can be contaminated. If test data appeared in a model’s training data, reported accuracy can be too optimistic.
- Cost scales with volume. Every review sent to a hosted model has a price, and how the prompt is structured can change it considerably.
- Summaries can drop small groups. A fluent overview of thousands of comments can leave out issues raised by only a few people.
The rule that does not change: validate
Every approach above produces numbers that look precise. Whether they measure what they claim to measure is a separate question, and it has to be answered with evidence rather than assumed.
01 Define what counts Write down the categories and what belongs in each before looking at results. apply to a sample
02 Compare with people Check model labels against human coders on examples the method has not seen. measure agreement
03 Look at rare categories Make sure small categories have enough examples to be checked at all. freeze the setup
04 Record the setup Keep the model, prompt, and settings so the result can be reproduced.
A clear prompt is not the same thing as a valid measure. Comparison with human judgment remains the standard, including for the small categories that are easiest to overlook.
What the articles on this site test
The articles here take the language-model approach and examine it piece by piece, using recorded runs rather than impressions.
- From 113,518 Reviews to 23 Topics shows how a topic list can be discovered from the reviews themselves and reviewed before use.
- Reading 200,000 Reviews with LLMs: Without Overpaying measures what output format, reasoning settings, caching, and batch size do to the cost of labeling.
- Reducing LLM Costs with Prompt Caching shows why the order of a prompt can change the bill for the same work.
- What LLM Summaries Miss in Customer Feedback tests how often summaries lose rare topics compared with labeling each document.
The analyses apply the same methods to a single question.
The example review was written for illustration and does not come from any dataset.