Case study · consumer insights

What makes drinkware buyers unhappy, and can an AI sort the reviews?

I took 2,000 Amazon reviews of insulated drinkware from four brands (YETI, Stanley, Hydro Flask, Owala) and asked two questions: what do unhappy buyers write about, and can an AI read reviews well enough to watch them every month? An open AI model read every review, an LLM (Claude) named the main complaint in the 300 unhappy ones, and I checked the AI against the reviewers' own stars before using any of it.

14 of 76YETI one- and two-star reviews say a lid piece was missing, 7 of them the MagSlider

16.0%of YETI's reviews are 1 or 2 stars (Stanley 15.0%, Hydro Flask 18.7%, Owala 12.2%)

86%of reviews the AI sorts on its own at 96.7% accuracy; a person reads the rest

What I found

YETI's unhappy reviews split three ways, and one is easy to fix. Wrong or missing parts (26%), doesn't keep cold or hot (24%) and leaks or lid problems (24%). The missing parts stand out: 14 of those 20 reviews say a lid piece was missing, 7 of them the MagSlider.
Each brand has its own weak spot. Stanley: damaged, dented or broken (34%) and doesn't keep cold or hot (27%); Hydro Flask: leaks or lid problems (36%) and damaged, dented or broken (21%); Owala: damaged, dented or broken (41%) and leaks or lid problems (23%).
The AI is good enough to read every review. It read the mood the way the stars say 94.4% of the time (kappa 0.82), and after a one-number fix its confidence means what it says, so a cut-off can decide which reviews a person reads.

What YETI could do with it

Find out where the lid pieces go missing. Packing, returned units sold again as new, or third-party sellers: each has a different fix, and the review dates and listings help tell them apart.
Watch the temperature complaints. Keeping drinks cold or hot is the core promise, and it is 24% of YETI's unhappy reviews. A monthly view shows whether that share moves after a product or lid change.
Let the AI read every review, every month. At a 90% cut-off it sorts 86% on its own and is right 96.7% of the time; a person reads the other 14%, the hard cases.

The main finding

What unhappy buyers write about, by brand

Each one- and two-star review got one main complaint, named by the LLM with a short codebook. Each row adds up to 100%.

	Leaks or lid problems	Doesn't keep cold or hot	Damaged, dented or broken	Wrong or missing parts	Size, weight or design	Cleaning, smell or taste	Too expensive	Something else
YETI76 of 476 reviews (16%) are 1 or 2 stars	24%	24%	13%	26%	4%	4%	1%	4%
Stanley71 of 473 reviews (15%) are 1 or 2 stars	21%	27%	34%	8%	4%	0%	1%	4%
Hydro Flask92 of 492 reviews (19%) are 1 or 2 stars	36%	16%	21%	13%	4%	5%	3%	1%
Owala61 of 500 reviews (12%) are 1 or 2 stars	23%	7%	41%	13%	3%	8%	0%	5%

So what. YETI: wrong or missing parts (26%) and doesn't keep cold or hot (24%); Stanley: damaged, dented or broken (34%) and doesn't keep cold or hot (27%); Hydro Flask: leaks or lid problems (36%) and damaged, dented or broken (21%); Owala: damaged, dented or broken (41%) and leaks or lid problems (23%). With 61 to 92 unhappy reviews per brand, these are directions to check, not final numbers; a monthly tracker would confirm them.

Reading behind YETI's missing parts

I read the 20 YETI reviews about a wrong or missing part. 14 say a lid piece was missing, and 7 of those name the slider (the MagSlider). The other 6: a straw lid that didn't fit, a handle that didn't match the listing, and items that seemed used or fake. So the fix sits with packing, returns and sellers more than with product design.

The data

2,000 real reviews, checked before use

Source: Amazon Reviews 2023 (McAuley Lab, UC San Diego), the newest public release of Amazon reviews. I kept insulated tumblers, mugs and bottles from the four brands, the newest 21 months in the release (2022-01-01 to 2023-09-10), and drew 500 reviews per brand at random. A title filter dropped coolers and most accessories; a few lids and accessories sold on their own (mostly Hydro Flask's press-in lid) got through and are kept.

2,000reviews, 500 per brand

0 / 0duplicate IDs / empty reviews

59Spanish reviews set aside (this model reads English)

70%are 5 stars; 309 are 1 or 2 stars

Limits: Amazon reviewers are not all buyers, and the release ends in September 2023. The method is the point: the same steps run on a brand's own reviews or survey comments.

How the AI read the reviews

A fast model for every review, an LLM for the complaints

Step 1. Laya is an open "decision model" (Apache 2.0, 421M parameters, runs on a laptop). Instead of writing text, it answers typed questions with a probability for each answer, so there is nothing to parse and nothing it can make up. Each review got these questions:

Is the reviewer happy with the product? A: yes · B: no
For 1-2 star reviews, seven yes/no questions: does the reviewer complain that it leaks or the lid is a problem · it doesn't keep drinks cold or hot · it arrived damaged or broke · a part was wrong or missing · the size, weight or design · cleaning, smell or taste · the price?

Step 2. Used as is, the decision model is very good at mood but mixes up complaint types: it agreed with the LLM on only 61% of the unhappy reviews. So the 300 unhappy reviews also went to Claude with a one-paragraph codebook: the same complaint types and the rule "pick the one the reviewer stresses most". The complaint numbers on this page come from Claude.

Check 1 · Does it read the mood the way the reviewer rated it?

The stars are the reviewer's own verdict: 4-5 stars = happy, 1-2 = unhappy (3 stars are mixed and left out).

	Model: happy	Model: unhappy
Stars: happy (4-5)	1,404	91
Stars: unhappy (1-2)	9	291

So what. It agreed 94.4% of the time (kappa 0.82) and missed only 9 of 300 unhappy reviews. Reading those misses: most are calm reports of a missing piece or a dent from buyers who still like the product, and one reads like a five-star review.

Check 2 · When it says "90% sure", is it right 90% of the time?

I grouped the reviews by how sure the model said it was and checked how often it was right. Then I fixed it with one number (a "temperature", T = 0.7), learned on one half of the reviews and tested on the other half.

Before the fixAfter the fix · dashed line = perfectly calibrated · dot size = number of reviews · bands with fewer than 20 reviews are left off the chart (too few to judge)

So what. Calibration error on the test half went from 0.037 to 0.014. Before the fix the model was right more often than it claimed (under-confident), so the fix sharpened its probabilities. Now a cut-off on its confidence means something.

The decision · Which reviews can it handle alone?

Cut-off (model at least this sure)	Handled by the model	Right when it handled them	Sent to a person
60%	98%	94.9%	16 of 875
70%	97%	95.6%	30 of 875
80%	94%	96.0%	55 of 875
90%	86%	96.7%	122 of 875
95%	74%	97.7%	223 of 875

So what. At a 90% cut-off the model handles 86% of reviews and is right 96.7% of the time; a person reads the rest. That is the human-in-the-loop setup: the model does the volume, people do the hard cases.

A review tracker. Every new review scored the same way: mood, complaint type, model confidence. One table by product and month.
A check set. 100 reviews labeled by a person, re-run whenever the model or the questions change, so a drop in agreement shows up before the numbers move. It also measures how far to trust the complaint types.
A routing rule with a budget. The cut-off set by how many reviews a person can read each week.
Alerts. When one complaint type jumps for a product (for example leaks after a lid change), flag it to the product team with five example reviews.

How it's built (code, files, limits)

fetch_data.py streams the public review files (41 GB) and draws the sample · run_model.py asks the decision model the questions and saves its answers (ai_answers.csv) · llm_code.py has Claude name the main complaint of each unhappy review (llm_complaints.csv) · analysis.py runs the checks and the brand comparison and writes results.json · build_page.py builds this page from results.json. All files: code.

m["ai_happy"] = (m["p_happy"] >= 0.5).astype(int)
accuracy = (m["ai_happy"] == m["happy_by_stars"]).mean()
p_fixed = p**(1/T) / (p**(1/T) + (1-p)**(1/T))      # the one-number fix (temperature)

Limits: one marketplace and one period; Amazon reviews skew to strong opinions; the complaint types are one LLM's reading and still need a person's check on a labeled sample (the check set above); the decision model was used as is, with no training on this data.