"""Build the case study page (page/index.html) from results.json. Every number on the page comes from results.json. Run: python3 build_page.py (after python3 analysis.py) The page is self-contained (no scripts), in the tabrizianalytics.com look.""" import html import json from pathlib import Path R = json.loads(Path("results.json").read_text()) NAMES = R["names"] TYPES = ["leaks_lid", "temperature", "damage", "wrong_item", "size_design", "cleaning_taste", "price"] BRANDS = [b["brand"] for b in R["by_brand"]] E = html.escape def pct(x, d=0): return "n/a" if x is None else f"{x * 100:.{d}f}%" def reliability_svg(before, after): W, H, L, B, T, Rm = 460, 300, 46, 40, 14, 14 X = lambda v: L + (v - 0.5) / 0.5 * (W - L - Rm) Y = lambda v: H - B - (v - 0.5) / 0.5 * (H - B - T) s = [f''] for v in [0.5, 0.6, 0.7, 0.8, 0.9, 1.0]: s.append(f'' f'{round(v*100)}%' f'{round(v*100)}%') s.append(f'') for pts, cls in [(before, "b"), (after, "a")]: pts = [p for p in pts if p["n"] >= 20] # bands with under 20 reviews are too few to judge d = " ".join(f"{X(p['said']):.0f},{Y(max(p['right'], 0.5)):.0f}" for p in pts) s.append(f'') for p in pts: s.append(f'') s.append(f'How sure the model said it was' f'How often it was right') s.append("") return "".join(s) def heat_table(): rows = [] head = "".join(f"{E(NAMES[t])}" for t in TYPES + ["other"]) for b in R["by_brand"]: cells = [] for t in TYPES + ["other"]: v = b[t] alpha = min(1, v / 0.6) cells.append(f'{pct(v)}') rows.append(f'{E(b["brand"])}{b["unhappy"]} of {b["reviews"]} reviews ({pct(b["unhappy_share"])}) are 1 or 2 stars{"".join(cells)}') return f'
{head}{"".join(rows)}
' def top_complaints(b, k=2): pairs = sorted(((b[t], t) for t in TYPES), reverse=True)[:k] # "other" is not a complaint type return " and ".join(f"{NAMES[t].lower()} ({pct(v)})" for v, t in pairs) mood, cal, routes, human = R["mood"], R["calibration"], R["routing"], R["human_check"] chk = R["checks"] pick = next((r for r in routes if r["cut"] == 0.9), routes[-1]) yeti = next(b for b in R["by_brand"] if b["brand"] == "YETI") others = [b for b in R["by_brand"] if b["brand"] != "YETI"] lid = R["yeti_missing_lid_part"] top3 = sorted(((yeti[t], t) for t in TYPES), reverse=True)[:3] # The YETI lines below are written for these three complaint types leading YETI's list; stop if new data changes that. assert {t for _, t in top3} == {"wrong_item", "leaks_lid", "temperature"}, top3 stars_line = ", ".join(f"{b['brand']} {pct(b['unhappy_share'], 1)}" for b in others) few, many = min(b["unhappy"] for b in R["by_brand"]), max(b["unhappy"] for b in R["by_brand"]) if human: # filled in once data/my_labels.csv exists (analysis.py Step 7) human_card = f"""

Check 3

Does the LLM agree with a person on what the complaint is?

I labeled {human['n']} one- and two-star reviews myself, without seeing either AI's answers, and compared both with my labels.

{pct(human['llm_agree'])}of {human['n']} reviews: the LLM's main complaint matched mine (kappa {human['llm_kappa']:.2f})
{pct(human['dm_agree'])}the decision model's match on the same reviews (kappa {human['dm_kappa']:.2f})
{''.join(f"" for p in human['per_type'])}
Main complaint (my label)ReviewsLLM agreedDecision model agreed
{E(p['name'])}{p['mine']}{p['llm_matched']}{p['dm_matched']}
""" else: human_card = "" page = f""" What makes drinkware buyers unhappy · Tabrizi Analytics
← Tabrizi Analytics  ·  Drinkware reviews  ·  Code

Case study · consumer insights

What makes drinkware buyers unhappy, and can an AI sort the reviews?

I took {chk['rows']:,} Amazon reviews of insulated drinkware from four brands ({', '.join(BRANDS)}) and asked two questions: what do unhappy buyers write about, and can an AI read reviews well enough to watch them every month? An open AI model read every review, an LLM (Claude) named the main complaint in the {R['unhappy_total']} unhappy ones, and I checked the AI against the reviewers' own stars before using any of it.

{lid['n']} of {yeti['unhappy']}YETI one- and two-star reviews say a lid piece was missing, {lid['slider']} of them the MagSlider
{pct(yeti['unhappy_share'], 1)}of YETI's reviews are 1 or 2 stars ({stars_line})
{pct(pick['auto_share'])}of reviews the AI sorts on its own at {pct(pick['auto_right'], 1)} accuracy; a person reads the rest

What I found

What YETI could do with it

  1. Find out where the lid pieces go missing. Packing, returned units sold again as new, or third-party sellers: each has a different fix, and the review dates and listings help tell them apart.
  2. Watch the temperature complaints. Keeping drinks cold or hot is the core promise, and it is {pct(yeti['temperature'])} of YETI's unhappy reviews. A monthly view shows whether that share moves after a product or lid change.
  3. Let the AI read every review, every month. At a 90% cut-off it sorts {pct(pick['auto_share'])} on its own and is right {pct(pick['auto_right'], 1)} of the time; a person reads the other {pct(1 - pick['auto_share'])}, the hard cases.

The main finding

What unhappy buyers write about, by brand

Each one- and two-star review got one main complaint, named by the LLM with a short codebook. Each row adds up to 100%.

{heat_table()}
So what. {'; '.join(f"{b['brand']}: {top_complaints(b)}" for b in R['by_brand'])}. With {few} to {many} unhappy reviews per brand, these are directions to check, not final numbers; a monthly tracker would confirm them.

Reading behind YETI's missing parts

I read the {lid['of']} YETI reviews about a wrong or missing part. {lid['n']} say a lid piece was missing, and {lid['slider']} of those name the slider (the MagSlider). The other {lid['of'] - lid['n']}: a straw lid that didn't fit, a handle that didn't match the listing, and items that seemed used or fake. So the fix sits with packing, returns and sellers more than with product design.

The data

{chk['rows']:,} real reviews, checked before use

Source: Amazon Reviews 2023 (McAuley Lab, UC San Diego), the newest public release of Amazon reviews. I kept insulated tumblers, mugs and bottles from the four brands, the newest 21 months in the release ({chk['first_day']} to {chk['last_day']}), and drew 500 reviews per brand at random. A title filter dropped coolers and most accessories; a few lids and accessories sold on their own (mostly Hydro Flask's press-in lid) got through and are kept.

{chk['rows']:,}reviews, 500 per brand
{chk['duplicate_ids']} / {chk['empty_text']}duplicate IDs / empty reviews
{chk['spanish']}Spanish reviews set aside (this model reads English)
{pct(chk['stars'][4] / chk['rows'])}are 5 stars; {chk['stars'][0] + chk['stars'][1]} are 1 or 2 stars

Limits: Amazon reviewers are not all buyers, and the release ends in September 2023. The method is the point: the same steps run on a brand's own reviews or survey comments.

How the AI read the reviews

A fast model for every review, an LLM for the complaints

Step 1. Laya is an open "decision model" (Apache 2.0, 421M parameters, runs on a laptop). Instead of writing text, it answers typed questions with a probability for each answer, so there is nothing to parse and nothing it can make up. Each review got these questions:

Is the reviewer happy with the product? A: yes · B: no
For 1-2 star reviews, seven yes/no questions: does the reviewer complain that it leaks or the lid is a problem · it doesn't keep drinks cold or hot · it arrived damaged or broke · a part was wrong or missing · the size, weight or design · cleaning, smell or taste · the price?

Step 2. Used as is, the decision model is very good at mood but mixes up complaint types: it agreed with the LLM on only {pct(R['coders_agree'])} of the unhappy reviews. So the {R['unhappy_total']} unhappy reviews also went to Claude with a one-paragraph codebook: the same complaint types and the rule "pick the one the reviewer stresses most". The complaint numbers on this page come from Claude.

Check 1 · Does it read the mood the way the reviewer rated it?

The stars are the reviewer's own verdict: 4-5 stars = happy, 1-2 = unhappy (3 stars are mixed and left out).

Model: happyModel: unhappy
Stars: happy (4-5){mood['table']['stars_happy_ai_happy']:,}{mood['table']['stars_happy_ai_unhappy']:,}
Stars: unhappy (1-2){mood['table']['stars_unhappy_ai_happy']:,}{mood['table']['stars_unhappy_ai_unhappy']:,}
So what. It agreed {pct(mood['accuracy'], 1)} of the time (kappa {mood['kappa']:.2f}) and missed only {mood['unhappy_missed']} of {mood['unhappy_n']} unhappy reviews. Reading those misses: most are calm reports of a missing piece or a dent from buyers who still like the product, and one reads like a five-star review.

Check 2 · When it says "90% sure", is it right 90% of the time?

I grouped the reviews by how sure the model said it was and checked how often it was right. Then I fixed it with one number (a "temperature", T = {cal['T']}), learned on one half of the reviews and tested on the other half.

{reliability_svg(cal['bands_before'], cal['bands_after'])}
Before the fixAfter the fix · dashed line = perfectly calibrated · dot size = number of reviews · bands with fewer than 20 reviews are left off the chart (too few to judge)
So what. Calibration error on the test half went from {cal['ece_test_before']:.3f} to {cal['ece_test_after']:.3f}. {"Before the fix the model was right more often than it claimed (under-confident), so the fix sharpened its probabilities." if cal['T'] < 1 else "Before the fix the model claimed more than it delivered (over-confident), so the fix softened its probabilities."} Now a cut-off on its confidence means something.

The decision · Which reviews can it handle alone?

{''.join(f"" for r in routes)}
Cut-off (model at least this sure)Handled by the modelRight when it handled themSent to a person
{pct(r['cut'])}{pct(r['auto_share'])}{pct(r['auto_right'], 1)}{r['to_person']} of {cal['test_n']}
So what. At a 90% cut-off the model handles {pct(pick['auto_share'])} of reviews and is right {pct(pick['auto_right'], 1)} of the time; a person reads the rest. That is the human-in-the-loop setup: the model does the volume, people do the hard cases.
{human_card}

Next

What I'd set up to run every month

  1. A review tracker. Every new review scored the same way: mood, complaint type, model confidence. One table by product and month.
  2. A check set. 100 reviews labeled by a person, re-run whenever the model or the questions change, so a drop in agreement shows up before the numbers move. It also measures how far to trust the complaint types.
  3. A routing rule with a budget. The cut-off set by how many reviews a person can read each week.
  4. Alerts. When one complaint type jumps for a product (for example leaks after a lid change), flag it to the product team with five example reviews.
How it's built (code, files, limits)

fetch_data.py streams the public review files (41 GB) and draws the sample · run_model.py asks the decision model the questions and saves its answers (ai_answers.csv) · llm_code.py has Claude name the main complaint of each unhappy review (llm_complaints.csv) · analysis.py runs the checks and the brand comparison and writes results.json · build_page.py builds this page from results.json. All files: code.

m["ai_happy"] = (m["p_happy"] >= 0.5).astype(int)
accuracy = (m["ai_happy"] == m["happy_by_stars"]).mean()
p_fixed = p**(1/T) / (p**(1/T) + (1-p)**(1/T))      # the one-number fix (temperature)

Limits: one marketplace and one period; Amazon reviews skew to strong opinions; the complaint types are one LLM's reading and still need a person's check on a labeled sample (the check set above); the decision model was used as is, with no training on this data.

""" Path("page").mkdir(exist_ok=True) Path("page/index.html").write_text(page) print("Wrote page/index.html")