The first pass, by the numbers

When people hear "AI triages customer feedback," they tend to imagine one of two things: a toy that writes summaries nobody reads, or a machine that replaces the operations team. The real shape is a compression story, and it's easiest to see with numbers.

Suppose a company receives 20,000 pieces of feedback in a week — support tickets, sales-call transcripts, Slack threads, community posts, early-access notes, telemetry flags. Those are 20,000 records. They are nowhere near 20,000 insights. Inside that pile there might be 8,000 duplicates, 4,000 complaints too vague to act on, 3,000 support problems that belong in a help queue, 2,000 documentation gaps, 1,000 feature requests, a few hundred patterns that point at what the product is fundamentally weak at, a couple hundred serious blockers, a handful of safety escalations — and a few genuinely new signals that should shape the roadmap. The job of a first pass is turning that chaos into a small, structured, evidence-attached set of issues that humans can judge. I built Cherry around this idea, and the eight-stage pipeline on its homepage is this essay in diagram form.

What the machine should and shouldn't touch

A useful first pass divides labor strictly. Ordinary software retrieves and calculates: ticket counts, customer counts, revenue on the affected accounts, dates, plan tiers, existing ticket IDs, who owns which product area. The model interprets: what the customer was trying to do, what actually went wrong, whether two complaints share a mechanism, which evidence is representative, who should own the fix. The rule I hold: the model should never invent what a database can answer. When a cluster says "47 reports across 29 independent accounts," those numbers came from joins, never from generation — the model's contribution is the claim that those 47 reports describe the same underlying thing.

That claim is where the interesting failures live, and there are two of them, pulling in opposite directions.

Missed duplicates are the obvious one: "loses track of instructions in long sessions," "my configuration rules disappear," "it stops following our conventions halfway through" — five phrasings, one issue. Semantic clustering catches most of this.

False duplication is the quieter killer. Take five complaints that all mention the product "forgetting": one is a user expecting memory that was never saved, one is an admin whose configured policy isn't applying, one is a context-length limitation, one is a genuine bug, one is a product-education gap. Bad synthesis merges them into a giant cluster called "the product forgets things" — which sounds organized and is useless, because it has five different mechanisms, five different owners, and no single fix. Good synthesis separates by mechanism and use case. This is precisely why the first pass needs evals and human correction: clustering isn't correct just because the output sounds tidy.

One more quiet job before anything gets created: check what already exists. Does this issue have a ticket? Did leadership already rule it an intentional tradeoff in March? Is engineering mid-investigation? A first pass that skips this step becomes a ticket-duplication machine — industrializing the very noise it was built to reduce.

The economics

Run the funnel on 10,000 records: perhaps 1,500 contain no actionable feedback, 3,000 are near-duplicates, and the remaining substance condenses into a couple hundred candidate clusters — most merging into known issues, a few dozen genuinely new, a small number routed with high confidence, an ambiguous handful sent to human review, and one or two escalated immediately. The humans deeply read: the escalations, the ambiguous clusters, samples from the high-volume ones, and every low-confidence call. Everything else, they audit.

That's the entire trade. Human attention stops being spread across everything and lands exactly where judgment changes the outcome. The volume of manual work shrinks, and the team's judgment moves up the stack: from reading every submission to auditing samples and hard cases; from applying tags to designing the taxonomy and evaluating the tagging; from writing weekly summaries to interpreting changes for decision-makers; from routing every ticket to defining ownership rules and adjudicating exceptions; from counting feedback to deciding what counts as signal. The first pass compresses the analyst layer. Judgment doesn't compress.

What "learning from corrections" means, concretely

When a reviewer changes a route or splits a cluster, "the system learns" can mean five different things, in escalating order: the correction is stored against the issue; similar past corrections are retrieved as context for future calls; a recurring error pattern prompts a change to the taxonomy or the instructions; the corrected case becomes a permanent labeled test in the eval suite; and, far downstream, enough high-quality corrections might inform actual model training. The first four are operations work — taxonomy, prompts, evals, governance. Only the fifth touches the model itself, and it comes last, after the cheaper levels have been exhausted. I wrote about the measurement side of this in why the human stays in the loop; the short version is that a correction only becomes learning once it's measured.

The job that remains

So does the first pass replace the operations function? It automates the portion dominated by reading, tagging, and copying between systems. What it creates is a harder and more interesting job: design the operating model, define what quality means, build the evals that keep the machine honest, govern the sensitive data, resolve ambiguous ownership, and answer the only metric that ultimately matters — whether any of this changes what gets built. A system can do analyst-scale first-pass work all day. Someone still has to be responsible for whether it's doing it well, and for what happens next.