Part of Fin Channel Communication Standards
A 250-conversation audit of whether Fin's chat replies actually read like chat — and the 9% that proved it already could
A chat window sets a specific expectation: short exchanges, one thing at a time, something that feels like a real back-and-forth rather than a document. The question was whether Fin's actual replies in the chat widget matched that expectation, or whether they read more like an email pasted into a chat box. Nobody had measured this directly. The only way to find out was to read a real sample of what Fin was actually sending.
The chat conversation population over a 60-day window ran to 7,051 conversations where Fin participated. A random sample of 250 of those got read in full, turn by turn, and scored against five criteria:
The results were consistent and specific, not vague. 91% of sampled conversations failed on chunking, meaning Fin was overwhelmingly front-loading multi-step answers into a single message instead of pacing them out. The same 91% failure rate showed up on template language, with phrases like "Is that what you were looking for?" and "Did that answer your question?" appearing in the vast majority of replies regardless of context. 69% failed on reply length. Only about 9% of the sample, 22 conversations, hit the ideal pattern: short, chunked, collaborative.
That 9% turned out to be the most useful data point in the whole audit. Nearly all of those well-behaved conversations were browser and technical troubleshooting conversations specifically, not a random scatter across topics. Tracing why led straight to the cause: the only guidance in the entire configuration that explicitly instructs chunking, checking in, and collaborative framing is scoped to a single troubleshooting procedure. It's not that Fin can't communicate the right way, it clearly can, it's that the instruction to do so was never generalized past one narrow use case. The other 91% of conversations, covering course setup, billing, and account questions, had no equivalent instruction at all.
A second, separate finding surfaced during the same review: in one conversation, a customer explicitly typed "Stop chatting" and then an all-caps demand to be left alone, and Fin nonetheless sent two more follow-up messages afterward. That conversation scored the lowest possible customer satisfaction rating. The follow-ups turned out to be generated by an automated idle-timer workflow running independently of Fin's own reasoning, not something guidance alone could suppress, which meant the fix had to be a workflow-level change, not just a wording change.
The audit itself is complete: a real sample read and scored against defined criteria, a specific root cause identified for the two dominant failure patterns (chunking and template language), and a set of concrete recommendations drafted for each, including exact proposed guidance language and one workflow-level fix for the "stop chatting" failure mode. Whether those recommendations have since been implemented isn't confirmed as of this audit.
The fix a customer-facing AI agent needs is rarely "teach it something new." Fin already knew how to communicate the right way, the proof was sitting in 9% of its own conversation history. The actual problem was a scoping mistake: a good instruction that existed in exactly one place instead of everywhere it was needed. Finding that kind of gap requires reading real conversations end to end rather than trusting an aggregate pass rate, since the aggregate alone wouldn't have pointed at a single narrow procedure as the explanation.