Part of Fin Channel Communication Standards
A 241-conversation email audit, cross-validated against CX Score data, that turned into thirteen shipped changes — each traceable to a specific number
Email is a fundamentally different promise to a customer than chat. Nobody is sitting there waiting for the next message to arrive in thirty seconds, which means an email reply needs to be complete, well-structured, and self-contained the first time, not a quick conversational volley. The question was whether Fin's email behavior actually matched that expectation, and the only way to know for certain was to read a real, large sample of what Fin was actually sending customers by email.
The population covered 9,137 email conversations where Fin participated over a 90-day window. A random sample of 250 got pulled, and 241 were successfully scored across five criteria:
The scores broke down sharply by conversation type. Billing conversations scored 60% complete. Technical conversations scored 0% complete, the worst of any category. Account conversations scored 33%. The pattern behind the Technical failure was consistent across every case: Fin was being routed straight to a human handoff before it ever got a chance to attempt a substantive reply, even in cases where a partial, helpful answer was possible. The single most common failure across every category was a generic, chat-style auto-escalation message, emoji included, that carried no personalization, no acknowledgment of the actual question, and no explanation of what happens next or when. A second recurring pattern: student-specific requests, refunds and email changes that are actually the responsibility of the individual school rather than the company directly, were being escalated to the support team instead of redirected to the right party, adding unnecessary handling time on both ends.
A follow-up pass layered CX Score data from the full 9,137-conversation population directly against the audit's own findings, and the two data sets, gathered completely independently of each other, pointed at the same conclusion. Technical conversations had both the lowest completeness score in the audit (0%) and the lowest CX Score of any high-volume topic (51.8%). That cross-validation is what turned a sample-based finding into something with real confidence behind it. It also surfaced a finding the original audit hadn't been looking for: customers writing in simply to ask how to reach support scored the single lowest CX rating of any topic (44.4%), even though the audit itself had scored those same replies as 100% complete, revealing that being correctly redirected and feeling well-treated are not the same thing.
That combined picture turned into thirteen concrete changes to how Fin handles email, each one traceable back to a specific number from the audit or the CX data, not a general sense that things could be better:
Seven new email-specific rules:
One new procedure, an eight-step flow covering a refund request end to end, from checking eligibility through collecting proof to escalating, aimed at the single largest category of dissatisfied email conversations in the whole 90-day window (188 of them). That procedure's actual before/after impact, including a real, sometimes counterintuitive read on what "working" looks like, is its own story: see Automating the Triage Step on Student Refund Requests.
Five existing rules updated or turned back on:
All thirteen changes shipped. Each one maps back to a specific, named number from the audit, whether that's a completeness score, a CX Score, a resolution rate, or a count of dissatisfied conversations in a category, rather than a general impression that something needed fixing. What hasn't happened yet is the follow-up measurement: rerunning the same scoring against a post-change sample to confirm the baseline numbers (the 0% Technical completeness, the 44.4% CX Score on contacting support, the overall resolution rate) actually moved. That re-audit is the open piece.
An audit is only as useful as the traceability between what it finds and what changes because of it. Every one of the thirteen changes here can be pointed back to the exact metric that justified it, which is a different, stronger claim than "we reviewed some conversations and made improvements." The CX Score cross-validation step matters for the same reason: two independently-gathered data sets landing on the same weak spot is a much higher-confidence signal than either one alone, and it's also what surfaced the counterintuitive finding, that a "successfully redirected" customer can still be the least satisfied customer in the dataset, that a single-pass audit would have missed entirely.