RepSignal

Problem

Sales conversation-intelligence tools listen to calls and tell a rep what to do better next time. The interesting forward-deployed problem is not the model; it is the integration: taking an unstructured transcript that arrives over a webhook and turning it into structured, schema-validated coaching a downstream system can actually trust. RepSignal is a one-day build of exactly that integration. A sales-call transcript arrives at a Node.js webhook, gets validated, goes to an LLM under a forced tool-use contract, and comes back as a typed coaching scorecard.

Why it mattered

I built RepSignal to close a specific, recurring gap in my portfolio: TypeScript. Most of my shipped work is Python, and forward-deployed and integrations roles keep asking for a TypeScript and Node service that turns messy customer data into validated structured output. The honest claim I wanted to be able to defend is narrow: I can stand up a strict-TypeScript monorepo, put a shared Zod schema at the center as the single source of truth, and run an LLM where the failure mode is a schema-invalid field rather than a bad vibe. The eval is the point, and I ran it live on my own key rather than reporting a number I assumed.

Architecture

• A sales-call transcript arrives at a Node.js and Express webhook endpoint (POST /webhooks/transcripts).
• The payload is validated against a shared Zod schema before anything else touches it. That schema lives in its own package and is the single source of truth: both the API and the dashboard import it, and the inferred TypeScript types come from it rather than being written twice.
• The talk/listen ratio, the one field that is a pure computation rather than a judgment, is computed in code by a single deterministic function (computeTalkListenRatio) that both the API and the eval import. The model never sees it.
• The validated transcript is passed to Claude (claude-sonnet-5) with forced tool-use, so the model must return a structured coaching scorecard rather than prose. The model judges only the five fields that need judgment: discovery questions asked, objections raised, whether a next step was secured with an evidence quote, risk flags, and coaching tips.
• The scorecard is validated against the same schema on the way out, with the deterministic talk/listen ratio merged in, so the served object carries the full conversation-intelligence signal.
• The scorecard is served by a typed REST API (GET /scorecards/:id, GET /health) and rendered by a React and TypeScript (Vite) dashboard.
• A side rail runs the eval harness, a self-authored synthetic 14-transcript set that scores the five LLM-judged fields field by field and checks the deterministic ratio separately, plus 47 offline Vitest tests under ESLint and GitHub Actions. CI runs with no API key, because the LLM responses in tests are fixtured.
• The whole thing is a Turborepo monorepo, so the schema package, the API, and the dashboard build and test as one graph.

Technical decisions

The computable metric belongs in code, not in the model. The talk/listen ratio is arithmetic: it is speaker-time divided by total time, and it has one correct answer for a given transcript. My first version let the LLM estimate it alongside the genuine-judgment fields, and the eval showed exactly why that was wrong, since the model's estimates drifted just outside the golden tolerance. So I moved that metric out of the LLM into a single deterministic function shared by the API and the eval, and now the model only judges what actually needs judgment. This eliminated a whole class of misses at a stroke and made the ratio exact by construction, while shrinking the surface the LLM is accountable for from six fields to five.

One Zod schema as the single source of truth, not types written twice. Alternative: hand-write TypeScript interfaces and validate separately. A conversation-intelligence integration sits at a trust boundary in both directions, untrusted transcript coming in and model-generated scorecard going out, so a single Zod schema validates both and the TypeScript types are inferred from it. There is one place a field can change.

Forced tool-use, so the model returns a scorecard, not prose. Alternative: prompt for JSON and hope. The risk in an extraction step is a plausible-sounding malformed field, so the model is constrained to the tool schema and the result is Zod-validated before it is stored or served. A bad field is rejected, not rendered.

Offline, fixtured CI with no API key. Alternative: call the live model in CI. Tests that hit a paid nondeterministic API are slow, flaky, and leak keys, so the 47 tests run against fixtured LLM responses and stay deterministic. The live model is exercised separately, as an eval I run by hand.

A Turborepo monorepo instead of three loose packages. Alternative: separate repos or a single flat app. Keeping the schema package, the API, and the dashboard in one Turborepo graph is what makes "the dashboard uses the same types as the API" a build guarantee rather than a hope.

Interesting challenges

The sharpest part of this build was the live eval, and specifically what it caught. I scored the extractor against a self-authored synthetic 14-transcript set covering strong discovery, missed next steps, heavy objections, and monologue-heavy bad talk ratios. The model judges five fields per transcript, so the LLM-judged score is out of 70, and it got 66 of 70 LLM-judged field checks correct on this run. The talk/listen ratio sits outside that number: it is computed deterministically, so it is 14 of 14 by construction and I report it separately rather than folding it into the 70. The 66 of 70 is representative, not fixed: the LLM varies run to run, so a rerun lands near it rather than exactly on it. The number matters less than how I got it. Running the eval live surfaced three real bugs that offline mocks would have hidden. In the monorepo, dotenv was loading from the workspace directory instead of the repo root, so the key was never found. The API's default port collided with a local Next.js server. And on Node 24, undici was dropping long model responses with a "Premature close" error, which I fixed by pinning Node 20 LTS and hardening the streaming path with retries. Each was fixed, and the first two got regression tests.

Moving the talk/listen ratio into deterministic code did more than make one field exact: it removed a whole class of misses. In the earlier six-field version, the ratio estimates drifted just outside the golden tolerance band and cost several checks, and those misses are now gone. The sole remaining disagreement is that the model is conservative about risk. On 4 of the 14 transcripts it raises a risk flag where the golden label had none, over-flagging on ambiguous lines rather than under-flagging. The golden labels were written to be right, not tuned to make the score look better.

Results

Scope and honest limits

RepSignal is an independent one-day project. Every transcript is synthetic, self-authored, and self-scored; there is no real call data anywhere in it. It is not a real integration: it is not connected to any CRM, telephony provider, or conversation-intelligence platform, and it is not affiliated with or endorsed by any company in that space.

The eval is a self-authored synthetic 14-transcript set. The result of 66 of 70 LLM-judged field checks is a clean-enough score on that small set from a single live run, and because the LLM varies, it is representative rather than a fixed benchmark. The talk/listen ratio is a separate deterministic number, 14 of 14 by construction, and it is never folded into the 70. A larger or independently authored set would move the LLM-judged number.

The infra/ directory holds a minimal Terraform config as a demonstration of how the service would deploy. It was never applied, so there is no live backend. The live demo at https://armaangulati1.github.io/repsignal/ is a static demo-mode build of the dashboard showing one synthetic sample scorecard (call-0003 from the eval set); it does not call the API and it uses no real data.

What this project does not claim: no TypeScript or Node.js at production scale or over multiple years, because this is one project; no real customer-data, telephony, or CRM integration; no distributed data systems; and no Terraform beyond the single never-applied demo config.

Lessons

Running an eval live is a different activity from reporting one. The 66 of 70 LLM-judged field-check result on that synthetic 14-transcript set is only trustworthy because producing it broke the pipeline three times, at the environment, the port, and the HTTP layer, and I fixed each break before trusting the number.

The eval also told me where a metric belonged. Watching the talk/listen ratio miss on a value that has one correct answer was the signal to move it out of the model and into a deterministic function. The lesson is to let the model judge only what needs judgment, and to compute everything computable in code where it can be exact.

A schema at the center pays for itself at both boundaries. The same Zod definition catches a malformed transcript coming in and rejects a malformed scorecard going out, and because the TypeScript types are inferred from it, the API and the dashboard cannot silently disagree about a field.

Future work

• Expand the eval beyond 14 self-authored synthetic transcripts, ideally with transcripts written by someone other than me, before trusting the field-check number past a demo.
• Calibrate the risk-flag threshold, the sole remaining miss type, against a labeled tolerance so the model stops over-flagging on ambiguous lines.
• Connect a real transcript source behind the same webhook contract to test whether the schema holds against messy real-world input rather than clean synthetic transcripts.