Case Studies

The systems I've architected and owned — the problem, the architecture, the numbers each one runs at, and what my actual role was.

1. Chargebee AI Platform & Backstage

Apr 2025 – present · AI Platform team

Problem

AI features were being built ad hoc, each team wiring its own prompts, retrieval and tooling. Once Copilot moved past single-turn RAG into multi-agent execution, the harder problem stopped being “can it answer?” and became “why did it answer that, and how do we know it got better?” — unanswerable without per-step visibility into every agent run.

Solution & Architecture

  • Built the internal AI platform every AI feature at Chargebee now runs on — shared agent runtime, tool contracts, retrieval and model access.
  • Built Backstage, the control plane for managing and evaluating AI interactions.
  • Full interaction traces: user query → each assistant turn → every agent invoked, with per-step latency captured end to end.
  • Skill/agent registry, so product teams register capabilities against the platform instead of forking Copilot.
  • Evaluation over recorded real interactions, so behaviour changes are measured rather than eyeballed. Every LLM and human interaction is reviewable, with random human sampling and an LLM judge scoring quality — the mechanism that makes agent regressions visible before customers find them.

From internal platform to external product

The platform stopped being internal-only. Chargebee’s MCP server now exposes our toolsets to any MCP client — Claude, ChatGPT, Codex CLI — which meant solving the problems that come with being someone else’s dependency:

  • OAuth 2.0 end to end: client pre-registration for both tenant-specific and public MCP URLs, refresh tokens, per-agent client credential management, and a user-facing consent flow where people pick which agent/MCP server they’re authorising and can revoke past consents.
  • Custom MCP servers: merchants compose their own server from available toolsets, with per-tool argument controls (defaults, validation, hiding params) rather than all-or-nothing access.
  • Multi-tenant rate limiting scoped by tenant × user/API/IP, consistent across CLI, desktop and web clients.
  • YAML-defined agents, servers and tools, so a new agent no longer requires a service code change or deploy.
  • End-to-end MCP test suite covering every toolset per server, positive and negative cases, run automatically against each newly published client.

How adoption actually happened

An internal platform only becomes a platform when other teams choose it over building their own. We never had a mandate, so the approach was to make the alternative obviously more expensive:

  • We built the tooling first, not the pitch. Backstage gave any team monitoring, stats, tracking and evaluation on day one — the unglamorous work every team would otherwise have redone badly and separately.
  • We made quality measurable, not arguable. Every LLM and human interaction goes through review: random human sampling plus an LLM judge. That turned “is the agent any good?” from an opinion into a number, which is what a team needs before putting an agent in front of their customers.
  • We went first, on the hardest surface. We built the Copilot orchestrator and the foundation agents — support, data lookup, knowledge base — against the billing app ourselves. The platform’s abstractions were proven on real production load before anyone else was asked to trust them.
  • Then we demonstrated the cost. Showing a team how quickly a new agent could be built and evaluated on the platform did the convincing. Adoption followed the demo, not a memo.

That sequence is why six product teams now build on it. The technical work bought the credibility; the tooling removed the excuse.

Scale & Impact

  • The platform is the substrate for AI across Chargebee — Copilot and downstream AI features build on it.
  • Six product teams — support, onboarding, billing, integrations, receivables and docs — ship their own skills without going through my team, which is what unblocked Copilot’s coverage growth.
  • Published to the Claude marketplace, and adoption has been entirely organic — no outbound push. It reached 90+ live sites within weeks of rollout, growing 125% week over week, and runs on 100+ live sites today.
  • Agent behaviour became debuggable: a slow or wrong answer resolves to a specific agent and step.

Notable Challenge — one URL, three regions

Marketplace listings accept a single MCP URL. We run region-specific endpoints (mcp.chargebee.com, mcp.eu., mcp.au.) for data residency, so satisfying a single-URL registration without breaking regional isolation was a genuine constraint rather than a configuration detail.

Role

  • Architect and builder of both the platform and Backstage.
  • Set the agent/skill contract other teams build against.

2. Chargebee Copilot – Autonomous Agent System

Nov 2024 – present · GA Feb 2025 · AI Platform team

Problem

Support teams were overwhelmed with repetitive billing queries, leading to high ticket inflow and delayed response times.

Solution & Architecture

Copilot runs as a fully autonomous, agent-to-agent (A2A) system:

  • The main Copilot agent acts as an orchestrator, routing work across specialised agents and holding the bar on answer quality and resolution — not just retrieval.
  • Specialised agents include a data lookup agent, a knowledge base agent, and a support first-response agent that works the first-response SLA on the support desk.
  • Opened up as a platform: Copilot was extended beyond a single module, and every team can now contribute skills, so coverage grows without my team as a bottleneck.
  • It resolves L2+ queries automatically — the tier that previously required human investigation.

In production, support has seen it independently handle cases such as correcting an incorrect refund, troubleshooting invoice payments by reading configuration and email logs, customer spend analysis, and attaching accept-quote links to quote emails. The refund case is a representative L2+ investigation: the kind of query that would otherwise have taken several days of human work.

Where it started

The first version, GA in Feb 2025, was a single RAG pipeline: billing-domain retrieval over Pinecone, custom agents mapped to billing primitives (subscriptions, invoices, dunning), built on LangChain, LangGraph, Langfuse, Splunk and AWS. It worked, and it capped out where retrieval-only assistants cap out — it could answer, but it could not investigate. Getting past that ceiling is what forced the move to orchestration.

Where it runs, and what it can do

  • Embedded across every Chargebee application — billing, docs, RevRec, retention, growth — as a resizable sidebar, with conversation history available from the Copilot menu.
  • Permissions come from the user’s own RBAC. Copilot Skills were decoupled from MCP servers and merchant-managed role assignment was removed, so a skill can never exceed what the signed-in user is allowed to do.
  • File attachments — images, PDF, CSV and XLSX — so agents get real context instead of pasted fragments.
  • Interactive UI components as tools, letting the model ask structured multiple-choice questions rather than guessing intent, with human-in-the-loop approval for write actions.
  • Page context from detail pages is passed to the agent and can be dismissed by the user.

Operating it

Model upgrades are treated as measurable changes, not swaps. Moving GPT-5.2 → 5.4 was benchmarked and adopted for faster responses through fewer unnecessary tool calls, better tool-selection accuracy and less verbosity; the current generation runs GPT-5.5. Negative reviews are analysed systematically — conversations are tagged by originating application so each module team reviews only what’s theirs, with a dashboard tracking review sentiment and response quality. Concrete fixes have come out of that loop: pagination exhaustion in the resource filter agent, parameter confusion between product-catalog identifiers, and database IDs leaking into tool responses.

Safety and correctness work has been part of the same effort — SSRF mitigation in the onboarding and knowledge-base toolsets, CSRF mitigation in the full-page onboarding agent, and card-number masking on submitted input.

Scale & Impact

  • ~5,000 queries/day.
  • Latency: 10–30s per query.
  • Reduced ticket inflow by 30%; improved first-response time by 50%.
  • Resolves L2+ tickets end to end — work that previously took days of human investigation.

Role

  • Tech Lead, 3-person team.
  • End-to-end ownership: infra, pipeline design, monitoring, rollout.
  • Collaborated with product/support teams to refine Copilot behaviour.

Architecture Diagram

flowchart TB
  U["User · any Chargebee app"] --> ORCH
  ORCH["<b>Copilot orchestrator</b><br/>routes · judges sufficiency · holds the bar"]

  ORCH --> DL["Data lookup agent"]
  ORCH --> KB["Knowledge base agent"]
  ORCH --> FR["First-response agent"]

  DL --> RBAC
  KB --> RBAC
  FR --> RBAC
  RBAC["User RBAC gate — a skill can never<br/>exceed what the caller may do"]

  RBAC --> TOOLS["Toolsets · billing data · docs · catalog · events"]
  TOOLS -. results .-> ORCH
  ORCH -. "answer or resolution" .-> U

  SK["Skill registry · any team contributes"] -. registers .-> ORCH
  ORCH -. "per-step trace, latency, tokens" .-> BS[("Backstage")]

3. AI-First Accounts Receivable

Apr 2026 – present · Billing Desk, which I own

Status: shipping in stages. The dashboard is live for all customers; dunning automation and the collector queue are rolling out to production sites. Newer surfaces are still in build.

Problem

Receivables is still run out of queues and forms. Collectors chase invoices manually, dunning is a static schedule rather than a decision, customer replies get triaged by hand, and the context needed to act — what was promised, what’s disputed, what’s actually owed — is scattered across systems. It’s a domain built for humans doing repetitive lookups, which is precisely the shape of work software should be absorbing.

Solution & Architecture

Receivables dashboard — a 12-month position on outstanding, overdue and not-overdue balances, plus an aging report. Live for all customers. Metrics were originally served by RevenueStory APIs and are being migrated onto Billing Desk’s own metrics APIs, so every receivables number resolves to one system rather than two.

Dunning sequence automation — built deliberately as a generic workflow executor rather than an email feature. The current implementation sends reminder sequences; the same engine takes other action types without redesign. The end-to-end path is: invoice created in Chargebee → event consumed by Billing Desk → local invoice projection updated → sequence run initialised from config → step executed → notification scheduled through the Chargebee notification service → email sent on the merchant’s template.

Two decisions worth calling out:

  • Reused the existing Chargebee notification service behind a thin Billing Desk wrapper, rather than building a parallel sending path — the templates, deliverability and merchant configuration were already solved.
  • A dedicated default SMTP account for receivables, so receivables volume can’t disturb existing email setup or counters.

Authoring is draft-and-publish with auto-save: sequences are configured iteratively and published when ready, not edited live.

Collector Queue — the sync engine

The queue needs a fast, queryable local view of unpaid invoices and customers. Sync is deliberately layered, because any single mechanism eventually misses something:

  • Kafka events as the primary path — near-real-time updates as invoices change.
  • A reconciliation job every two hours to detect and repair anything the event path missed.
  • On-demand admin sync for targeted correction.
  • Failed-entity tracking with automatic retries, escalating to a notification once retries are exhausted — so failures surface rather than silently rot.
  • Bulk migration to backfill all historical unpaid invoices into the queue.
flowchart TB
  CB["Chargebee invoices<br/>and customers"]

  CB -->|"Kafka events<br/>primary path, near real-time"| PROJ
  CB -->|"reconciliation sweep<br/>every 2h, catches misses"| PROJ
  CB -->|"on-demand admin sync<br/>targeted repair"| PROJ
  CB -->|"one-off bulk backfill<br/>historical unpaid invoices"| PROJ

  PROJ[("Collector Queue projection")]

  PROJ --> FAIL{"Entity synced?"}
  FAIL -->|no| RETRY["Failed-entity tracking<br/>automatic retries"]
  RETRY -->|"retries exhausted"| ALERT["Notify — failures surface,<br/>they don't rot silently"]
  RETRY -.->|retry| PROJ

  FAIL -->|yes| AGG["Precompute aggregates<br/>aging, outstanding"]
  AGG --> DAILY["Daily refresh job"]
  AGG --> LISTS["Customer + invoice lists<br/>no aggregation at request time"]

Aggregates (aging buckets, outstanding amounts) are precomputed during sync and refreshed by daily jobs, which keeps expensive aggregation off the request path for every list page.

Notable Challenge — reply threading across email providers

Supporting a real inbox means mapping an inbound reply back to the message that caused it. The standard answer is the Message-ID header, which turned out not to be dependable here: it isn’t stored today by SendGrid, isn’t retrievable through generic SMTP, and behaves inconsistently across providers — including merchants’ own custom SMTP. Chargebee’s email system was built to send notifications, with replies going to the merchant’s configured from-address, not to be an inbox.

So the threading model has to be provider-agnostic rather than relying on a header we can’t guarantee. That, rather than the UI, is the real design problem in this area.

Why this sits with me

It’s the intersection of the two things I’ve spent the most time on: deep AR and billing domain expertise from years of NetSuite and ERP integration work, and the agent platform I built. The domain knowledge is what makes the automation correct; the platform is what makes it shippable.

Role

  • Leading the architecture and delivery of the vertical, end to end.
  • Owned the sync engine, sequence framework, notification integration and metrics APIs.

4. Chargebee Jobs Framework – Distributed Job Processing

Late 2021 – early 2022 · Core Platform team

Problem

Chargebee needed a fault-tolerant async system to run subscription renewals, invoicing, reporting, and integrations — at tens of millions of jobs/day scale.

Solution & Architecture

  • Built a DB-driven distributed job framework (no Kafka/SQS).
  • Components: Job Picker, Executor, Throttler, Config Manager (Vinyas).
  • Job types: recurring, site-segmented, transactional, SLA-based, idempotent (WIP).
  • Observability: Splunk dashboards, CloudWatch metrics, SLA breach alerts.

Scale & Impact

  • 40M+ jobs/day, and it is what Chargebee’s core billing modules run on — subscriptions, invoicing, renewals, dunning.
  • Avg exec < 1.5s, 95% < 3s.
  • Multi-tenant isolation + fairness (throttling).

Notable Incident

  • Duplicate execution SEV due to race conditions.
  • Solved with a two-step picking protocol: atomic scheduled→picked in DB, then atomic picked→started before execution. Eliminated duplicates.

Later: attacking queue wait, not execution time

Average execution time had never been the problem — jobs were fast once they started. The latency users actually felt was waiting to be picked up, and it was invisible, because total duration conflated the two. Splitting the measurement into pick-up delay vs execution time vs total made the real distribution legible for the first time, which mattered increasingly as LLM invocations and agent runtime loops began running as jobs.

With that visible, the fixes were targeted:

  • 40–60% reduction in queue wait time, with critical jobs brought down to sub-second pick-up.
  • Priority work isolated from internal work, so background processing can’t starve latency-sensitive jobs.
  • Every non-DB-backed task migrated onto the durable jobs framework, so in-flight work survives deploys instead of being dropped.
  • Long-running job support to power bulk operations and one-time migrations, with batching for large queries.
  • Read-replica routing for internal read endpoints, taking load off the master.
  • Operational tooling: a Superset dashboard for load across servers, plus admin actions to reschedule, cancel, flush and auto-delete jobs.

The general lesson: an aggregate metric that averages two different phenomena will hide the one that’s hurting. The instrumentation change was worth more than any single optimisation that followed it.

A second job system, deliberately kept separate

When agent workloads arrived, the obvious move was to run them on the framework that already existed. We built a separate job system for agent inference and interaction handling instead — now over a million jobs a day — with the core billing framework left untouched.

The two workloads have almost nothing in common operationally. Billing jobs are short, predictable and revenue-critical: an invoice run that slips is a customer-visible failure. Agent jobs are long, latency-bound on an external model provider, and bursty in a way we don’t control. Putting them in one pool means the slowest dependency in the system gets a vote on whether renewals go out on time.

Keeping them separate costs some duplicated infrastructure. What it buys is a blast radius: a model provider degrading, a runaway agent loop or a bad prompt deploy cannot contend with subscription billing for workers. For a system the business runs on, that trade is not close.

Role

  • Tech Lead alongside Architect.
  • Owned schema design, idempotency semantics, and contention resolution.
  • Partnered with SREs for scaling/sharding strategy and monitoring.

Architecture Diagram

flowchart LR
  subgraph PICK["Two-step atomic pick — the fix for duplicate execution"]
    direction LR
    S(["scheduled"]) -->|"atomic CAS #1"| P(["picked"])
    P -->|"atomic CAS #2"| R(["started"])
  end

  DB[("sch_jobs")] --> PICK
  PICK --> EX["Executor pools"]
  THR["Throttler<br/>per-tenant fairness"] -.->|admits| EX
  CFG["Config manager"] -.-> THR
  EX --> AUD[("sch_jobs_audit")]
  EX -.-> OBS["Splunk · CloudWatch · Superset"]

The measurement split that mattered later — total duration hid the half that was actually hurting:

flowchart LR
  A(["Job scheduled"]) -->|"queue wait<br/>cut 40–60%"| B(["Picked up"])
  B -->|"execution &lt; 1.5s avg"| C(["Complete"])

5. Catalyst Sync Framework – Unified Integration Platform

2020 · Integrations team

Problem

Every integration — NetSuite, QuickBooks, HubSpot, Intercom — had been built as its own codebase. Each one re-solved the same problems in a slightly different way: scheduling, batching, field mapping, retry semantics, failure visibility. The per-connector cost was high, but the real damage was compounding: a fix to retry logic in one connector didn’t reach the other nine, so reliability varied by which integration a merchant happened to use, and every new connector added a maintenance surface rather than reusing one.

Solution & Architecture

The bet was that the differences between connectors are narrower than they look. Auth, endpoints, field names and rate limits vary; scheduling, batching, checkpointing, retry and reconciliation do not. So Catalyst separates the two: the engine is written once, and each connector supplies only what is genuinely specific to it.

  • Config-driven sync framework — integrations are defined in JSON rather than code, so adding a connector means describing a mapping, not writing a pipeline.
  • Pluggable pipeline engine on Apache Camel + Java, providing scheduling, parallelism and batching to every connector uniformly.
  • A connector contract covering auth, schema and rate limits — the only surface a new integration has to implement.
  • Isolated integration servers and schema, sharded so one noisy tenant or one slow third-party API cannot degrade the rest.
  • Fault tolerance built into the engine, not the connector: checkpointing, retries and DLQs, so reliability improvements land everywhere at once.
  • Operational visibility through Splunk dashboards, starvation reports and SLA dashboards.

The 80% figure is a direct consequence of that split: the engine work is amortised across every connector, so the marginal integration costs roughly what its genuinely unique parts cost — and nothing more.

Scale & Impact

  • Unified ~10 integrations onto Catalyst.
  • 90k records/min throughput (well above API limits).
  • Reduced new integration dev effort by 80%.

Notable Challenge — config at the wrong altitude

NetSuite tax sync failed intermittently, and only for some merchants. The cause was that SuiteTax provider configuration was modelled at the integration level when it is genuinely a tenant property — two merchants on the same connector can run different tax providers. Moving that config from integration-level to tenant-level fixed it.

The interesting part is the class of bug rather than the instance: in a config-driven framework, most defects aren’t logic errors, they’re settings modelled at the wrong altitude. Getting the ownership boundary of a config value right is the design work.

Role

  • Core Contributor & Tech Lead.
  • Architected pipeline, config-driven model, monitoring.
  • Owned accounting integrations (NetSuite, Xero, QuickBooks).
  • Collaborated with SREs on sharding + operational SLAs.

Architecture Diagram

flowchart TB
  CB["Chargebee<br/>source of truth"] --> ENG

  subgraph ENG["Catalyst sync engine — written once"]
    direction TB
    SCHED["Scheduler + batching"] --> MAP["Field mapping<br/>from JSON config"]
    MAP --> RETRY["Retry, DLQ, reconciliation"]
  end

  ENG --> CONTRACT["<b>Connector contract</b><br/>auth · schema · rate limits"]

  CONTRACT --> NS["NetSuite"]
  CONTRACT --> QB["QuickBooks"]
  CONTRACT --> ZH["ZohoCRM"]
  CONTRACT --> HS["HubSpot"]
  CONTRACT --> NEW["+ new connector<br/>config, not a codebase"]

  ENG -.-> OBS["Splunk dashboards"]

6. Revmonk – Startup (Lead Intelligence Platform)

Problem

Sales teams lacked real-time lead intelligence. Existing tools required manual enrichment and lacked automated outreach.

Solution & Architecture

  • Co-founded and built Revmonk from scratch.
  • Tracking Layer: ~10k–50k events/tenant/day.
  • Queueing: Redis + SQS, retries with backoff, DLQ for failures.
  • Enrichment: Clearbit, LinkedIn APIs with caching in Postgres + Redis.
  • Outreach Automation: SendGrid workflows, >95% deliverability.
  • Frontend: React + Next.js dashboard for sales teams.

Scale & Impact

  • 50+ trial users, 2 paying customers.
  • Pipeline processed hundreds of thousands of events/day.
  • Event → Outreach trigger in seconds.

Notable Challenges

  • API cost → solved with caching + tiered enrichment.
  • Deliverability issues → fixed with DKIM/SPF, warmed IPs, personalization.

Role & Ownership

  • Co-Founder & Sole Engineer (with 1 intern).
  • Owned end-to-end stack: backend, frontend, infra, outreach.
  • Took part in customer calls, demos, onboarding.
  • Partnered with co-founder (sales/marketing).

Why we shut it down

We had a working product and paying customers, and stopped anyway. The reason was simple and unglamorous: operational cost. Running the tracking, enrichment and outreach pipeline cost more than the business was bringing in, and engineering could only narrow the gap — caching and tiered enrichment trimmed the bill without changing the conclusion.

Deciding to stop while revenue was still coming in was harder than building the product had been.