Rebuilding a carrier's provisioning platform under a compliance deadline
We reverse-engineered an undocumented carrier-appliance API and shipped a desired-state provisioning control plane in two months, replacing a stalled nine-month effort. It has run three years without ever needing to change for a fault of its own.
Anonymized case study. A regional VoIP / telephony carrier; client name withheld. Upstream providers (Bandwidth, Telnyx) are shared carrier infrastructure, not the client. Engagement led by Ryan Luccioni, principal.
Architecture
The state store is the single source of truth; the reconciliation service is the only thing that pushes configuration to the appliances. The API does the other integration work, including tying into the carrier's existing billing system, reading client info and line saturation from it and writing location addresses back. What the API does not do is own SBC convergence. That separation is deliberate, so there's no side channel that can put an appliance out of sync.
Context
A regional telephony carrier ran three Session Border Controllers: the appliances at the edge of a voice network, notoriously hard to configure reliably. A new federal regulation (STIR/SHAKEN caller-ID authentication) forced every operator to get compliant fast: provisioning and maintaining configuration across thousands of numbers and many downstream business clients, kept in step with billing. Manual configuration across three appliances was not survivable at that volume.
An outside contractor had been building the admin platform for about nine months without shipping. We scrapped it and shipped a replacement in about two.
The problem the contractor couldn't get past
The Session Border Controller has no documented API. The vendor doesn't want you driving it programmatically. The real problem was never the admin UI; it was getting reliable programmatic control of an appliance built to resist it, then keeping three of them in agreement with each other and with billing, indefinitely.
How we got the API. Our principal started in the browser dev console watching the web UI's own traffic; performed each configuration action by hand, recorded the requests, and inspected them, using an intercepting proxy where he needed to see more. Then, on a test system, he replicated each operation directly against the web API, poking through carefully to avoid race conditions, until we had reliable programmatic control of every operation the UI could do.
What we built
A desired-state control plane, not a config script. "Provision this number / route / 911 record" is a statement of desired state; the hard part is convergence, not the individual write.
- A Postgres source of truth: one primary plus two read replicas for availability and read scaling. Staff describe what the world should look like (create a network access point, assign it to a customer, give customers locations, assign numbers to locations, wire up 911); nothing else is authoritative.
- A reconciliation service continuously drives the three SBCs toward that state. It detects drift and applies changes as an atomic, all-or-nothing operation: a failed apply leaves the appliance in its prior good state and the next pass reconverges. It rolls changes so one appliance is always online. The apply is idempotent: re-emitting the same desired state is a no-op, so the system converges rather than corrupting on a partial failure.
- Custom routing lives inside the appliance as Ruby scripts that route on live SIP traffic properties, the mechanism behind call-set rotation (below) and other bespoke routing.
The pattern we arrived at: desired state in a consistent store, a controller reconciling the fleet toward it, rolling and idempotent. It is the same control-loop model that runs modern cloud infrastructure. We built it from the problem, not from the playbook.
The invention: call-set rotation
High-volume outbound from a single number gets flagged as spam by downstream carriers. We first saw it hurt a healthcare client: legitimate callbacks showing as "spam likely," a safety problem, not a marketing one. The fix: round-robin outbound across 10–20 numbers so no single line trips the flagging threshold, implemented as SIP-aware routing logic inside the appliance. It generalized cleanly to a high-volume outbound campaign client.
Outcome
- Shipped in ~2 months, replacing a stalled ~9-month effort.
- Three years in production. The measure that matters: in three years the control plane never once had to change to fix a fault of its own. When things broke around it, it was upstream systems breaking their API contracts, not this. Layered safeguards meant it was never down, and "down" would only ever have meant staff can't configure. Call service was never affected. That stability wasn't luck; it was tested for up front.
- Still the platform the carrier runs its compliance posture on today.
Reliability, stated honestly: not "zero incidents," but a system that stayed correct and untouched while everything around it churned. A component you never have to open again is worth more than one that never technically failed.