Usage Billing Engine

A high-growth SaaS platform was experiencing compounding issues with their billing infrastructure. As customer count scaled from hundreds to thousands of tiered accounts, the existing monolithic billing pipeline began to buckle under load. Usage telemetry — API calls, storage reads, compute minutes — was being ingested into a single processing queue that could not keep pace with incoming volume.

The core issue was a synchronous processing model. Every usage event triggered an immediate database write, balance recalculation, and invoice line item generation — all within the same transaction. Under normal load this worked, but during peak periods (month-end reconciliation, product launches, promotional campaigns), the queue depth grew faster than it could be drained. Events began timing out, and partially processed batches left balances in inconsistent states.

Race conditions emerged on customer-tiered accounts where multiple usage streams (API, storage, compute) competed for the same balance record. Two concurrent writes could overwrite each other, silently dropping usage events or double-counting them. The finance team discovered discrepancies only after generating invoices, leading to manual correction workflows that consumed hours each billing cycle.

Support tickets spiked. Customers disputed invoices that didn't match their dashboard usage. The dashboard itself lagged behind actual consumption by 4–6 hours during peak periods, eroding trust. The engineering team was spending more time firefighting billing incidents than building product features.

Rather than patching the existing pipeline, I proposed a ground-up redesign centered on two principles: decouple ingestion from reconciliation, and make every state transition idempotent. The goal was a system that could absorb usage telemetry at any volume without blocking, reconcile balances asynchronously in bounded time, and guarantee correctness even under concurrent write conditions.

The approach introduced a buffered event stream between the platform's telemetry producers and the billing database. Usage events would land in an append-only log first, then be consumed by a separate reconciliation worker that processes them in controlled batches. This separated the concern of "capturing usage" from "calculating billing" — the platform no longer needed to wait for balance calculations before acknowledging a usage event.

I also introduced a deterministic ordering mechanism. Each customer account's usage events are partitioned by account ID, ensuring that events for the same account are processed sequentially even though different accounts can be processed in parallel. This eliminated the race conditions entirely without requiring distributed locks.

The system is composed of four primary layers, each with a clearly defined responsibility:

The buffer layer is the critical innovation. By partitioning by account ID and processing each partition sequentially, we guarantee that two events for the same customer can never be reconciled concurrently. Different accounts process in parallel, giving us horizontal scaling without sacrificing correctness.

The ledger is append-only by design. Rather than updating a balance in place (which creates race conditions), each reconciliation run appends a new balance snapshot with a hash of the previous entry. This creates a tamper-evident chain that the finance team can verify independently.

The ingestion layer was implemented as a lightweight HTTP endpoint that validates incoming telemetry against a schema, assigns a monotonically increasing sequence number per account, and appends to the event log. The endpoint returns a 202 Accepted immediately — the platform doesn't wait for processing to complete. This alone reduced platform latency during peak usage by 340ms per request.

The buffer uses a partitioned queue with per-account ordering guarantees. Each partition is a FIFO queue backed by a database transaction log. Consumers claim a partition, process all available events for that account, then release it. If a consumer crashes mid-processing, uncommitted events remain in the partition and are retried by the next consumer. No events are lost.

The reconciler is the most complex component. It applies tiered pricing rules, handles proration for mid-cycle plan changes, calculates overage charges against included quotas, and generates daily invoice line items. Every calculation is idempotent — running the same reconciliation twice produces the same result. This is critical for crash recovery and allows us to replay any partition from any point in time.

The ledger stores balance snapshots with cryptographic checksums. Each entry references the previous entry's hash, creating a chain. If any entry is tampered with, subsequent checksums fail. The finance team runs a nightly verification script that walks the chain and flags any inconsistencies. In six months of production, zero discrepancies have been found.

Error handling follows a dead-letter pattern. Events that fail reconciliation after three attempts are moved to a dead-letter queue with full context (event payload, error reason, partition state). Engineers can inspect, fix, and replay individual events without affecting the live pipeline.

Invoice reconciliation times dropped from 45 minutes to under 12 seconds. The system now processes 2.3 million usage events per day with a p99 latency of 800ms from ingestion to ledger entry. During a recent product launch that drove 4x normal traffic, the pipeline absorbed the spike without any degradation to customer-facing dashboards.

Support tickets related to billing discrepancies fell by 94%. The dashboard now reflects usage within 2 seconds of the event occurring, up from the previous 4–6 hour lag. Customers trust the numbers they see, and the finance team no longer needs to manually verify invoice accuracy before sending them.

The engineering team reclaimed an average of 12 hours per sprint that was previously spent on billing incident response. That time is now invested in product development. The system has required zero emergency interventions since deployment.