When our four-vCPU box stopped being enough

A customer could request a page that CloudFront had never seen. That single page sent Orbit through Nexus, MySQL and Redis before there was anything to cache. At the same time, background jobs and the legacy operations backend shared the same four vCPUs. When the host began returning 429s, “add more workers” sounded like the obvious answer. It helped threads spend less time idle, but it could not give the machine another CPU.

This is the story of how we moved production from an x86 m6i.xlarge to an ARM c7g.2xlarge: what we found running, what our 40-request-per-second trial proved, the three times our cutover rehearsal stopped safely, and why switching a load balancer was slower than moving the data.

A cache miss did not mean one unit of origin work. The old host carried the request path and the jobs around it.

First we stopped making the host repeat itself

We traced cold SEO renders instead of assuming every 429 was a rate-limit problem. Some pages fetched the same data more than once. We consolidated those calls into a Nexus SEO response and shared its six-hour derived result in Redis, rather than maintaining a different cold cache inside each Orbit worker. CloudFront kept approved anonymous HTML near readers; malformed and retired URLs were rejected earlier. MySQL stayed the source of truth.

That work mattered before choosing a larger instance. Otherwise a faster CPU would simply have repeated unnecessary calls faster. It also clarified the limit: threads can overlap waiting on I/O, but Orbit, Nexus, MySQL, Redis and the other processes still compete for the same four cores.

Then we walked the old host. Our architecture diagram described the main request path. The running machine told a longer story: Orbit, Nexus, Concierge, GMC Backend, Celery and recovery workers, MySQL, three Redis workloads, Nginx, telemetry, cron jobs and GitHub deployment runners. An apparently idle service in the diagram could still own a systemd unit, a file path, a scheduled task or a deployment dependency. The migration inventory had to come from processes, units, mounts, listeners and workflows, not from a drawing alone.

Even the code revision was less obvious than a checkout directory suggested. Orbit and Nexus were served by blue/green systemd slots; an updated directory on x86 was not necessarily the slot taking traffic. We resolved the active unit and exact revision. On the detached ARM candidate, an airport page then exposed two more assumptions: a logical application dump had not carried MySQL system users, and a later environment file overrode the Nexus URL we thought we had configured. We fixed both before ARM could receive customer traffic. The old frontend process, conversely, was already stopped and did not need migrating.

A capacity trial, with a narrow claim

The ARM candidate doubled available vCPUs while retaining 16 GiB of memory. We sent 40 requests per second for 30 minutes through the candidate workload. Out of 72,000 attempts, 71,993 returned HTTP 200 and seven ended in client connection errors. Median latency was 135.4 ms; p95 was 286.92 ms and p99 was 489.76 ms. CPU averaged 46.57%, with a p95 of 55.33%; available memory never fell below 9.5 GiB. MySQL handled roughly 1,204 queries per second in this run without observed aborted connects, slow queries or disk temporary tables.

The trial measured this workload on this candidate. It was not a general ARM-versus-x86 benchmark, nor a promise that every future traffic mix would behave the same way.

The numbers gave us room to proceed. They did not tell us whether the stateful handoff would work. That was a different proof.

Moving the data meant moving authority

Our first database copy exposed the risk: it had 233 tables while the live database had 235. “Restore completed” was not the same as “restore is current.” We took a fresh coordinate-bearing MySQL snapshot, applied file-and-position replication over TLS, and kept ARM as a read-only replica until the schemas matched and lag reached zero.

At cutover we needed one writable side, not two. We froze writes and side-effecting workers, let replication catch up, transferred the Orbit session Redis archive, promoted ARM, and set up reverse replication toward x86 during observation. The session transfer mattered as much as the database: a healthy page load would have concealed customers losing active sessions. We held the old host as a recovery path while ARM served traffic.

The rollback path changed at each gate. Before promotion, x86 was still authoritative. After promotion, a return to x86 required the reverse replication path to be healthy.

We rehearsed the controller, and three failures were useful:

1. A graceful Supervisor stop exceeded its time window and returned a failure even though the process had stopped. The controller began checking the actual postcondition, then safely resumed x86.
2. Orbit’s restricted Redis account could not run DBSIZE. We used the permitted INFO keyspace check instead; access control remained restricted.
3. A generated reverse-replication password was 64 characters, but the MySQL API accepted at most 32. The controller stopped with ARM still replica-ready, x86 active and the load balancer unchanged. We bounded the credential length and rehearsed again.

Each failure was a reminder that a cutover script must distinguish a command’s exit code from the state it left behind. A safe refusal can be a successful rehearsal.

The slowest part was the switch

Once ARM was active, six concurrent load-balancer modify-rule calls took 153 seconds. A later read-only describe-rules call took 709 seconds. Application response time was not our only latency; the cloud control plane could dominate the operational timeline.

We changed the routing design for the next switch: prepare shadow rules in advance, then use one set-rule-priorities operation to move traffic, with the inverse operation ready for rollback. That design was a response to what the initial cutover taught us; we did not retroactively use it for the first switch.

During an observed production window after cutover, the origin received 57,300 requests. Eighty-four were 429 (0.147%) and seven were 5xx (0.012%). Those are window-specific counts, not a permanent availability claim. We checked customer and admin paths, Orbit, Nexus, Concierge, MySQL, Redis and systemd health while the old host remained recoverable.

Retiring the old machine was another migration step

After roughly 18 hours of ARM observation, we took a 100 GiB root-volume snapshot and terminated x86 with the owner’s authorization. The first termination attempt met instance termination protection. A second read immediately after clearing protection still saw stale state. We waited for the control plane to reflect the change, then the third attempt succeeded. The old volume was removed with the instance.

That snapshot preserves configuration and forensic evidence; it is not a current database rollback. We rechecked the live flows after termination because “old machine gone” is a dependency test of its own. The production deployment path, databases, cache and customer routes still had to be independent of that box.

We would run the next migration with the same order of evidence: inventory the running host, remove avoidable request work, prove candidate capacity, prove data authority transfer, rehearse refusals, keep the traffic switch small, and verify again after retirement. The most valuable rehearsals were not the ones that looked smooth. They were the ones that told us exactly where it was still safe to stop.