Incident response and capacity

Native-memory investigation

Frontend hosts were exhausting memory even though Java heap dumps looked clean. I built a local reproduction and traced retained memory to a native allocator used by a library on the request path.

I tested the mechanism on a staging host and compared the model with production. It predicted 3.67 GB of retained memory for a host that had lost 3.66 GB. I documented the reproduction and handed the owning team the mechanism and code issues.

Containment came from a rollback. The permanent library fix belonged to the owning team.

Scaling policy

I investigated recurring capacity alarms and found that the first scale-up step could hit the group's maximum before the service needed that much capacity. I evaluated alternative explanations before changing the initial step from 20% to 5%, preserving the larger steps for surges.

Across 773 matched five-minute production periods, the smaller step used 16% fewer hosts. Average decision latency was 98.92 ms before and 98.33 ms after. The comparison covered one production workload.

Operational follow-through

I turn incident findings into code, monitoring changes, and runbooks. Other work includes a shared canary-ordering fix, host-management procedures, and deployment rollback guidance.