Monitoring on GeekyRyan

Building an Operational Strategy for Understanding and Responding to Alerts

Thu, 05 Sep 2024 09:00:00 +0000

You can have all the alerts configured you want, but if the team doesn’t agree on what they mean, who’s responsible for them, and what a good response looks like, they’re just noise. This was about building that operational strategy - making sure every alert had a clear owner, a documented playbook, and an agreed severity/escalation path.

It’s the human side of alerting, not the technical side. Once you’ve got the right metrics feeding into alerts, you need the operational discipline to make sure those alerts actually get owned and responded to consistently, whoever’s on call.

Closing Gaps in Infrastructure and Application Monitoring

Fri, 15 Mar 2024 09:00:00 +0000

Infrastructure evolves fast, and monitoring coverage always lags behind. Full review of what we were actually monitoring to find and close the gaps in alerting and visibility.

Problem was simple: as our footprint grew, metrics coverage didn’t keep up. Blind spots on CPU, memory, disk, network throughput on various resources. Assessed everything, implemented missing alerting, wrote response procedures.

Pairing every new alert with “here’s what you do when this fires” was as important as the alert itself. An alert with no playbook is just noise.

Improving Release Environment Stability Through Proactive Monitoring

Thu, 30 Nov 2023 09:00:00 +0000

Release environments get less monitoring love than production, which is backwards - instability there slows down every team trying to validate their work.

Hit a 95% stability target in the release/daily environments by monitoring proactively instead of waiting for complaints. Built visibility into the full change set landing in these environments, so when something broke we could correlate it against what actually changed instead of guessing.

Dashboard showing deployment metrics over time - failure rates, time-to-recovery, what kinds of changes cause instability. Once the visibility existed, we caught issues before they became “developer needs to file a ticket” problems. Real win was reducing how much environment instability was dragging on everyone’s velocity.