How much are your 9's worth?
Metadata
Highlights
All nines are not created equal. Most of the time I hear an extraordinarily high availability claim (anything above 99.9%) I immediately start thinking about how that number is calculated and wondering how realistic it is.
Human beings are funny, though. It turns out we respond pretty well to simplicity and order.
Having a single number to measure service health is a great way for humans to look at a table of historical availability and understand if service availability is getting better or worse. It’s also the best way to create accountability and measure behavior over time…
… as long as your measurement is reasonably accurate and not a vanity metric.
Cheat #1 - Measure the narrowest path possible.
This is the easiest way to cheat a 9’s metric. Many nines numbers I have seen are various version of this cheat code. How can we create a narrow measurement path?
Cheat #2 - Lump everything into a single bucket.
Not all requests are created equal.
Cheat #3 - Don’t measure latency.
This is an availability metric we’re talking about here, why would we care about how long things take, as long as they are successful?!
Cheat #4 - Measure total volume, not minutes.
Let’s get a little controversial.
In order to cheat the metric we want to choose the calculation that looks the best, since even though we might have been having a bad time for 3 hours (1 out of every 10 requests was failing), not every customer was impacted so it wouldn’t be “fair” to count that time against us.
Building more specific models of customer paths is manual. It requires more manual effort and customization to build a model of customer behavior (read: engineering time). Sometimes we just don’t have people with the time or specialization to do this, or it will cost to much to maintain it in the future.
We don’t have data on all of the customer scenarios. In this case we just can’t measure enough to be sure what our availability is.
Sometimes we really don’t care (and neither do our customers). Some of the pages we build for our websites are… not very useful. Sometimes spending the time to measure (or fix) these scenarios just isn’t worth the effort. It’s important to focus on important scenarios for your customers and not waste engineering effort on things that aren’t very important (this is a very good way to create an ineffective availability effort at a company).
Mental shortcuts matter. No matter how much education we try, it’s hard to change perceptions of executives, engineers, etc. Sometimes it is better to pick the abstraction that helps people understand than pick the most accurate one.
Data volume and data quality are important to measurement. If we don’t have a good idea of which errors are “okay” and which are not, or we just don’t have that much traffic, some of these measurements become almost useless (what is the SLO of a website with 3 requests? does it matter?).
What is your way of cheating nines? ;)