
Reliability isn’t perfection—it’s predictability, even in chaos.

- Carl Eubanks, The 22nd Time He Told Me He Wanted Veggie Straws On Tuesday (2025).
Now, before I get into anything, there’s some important context I need to provide. What are veggie straws? Well, if we do a quick google search, you may get an answer that says the following: “Veggie straws are a type of snack food made from a combination of…”. This is a lot of unnecessary information, so I’ll save you the trip. They are a processed bag of dehydrated vegetables and potato starch/flour that look like rectangular straws.

The image illustrates an average night in the Eubanks household where I am strapped to the ceiling in order to eat veggie straws without someone taking them.
The veggie straw principle
Every night, right before bath time, my 3.5-year-old son hits me with a familiar request:

We want veggie straws.

Not sometimes. Not occasionally. Every. Single. Time. I can rely on it—just like I can rely on the sun rising, a demo going wrong that was “working 10 minutes ago”, or my son saying “we” when he most definitely does not mean “we”. That’s reliability in its most human form: given the same input, you get the same output. It’s not about whether it’s the “right” output. It’s about consistency.
Reliability in systems

The image depicts a father who, for the 9th time today (it’s only 11:00am), has mentioned something that he read in the book Designing Data-Intensive Applications, as well as a starfish looking—staring at an alpaca and a llama who just spoke.
In chapter 1 of Designing Data-Intensive Applications, reliability is defined as:

A system’s ability to function correctly even when things go wrong.

That breaks down into a few things, but most notably: correctness, fault tolerance, and predictable behavior. First off, we must think of these things as they pertain to the system we are observing or using. With that in mind, correctness refers to the right result being returned when it works. What is this right result? Hint: make sure not to think too hard on that. It’s the predictable behavior. Under the same conditions, you know what will happen. And it continues to give you this favorable—or unfavorable (can’t count out the amazing UX you’ve encountered that may or may have not driven you into a brief state of fight or flight)—outcome even when parts of the system fail.
In other words, a reliable system doesn’t surprise you—unless it surprises you in a consistent way. My toddler, like everyone in the world who stays up rather than creating a healthy bedtime routine, dislikes going to bed. Although it should not, it still surprises me how he never fails to show the utmost resistance when I say the word “Alright” after I have read (or sang) the last of a book or two or five before bed.
On the flip-side, there’s times when something is unreliable. This is why, in my perspective, you must think of reliability as a state of being for a system, with different factors at play. Even if a system is fault-tolerant, it cannot tolerate every fault. Under certain load parameters, a system can be reliable, while with others, it becomes less and less reliable until it can be considered unreliable… under those conditions. So the question becomes: when can I rely on you?
Faults and failures
As aforementioned, reliability is built on fault tolerance, not fault avoidance. We define a fault as something that goes wrong internally (server crash, packet drop, hardware glitch), and a failure as the fault becoming visible to the end user because the system couldn’t handle it.
A fault-tolerant system:

detects the fault (observability, graceful and intentional error handling)

contains the damage (isolation, dead-letter queues, retries with jitter backoff, fallbacks and circuit breakers)

recovers automatically (self-healing, failover)


Why averages lie: the tail latency problem
Averages can make unreliable systems look great on paper (bare with me as I give meaningless numbers with zero context):

“Average latency: 300ms”

…but 1% of users are waiting 9 seconds.


That’s tail latency, and it’s where the real pain lives. Whether we choose to care about that tail is up to the business (fyi: most, in fact, do not). Thus, it’s imperative that you keep those separate from the lived experiences of the commoner, the unremarkable, the quotidian, the 99%.
Percentiles tell the truth:

P50 (median): half of the requests are faster than this.

P95: 95% are faster than this—5% are slower.

P99: only 1% are slower than this.


A system that’s “fast most of the time” but occasionally glacial can still be functionally unreliable.
Building for fault tolerance
If veggie straws are inevitable, plan around them. If faults are inevitable, design around them.
Redundancy: keep spares (replication, standby nodes, a 2nd box of veggie straws from Costco).

The image depicts a father purchasing multiple boxes of veggie straws in an ideal world where his sons will not change their mind the next day on what snacks they “like”. He, without regard for the money in his pocket and the unhinged, extremely malleable brains of his little ones, plans for the “inevitable” case of them eating more, and not less. He is unbothered by the inaccurate sign placed around the alpaca, nor the smiling starfish with shorts who had given him a box of Not Again’s.
Isolation: don’t let one failure take down the whole system.Graceful degradation: better partial service than total outage.Monitoring: faults you don’t see are just failures waiting to happen.

The image depicts a father who has just setup a new, state-of-the-art, monitoring system strictly for veggie straw inventory, that are stored in containers reinforced with glass-clad polycarbonate.
Real-world example: Google’s “whoops” moment
One famous outage happened when there was a faulty automated quota update to the API management system (Google Cloud’s Service Control component).
Fault: An update that contained corrupted data, including blank fields, which were then distributed globally.Failure: Blast radius was extreme. Caused binaries responsible for validating API traffic to enter a crash loop, rejecting all API requests. The lack of proper redundancy meant the failure cascaded throughout the system.
Lesson: Faults are normal. Failures are preventable.
The reliability mindset
A reliable system doesn’t promise nothing will go wrong. Rather, that we’ve thought about what will go wrong, we’ve contained it, and we’ve kept it from ruining the whole experience.
To close, below is a thought experiment that you can take to many situations, not just technical ones:
For the people who stay up late, go to bed at different times each night, and regret it in the morning (hey, some people don’t regret it): what fault(s) did you encounter, and what is the failure that occurred? How can your “system” become fault tolerant in this scenario?

Although there are no wrong answers, there are, most certainly, better ones.

- Carl Eubanks, Looking Both Ways on a One Way Street (2022)

Next in the Series: Maintainability — Choose your hard


What my toddler taught me about fault tolerance

Most systems don’t break because of a missing feature. They break because they weren’t designed to survive growth, chaos, or time—mainly chaos.
That’s the first thing Designing Data-Intensive Applications wants you to understand. Chapter 1 isn’t about tech stacks or tuning knobs. It’s about how to build systems that don’t collapse under their own weight.
And you know what popped into my head? A bathroom scale.
Scalability: What my bathroom scale taught me about load
Scalability isn’t just a buzzword. It’s a design constraint. And I realized this while staring at my bathroom scale.
That little device does one job: measure weight. But it has a maximum load—mine tops out at 350 lbs. Go over that, and it stops being useful. The reading becomes inaccurate, or worse, nonexistent. Software is the same way.
Every system, from web servers to queues and databases, has its equivalent of that weight limit. But in software, load isn’t just how much, it’s how fast and how often.
You can have 100K users, and that might be fine. But if 50K show up at once during a ticket drop? That’s rush hour. And rush hour is what tests your scale.

No, no, not that rush hour!

That’s better.

Scalability isn’t about handling more users. It’s about handling more users at once.

That’s why we ask:

Can our DB handle fan-out writes to 10M timelines?

Can our app layer handle 500 RPS bursts?

Can workers scale up fast enough when the queue spikes?

WHAT IF ARIANA GRANDE POSTS A VIDEO? THE COMMENTS. WILL MY YOUTUBE COMMENTS DISAPPEAR?


Scalability is architectural. It's built around two questions: Can it carry more load than the cognitive load I’m carrying now? What’s the real load this system needs to carry?
🚦 Rush hour (traffic) and the twitter timeline
Let’s apply this to a real-world problem: Twitter’s timeline architecture. Now, every system has a set of critical business problems that make heads hurt. For payment processing, it’s preventing double processing (let’s not mess with people’s money). For a toddler’s bedtime routine, it’s getting them to bed. So, of course, news feeds will have their own set of issues.
Let’s start big. When a celebrity tweets:

Do you push that tweet to every follower’s timeline immediately (fan-out on write)?

Or compute timelines per user when they open the app (fan-out on read)?


Each has tradeoffs:

Fan-out on write = massive burst load, but fast reads

Fan-out on read = consistent writes, but slower reads


Neither scales forever.
That’s why Twitter uses hybrid models with caching, deferred writes, async workers (you gotta handle computationally-heavy timelines somehow, right), priority queues, x, y, z, this, that, and the third in order for you, the hopefully-didn’t-just-dust-off-the-old-iPhone-and-haven’t-touched-twitter-in-3-years user, to open up the app and obtain a fresh timeline (with async fetches and recommended posts so you never put down your phone).

You can’t just buy a bigger scale. You need different kinds of scales for different kinds of load.

This is the key lesson from scalability: you can’t optimize for the average (which ultimately ignores the lived experience of many, many people). There are many different philosophies and approaches to this, but, in my opinion, if you think it’ll get enough people to write bad reviews and skew your J curve, you should probably design with it in mind.
A couple questions to keep your mind busy on your days off:
What breaks first?Why does one person have 5 variations of hydroflasks?

Next in this series: What my toddler taught me about fault tolerance


Tag: Technology

What my toddler taught me about fault tolerance

The bathroom scale and the myth of scalability: Why most systems fail at rush hour—not at rest