mediumSystem Design

How do distributed systems detect node failures?

Q: How do distributed systems detect node failures?

## Failure detection Nodes signal liveness with **heartbeats**: each node periodically says 'I'm alive'. If a node misses heartbeats for longer than a **timeout**, it's marked **suspect**, then **dead**. The fundamental catch: you **cannot distinguish a slow node from a dead one** over an unreliable network — so detectors trade off **speed vs false positives**. Node 1 Node 2 <text class='d-su

979 views

Understand the problem

Explain heartbeats.

heartbeatfailure-detection

Attempt it yourself

Sketch your approach before reading the solution — that's what interviews test.

Stuck? AI Nudge Available

Get a conceptual hint to guide your logic without spoiling the final implementation.

Study the solution

The solution is waiting

Give it an honest attempt first — then compare your thinking with the full walkthrough.

Read the code

Heartbeat + timeout monitor

Run Playground

TIMEOUT = 3.0   # seconds without a heartbeat -> suspected dead
last_seen = {"n1": 0.0, "n2": 0.0, "n3": 0.0}

def heartbeat(node, now):
    last_seen[node] = now

def check(now):
    return {n: ("alive" if now - t <= TIMEOUT else "DEAD")
            for n, t in last_seen.items()}

now = 5.0
heartbeat("n1", now)
heartbeat("n3", now)        # n2 stopped sending heartbeats
print(check(now))           # n2 -> DEAD (silent past the timeout)

Join the discussion

Discussion (0)

No responses yet. Be the first to share what you think.

What is idempotency and why does it matter?

How do you approach back-of-the-envelope capacity estimation?