Question presented to candidate: "A request to your API is slow, and it involves your service calling two other internal services in sequence. Individual logs from each service don't make it obvious WHICH specific downstream call is the actual bottleneck for THIS particular slow request, among thousands of concurrent requests. What does distributed tracing add that separate, per-service logs don't?"
What a strong answer should cover:
- OpenTelemetry creates a span for each meaningful operation (an incoming request, an outgoing call to a downstream service) — every span belonging to the identical logical request shares the same real trace ID, directly answering the prompt's "which call, for THIS request" problem: separate log lines from different services have no inherent way to be correlated together; spans sharing a trace ID do, by design.
- 📌 Verified, not assumed: a real, nested
handleRequest->fetchUserspan pair genuinely shared the identical real trace ID — direct, concrete correlation across what would, in a real distributed system, be two genuinely separate services. Each span's real, measured duration was also captured accurately — thefetchUserspan's real duration (~21.8ms) genuinely matched its real, simulated 20ms delay. - 📌 Interview term: parent-child span relationship — verified directly: the child (
fetchUser) span's realparentSpanIdgenuinely matched the parent (handleRequest) span's own realspanId— this is the exact, concrete mechanism that lets a tracing UI reconstruct the real, actual call hierarchy and show precisely how much of a slow request's total time each specific nested operation consumed, directly answering the prompt's bottleneck-identification need. - A precise answer names the real, practical setup shape beyond the verified demo's manual span creation: real production Node.js OpenTelemetry setup commonly uses auto-instrumentation packages that automatically wrap common libraries (HTTP clients, database drivers) to create the identical kind of real spans verified above without requiring a developer to manually call
tracer.startSpan()/span.end()at every single call site — manual spans (verified directly in this demo) remain useful for wrapping custom, application-specific logic auto-instrumentation can't know about. - The precise, honest scope: OpenTelemetry's real value is specifically correlating and timing operations across a request's actual execution path — it's a genuinely different tool than structured logging (recording discrete events, covered in this bank's own dedicated question) or CPU profiling (finding a hot function within one process, covered in its own dedicated question) — a complete observability setup typically uses all three together, each answering a genuinely different diagnostic question.
Clarifying questions expected:
- "Do the downstream services this request calls already propagate trace context correctly (passing the trace ID across the actual network call), or would that require real, additional integration work?" — real, cross-service trace propagation needs each service to correctly forward context, not something automatic without any setup.
- "Is auto-instrumentation available and sufficient for the specific libraries this service uses, or does the actual bottleneck logic need manual, custom spans (verified above) to be genuinely visible in a trace?"
Code / implementation expected: Yes — real, nested spans genuinely sharing a trace ID, with a real, confirmed parent-child relationship and accurately measured real durations, is the concrete, convincing proof of exactly how distributed tracing answers the prompt's "which call, for this specific request" question.