Designing Telemetry with Questions in Mind

Question-driven instrumentation starts from what operators need to know, not from what a telemetry platform happens to collect easily. This is one of the clearest differences between mature observability and accidental observability. Mature systems ask: which questions appear during failures, latency regressions, release reviews, and customer escalations, and what evidence must exist so those questions can be answered quickly?

Teams that skip this step often instrument by signal type instead:

• add logs everywhere
• publish generic request metrics
• turn on tracing
• create some dashboards

That may produce activity, but it does not guarantee diagnosability. A better approach maps each operational question to the evidence needed to answer it.

Start With Real Operational Questions

Useful starting questions are concrete:

• Which dependency made this route slower?
• Which tenant, region, or feature flag is affected?
• Did the workflow fail before or after the queue handoff?
• Is the issue broad enough to consume error budget meaningfully?
• Did a deployment, config change, or traffic-shape shift precede the symptom?

These questions are strong because they force clarity about scope, causality, and decision-making.

    flowchart LR
	    A["Operational question"] --> B["Choose evidence"]
	    B --> C["Logs"]
	    B --> D["Metrics"]
	    B --> E["Traces"]
	    B --> F["Events"]
	    C --> G["Dashboards, triage, and incident response"]
	    D --> G
	    E --> G
	    F --> G

One Question Usually Needs Several Signals

A common mistake is trying to map each question to exactly one telemetry family. Real questions usually need more than one:

• metrics establish whether a problem is broad, urgent, or worsening
• traces identify where the path changed
• logs explain the detailed local outcome
• events show state transitions or workflow progression

This is why instrumentation design should think in evidence bundles rather than in one-signal answers.

 1question_map:
 2  - question: "Which dependency caused checkout latency?"
 3    evidence:
 4      - latency_histogram_by_route
 5      - dependency_trace_spans
 6      - structured_timeout_logs
 7  - question: "Which customers are affected?"
 8    evidence:
 9      - route_error_rate_by_region
10      - tenant-aware logs
11      - workflow state events
12  - question: "Did the async retry path succeed later?"
13    evidence:
14      - queue_age_metric
15      - retry_scheduled_event
16      - worker completion log

What to notice:

• the map begins with the question, not the tool
• each answer uses only the signals needed to remove uncertainty
• async questions pull in event and queue evidence that request metrics alone cannot provide

Questions Protect Teams From Vanity Telemetry

Question-first design also helps prevent vanity dashboards and low-value logs. If a metric, field, or trace attribute cannot be tied to a meaningful operational question, it should face a high bar before being added. This does not mean every signal needs a unique question written in a document. It means signals should exist because they help people reason, not because they are easy to emit.

This is especially important in large systems, where every extra label, record, or span attribute has cost and complexity.

Good Questions Are Decision-Oriented

The best observability questions are not trivia questions. They are decision questions:

• page now or watch longer?
• roll back or keep investigating?
• route traffic away from one dependency?
• treat this as one tenant issue or a platform incident?
• slow delivery work and spend error budget intentionally?

If a signal does not improve one of those choices, it may not deserve long-term collection.

Design Review Question

If a team says “we have a lot of telemetry” but cannot list which operational questions that telemetry is meant to answer, what risk is it taking?

The stronger answer is that it is collecting evidence without a diagnosis model. That usually leads to noisy dashboards, missing context, and expensive data that does not support better decisions.

Quiz Time