Observability That Answers Questions, Not Dashboards
Dashboards are how you OBSERVE, not what you're building. Design observability around golden signals, SLOs, and concrete questions.
Most observability setups fail for the same reason as most dashboards: they instrument metrics but never ask questions. "Do we have a dashboard for CPU?" is the wrong question. "How do we know a deploy is bad within 5 minutes?" is a right one.
Start from the golden signals
Every service answers four questions. If your stack answers these, you have observability; everything else is decoration.
| Signal | Question it answers |
|---|---|
| Latency | How slow are we, per request, per processor? |
| Traffic | How much demand are we serving? |
| Errors | What fraction of requests fail, explicitly and silently? |
| Saturation | How full is the resource that limits us? |
Instrument with the "3 magical numbers" in mind
For every operation, record: count, sum, and sum of squares (for histograms). Prometheus histograms encode this naturally:
http_request_duration_seconds_bucket{le="0.1"} 240
http_request_duration_seconds_bucket{le="0.5"} 980
http_request_duration_seconds_bucket{le="1"} 1200
http_request_duration_seconds_bucket{le="+Inf"} 1300
http_request_duration_seconds_sum 410.2
http_request_duration_seconds_count 1300
This gives you percentiles without storing every request. Average latency is a lie the histogram doesn't need to tell.
The SLO loop
An SLO is a contract you actually verify. Pick an error budget for the 10% of services that matter:
service: checkout-api
slo:
objective: 99.9% of requests under 300ms over 30 days
burn_rate_alerts:
fast: 1 hour of budget per 60s window
slow: 6 hours per 3600s window
Burn-rate alerts are how you get paged for budget burn, not for every red graph — which is the difference between an on-call that sleeps and one that doesn't.
Logs and traces: the read-me-later layer
Metric says something is wrong; logs and traces say what. Keep three rules:
- Structured JSON, always. Field names stable across services.
- Logs are for events (requests, jobs, decisions), not for debugging dumps.
- Trace every request that crosses a service boundary. Correlate via
trace_idin logs so "find the trace" has a one-liner.
{"level":"warn","trace_id":"4f3a...","service":"checkout","event":"retry","attempt":2,"ms":412}
The anti-checklist
- Don't add a dashboard for every metric. A dashboard you never look at is debt.
- Don't page on static thresholds you can't explain ("CPU > 80%"). Page on SLO burn.
- Don't store raw logs for everything forever. Define retention by value, not by "we can."
Summary
Instrument the golden signals, write an SLO you actually monitor, alert on budget burn, and keep logs/traces correlated. When someone asks "can you observe the system?", the answer should be: "I can answer questions about it." That's the whole goal.