DrugMobthe dealer is in

p01 · the problem A web service answers 97% of requests in 300 ms and 3% in over 8 seconds. The slow ones are spread across all endpoints and all hours. Each request does about a dozen sequential database queries and one call to an external API. Adding timing logs around every query showed nothing individually slow in the samples taken. It cannot be reproduced locally.

before Add more timing logs, this time around the external API call as well, and raise the client timeout so the slow requests at least succeed.

after Instrument external API with p50/p95/p99. Check DB connection pool saturation on the slow 3%. Enforce a hard 2s budget for the whole request chain and fail fast on the external call. Sample the slow requests for concurrent load on the external service.

p01 · the problem A web service answers 97% of requests in 300 ms and 3% in over 8 seconds. The slow ones are spread across all endpoints and all hours. Each request does about a dozen sequential database queries and one call to an external API. Adding timing logs around every query showed nothing individually slow in the samples taken. It cannot be reproduced locally.

before Add more timing logs around every query and the external API, raise client timeout so slow requests at least succeed.

after Stop staring at individual queries. Treat the 12 sequential DB calls + 1 external call as a single span. Add distributed-trace style timing for the whole request and specifically for connection acquisition and external call. Hypothesis: the tail is either pool exhaustion or external latency that only appears under real load. Measure correlation of the 3% with those two spans first.

p01 · the problem A web service answers 97% of requests in 300 ms and 3% in over 8 seconds. The slow ones are spread across all endpoints and all hours. Each request does about a dozen sequential database queries and one call to an external API. Adding timing logs around every query showed nothing individually slow in the samples taken. It cannot be reproduced locally.

before Add more timing logs, this time around the external API call as well, and raise the client timeout so the slow requests at least succeed.

after Stop inspecting queries: they are fine, so the 8 s sits in time nobody timed. For each slow request compute total time minus the sum of query times, and plot the slow durations: a tight cluster just above 8 s means a timeout plus a retry, not slow work. Then time the three unmeasured gaps: waiting for a pool connection, the external API call split into DNS/connect/TLS/response, and queueing before the handler. Group slow requests by host, DB connection and upstream IP; 3% across every endpoint suggests one bad instance or stale connections. Do not raise the client timeout; it hides the signal.

p01 · the problem A web service answers 97% of requests in 300 ms and 3% in over 8 seconds. The slow ones are spread across all endpoints and all hours. Each request does about a dozen sequential database queries and one call to an external API. Adding timing logs around every query showed nothing individually slow in the samples taken. It cannot be reproduced locally.

before Add more timing logs, this time around the external API call as well, and raise the client timeout so the slow requests at least succeed.

after Stop timing queries; time the gaps between them. A jump from 300 ms to 8 s with nothing in between, on every endpoint at every hour, looks like a wait on something shared that ends in a timeout and retry, not slow work. 1) In production, trace every request over 2 s end to end: load balancer time, handler time, sum of spans. The unaccounted part is the culprit: pool acquire, DNS, TCP/TLS connect, GC, queueing. 2) Histogram the slow durations; a spike at a fixed value names a timeout in config. 3) Group slow requests by instance, DB host, API IP; 3% may be one bad node. Leave timeouts alone.

p01 · the problem A web service answers 97% of requests in 300 ms and 3% in over 8 seconds. The slow ones are spread across all endpoints and all hours. Each request does about a dozen sequential database queries and one call to an external API. Adding timing logs around every query showed nothing individually slow in the samples taken. It cannot be reproduced locally.

before Add more timing logs, this time around the external API call as well, and raise the client timeout so the slow requests at least succeed.

after Stop looking at queries: the cause is common to every endpoint and hour, and a jump from 300 ms straight to 8 s+ looks like a fixed timeout followed by a retry, not slow work. Suspects: waiting for a DB pool connection, DNS or connect to the external API, one bad instance or upstream host. One change: on any request over 2 s, log total time against the sum of timed spans, plus pool-acquire wait, DNS/connect/response times for the API call, instance id and upstream IP. Plot the slow durations; if they cluster on a constant, find that timeout. Don't raise the client timeout; it hides the signal.

p01 · the problem A web service answers 97% of requests in 300 ms and 3% in over 8 seconds. The slow ones are spread across all endpoints and all hours. Each request does about a dozen sequential database queries and one call to an external API. Adding timing logs around every query showed nothing individually slow in the samples taken. It cannot be reproduced locally.

before Add more timing logs, this time around the external API call as well, and raise the client timeout so the slow requests at least succeed.

after Stop inspecting queries: if none is slow, the 8 s lives between them. First action, with logs you already have: for each slow request, subtract the sum of its query timings from total wall time. The gap is the culprit. Then trace slow requests in production with spans for the waits never timed: DB pool acquire, and the external call split into DNS, connect, TLS, response. Check whether slow times cluster at a fixed value (a timeout plus retry) or on one host. Likely fixes: a 1 s timeout with fallback on the external call, or a pool sized so requests never queue. Don't raise the client timeout.

p01 · the problem A web service answers 97% of requests in 300 ms and 3% in over 8 seconds. The slow ones are spread across all endpoints and all hours. Each request does about a dozen sequential database queries and one call to an external API. Adding timing logs around every query showed nothing individually slow in the samples taken. It cannot be reproduced locally.

before Add more timing logs, this time around the external API call as well, and raise the client timeout so the slow requests at least succeed.

after Stop timing queries; the 8 s is likely lost between them. In production: 1. For slow requests only, log wall time minus the sum of timed spans. A big gap means waiting: pool checkout, DNS, TCP/TLS connect, or queueing before the handler starts. 2. Histogram the slow durations. A spike at one value means timeout plus retry, not load. 3. Group them by host, connection id and upstream IP. 3% is about 1 in 32: one bad instance, connection or resolver. With 12 checkouts per request, a 0.25% stall rate gives 3%. Dump the stack of any request past 2 s. Don't raise the timeout; it hides the signal.

p01 · the problem A web service answers 97% of requests in 300 ms and 3% in over 8 seconds. The slow ones are spread across all endpoints and all hours. Each request does about a dozen sequential database queries and one call to an external API. Adding timing logs around every query showed nothing individually slow in the samples taken. It cannot be reproduced locally.

before Add more timing logs, this time around the external API call as well, and raise the client timeout so the slow requests at least succeed.

after Stop timing queries: the missing time is probably between them. 300 ms or 8 s with nothing in between looks like a wait that expires, not slow work, and every endpoint at every hour points at something all requests share. The external call has never been timed. Next: 1) Plot the slow durations; a spike at one value names a timeout. 2) Trace one slow request end to end: load balancer to handler, pool wait, connect, DNS, each query, the external call, the gaps. 3) Group slow requests by instance, DB host and API IP. Do not raise the client timeout; it hides the signal. Cause not yet known.