TroubLess

From alert to diagnosis.

Choose an incident and see the difference in Slack.

SlackSearch SRE Operations
incidents
MessagesFiles
Today
New message
TroubLessAPP

CPU saturated on worker-03

Evidence

export-worker accounts for 82% CPU usage. The spike starts at 14:31 with the export batch; memory remains at 54%.

Impact

Export jobs accumulate 240 pending items. Other host processes compete for CPU and take longer to run.

Probable cause

Export batch concurrency saturated CPU: eight workers started together on the same host, according to process metrics and batch logs.

Action

Reduce batch concurrency and monitor CPU and job backlog. Confirm a return to normal before restoring volume.

Observed metrics · 5 minutes:
  • CPU 96%
  • normal for this hour 28%
  • process export-worker
worker-03 · 5 minutes · Fictional example

Dev

I applied the fix based on the TroubLess diagnosis. I am validating the result.

SRE

I am monitoring the metrics after the fix. Let’s confirm recovery before closing the incident.

Message #incidents · preview