Slow search queries — search-api

search-api p99 latency degraded to 2.8s over 20 minutes. Salvo correlated with an index rebuild on the search-ES cluster that had been running since 18:40. Resolved on its own.

1. ×12
   search-api p99 > 2s

   grafana · 3d ago
   grafana
2. ×6
   elasticsearch heap > 85%

   prometheus · 3d ago
   prometheus

1. SALVO06:40:00 PM UTC
   Index rebuild job started
2. SALVO07:11:00 PM UTC
   First p99 alert
3. SALVO07:12:02 PM UTC
   INC-2340 opened · cause candidate: index rebuild (88%)
4. SALVO07:46:00 PM UTC
   Alert resolved · job completed