Testnet Explorer Alerter Liveness

Polls /api/health/alerter on the testnet explorer and checks that its module-scoped Telegram delivery state shows a successful send within the last 30 minutes. If the alerter has been silent for longer, this probe trips DEGRADED (>30 min) or DOWN (>120 min), and integra-status sends its own Telegram alert — using a separate bot+token, on a separate host (Vultr) from the explorer (Hetzner). This is the meta-watchdog called out by GOAL_TESTNET_STABILITY.md §6.D4. The thresholds were relaxed from 15min/60min on 2026-05-13 to eliminate self-noise during healthy quiet periods; they will be tightened back once the explorer ships a periodic Telegram heartbeat that updates lastSuccessAt every ~5 min regardless of whether real alerts are being sent.

Operational

Environment

dev

Response Time

Average

257ms

Min

17ms

Max

1911ms

p99

1457ms

Uptime History

120 checks agoNow

Direct Links

Troubleshooting

Possible causes:

Explorer auto-disabled after 10+ consecutive RPC failures (shouldSync=false) — sync-status API returns empty explorers array— Fix: Re-enable in DB: UPDATE explorers SET "shouldSync"=true, "syncFailedAttempts"=0, "syncDisabledAt"=NULL WHERE id=1; then check worker logs for 'auto-disabled due to RPC failures' to confirm root cause
BullMQ blockSync job stuck in active state— Fix: SSH to server, clear queue: docker exec <backend> node -e "...Queue('blockSync')...obliterate()..."; restart workers
PM2 sync process crashed— Fix: Check PM2 logs: docker exec <pm2> pm2 logs --lines 50; restart: docker compose restart pm2
EVM indexer disabled on chain node— Fix: Set enable-indexer = true in app.toml, restart node, run: intgd index-eth-tx backward && intgd index-eth-tx forward
Workers disconnected from Redis/RPC— Fix: docker compose restart high_priority_worker medium_priority_worker low_priority_worker
Missing DB columns causing silent INSERT failures— Fix: Run migration: docker exec <backend> npx sequelize-cli db:migrate

Dependencies

Depends on