Type “help” for hints, ↑ for history.
nginx was healthy: it was only reporting that nothing was listening on port 3000. The boutique service
was crashing at startup with ENOSPC, because /var was full. The culprit: LOG_LEVEL=debug left on in
production, which produced a 38 GB file.
Incident resolved
Analysis and fix
Immediate fix: empty the log (truncate -s 0 /var/log/boutique/debug.log), then restart
the service and check with curl.
To keep it from happening again:
- set
LOG_LEVEL=infoback in production; - add a logrotate rule for
/var/log/boutique/*.log(daily rotation, compression, 7 days); - monitor disk space (alert at 80%);
- write a short post-mortem: timeline, cause, actions.