Build Monitoring System 2026-06

Context

Observability issue surfaced as a common problem across departments.
the team was about to treat the issue by adopting metric visualization and alerting.
Since I had experience building a monitoring system, I volunteered and took on the task.

Design

I designed the whole architecture featuring the backend system and alarm channels within resource constraints given by the team leader.
I specified target metrics to be managed (SLI) and applied general thresholds for them.

Contributions

• Built a system containing Grafana and Prometheus based on a container environment using Docker Compose.
• Provisioned and integrated Grafana alerting with SMTP and Microsoft Teams.
• Conducted E2E test under simulated stress conditions.
• Staged and finalized production deployment.
• Controlled access via limited authorized account registration.
• Documented and shared knowledge with colleagues.

Challenges

I suggested a design for central monitoring system on an independent node for availability but it was not accepted due to team infrastructure constraints.
I also suggested adopting Loki for log aggregation but it was not accepted due to lack of consensus among reviewers.