← Home
· 6 min

Quiet Chatter 2 : Running Microservices on a Small AWS Budget

AWSArchitectureQuiet Chatter

Quiet Chatter is closed now. This post describes its infrastructure as it ran.

This is the infrastructure of Quiet Chatter, a one-person service. Most decisions here trade a managed service for something I ran myself, because the budget was small.

How a Request Travels

Cloud infrastructure architecture Cloud infrastructure: Cloudflare in front, a VPC with public and private subnets on AWS.

Nodes and pods inside the k8s cluster Inside the cluster: the gateway node, the worker nodes, and the control-plane node.

  1. The client is a static React build on Cloudflare Pages.
  2. When the browser calls the backend API, Cloudflare DNS proxies the request to the gateway node's Elastic IP. This leg uses a Cloudflare Origin Certificate, so TLS is end to end.
  3. Nginx on the gateway node listens on :443. It runs in hostNetwork mode, holding the node's IP directly, terminates TLS, and passes the request to the API gateway pod through cluster networking (ClusterIP and kube-proxy).
  4. The API gateway (Spring Cloud Gateway) routes by URI path to a microservice. It verifies the JWT and handles authentication here.
  5. Each microservice handles the request, using Redis (cache), Redpanda (event streaming), and RDS (database) as needed.
  6. The API gateway wraps the response in a common response structure and returns it.

Infrastructure as Code

The whole AWS setup is Terraform, in three layers (repository).

  • 01-base: VPC, subnets, security groups, IAM, S3, Secrets Manager, the NAT instance
  • 02-platform: the control-plane EC2 (k3s server, Redis, Redpanda) and RDS PostgreSQL
  • 03-apps: the gateway EC2 with a fixed EIP, and the worker Auto Scaling Group on Spot

Layers pass data through S3 terraform_remote_state. For example, 03-apps needs the control plane's private IP and security group ID to create the worker ASG, and reads them from 02-platform's remote state.

Keeping the Cost Down

Compute

  • Worker nodes on Spot instances in an ASG
  • Memory limits on every pod, to prevent OOM and node over-commit
  • Explicit JVM heap bounds (JAVA_OPTS: -Xms64m -Xmx192m) to keep containers small
  • k3s run by hand instead of EKS: no managed Kubernetes fee, less memory
  • Redpanda instead of Kafka: the same API with less memory

Network

  • No ALB: Nginx on hostNetwork receives traffic directly
  • No NAT Gateway: an EC2 instance does NAT with iptables masquerade

Operations

  • Ghost nodes are cleaned up automatically
  • Grafana Cloud's free tier for monitoring, so I run no Prometheus or Loki storage of my own

CI/CD

Code moves through dev → main → prod branches in each repository.

CI/CD flow Front end through Cloudflare Pages; back end through GitHub Actions, Docker Hub, S3, and sync.sh, with Discord notices.

Frontend. A push to dev or prod builds and deploys on Cloudflare Pages.

Backend.

  1. GitHub Actions builds the Docker image, tags it with the commit SHA (sha-xxxxxxx), and pushes it to Docker Hub.
  2. GitHub Actions updates the image tag in the Kubernetes manifests stored in S3.
  3. sync.sh, run periodically on the control-plane node, notices the new manifests and applies them with kubectl apply.
  4. Kubernetes performs a rolling update.

sync.sh is run by a systemd timer every five minutes. Besides syncing secrets and applying manifests, it removes ghost nodes: node records left NotReady for more than 15 minutes after a Spot instance is reclaimed. More specialized deployment tools exist, but at this size a timer and an S3 script are enough. A webhook reports each rollout to Discord as it happens.

Security

Inbound requests

  • Nginx rejects any request whose Host is not *.quiet-chatter.com, and rate-limits each IP to 30 requests per minute.
  • Spring Cloud Gateway rejects origins outside the CORS allow list and blocks /internal/** from outside (403). That path is for calls between microservices, which must carry an X-Internal-Secret header.
  • SSH: port 22 is closed. Access goes through AWS SSM Session Manager, with no key files to leak, controlled by IAM.

User authentication. JWT. The member service issues tokens; the API gateway verifies them by parsing.

  • Tokens live only in cookies on the backend domain, with httpOnly, secure, and sameSite=Lax. JavaScript cannot read them. The access token lasts 30 minutes.
  • The refresh token lasts 30 days, with rotation: on reissue or logout the old one is destroyed in Redis and cannot be reused. Storing it in Redis lets the server revoke it at once.
  • The gateway only parses the access token and does not depend on Redis. The member service alone owns the token lifecycle and the cookies.
  • After verification the gateway adds an X-Member-Id header with the member's UUID. Internal services trust it without checking the token again, because the cluster network is isolated from outside.

OAuth. Naver is the OAuth 2.0 provider. The only thing collected at sign-up is Naver's user identifier. It is a secondary identifier used only inside the service; a generated UUID is the primary one.

Secrets. Ten sensitive values (database password, JWT key, OAuth client secret, and so on) are one JSON secret in AWS Secrets Manager, created by Terraform.

AWS Secrets Manager (JSON)
  → sync.sh parses it with jq
  → kubectl create secret
  → pods read it through the Deployment's secretKeyRef

Network and IAM. Worker nodes and RDS sit in private subnets; outside traffic enters only through the gateway node. At my previous job I used an ALB and a bastion host with public keys. Here SSM gives the same level of security with less to run.

  • Worker SG: NodePort range only from the gateway SG
  • Control-plane SG: Redis (6379) and Redpanda (9092) only from the worker SG
  • RDS SG: PostgreSQL (5432) only from the worker and control-plane SGs
  • Every EC2 role has three permissions: SSM access, GetSecretValue on the one secret, and read access to the one infrastructure bucket

Monitoring

Logs are collected with Grafana Alloy, the successor to Promtail, and stored in Grafana Cloud (Loki). ELK is the industry standard, but the Loki stack uses less memory. Separating the log node from the API nodes is the textbook answer; at this size, keeping it in the same cluster is the efficient one.

A batch server also aggregates the logs once a day, has Gemini brief them, and pushes a daily report to Discord.

Daily Discord report A daily report. The 404s are requests from search portals, such as /robots.txt.

Next: five backend design cases.

Comments 0

Be the first to comment.