Skip to content

fix(approuter): bump memory 512M→2048M to stop prod OOM crash-loop - #1587

Merged
jung-thomas merged 1 commit into
mainfrom
worktree-fix-approuter-oom-memory
Aug 10, 2026
Merged

jung-thomas merged 1 commit into
mainfrom
worktree-fix-approuter-oom-memory

Conversation

@jung-thomas

Copy link
Copy Markdown
Contributor

Problem

The prod approuter tutorials-prod-approuter (serves developers.sap.com) crash-looped in production on 2026-08-10 with exit status 137 — container cgroup OOM kill at its 512M memory limit.

Root cause

  • Fresh deploy at 10:40 EDT → developers.sap.com vanity route mapped at 13:10 EDT → OOM crash-loop 13:37–13:55 EDT once real prod traffic arrived.
  • Exit 137 is a container-level cgroup OOM (total RSS > limit), distinct from the srv module's earlier exit-134 V8 heap SIGABRT — but the same class of under-provisioning. The srv module was already bumped 1G→2G for this reason (mta.yaml:96); the approuter was never given headroom.
  • 512M was too little for concurrent request buffering + the large static-asset surface (admin / analytics / explore / scanner / data-inspector bundles copied into the approuter at build).

Fix

  • tutorials-approuter memory: 512M → 2048M in .deploy/mta.yaml.
  • MTA version 1.13.1 → 1.13.2 (patch — config fix).
  • Already live in prod via emergency cf scale ... -m 2048M; this bakes it into the descriptor so the next deploy doesn't silently revert to 512M.

Disk

Checked per request: disk-quota is already 2048M and steady-state usage is ~33% (682M). Approuter logs stream off-container to stdout/stderr, so log growth is not a disk/OOM vector — no disk change needed.

Verification

  • Emergency scale to 2G stopped the crash loop; app running since 17:56Z at ~261M / 2G (flat, not climbing → load pressure, not a leak).
  • yq confirms descriptor parses and reports memory: 2048M, disk-quota: 2048M, version: 1.13.2.

The prod approuter (tutorials-prod-approuter, serves developers.sap.com)
crash-looped on 2026-08-10 with exit 137 (container cgroup OOM) once the
vanity route was mapped and real traffic arrived. 512M was insufficient
headroom for concurrent request buffering + the large static-asset surface.
Mirrors the srv module's earlier 1G→2G OOM fix. Matches the emergency
cf scale that stabilized prod; steady-state usage is ~260M.

MTA version 1.13.1 → 1.13.2 (patch: config fix).
@jung-thomas
jung-thomas merged commit a32b298 into main Aug 10, 2026
4 checks passed
@jung-thomas
jung-thomas deleted the worktree-fix-approuter-oom-memory branch August 10, 2026 18:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant