Exit Code 137 Isn't Always Out of Memory

I have seen exit code 137 on one project for two unrelated reasons. One was a real out-of-memory kill during a deploy. The other happened on my own laptop, used no memory at all, and I blamed memory for it for months.

I wrote earlier about scheduled posts on this site without a database. This one is about the two times the build killed itself.

What 137 tells you

Exit codes above 128 follow a POSIX convention: 128 plus the number of the signal that ended the process. 137 is 128 + 9, and signal 9 is SIGKILL. The process didn’t exit on its own. Something sent it a kill that it couldn’t catch or clean up after.

That’s all the number says. It tells you which signal arrived. It doesn’t tell you who sent it or why. Most pages that explain 137 are about Docker and Kubernetes, where the usual sender is the kernel’s out-of-memory killer, so the advice jumps straight to “give the container more memory.” That advice is right a lot of the time. It was right for my first case and wrong for my second.

Cause 1: the compile really ran out of memory

The site has no database. Posts live in priv/content/site_content.json, and the app reads that file at compile time into module attributes inside SublimeCoding.Site.Content. At runtime there’s nothing to query, just terms already sitting in the compiled module.

The catch is how Elixir treats a module attribute inside a function body. Every reference to @attr inlines the full term into the code generated for that function. If one function reads the attribute, the corpus is in the compiled code once. If five functions read it, it’s in there five times.

Several functions in my content module referenced the big corpus attributes directly. At 113 posts, mix compile on Fly’s remote builder ran out of memory partway through a deploy and died with exit 137. This one really was memory, but the memory was going to copies of the same corpus in the generated code.

The fix, in commit 5573a0c on July 23, was to reference each big constant from exactly one function body. Every other use calls a private accessor:

defp published_constant, do: @published_posts
defp all_tags_constant, do: @all_tags
defp posts_by_tag_constant, do: @posts_by_tag_slug

and the functions that used to read the attribute call the accessor instead (wrapped to fit):

defp visible_posts(now) do
  published_constant()
  |> Enum.reject(&scheduled?(&1, now))
end

The peak memory of a local prod compile went from 4.5GB to 2.45GB. That’s the before and after recorded in the commit, measured on my machine.

There’s a standing rule now, written as a comment above the accessors in content.ex: a new function that reads a big attribute calls the accessor, never the attribute. The corpus only grows. It was 113 posts when the builder died, and it’s past 180 now.

Cause 2: a binary that was killed before it ran

The second 137 was mix assets.build on my Apple Silicon Mac. It had never worked on that machine. It died with exit 137 every time, and I assumed it was memory because that’s what 137 means in every article. I treated it as a quirk of the laptop and let Fly build the CSS on deploy, which worked fine.

The real cause had nothing to do with RAM. The Tailwind standalone binary for macOS arm64 that I downloaded came with an ad-hoc code signature that failed validation. Apple Silicon refuses to run a Mach-O file whose signature doesn’t verify, and the way it refuses is SIGKILL. The process is killed before it uses any memory.

Checking the binary showed it: codesign -v reported an invalid signature.

My first thought was local corruption, a bad download or a disk problem. It wasn’t. A fresh download of the same binary from the project’s GitHub release was byte-identical to the copy in _build, and it failed the same way. The binary as published doesn’t pass the check on my machine.

Fly’s builds run on Linux, where there’s no signature check, so only the macOS arm64 binary ever trips it.

The fix

codesign --force --sign - <binary> signs the binary again, ad hoc, over its actual bytes. After that the signature verifies and the binary runs. I didn’t want to do that by hand, because tailwind.install would download it again, bad signature and all, on the next version bump. So I wired it into the assets.setup alias with a function in mix.exs, resign_macos_binaries/1. The core of it checks the signature and re-signs only when the check fails:

case System.cmd("codesign", ["--verify", bin],
       stderr_to_stdout: true) do
  {_, 0} -> :ok
  {_, _} ->
    System.cmd("codesign",
      ["--force", "--sign", "-", bin],
      stderr_to_stdout: true)
end

On Linux and in CI the step doesn’thing, which is also the reason this stayed hidden. Fly deploys always worked because Linux has no signature check. Local builds on the Mac failed every time. Both facts fit the story “the Mac doesn’t have enough memory,” which is why I believed it.

What it cost

Because the local asset build had never succeeded, priv/static/assets/ on my machine had been stale for months. CSS was only ever built on Fly. Anything I ran locally that read the built bundle was reading an old copy. I never saw it, because the command that should have refreshed it kept failing.

Where I’d look the next time

Both of my cases printed the same number. They had nothing else in common, and the order below is how I’d sort them now.

1. Was it an OOM kill? On Linux, look for kernel messages about the killer. dmesg or journalctl -k will show oom-kill lines if the kernel did it. In a container, check the platform’s own OOM event. I hit three of those running agents in parallel, in Claude Code filled /tmp and froze my Linux box. If you find one, this is memory, and the question is whether the process needs that much or is wasting it. My first case was wasting it.
2. Did it die instantly? A process that gets killed in the first second, before it has done any work, was probably not killed for memory use. It never had time to use any. On macOS, run codesign -v on the binary. That’s how I found mine.
3. Did something else send kill -9? A process supervisor, a timeout wrapper, or a CI step with a time limit can all end a job with SIGKILL. The exit code looks identical to an OOM kill. The place to look is the config of whatever launched the command.
4. Only then, tune memory. Raising the limit’s the cheapest change, and it also hides a problem like the one in cause 1, where the real fix was removing the duplicated data. Check that the sender is the OOM killer before spending time here.

The order is cheapest first. Step 1 is a one-line command. Step 4 changes your build or your infrastructure bill, and it’s the one I’d have reached for first both times.

What I changed

The compile now has headroom, and a rule that stops the next function from undoing it. The Mac builds assets, and the codesign step in mix.exs handles the part that breaks the next time Tailwind’s binary is updated. Next in the same vein: Phoenix’s default cache-control and the check it breaks.