Explained
Your server ran out of memory — what that actually means
Real server problems, translated into plain English by Onserva.
What you're seeing
An app died without a goodbye. The logs just stop, or end with a single word: Killed. If it runs in a container, you might see it exited with code 137, or simply notice it restarting over and over. Nothing in your code explains it — because the cause wasn't in your code.
What it actually means
When a Linux machine truly runs out of memory — fast memory gone, emergency reserve gone — it faces a choice: freeze entirely, or sacrifice something. It sacrifices. A part of the system nicknamed the out-of-memory killer picks the process it judges most expendable — usually the biggest — and ends it instantly. No shutdown, no goodbye, no final log line. That is what “Killed” means, and exit code 137 is the same event wearing a number.
It feels brutal, but it is the machine protecting everything else you run. The trouble is the choice of victim: the biggest process is often your database or your main application — precisely the thing you least wanted sacrificed.
Why it happens
- A slow leak finally mattered. Some apps grow a little every day and never give it back. Weeks of quiet growth, then one busy hour tips the machine over.
- A burst on a full machine. A build, an import, a report — something asked for a lot at once, and the machine had nothing left to give.
- A container hit its own ceiling. Containers can carry individual memory limits. One that reaches its ceiling is killed even while the machine has plenty free — which is why the numbers can look fine while an app keeps dying.
- No emergency reserve. A machine without swap goes from “tight” to “killing” with no gradual warning stage in between.
What to do right now
- Confirm it was the machine, not your app:
journalctl -k | grep -i "out of memory"— the killer signs its work in the system log, naming the victim. - See where memory stands now:
free -h— if available memory is still nearly zero, the next kill is already queued. - Find the grower:
docker stats --no-stream(ortop, sorted by memory) — the app far above what it plausibly needs is your suspect. Restart it deliberately, on your terms, rather than waiting for the killer to do it on its own. - If the machine has no swap, add a modest swapfile. It will not fix a leak, but it turns “sudden killing” into “visible slowing” — which is the warning stage you were missing.
When to worry
One kill after an unusual burst is an event. Kills recurring daily are a leak or a machine that is simply too small — and each one risks landing on your database next. If the same app dies and returns endlessly, you are watching the killer and a restart policy take turns: that loop has its own page — your app keeps restarting itself.
How Onserva handles this
The kill is the last frame of a story that took hours to tell — memory climbing, the emergency reserve filling, response times sagging. Onserva watches the whole story: memory and swap trends judged on their climb, each container's memory individually, and alerts in plain English while there is still time to choose your own remedy. When a restart genuinely is the right fix, it is offered as one press — and when the readings point at an application bug instead, it says so rather than restarting things for the sake of activity.