If you run Python inside containers, chances are you have seen Linux’s OOMKiller working at least a couple of times.
This happens because Python sees the entire host’s resources as if they were available for its use.
Then, it may try to allocate more memory than it is allowed to, which causes Linux to kill the process.
To fix that, we may read the actual limits from
/sys/fs/cgroup/memory/memory.limit_in_bytes and set it as the process max address space area.
import resource import os import time if os.path.isfile('/sys/fs/cgroup/memory/memory.limit_in_bytes'): with open('/sys/fs/cgroup/memory/memory.limit_in_bytes') as limit: mem = int(limit.read()) resource.setrlimit(resource.RLIMIT_AS, (mem, mem)) x = bytearray(900*1024*1024) # allocate 900mb print('ok') time.sleep(20)
Save it as
mem.py, and lets run some tests:
$ docker run \ --rm \ -m 1G \ -v $PWD:/tmp \ python:rc-alpine \ python /tmp/mem.py
If you run
docker stats in another container, you’ll see it takes ~900Mb of RAM, and everything is fine.
This should work with or without setting the
Now, let’s try to change:
-x = bytearray(900*1024*1024) # allocate 900mb +x = bytearray(4000*1024*1024) # allocate 4000mb
And we’ll see:
$ docker run --rm -m 1G -v $PWD:/tmp python:rc-alpine python /tmp/mem.py Traceback (most recent call last): File "/tmp/mem.py", line 10, in <module> x = bytearray(4000*1024*1024) # allocate 4000mb MemoryError
OK, but what if we don’t set
RLIMIT_AS? Lets test:
docker run \ --rm \ -m 1G \ -v $PWD:/tmp \ python:rc-alpine \ python -c "x = bytearray(4000*1024*1024); print('ok')"
You’ll see the container gets killed without printing ‘ok’.
I think most of the time you’ll rather have a
MemoryError instead of a
SIGKILL, if that’s the case, running the first snipped in the
__init__ of your code might be a good workaround.
I think Python should probably copy something from Java, and either make this the default behavior or hide it behind a flag.
In my perception, most people would expect this to be the default… as well as a lot of other tools (
top for example, but that’s a matter for another post).
As stated before, this is just a workaround.
Its possible it doesn’t work on all distributions, or that it stops working in the future.
Use it carefully.