modules

Introduction

When I tried to execute my gopher daemon, pygopher in a chroot, I faced some problems and thus tried to understand how the memory management of the modules in Python works.

This page describres my observations and is not a correct nor complete description of the memory management in Python.
They are probably incomplete and wrong and are not based on the official documentation (which I probably can’t understand).

The problem

When a Python program is started, I can observe that Python:

In the case of a chrooted process, the last point is problematic: Python does not update the path when the process goes in the chroot for already loaded modules.

The attempts

I tried to force the unload/reload (importlib.reload) of a module but:

virtualenv won’t help: it only copies a subset of the core modules.
You should juggle (symbolic links) with absolute paths that point:

And virtualenv doesn’t allow to manager different versions of Python.

The solution

The solution is to use pyenv. It allows to build every version of Python in any directory.
It thus simplified the management of the paths.

To overcome the problem of the paths kept in memory for core modules, I simply created a symbolic link inside the chroot in order to fake the chrooted processus and make it believes the path still exists.

Differences with Perl (or others)

I designed my finger daemon, pfinger, with the same goals.

I didn’t face the same problems with Perl because it loads all modules in memory once and fo all (except for dynamic loaded ones): the memory management is totally different (no GC).

Conclusion

I don’t have a lot of experiences with Python but this problem helps me to learn quite useful things.

I also had to write my code in a more reusable design: