Python

Python on IDUN cluster is available in different forms.

NOTE: There can be naming confusion because of:
– “modules in Python” – files containing Python code that can be imported inside another Python program. Link
– “modules on cluster” – “Environment Modules”. A tool to dynamically change the users’ environment. Link

1. Use module with Python and preinstalled libraries/modules

IDUN has multiple (Environment) modules with Python that includes different set of Python packages. These modules are more popular because they have a lot of preinstalled libraries/modules. For example:

2. Use module Python

You will get plain Python: not customized, no extra libraries.

This command can list available Python releases (at the moment of writing).

$ module avail Python
. . .
   Python/3.9.6-GCCcore-11.2.0
   Python/3.10.4-GCCcore-11.3.0
   Python/3.10.8-GCCcore-12.2.0
   Python/3.11.3-GCCcore-12.3.0

To start using one of these modules use this command:

$ module load Python/3.11.3-GCCcore-12.3.0
$ python -V
Python 3.11.3

2.1. Install packages with pip

Example: Install "numpy" package in your home directory:

pip install --user numpy

These python modules will be installed to your cluster home directory: /cluster/home/[USERNAME]/.local/lib/python3.11/

2.2. Use a Virtual Environment to install packages

Virtual environments are useful when you need to use different versions of Python packages.
There are different implementations of "Virtual environments" in Python. This is example with "venv".
"venv" is included in the standard library starting from Python 3.3

This is an example with 2 Python virtual environments and different "numpy" releases and the first virtual environment will get the latest "pandas" package as well.

Load the Python module and create 2 virtual environments:

$ module load Python/3.11.3-GCCcore-12.3.0
$ python -m venv /cluster/home/[USERNAME]/PyEnv1
$ python -m venv /cluster/home/[USERNAME]/PyEnv2

Activate first environment and install numpy 1.24.3

$ source /cluster/home/[USERNAME]/PyEnv1/bin/activate
(PyEnv1) $ pip install numpy==1.24.3
(PyEnv1) $ pip install pandas

Deactivate first environment:

(PyEnv1) $ deactivate

Activate the second environment and install numpy 1.22.0

$ source /cluster/home/[USERNAME]/PyEnv2/bin/activate
(PyEnv2) $ pip install numpy==1.22.0
(PyEnv2) $ deactivate

2.3. Reproducing a Python Virtual Environment

This is an example to reproduce the Virtual Environment created above "PyEnv1".

Load Python (if not loaded), activate the environment, generate a requirements.txt file, and deactivate the environment. 

$ module load Python/3.11.3-GCCcore-12.3.0
$ source /cluster/home/[USERNAME]/PyEnv1/bin/activate
(PyEnv1) $ pip freeze > requirements.txt
(PyEnv1) $ deactivate

This is just a text file with package names and release versions. You can edit it if needed.

$ cat requirements.txt 
numpy==1.24.3
pandas==2.1.4
python-dateutil==2.8.2
pytz==2023.3.post1
six==1.16.0
tzdata==2023.4

Create a new virtual environment, activate, and install packages listed in the file requirements.txt

$ python -m venv /cluster/home/[USERNAME]/PyEnv3
$ source /cluster/home/[USERNAME]/PyEnv3/bin/activate
(PyEnv3) $ pip install -r requirements.txt

Virtual Environment "PyEnv1" and "PyEnv3" are identical now.

Scroll to Top