Installing your own Python modules on the cluster [DEPRECATED]

These instructions are out-of-date, but retained for reference.
 
*****************
** DEPRECATED **
*****************
These instructions are out-of-date, but retained for reference.

Please use "module load anaconda/2.3.0" to get the supported Python distribution
and follow instructions here to create environments and install packages.

 

There are several options for installing your own local Python modules. The first uses the standard cluster Python installation (2.6) and extends it via the $PYTHONPATH environment variable, the second uses the virtualenv and pip tools with Python 2.7 (/iplex/01/sys/applications/bin/python2.7) to create a personalised local Python environment that can also be recreated by other users from a list of required packages. A third way is to use pip to install local modules without virtualenv.

Option 1. Local modules using the standard installation

The general procedure is to override the module's default installation location (usually /usr) and configure your $PYTHONPATH variable to include the new path as well as the standard path.

(This applies to Python packages which are installed with "python setup.py install". C-based packages which use "./configure" can be prepared similarly)

a) (This step is only necessary if the installer complains about the install location not supporting .PTH files)

Add to your PYTHONPATH:

export PYTHONPATH=$PYTHONPATH:/home/<your username here>/lib64/python2.6/site-packages/

b) Download the source package, e.g.

curl -O http://pypi.python.org/packages/source/P/PrettyTable/prettytable-0.6.1.tar.gz

and unpack.

c) Install with a new prefix, for example:

python setup.py install --prefix=/home/<your username>

d) If not already done, do step a). This will need to be set every time you want to use this module, so it makes sense to add this line to your .profile

e) test the installation:

python -c "import prettytable"

This should complete without an error message.

Option 2. virtualenv and pip

You will create a local, virtual, Python environment separate from the main Python installations. You can add and remove modules as you like, the changes will only be made to your copy.

a) Create a new virtualenv called 'dev', or whatever you'd like to call it:

virtualenv dev

This will create a subdirectory called 'dev'

linstead@login01:~$ virtualenv dev
New python executable in dev/bin/python2.7
Also creating executable in dev/bin/python
Installing setuptools............done.
Installing pip...............done.
linstead@login01:~$

To use this environment, you must activate it:

source dev/bin/activate

e.g.

linstead@login01:~$ source dev/bin/activate
(dev)linstead@login01:~$

Note the new prompt, and the new default Python:

(dev)linstead@login01:~$ which python
/home/linstead/dev/bin/python
(dev)linstead@login01:~$

b) install packages with 'pip':

(dev)linstead@login01:~$ pip install graph
Downloading/unpacking graph
  Downloading graph-0.4.tar.gz
  Running setup.py egg_info for package graph
Installing collected packages: graph
  Running setup.py install for graph
Successfully installed graph
Cleaning up...
(dev)linstead@login01:~$
(dev)linstead@login01:~$ python -c "import graph"
(dev)linstead@login01:~$

c) Optional. You can share your environment by creating a requirements.txt file. Other users can recreate your environment with pip and this file:

pip freeze > requirements.txt
pip install -r requirements.txt

Option 3: pip without virtualenv

This option uses pip with the '--user' argument thus:

pip install --user cherrypy

If you get an error like this:

error: can't combine user with with prefix/exec_prefix/home or install_(plat)base

you should create a file .pydistutils.cfg in your home directory with the following contents:

[install] prefix= 

This overrides the system-wide "prefix" setting for installing Python modules. You may need to add ~/.local/lib/python2.6/site-packages/ to your $PYTHONPATH to access the new modules.

 

Errors:

If you see an error like this:

  Complete output from command /home/linstead/parallelpython/bin/python2.7 -c "#!python
\"\"\"Bootstra...sys.argv[1:])

" /iplex/01/sys/applic...ols-0.6c11-py2.7.egg:
  error: can't create or remove files in install directory

The following error occurred while trying to add or remove files in the
installation directory:

    [Errno 2] No such file or directory: '/lib/python2.7/site-packages/test-easy-install-24460.pth'

try modifying or deleting the file $HOME/.pydistutils.cfg