In this tutorial, you will learn how to:
A custom Jupyter notebook to meet the needs of your computing environment can be provisioned at CSC through Puhti web interface. Here are minimal instructions for setting up your Jupyter notebook on Puhti supercomputer:
Tykky container wrapper installs your software tools/packages as a container image for improved performance metrics such as faster startup times, reduced I/O load, and fewer number of file creations on Lustre parallel filesystems. For the purpose of this tutorial, we use NMRLipids course set up which needs installation of several python packages for Jupyter notebook. Please install all needed computing environment in the projappl directory.
Install python packages for course using tykky wrapper as below:
# Navigate to the scratch area of your course project
cd /scratch/project_xxxx/ # Make sure to replace project_xxxx with correct project number
# As a starting point, you can download all necessary template files for the
# customising course environment from allas object storage (as shown below).
# The downloaded template files include a module environment file namely, NMRLipids-course.lua
# and resource specifications file, NMRLipids-course-resources.yaml files. You can find more information about these files later.
wget https://a3s.fi/CSC_training/Puhti_web_helper.tar.gz && tar -xavf Puhti_web_helper.tar.gz
cd Puhti_web_helper
module load tykky
mkdir -p /projappl/project_xxxx/$USER and mkdir -p /projappl/project_xxxx/$USER/NMRLipids
# You can write all needed python packages in yaml as in env_nmr.yml for NMRLpids course and install with tykky
conda-containerize new --prefix /projappl/project_xxxx/$USER/NMRLipids env_nmr.yml # Installation can take for a while
Tykky would install all needed packages (as listed in the file, env_nmr.yml) to the directory /projappl/project_xxxx/$USER/NMRLipids.
Please note that you have to provide absolute path of bin directory (which exists inside the folder of /projappl/project_xxxx/$USER/NMRLipids) in the “.lua” file
The template files for custom notebook environment have to be added under the directory /projappl/project_xxxx/www_puhti_modules/ in order te be visible in Puhti web interface. The www_puhti_modules directory can be created if it does not exist.
The two files needed for setting up the course modules are:
<<course_name>>.lua file that defines the module that sets up the Python environment. Only files containing the text Jupyter will be visible in the app.<<course_name>>-resources.yml that defines the default resources used for Jupyter.In this NMRLipids course example, the above-mentioned two template files (NMRLipids-course-resources.yml and NMRLipids-course.lua) are available in the folder ‘Puhti_web_helper’ that you have downloaded from allas. so just modify cautiously for your own installation and place them in www_puhti_modules folder as below:
# Make sure to use correct project number (in two places in NMRLipids-course.lua file) and your CSC username in the the copied files in
# /projappl/project_xxxx/www_puhti_modules.
mkdir -p /projappl/project_xxxx/www_puhti_modules && cp NMRLipids-course-resources.yml NMRLipids-course.lua /projappl/project_xxxx/www_puhti_modules
As the name of the “.lua” file will appear as custom notebook environment on Puhti web inaterface, please use some unique name for .lua and .yaml file (e.g., add your CSC user name in the file name e.g., NMRLipids-course-yourcscusername.lua) in this course environment.
cd /projappl/project_xxxx/www_puhti_modules
# add your CSC username in .lua and .yaml file names
mv NMRLipids-course-resources.yml NMRLipids-course-yourcscusername-resources.yml
mv NMRLipids-course.lua NMRLipids-course-yourcscusername.lua