SHPCP February Lunch & Learn

Date/Time
Date(s) - February 24, 2022
12:00 pm - 1:00 pm

Add to Calendar
iCal

About the Event

When working within teams and organizations managing and keeping track of the software packages used in a workflow or project can be very important. In HPC environments this used to be relatively straightforward, the administrators packaged a set of available software packages and compilers and provided a mechanism (i.e. module load etc) for end users to load those packages in a convenient manner. Unfortunately, with the rise of the modern scientific software ecosystem the sheer variety of packages available and the speed at which those packages get updated makes it impractical for a centralized administration paradigm to keep up. On the flip side, often end users install packages in an ad-hoc manner making reproducibility and collaboration near impossible.

Conda-Store provides a way for end users to quickly create and modify environments while at the same time capturing those environments in a reproducible manner that can be shared within teams and managed by IT. We will walk through the architecture of Conda-Store and show how it balances the three corners of the IT  / User / Reproducibility triangle. Conda-Store packages a given environment in many formats including filesystem, lockfiles, tarballs, and docker images. This would enable nodes within a compute cluster to download tarballs of the environment to avoid a common issue in HPC distributed file systems being overloaded by small file accesses. In the future there are plans for Conda-Store to additionally serve singularity containers of environments. We will also briefly demonstrate Conda-Store via QHub-HPC, a lightweight way to deploy a small slurm based HPC cluster in the cloud or on bare-metal with JupyterHub, Dask, Dashboard sharing, monitoring, flexible authentication, and many other features integrated. Related to QHub-HPC we would like to show how using jupyterhub and jupyterhub-ssh can remove the need for a user login node and ensure that all users are launched inside of dedicated Slurm allocations.

[1] https://conda-store.readthedocs.io/en/latest/
[2] https://github.com/Quansight/qhub-hpc

About the Speaker

Christopher Ostrouchov, Ph.D.
Senior Software Architect at Quansight

Chris is a scientific software architect at Quansight and the technical lead on QHub and Conda-Store.  He is passionate about numerical computing, reproducibility, and making open source sustainable. He is a contributor to open source software in the pydata ecosystem mainly around JupyterHub, Dask, and NixOS.

Chris earned his Ph.D. in Computational Material Science and Engineering from the University of Tennessee, Knoxville, including a Chancellor’s Fellowship scholarship, and a BS in Computational and Applied Mathematics from Clemson University. In his spare time he enjoys running and contributing to reproducible software via nixpkgs and nixos..

A reminder email with Zoom join info will be sent the day prior to the meeting to all registrants.
Note: please check your spam folder if you don’t see it and consider adding the domain hpcsociety.org to your approved list.

Members: remember to login to see your free ticket option.
Note: if you see a message to renew, we have seen a glitch on some browsers. Click the renew link, which will take you to your account page, where you will see if you are current. If you are, simply navigate again to the event page and you should see your free ticket option.

Bookings