Date(s) - February 24, 2022
12:00 pm - 1:00 pm
Add to Calendar
The Society of HPC Professionals lunch and learn event
Lunch & Learn – February 2022
Serving and Managing Reproducible HPC Environments via Conda-Store
Chris Ostrouchov, Quansight
24 February 2022 | Live stream
Download presentations and/or watch videos
About the Event
When working within teams and organizations managing and keeping track of the software packages used in a workflow or project can be very important. In HPC environments this used to be relatively straightforward, the administrators packaged a set of available software packages and compilers and provided a mechanism (i.e. module load etc) for end users to load those packages in a convenient manner. Unfortunately, with the rise of the modern scientific software ecosystem the sheer variety of packages available and the speed at which those packages get updated makes it impractical for a centralized administration paradigm to keep up. On the flip side, often end users install packages in an ad-hoc manner making reproducibility and collaboration near impossible.
Conda-Store provides a way for end users to quickly create and modify environments while at the same time capturing those environments in a reproducible manner that can be shared within teams and managed by IT. We will walk through the architecture of Conda-Store and show how it balances the three corners of the IT / User / Reproducibility triangle. Conda-Store packages a given environment in many formats including filesystem, lockfiles, tarballs, and docker images. This would enable nodes within a compute cluster to download tarballs of the environment to avoid a common issue in HPC distributed file systems being overloaded by small file accesses. In the future there are plans for Conda-Store to additionally serve singularity containers of environments. We will also briefly demonstrate Conda-Store via QHub-HPC, a lightweight way to deploy a small slurm based HPC cluster in the cloud or on bare-metal with JupyterHub, Dask, Dashboard sharing, monitoring, flexible authentication, and many other features integrated. Related to QHub-HPC we would like to show how using jupyterhub and jupyterhub-ssh can remove the need for a user login node and ensure that all users are launched inside of dedicated Slurm allocations.
About the Speaker
Christopher Ostrouchov, Ph.D.
Senior Software Architect at Quansight
Chris is a scientific software architect at Quansight and the technical lead on QHub and Conda-Store. He is passionate about numerical computing, reproducibility, and making open source sustainable. He is a contributor to open source software in the pydata ecosystem mainly around JupyterHub, Dask, and NixOS.
Chris earned his Ph.D. in Cneering from the
A reminder email with Zoom join info will be sent the day prior to the meeting to all registrants.
Note: please check your spam folder if you don’t see it and consider adding the domain hpcsociety.org to your approved list.
Members: remember to login to see your free ticket option.
Note: if you see a message to renew, we have seen a glitch on some browsers. Click the renew link, which will take you to your account page, where you will see if you are current. If you are, simply navigate again to the event page and you should see your free ticket option.
Bookings are closed for this event.