diff --git a/docs/software/communication/openmpi.md b/docs/software/communication/openmpi.md index 018fecce..e1d21391 100644 --- a/docs/software/communication/openmpi.md +++ b/docs/software/communication/openmpi.md @@ -6,7 +6,7 @@ However, [OpenMPI](https://www.open-mpi.org/) can be used as an alternative in some cases, with limited support from CSCS. OpenMPI is available for use in both uenv and containers. -To use OpenMPI on Alps, it must be built against [libfabric][ref-communication-libfabric] with support for the [Slingshot 11 network][ref-alps-hsn]. +Support for the [Slingshot 11 network][ref-alps-hsn] is provided by the [libfabric][ref-communication-libfabric] library. [](){#ref-communication-openmpi-using} ## Using OpenMPI @@ -14,12 +14,7 @@ To use OpenMPI on Alps, it must be built against [libfabric][ref-communication-l [](){#ref-communication-openmpi-uenv} ### uenv -!!! under-construction - Building and using OpenMPI in uenv on Alps is work in progress. - - The instructions found on this page may be inaccurate, but are a good starting point to using OpenMPI on Alps. - -OpenMPI is provided through a [uenv][ref-uenv] similar to [`prgenv-gnu`][ref-uenv-prgenv-gnu]. +OpenMPI is provided in the [`prgenv-gnu-openmpi`][ref-uenv-prgenv-gnu-openmpi] uenv. Once the uenv is loaded, compiling and linking with OpenMPI and libfabric is transparent. At runtime, some additional options must be set to correctly use the Slingshot network. @@ -29,34 +24,72 @@ This is done with the `--mpi` flag of `srun`: srun --mpi=pmix ... ``` -Additionally, the following environment variables should be set: +There are two primary ways to configure OpenMPI and libfabric to use the Slingshot network: + +1. [Only using the CXI provider][ref-communication-openmpi-cxi]. + This method has been found to work in more applications but uses NICs for intra-node communication which can limit performance. +2. [Using the LINKx provider][ref-communication-openmpi-lnx] which combines the CXI provider for inter-node communication with the shared memory provider for intra-node communication. + This provider is newer, may not support all features, and more likely to contain bugs, but makes full use of intra-node bandwidth. + +We recommend trying the LINKx provider first as it provides better performance in the situations that it's supported. +If you encounter failures using the LINKx provider we ask you to [get in touch with us][ref-get-in-touch] so that we can evaluate whether upstream libfabric or OpenMPI need fixing. + +[](){#ref-communication-openmpi-cxi} +#### Using the CXI provider + +To use the CXI provider the following environment variables should be set: + ```bash export PMIX_MCA_psec="native" # (1)! export FI_PROVIDER="cxi" # (2)! -export OMPI_MCA_pml="^ucx" # (3)! +export OMPI_MCA_pml="cm" # (3)! export OMPI_MCA_mtl="ofi" # (4)! ``` 1. Ensures PMIx uses the same security domain as Slurm. Otherwise PMIx will print warnings at startup. 2. Use the CXI (Slingshot) provider. -3. Use anything except [UCX](https://openucx.org/documentation/) for [point-to-point communication](https://docs.open-mpi.org/en/v5.0.x/mca.html#selecting-which-open-mpi-components-are-used-at-run-time). The `^` signals that OpenMPI should exclude all listed components. +3. Use CM for [point-to-point communication](https://docs.open-mpi.org/en/v5.0.x/mca.html#selecting-which-open-mpi-components-are-used-at-run-time). 4. Use libfabric for the [Matching Transport Layer](https://docs.open-mpi.org/en/v5.0.x/mca.html#frameworks). -!!! info "CXI provider does all communication through the network interface cards (NICs)" +!!! info "The CXI provider does all communication through the network interface cards (NICs)" When using the libfabric CXI provider, all communication goes through NICs, including intra-node communication. - This means that intra-node communication can not make use of shared memory optimizations and the maximum bandwidth will not be severely limited. + This means that intra-node communication can not make use of shared memory optimizations and the maximum bandwidth will be severely limited. + Use the [LINKx][ref-communication-openmpi-lnx] provider to make full use of the available intra-node bandwidth. - Libfabric has a new [LINKx](https://ofiwg.github.io/libfabric/v2.1.0/man/fi_lnx.7.html) provider, which allows using different libfabric providers for inter- and intra-node communication. - This provider is not as well tested, but can in theory perform better for intra-node communication, because it can use shared memory. - To use the LINKx provider, set the following, instead of `FI_PROVIDER=cxi`: +[](){#ref-communication-openmpi-lnx} +#### Using the LINKx provider - ```bash - export FI_PROVIDER="lnx" # (1)! - export FI_LNX_PROV_LINKS="shm+cxi" # (2)! - ``` +The default configuration routes all communication through the NICs. +While performance may sometimes be acceptable, this mode does not make full use of the much higher intra-node bandwidth available on Grace-Hopper nodes. +In particular, GPU-GPU communication is significantly faster when using the appropriate intra-node links. + +The experimental [LINKx](https://ofiwg.github.io/libfabric/v2.3.1/man/fi_lnx.7.html) libfabric provider allows composing multiple libfabric providers for inter- and intra-node communication. +The CXI provider can be used for inter-node communication while the shared memory (`shm`) provider can be used to take advantage of xpmem for CPU-CPU communication and GDRCopy for GPU-GPU communication. + +!!! danger "The LINKx provider is experimental" + + While many basic tests work correctly using the LINKx provider we have had reports of applications failing to run with the LINKx provider. + Always validate your results to ensure MPI is working correctly. + +To use the LINKx provider set the following environment variables: + +```bash +export PMIX_MCA_psec="native" +export FI_PROVIDER="lnx" # (1)! +export FI_LNX_PROV_LINKS="shm+cxi:cxi0|shm+cxi:cxi1|shm+cxi:cxi2|shm+cxi:cxi3" # (2)! +export FI_SHM_USE_XPMEM=1 # (3)! +export OMPI_MCA_pml="cm" +export OMPI_MCA_mtl="ofi" +export OMPI_MCA_mtl_ofi_av=table # (4)! +``` - 1. Use the libfabric LINKx provider, to allow using different libfabric providers for inter- and intra-node communication. - 2. Use the shared memory provider for intra-node communication and the CXI (Slingshot) provider for inter-node communication. +1. Use the libfabric LINKx provider, to allow using different libfabric providers for inter- and intra-node communication. +2. Specify which providers LINKx should use. + Use the shared memory provider for intra-node communication and the CXI (Slingshot) provider for inter-node communication. + Choose one of the four available NICs on a node in a round-robin fashion. +3. Explicitly use xpmem for CPU-CPU communication. + The default is to use CMA. +4. The LINKx provider requires this option to be set. [](){#ref-communication-openmpi-ce} ### Containers @@ -85,6 +118,9 @@ Note that OpenMPI v5 is the first version with full support for libfabric, requi * The `--with-ofi` and `--with-ucx` flags configure OpenMPI with the libfabric and UCX back ends respectively. * The `--enable-oshmem` flag builds OpenSHMEM as part of the OpenMPI installation, which is useful to support SHMEM implementations like [NVSHMEM][ref-communication-nvshmem]. + Note that this example does not enable the [LINKx provider][ref-communication-openmpi-lnx] as in the uenv. + We do not currently provide instructions to enable the LINKx provider in manually built container images. + Expand the box below to see an example of a full Containerfile that can be used to create an OpenMPI container on the gh200 nodes of Alps: ??? note "The full Containerfile" @@ -105,14 +141,25 @@ The EDF file for the container should contain the following: ```toml [env] PMIX_MCA_psec="native" # (1)! +FI_PROVIDER="cxi" # (2)! +OMPI_MCA_pml="cm" # (3)! +OMPI_MCA_mtl="ofi" # (4)! ``` 1. Ensures PMIx uses the same security domain as Slurm. Otherwise PMIx will print warnings at startup. +2. Use the CXI (Slingshot) provider. +3. Use CM for [point-to-point communication](https://docs.open-mpi.org/en/v5.0.x/mca.html#selecting-which-open-mpi-components-are-used-at-run-time). +4. Use libfabric for the [Matching Transport Layer](https://docs.open-mpi.org/en/v5.0.x/mca.html#frameworks). + +Like with the uenv, the `--mpi=pmix` flag must be passed to `srun` to ensure PMIx is used for MPI initialization: +```bash +srun --mpi=pmix ... +``` [](){#ref-communication-openmpi-performance} -## OpenMPI Performance +## OpenMPI performance -We present some performance numbers for OpenMPI, obtained using the OSU benchmarks compiled in the above image. +We present some performance numbers for OpenMPI, obtained using the OSU benchmarks compiled in the above container image. !!! warning "no version information available" The following warning message was generated by each rank running the benchmarks below, and can safely be ignored. @@ -123,13 +170,17 @@ We present some performance numbers for OpenMPI, obtained using the OSU benchmar The first performance benchmarks are for the OSU point-to-point bandwidth test `osu_bw`. * inter-node tests place the two ranks on different nodes, so that all communication is over the Slingshot network -* intra-node tests place two ranks on the same node, so that communication is via NVLINK or memory copies in the CPU-CPU case +* intra-node tests place two ranks on the same node, but communication is still done over the Slingshot network -!!! note "impact of disabling the CXI hook" +!!! note + The container is configured to only use the CXI provider of libfabric, routing intra-node communication over NICs. + We currently only provide instructions on using the [experimental LINKx provider][ref-communication-openmpi-lnx], which can make use of higher intra-node bandwidth, for uenv. + +!!! note "Impact of disabling the CXI hook" On many Alps vClusters, the Container Engine is configured with the [CXI hook][ref-ce-cxi-hook] enabled by default, enabling transparent access to the Slingshot interconnect. - The inter node tests marked with `(*)` were run with the CXI container hook disabled, to demonstrate the effect of not using an optimised network configuration. - If you see similar performance degradation in your tests, the first thing to investigate is whether your setup is using the libfabric optimised back end. + The inter-node tests marked with `(*)` were run with the CXI container hook disabled, to demonstrate the effect of not using an optimised network configuration. + If you see similar performance degradation in your tests, the first thing to investigate is whether your setup is using the libfabric CXI provider. === "CPU-to-CPU inter-node" ```console @@ -255,8 +306,6 @@ The first performance benchmarks are for the OSU point-to-point bandwidth test ` 4194304 1802.75 Pass ``` - - === "CPU-to-CPU intra-node" ```console $ srun -N1 -n2 --mpi=pmix --environment=omb-ompi ./pt2pt/osu_bw --validation @@ -319,7 +368,6 @@ The first performance benchmarks are for the OSU point-to-point bandwidth test ` 4194304 23989.44 Pass ``` - Next is the all to all latency test `osu_alltoall`, for 8 ranks spread over nodes (4 ranks per node, 1 rank per GPU). === "CPU-to-CPU" @@ -437,3 +485,58 @@ Next is the all to all latency test `osu_alltoall`, for 8 ranks spread over node 524288 3107.62 Pass 1048576 5545.28 Pass ``` + +## Known issues + +Some asynchronous collectives are known not to work with GPU buffers, independent of the libfabric provider used. +For example, `MPI_Iallreduce` will fail with a segmentation fault. +Running the `osu_iallreduce` benchmark with GPU buffers results in: + +```console +$ srun -u --mpi=pmix -n4 osu_iallreduce -d cuda + +# OSU MPI-CUDA Non-blocking Allreduce Latency Test v7.5 +# Overall = Coll. Init + Compute + MPI_Test + MPI_Wait + +# Datatype: MPI_INT. +# Size Overall(us) Compute(us) Pure Comm.(us) Overlap(%) +[nid006549:31808] *** Process received signal *** +[nid006549:31808] Signal: Segmentation fault (11) +[nid006549:31808] Signal code: Invalid permissions (2) +[nid006549:31808] Failing at address: 0x4002da000000 +[nid006550:188198] *** Process received signal *** +[nid006550:188198] Signal: Segmentation fault (11) +[nid006550:188198] Signal code: Invalid permissions (2) +[nid006550:188198] Failing at address: 0x40029a000000 +[nid006549:31808] [ 0] linux-vdso.so.1(__kernel_rt_sigreturn+0x0)[0x400027ce07dc] +[nid006549:31808] [ 1] /user-environment/linux-neoverse_v2/openmpi-5.0.9-leskuw5dyswfdw3eaybcyfmsrbid3uuq/lib/libmpi.so.40(+0x19f1c8)[0x400029b0f1c8] +[nid006549:31808] [ 2] /user-environment/linux-neoverse_v2/openmpi-5.0.9-leskuw5dyswfdw3eaybcyfmsrbid3uuq/lib/libmpi.so.40(+0x12836c)[0x400029a9836c] +[nid006549:31808] [ 3] /user-environment/linux-neoverse_v2/openmpi-5.0.9-leskuw5dyswfdw3eaybcyfmsrbid3uuq/lib/libmpi.so.40(NBC_Progress+0x164)[0x400029a97bd4] +[nid006549:31808] [ 4] /user-environment/linux-neoverse_v2/openmpi-5.0.9-leskuw5dyswfdw3eaybcyfmsrbid3uuq/lib/libmpi.so.40(ompi_coll_libnbc_progress+0x8c)[0x400029a96a0c] +[nid006549:31808] [ 5] /user-environment/linux-neoverse_v2/openmpi-5.0.9-leskuw5dyswfdw3eaybcyfmsrbid3uuq/lib/libopen-pal.so.80(opal_progress+0x3c)[0x40002a23737c] +[nid006549:31808] [ 6] /user-environment/linux-neoverse_v2/openmpi-5.0.9-leskuw5dyswfdw3eaybcyfmsrbid3uuq/lib/libmpi.so.40(ompi_request_default_wait+0x50)[0x4000299f3810] +[nid006549:31808] [ 7] /user-environment/linux-neoverse_v2/openmpi-5.0.9-leskuw5dyswfdw3eaybcyfmsrbid3uuq/lib/libmpi.so.40(MPI_Wait+0x64)[0x400029a3df24] +[nid006549:31808] [ 8] /user-environment/env/default/libexec/osu-micro-benchmarks/mpi/collective/osu_iallreduce[0x40424c] +[nid006549:31808] [ 9] /lib64/libc.so.6(__libc_start_main+0xe8)[0x40002a073fa0] +[nid006549:31808] [10] /user-environment/env/default/libexec/osu-micro-benchmarks/mpi/collective/osu_iallreduce[0x404e98] +[nid006549:31808] *** End of error message *** +[nid006550:188198] [ 0] linux-vdso.so.1(__kernel_rt_sigreturn+0x0)[0x4000026a07dc] +[nid006550:188198] [ 1] /user-environment/linux-neoverse_v2/openmpi-5.0.9-leskuw5dyswfdw3eaybcyfmsrbid3uuq/lib/libmpi.so.40(+0x19f1c8)[0x4000044cf1c8] +[nid006550:188198] [ 2] /user-environment/linux-neoverse_v2/openmpi-5.0.9-leskuw5dyswfdw3eaybcyfmsrbid3uuq/lib/libmpi.so.40(+0x12836c)[0x40000445836c] +[nid006550:188198] [ 3] /user-environment/linux-neoverse_v2/openmpi-5.0.9-leskuw5dyswfdw3eaybcyfmsrbid3uuq/lib/libmpi.so.40(NBC_Progress+0x164)[0x400004457bd4] +[nid006550:188198] [ 4] /user-environment/linux-neoverse_v2/openmpi-5.0.9-leskuw5dyswfdw3eaybcyfmsrbid3uuq/lib/libmpi.so.40(ompi_coll_libnbc_progress+0x8c)[0x400004456a0c] +[nid006550:188198] [ 5] /user-environment/linux-neoverse_v2/openmpi-5.0.9-leskuw5dyswfdw3eaybcyfmsrbid3uuq/lib/libopen-pal.so.80(opal_progress+0x3c)[0x400004bf737c] +[nid006550:188198] [ 6] /user-environment/linux-neoverse_v2/openmpi-5.0.9-leskuw5dyswfdw3eaybcyfmsrbid3uuq/lib/libmpi.so.40(ompi_request_default_wait+0x50)[0x4000043b3810] +[nid006550:188198] [ 7] /user-environment/linux-neoverse_v2/openmpi-5.0.9-leskuw5dyswfdw3eaybcyfmsrbid3uuq/lib/libmpi.so.40(MPI_Wait+0x64)[0x4000043fdf24] +[nid006550:188198] [ 8] /user-environment/env/default/libexec/osu-micro-benchmarks/mpi/collective/osu_iallreduce[0x40424c] +[nid006550:188198] [ 9] /lib64/libc.so.6(__libc_start_main+0xe8)[0x400004a33fa0] +[nid006550:188198] [10] /user-environment/env/default/libexec/osu-micro-benchmarks/mpi/collective/osu_iallreduce[0x404e98] +[nid006550:188198] *** End of error message *** +srun: error: nid006549: task 0: Segmentation fault (core dumped) +srun: Terminating StepId=2243671.21 +[2025-12-17T12:59:34.342] error: *** STEP 2243671.21 ON nid006549 CANCELLED AT 2025-12-17T12:59:34 DUE TO TASK FAILURE *** +srun: error: nid006550: task 2: Segmentation fault (core dumped) +srun: error: nid006550: task 3: Terminated +srun: error: nid006549: task 1: Terminated +srun: Force Terminated StepId=2243671.21 +``` diff --git a/docs/software/prgenv/index.md b/docs/software/prgenv/index.md index 62b53a6c..847e7d81 100644 --- a/docs/software/prgenv/index.md +++ b/docs/software/prgenv/index.md @@ -10,6 +10,10 @@ CSCS provides "programming environments" on Alps vClusters that provide compiler Provides compilers, MPI, tools and libraries built around the GNU compiler toolchain. It is the go to programming environment on all systems and target node types, that is it is the first that you should try out when starting to compile an application or create a python virtual environment. +- :fontawesome-solid-layer-group: [__prgenv-gnu-openmpi__][ref-uenv-prgenv-gnu-openmpi] uenv + + Same as [`prgenv-gnu`][ref-uenv-prgenv-gnu] but with OpenMPI instead of Cray MPICH. + - :fontawesome-solid-layer-group: [__prgenv-nvfortran__][ref-uenv-prgenv-nvfortran] uenv Provides a set of tools and libraries for building applications that need the NVIDIA Fortran compiler, commonly required for OpenACC and CUDA-Fortran applications. diff --git a/docs/software/prgenv/prgenv-gnu-openmpi.md b/docs/software/prgenv/prgenv-gnu-openmpi.md new file mode 100644 index 00000000..bc9a985f --- /dev/null +++ b/docs/software/prgenv/prgenv-gnu-openmpi.md @@ -0,0 +1,56 @@ +[](){#ref-uenv-prgenv-gnu-openmpi} +# prgenv-gnu-openmpi + +The `prgenv-gnu-openmpi` uenv is otherwise similar to [`prgenv-gnu`][ref-uenv-prgenv-gnu] except it provides OpenMPI instead of Cray MPICH. + +!!! warning "OpenMPI is not officially supported on CSCS systems" + Cray MPICH is the preferred, and officially supported, MPI implementation on CSCS systems. + OpenMPI is provided on a best effort basis. + While most applications should work correctly with OpenMPI, there may be missing features, broken functionality, or bad performance compared to Cray MPICH. + Issues are best reported upstream, but CSCS is happy to help facilitate and coordinate issue reporting if you [get in touch][ref-get-in-touch]. + +Use of the uenv is similar to [`prgenv-gnu`][ref-uenv-prgenv-gnu]. +See the [OpenMPI documentation][ref-communication-openmpi] for important information on configuring OpenMPI to take advantage of the Slingshot network. + +### Versions + +=== "25.12" + 25.12 is the first release of the `prgenv-gnu-openmpi` uenv. + + ??? info "Packages exposed by the `default` and `modules` views" + * [aws-ofi-nccl@1.17.1](https://packages.spack.io/package.html?name=aws-ofi-nccl) + * [boost@1.88.0](https://packages.spack.io/package.html?name=boost) + * [cmake@3.31.9](https://packages.spack.io/package.html?name=cmake) + * [cuda@12.9.1](https://packages.spack.io/package.html?name=cuda) + * [fftw@3.3.10](https://packages.spack.io/package.html?name=fftw) + * [fmt@12.1.0](https://packages.spack.io/package.html?name=fmt) + * [gcc@14.3.0](https://packages.spack.io/package.html?name=gcc) + * [gmp@6.3.0](https://packages.spack.io/package.html?name=gmp) + * [gsl@2.8](https://packages.spack.io/package.html?name=gsl) + * [hdf5@1.14.6](https://packages.spack.io/package.html?name=hdf5) + * [kokkos@4.7.01](https://packages.spack.io/package.html?name=kokkos) + * [kokkos-kernels@4.7.01](https://packages.spack.io/package.html?name=kokkos-kernels) + * [kokkos-tools@develop](https://packages.spack.io/package.html?name=kokkos-tools) + * [libfabric@2.3.1](https://packages.spack.io/package.html?name=libfabric) + * [libtree@3.1.1](https://packages.spack.io/package.html?name=libtree) + * [lua@5.4.6](https://packages.spack.io/package.html?name=lua) + * [lz4@1.10.0](https://packages.spack.io/package.html?name=lz4) + * [meson@1.8.5](https://packages.spack.io/package.html?name=meson) + * [nccl@2.28.9-1](https://packages.spack.io/package.html?name=nccl) + * [nccl-tests@2.17.6](https://packages.spack.io/package.html?name=nccl-tests) + * [netcdf-c@4.9.3](https://packages.spack.io/package.html?name=netcdf-c) + * [netcdf-cxx@4.2](https://packages.spack.io/package.html?name=netcdf-cxx) + * [netcdf-cxx4@4.3.1](https://packages.spack.io/package.html?name=netcdf-cxx4) + * [netcdf-fortran@4.6.2](https://packages.spack.io/package.html?name=netcdf-fortran) + * [netlib-scalapack@2.2.2](https://packages.spack.io/package.html?name=netlib-scalapack) + * [ninja@1.13.0](https://packages.spack.io/package.html?name=ninja) + * [openblas@0.3.30](https://packages.spack.io/package.html?name=openblas) + * [openmpi@5.0.9](https://packages.spack.io/package.html?name=openmpi) + * [osu-micro-benchmarks@7.5.1](https://packages.spack.io/package.html?name=osu-micro-benchmarks) + * [papi@7.2.0](https://packages.spack.io/package.html?name=papi) + * [python@3.14.0](https://packages.spack.io/package.html?name=python) + * [squashfs@4.6.1](https://packages.spack.io/package.html?name=squashfs) + * [superlu@7.0.1](https://packages.spack.io/package.html?name=superlu) + * [xcb-util-cursor@0.1.5](https://packages.spack.io/package.html?name=xcb-util-cursor) + * [xpmem@2.8.2](https://packages.spack.io/package.html?name=xpmem) + * [zlib-ng@2.2.4](https://packages.spack.io/package.html?name=zlib-ng) diff --git a/docs/software/prgenv/prgenv-gnu.md b/docs/software/prgenv/prgenv-gnu.md index 779fedb8..303a40f0 100644 --- a/docs/software/prgenv/prgenv-gnu.md +++ b/docs/software/prgenv/prgenv-gnu.md @@ -10,6 +10,8 @@ It is the go to programming environment on all systems and target node types, th The [`linalg`][ref-uenv-linalg] environment is similar to prgenv-gnu, with additional linear algebra and mesh partitioning algorithms. + The [`prgenv-gnu-openmpi`][ref-uenv-prgenv-gnu-openmpi] is otherwise similar to `prgenv-gnu`, but provides OpenMPI instead of Cray MPICH. + [](){#ref-uenv-prgenv-gnu-versioning} ## Versioning