Omega_h not compatible with CUDA on Weaver

[ikalash]

I turned on Omega_h in the weaver nightlies and it looks like it's not compatible with the CUDA library:

CMake Error at CMakeLists.txt:136 (message):
  CUDA 11.2 does not support Omega_h, use an older or newer version

-- Configuring incomplete, errors occurred!
See also "/projects/albany/nightlyCDashWeaver/build/AlbBuild/tpls/omegah/Omega_h-prefix/src/Omega_h-build/CMakeFiles/CMakeOutput.log".
gmake[2]: *** [CMakeFiles/Omega_h.dir/build.make:92: Omega_h-prefix/src/Omega_h-stamp/Omega_h-configure] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:83: CMakeFiles/Omega_h.dir/all] Error 2
gmake: *** [Makefile:91: all] Error 2

CMake Error at cmake/GetOrInstallOmegah.cmake:115 (message):
  Die
Call Stack (most recent call first):
  CMakeLists.txt:753 (include)

https://sems-cdash-son.sandia.gov/cdash/build/53415/configure

I presume we will just punt on turning on Omega_h on weaver, or is there a different plan?

@jewatkins @mcarlson801

[jewatkins]

@cwsmith what versions of cuda are supported?

[cwsmith]

Hmmm. That check may be a bit conservative now that we have a 'pure' kokkos backend that doesn't rely on thrust; there were thrust bugs in some cuda releases. I'll run a test with the problematic cuda 11.2 and the new backend to confirm.

[cwsmith]

@jewatkins I'm running tests now (tracked here) and will keep you posted.

[cwsmith]

@jewatkins CUDA >= 11.4.4 works (with GCC 10.4.0 in this testing, newer/other versions are fine) in my testing.

[jewatkins]

@ikalash maybe it's best just to turn off omega_h for this build for now since we'll likely transition off of weaver and onto blake. I can test omega_h + cuda there

[ikalash]

We can definitely turn it off in the weaver nightlies. I will do this if there are no objections.

How about PM? That one is currently using gcc 11.2.0, which it sounds like might be problematic for omega_h. I was going to tell @mcarlson801 to try turning it on there once we got the weaver ones up, but it sounds like we may have to hold off.

[bartgol]

Why are we turning off weaver? Does blake feature V100 as well? I thought it didn't... Since Summit's life got extended by a year, I think it's best to keep V100 tested somewhere, so if blake does not feature V100, we should prob keep weaver.

[jewatkins]

We're not turning off weaver yet, just disabling omega_h. There's issues with the new module set on weaver (I sank a lot of time on it last FY) and there are open tickets which have not been resolved. blake has H100. Plan is to keep weaver online for as long as summit is online or if it takes too much work to maintain.

[jewatkins]

We can definitely turn it off in the weaver nightlies. I will do this if there are no objections.

How about PM? That one is currently using gcc 11.2.0, which it sounds like might be problematic for omega_h. I was going to tell @mcarlson801 to try turning it on there once we got the weaver ones up, but it sounds like we may have to hold off.

It makes sense to test omega_h on perlmutter so I'd go ahead and try to turn it on there.

[ikalash]

It makes sense to test omega_h on perlmutter so I'd go ahead and try to turn it on there.

Could you please try this @mcarlson801 ?

[jewatkins]

FYI, he's OOO this week