Home > GNU/Linux > CUDA-aware GROMACS 5.1.1 installation on Ubuntu 14.04.4 (Trusty)
CUDA-aware GROMACS 5.1.1 installation on Ubuntu 14.04.4 (Trusty)
How to deploy straightfully the cuda-aware GROMACS installation
Wednesday 16 December 2015, by
CUDA-aware GROMACS mode is very efficient in providing nanoseconds/day performance even on small workstations. I had to reinstall GROMACS recently and struggled in getting GROMACS aware of the CUDA capability of my graphic card. Since it works now, I’m trying to tell you the long story (short).
It seems that with very recent revisions of gromacs it is not possible anymore to use the bundled NVIDIA packages from the official Ubuntu repositories. Each combination of drivers and packages I checked proved to be problematic, since even if everything seemed to be installed properly, GROMACS failed to detect the hardware dynamically. The error message was:
NOTE: Error occurred during GPU detection:
no CUDA-capable device is detected
Can not use GPU acceleration, will fall back to CPU kernels
Disclaimer (updated on April, 22th, 2016) : up-to-now I have not been able to trace down where the problem lies, either in latest Ubuntu binary builds or in the detection code from GROMACS, I’ll update this post as I make progress on the topic. The explanations to get a properly working installation stand true.
Executive summary
In order to get a working GROMACS installation detecting properly the CUDA-aware graphic car you have to :
1 - download the cuda toolkit cuda_7.5.18_linux.run
2 - (optionnaly) download the GPU deployment kit (includes the NVML library)
3 - Switch to console mode to remove any existing packages and modules:
(quit any existing session)
Press simultaneously Ctrl+Alt+F1
Log in
sudo service lightdm stop
sudo rmmod nvidia
sudo apt-get --purge remove cuda* nvidia*
4 - Install the CUDA toolkit and the GPU deployment kit
sh cuda_7.5.18_linux.run --toolkitpath=/opt/apps/pkg/cuda/7.5.18
-> Accept default locations
-> Accept the DRIVER installation
sh gdk_linux_amd64_352_55_release.run --installdir=/opt/apps/pkg/nvidia_gdk/352.55 --silent
5 - Reconnect to your graphical session
sudo service lightdm start
exit
Press Ctrl+Alt+F1 simultaneously
6 - Connect to your session and install GROMACS using the path provided above:
tar -zxf gromacs-5.1.1.tgz
cd gromacs-5.1.1
mkdir build-cuda
cd build-cuda
cmake .. -DCMAKE_INSTALL_PREFIX=/opt/apps/pkg/gromacs/5.1.1 \
-DCMAKE_PREFIX_PATH=/opt/apps/pkg/fftw/3.3.4/ \
-DGMX_GPU=on \
-DGMX_BUILD_MDRUN_ONLY=on \
-DGMX_BINARY_SUFFIX=_cuda \
-DGMX_LIBS_SUFFIX=_cuda \
-DCUDA_TOOLKIT_ROOT_DIR=/opt/apps/pkg/cuda/7.5.18 \
-DNVML_INCLUDE_DIR=/opt/apps/pkg/nvidia_gdk/352.55/usr/include/nvidia/gdk/ \
-DNVML_LIBRARY=/opt/apps/pkg/nvidia_gdk/352.55/usr/src/gdk/nvml/lib/libnvidia-ml.so
make install
Note: this command is to build the "mdrun_cuda" executable only, this is why the "_cuda" suffix is used.
Note2: the NVML includes are not mandatory, this is an addition to better control NVIDIA hardware.
Detailed explanation
In the past i had to only install from the official Ubuntu repositories the NVIDIA driver and nvidia-cuda-toolkit and to compile gromacs as usual: everything was auto-detected correctly and I could run gromacs without further tweak.
It seems that the autodetection is now problematic, the latest cuda release needs the 352.39 driver while the latest NVIDIA driver in the repositories is 352.63. This slight difference in the "minor" version of the NVIDIA driver was not problematic in the past, and is not problematic for NVIDIA tools, since the commande nvidia-smi -a returns the expected output:
==============NVSMI LOG==============
Timestamp : Wed Dec 16 16:48:51 2015
Driver Version : 352.39
Attached GPUs : 1
GPU 0000:02:00.0
Product Name : Quadro K4000
Product Brand : Quadro
Display Mode : Enabled
Display Active : Enabled
Persistence Mode : Disabled
Accounting Mode : Disabled
Accounting Mode Buffer Size : 1920
Driver Model
Current : N/A
Pending : N/A
Serial Number : 0323314055322
GPU UUID : GPU-7b65de6b-4b92-6b56-d51b-031ed109c43e
Minor Number : 0
VBIOS Version : 80.06.76.00.0A
MultiGPU Board : No
Board ID : 0x200
Inforom Version
Image Version : 2030.0500.00.01
OEM Object : 1.1
ECC Object : N/A
Power Management Object : N/A
GPU Operation Mode
Current : N/A
Pending : N/A
PCI
Bus : 0x02
Device : 0x00
Domain : 0x0000
Device Id : 0x11FA10DE
Bus Id : 0000:02:00.0
Sub System Id : 0x097C10DE
GPU Link Info
PCIe Generation
Max : 2
Current : 2
Link Width
Max : 16x
Current : 16x
Bridge Chip
Type : N/A
Firmware : N/A
Replays since reset : 0
Tx Throughput : N/A
Rx Throughput : N/A
Fan Speed : 41 %
Performance State : P0
Clocks Throttle Reasons
Idle : Not Active
Applications Clocks Setting : Not Active
SW Power Cap : Not Active
HW Slowdown : Not Active
Unknown : Not Active
FB Memory Usage
Total : 3071 MiB
Used : 379 MiB
Free : 2692 MiB
BAR1 Memory Usage
Total : 256 MiB
Used : 4 MiB
Free : 252 MiB
Compute Mode : Default
Utilization
Gpu : 89 %
Memory : 4 %
Encoder : 0 %
Decoder : 0 %
Ecc Mode
Current : N/A
Pending : N/A
ECC Errors
Volatile
Single Bit
Device Memory : N/A
Register File : N/A
L1 Cache : N/A
L2 Cache : N/A
Texture Memory : N/A
Total : N/A
Double Bit
Device Memory : N/A
Register File : N/A
L1 Cache : N/A
L2 Cache : N/A
Texture Memory : N/A
Total : N/A
Aggregate
Single Bit
Device Memory : N/A
Register File : N/A
L1 Cache : N/A
L2 Cache : N/A
Texture Memory : N/A
Total : N/A
Double Bit
Device Memory : N/A
Register File : N/A
L1 Cache : N/A
L2 Cache : N/A
Texture Memory : N/A
Total : N/A
Retired Pages
Single Bit ECC : N/A
Double Bit ECC : N/A
Pending : N/A
Temperature
GPU Current Temp : 79 C
GPU Shutdown Temp : 101 C
GPU Slowdown Temp : 96 C
Power Readings
Power Management : Supported
Power Draw : 59.60 W
Power Limit : 87.00 W
Default Power Limit : 87.00 W
Enforced Power Limit : 87.00 W
Min Power Limit : 47.00 W
Max Power Limit : 87.00 W
Clocks
Graphics : 810 MHz
SM : 810 MHz
Memory : 2808 MHz
Applications Clocks
Graphics : N/A
Memory : N/A
Default Applications Clocks
Graphics : N/A
Memory : N/A
Max Clocks
Graphics : 810 MHz
SM : 810 MHz
Memory : 2808 MHz
Clock Policy
Auto Boost : N/A
Auto Boost Default : N/A
Processes
Process ID : 3255
Type : C
Name : mdrun_cuda
Used GPU Memory : 37 MiB
Process ID : 15576
Type : G
Name : /usr/bin/X
Used GPU Memory : 231 MiB
Process ID : 15998
Type : G
Name : compiz
Used GPU Memory : 96 MiB
To sum up
To tell a long story short:
avoid ubuntu packages and install upstream drivers and dependencies.
P.S.
Updated April, 22th.
1 Message