P100 GPU Nodes added to NewRiver

ARC is happy to announce the addition of 39 new GPU nodes to the NewRiver cluster. Each of these nodes is equipped with two Intel Xeon E5-2680v4 (Broadwell) 2.4GHz GPU (28 cores/node in all), 512 GB memory, and two NVIDIA P100 GPUs. Each GPU is capable of up to 4.7 TeraFLOPS of double-precision performance, so including CPU and GPU these nodes add over 400 TFLOPS of peak double-precision throughput to ARC's resources.

Usage notes:

  • Please restrict work on these nodes to GPU-enabled applications. These nodes represent the majority of ARC's GPU resources and we have run into GPU resource constraints in the past.
  • The P100 nodes are named nr127 through nr165.
  • The CPU architecture on these nodes (Broadwell) is newer than on the (Haswell) login nodes, so you should compile your code on the compute nodes for maximum performance.
  • We have created two queues for these nodes:
    • p100_normal_q for production jobs
    • p100_dev_q for code development and testing
  • You can start an interactive job on the nodes with this command (change yourallocation to the name of your allocation):

interact -q p100_dev_q -lnodes=1:ppn=28:gpus=2 -A yourallocation

  • We have updated the NewRiver example submission script to include information about the P100 nodes, and have also provided a simple example P100 submission script, in the Examples section.

Please let us know if you have questions or run into problems getting started.