HPC Service Overview

Introduction

The current cluster - an IBM / Lenovo NeXtScale based system - had been installed during summer of 2015 after an EU-wide competitive bidding and selection process (Wettbewerblicher Dialog mit Teilnahmewettbewerb). The system had been co-funded by the German federal government, the Land Brandenburg and the European Union.

According to the national supercomputer classification of the Deutsche Forschungsgemeinschaft (DFG) it is a tier-3 system. Its main purpose is to serve as a base for model development and capacity computing production runs. It ranked place 354 on the official June 2015 TOP500 list of the fastest supercomputers worldwide while installation still being in the final phase and will stay on the list at least until summer 2016.

The cluster computer is available to all scientists of the institute and to external scientists affiliated with the institute through co-operation agreements.  Registration with IT-Services is required prior to accessing the system.

The IBM / Lenovo NextScale Cluster (2015)

Photo cortesy of Lothar Lindenhan (PIK)

Cluster Highlights

• Intel Haswell and Broadwell processors with scalar frequencies of up to 3.6 GHz and 4 GByte DDR4 memory per core,
  
• A set of graphical co-processors to support development of new applications,
• High available parallel file system with 4 PByte capacity and 20 GBps max read/write bandwidth,
  
• Non-blocking high-performance FDR Infiniband network,
• Direct water-cooled processors and memory with waste heat used to heat office building(s) during the winter season.

Cluster Basic Metrics

Hardware

• A total of 6.624 CPU cores, to be used primarily for batch processing
  (5088 Intel Haswell + 1536 Intel Broadwell).
  
  
• A Mellanox SX6536 648-Port FDR InfiniBand Director Switch (56 Gbps per port).
  
  
• 4 PByte total net files system capacity.
  Based on eight IBM X Series / Lenovo GSS-24 systems, equipped with a total of 928 6TB disk drives and directly attached to the FDR Infiniband network.
  
  

• 12 support servers in high availability pairs to be used as:
• Interactive terminal servers [log-in required]
• Interactive visualization servers [log-in required]
• NFS / CIFS file system export [log-in required]
• Backup and archiving
• System administration
• Batch queue administration.
  
  
• 312 direct water-cooled Lenovo nx360 M5 compute servers each equipped with:
• 16 cores (2x Intel Xeon E5-2667 v3 8C, 3.4GHz, 20MB Cache, 135W processors) - Intel Haswell architecture.
• 64 GB DDR4 RAM @ 2133 MHz
• Mellanox Connect-IB FDR port (56 Gb/s)
  
  
• 48 direct water-cooled Lenovo nx360 M5 compute servers each equipped with:
• 32 Cores (2x Intel Xeon E5-2697A v4 16C, 2.6Ghz. 40MB Cache, 145W)  - Intel Broadwell architecture.
• 128 GB DDR4 RAM @ 2400 Mhz

• Mellanox Connect-IB FDR port (56 Gb/s)
  
  

• 6 air cooled compute servers each equipped with:
• 16 cores (2x Intel Xeon E5-2667 v3 8C, 3.2GHz, 20MB, 2133MHz) 135W  - Intel Haswell architecture.
• 256 GB DDR4 RAM @ 2133 MHz
• Mellanox Connect-IB FDR port (56 Gb/s)
• nVidia Kepler K40 accelerator (only on two nodes).
  
  

System Software

• Operating System: SUSE Linux Enterprise Server Version 12, Release Level 3
• Batch Queue and Resource Management: SLURM
• Cluster Administration: xCat
• Parallel Filesystem: IBM Spectrum Scale / GPFS  5.0
• Software Package Management: Environment Modules

Acknowledgments

Authors are encouraged to acknowledge the funding agencies, if their papers rely on numerical experiments conducted on the high performance computer of the institute. An example text is provided here [log-in required]. 

Cluster Access

Accessing the cluster requires additional authorisation which need to be applied for via this form by research domain secretaries and membership in a project group which is already registered for hpc job accounting.

Please note, that hpc access is required only for submitting and controlling compute jobs.  Data stored on the hpc cluster can be accessed remotely and a hpc cluster access is not necessary.

The hpc cluster is available via interactive login via secure shell (ssh) command using publickey authentication to interactive login host(s): cluster.pik-potsdam.de. A simple set-up for Linux client computers is available here.  A more complex set-up for Microsoft Windows client computers is provided here.  [log-in required].

Cluster Documentation

The complete set of user documentation is available here .

Cluster Support

Questions and comments should be sent to: this e-mail address  [log-in required].

Cluster Utilization 

A set of cluster statistics by user / group, month and year is available here  [log-in required].

Cost of Computing and Storage Capacity

Estimates for the cost of computing per CPU/h and for storage capacity per TByte are provided here [log-in required].