HPC Service Overview

The high-performance computer systems of the institute are used for scientific modeling, numerical simulation experiments, machine learning and scientific "big" data management.

Introduction

The current cluster - an IBM / Lenovo NeXtScale based system - had been installed during summer of 2015 after an EU-wide competitive bidding and selection process (Wettbewerblicher Dialog mit Teilnahmewettbewerb). The system had been co-funded by the German federal government, the Land Brandenburg and the European Union.

According to the national supercomputer classification of the Deutsche Forschungsgemeinschaft (DFG) it is a tier-3 system. Its main purpose is to serve as a base for model development and capacity computing production runs. It ranked place 354 on the official June 2015 TOP500 list of the fastest supercomputers worldwide while installation still being in the final phase and will stay on the list at least until summer 2016.

 

Rank Site System Cores Rmax (TFlop/s) Rpeak (TFlop/s) Power (kW)
354 Potsdam Institute for Climate Impact Research
Germany
HLR2015 - Lenovo NeXtScale nx360M5, Xeon E5-2667v3 8C 3.2GHz, Infiniband FDR14
Lenovo/IBM
5,040 212.8 258.0 128

 

The cluster computer is available to all scientists of the institute and to external scientists affiliated with the institute through co-operation agreements.  Registration with IT-Services is required prior to accessing the system.

The IBM / Lenovo NextScale Cluster (2015)

Five racks of the direct water cooled IBM NeXtScale Cluster installed at PIK in summer 2015 viewed from the back.

Photo cortesy of Lothar Lindenhan (PIK)

Cluster Highlights

  • Intel Haswell and Broadwell processors with scalar frequencies of up to 3.6 GHz and 4 GByte DDR4 memory per core,
  • A set of graphical co-processors to support development of new applications,
  • High available parallel file system with 4 PByte capacity and 20 GBps max read/write bandwidth,
  • Non-blocking high-performance FDR Infiniband network,
  • Direct water-cooled processors and memory with waste heat used to heat office building(s) during the winter season.

Cluster Basic Metrics

Hardware

  • A total of 6.624 CPU cores, to be used primarily for batch processing
    (5088 Intel Haswell + 1536 Intel Broadwell).

  • A Mellanox SX6536 648-Port FDR InfiniBand Director Switch (56 Gbps per port).

  • 4 PByte total net files system capacity.
    Based on eight IBM X Series / Lenovo GSS-24 systems, equipped with a total of 928 6TB disk drives and directly attached to the FDR Infiniband network.

  • 12 support servers in high availability pairs to be used as:
  • 312 direct water-cooled Lenovo nx360 M5 compute servers each equipped with:
    • 16 cores (2x Intel Xeon E5-2667 v3 8C, 3.4GHz, 20MB Cache, 135W processors) - Intel Haswell architecture.
    • 64 GB DDR4 RAM @ 2133 MHz
    • Mellanox Connect-IB FDR port (56 Gb/s)

  • 48 direct water-cooled Lenovo nx360 M5 compute servers each equipped with:
    • 32 Cores (2x Intel Xeon E5-2697A v4 16C, 2.6Ghz. 40MB Cache, 145W)  - Intel Broadwell architecture.
    • 128 GB DDR4 RAM @ 2400 Mhz
    • Mellanox Connect-IB FDR port (56 Gb/s)

  • 6 air cooled compute servers each equipped with:
    • 16 cores (2x Intel Xeon E5-2667 v3 8C, 3.2GHz, 20MB, 2133MHz) 135W  - Intel Haswell architecture.
    • 256 GB DDR4 RAM @ 2133 MHz
    • Mellanox Connect-IB FDR port (56 Gb/s)
    • nVidia Kepler K40 accelerator (only on two nodes).

System Software

Acknowledgments

Authors are encouraged to acknowledge the funding agencies, if their papers rely on numerical experiments conducted on the high performance computer of the institute. An example text is provided here [log-in required]. 

Cluster Access

Accessing the cluster requires additional authorisation which need to be applied for via this form by research domain secretaries and membership in a project group which is already registered for hpc job accounting.

Please note, that hpc access is required only for submitting and controlling compute jobs.  Data stored on the hpc cluster can be accessed remotely and a hpc cluster access is not necessary.

The hpc cluster is available via interactive login via secure shell (ssh) command using publickey authentication to interactive login host(s): cluster.pik-potsdam.de. A simple set-up for Linux client computers is available here.  A more complex set-up for Microsoft Windows client computers is provided here.  [log-in required].

Cluster Documentation

The complete set of user documentation is available here .

Cluster Support

Questions and comments should be sent to: this e-mail address  [log-in required].


Cluster Utilization 

A set of cluster statistics by user / group, month and year is available here  [log-in required].

Cost of Computing and Storage Capacity

Estimates for the cost of computing per CPU/h and for storage capacity per TByte are provided here [log-in required].