Slide 1

Size: px

Start display at page:

Download "Slide 1"

事禄
5 years ago
Views:

1 HPC 最佳实例 Tools, Compilers, and profiling 刘通 2010 年 10 月国际高性能计算咨询委员会中国区总监

2 Compilers and Libraries 免费编译器 GNU Compiler Open64 付费编译器 PGI Intel 數學函数庫 ACML Intel MKL MPI 库 Open MPI MVAPICH Platform MPI Intel MPI ParaStation MPI 2

3 系统基本测试 BIOS 设置 CPU/Memory 网络 Driver/Firmware 底层性能测试 MPI 性能点对点以及集群通信性能 Performance counters ( 性能计数器 ) File system ( 文件存储系统 ) Disk performance ( 硬盘的性能 ) Shared file system performance ( 共享文件系统的性能 ) 3

4 Profiling Tools ( 性能分析工具 ) Platform MPI (previously HP-MPI) -i file User time: 58.22% MPI time : 41.78% [Overhead:41.74% Blocking:0.04%] Routine Summary by Rank: Rank Routine Statistic Calls Overhead(ms) Blocking(ms) MPI_Allgather Open MPI IPM (LD_PRELOAD=~/ipm/lib/libipm.so ) Intel MPI # [time] [calls] <%mpi> <%wall> # MPI_Allreduce e # MPI_Waitall e genv I_MPI_STATS=10 Platform MPI (previously Scali MPI) SCAMPI_TRACE="-f arg;timing /opt/scali/bin/scanalyze -m trace tracefile.txt 4

5 NWChem NWChem 是量子化学计算软件美国太平洋西北国家实验室的环境分子科学研究室开发 NWChem 为分子学计算提供了多种方法 NWChem 能进行传统的分子动力学以及自由能模拟 5

6 NWChem 最佳实例编译环境设置 export ARMCI_NETWORK=OPENIB export MPI_LOC=/usr/mpi/gcc/openmpi export LIBMPI="-lmpi" export BLAS_LIB="-L/acml4.3.0/ifort64/lib -lacml -lacml_mv " export BLASOPT="-L/acml4.3.0/ifort64/lib -lacml -lacml_mv 运行工作路径应该是本地硬盘, 最好是并行文件存储系统 6

7 NWChem 性能测试结果 Input Dataset - H2O7 ACML 提供了更好的性能及可扩展性 300 NWChem Benchmark Result (H2O7 MP2) Wall time (Seconds) Lower is better Number of Nodes MVAPICH HP-MPI Open MPI MVAPICH + ACML HP-MPI + ACML OpenMPI + ACML InfiniBand DDR 7

8 NWChem 性能测试结果 Input Dataset - H2O7 InfiniBand 提供了更好的性能及可扩展性速度是以太网 2.36 倍 1800 NWChem Benchmark Result (H2O7 MP2) Performance (Jobs/day) Number of Nodes 136% Higher is better Ethernet InfiniBand DDR Open MPI + ACML 8

9 OpenAtom OpenAtom 是并行量子化学软件采用 the Car-Parrinello ab-initio Molecular Dynamics 方法解决 : Material science Chemistry Solid-state physics Biophysics 基于 Charm++ 并行编程开发框架 OpenAtom 是由美国 UIUC 大徐开发的开源软件 9

10 OpenAtom 性能分析通讯负载主要应用的 MPI 程式 MPI_Iprobe, MPI_Recv, MPI_Test, and MPI_Isend 点对点通信传输产生最大负载 16 Processes 32 Processes 64 Processes 128 Processes 10

11 OpenAtom 性能分析 MPI 消息大小大部分消息都在 1KB-16KB 128 Processes 11

12 OpenAtom 性能测试结果 - MPI 优化的 MPI 运行参数可以加速程序运行速度 Up to 7% higher performance with Open MPI --mca mpi_affinity_alone 1 --mca btl_openib_eager_rdma_threshold 8 Up to 15% higher performance with MVAPICH2 MV2_USE_RDMA_FAST_PATH=0 MV2_USE_RDMA_ONE_SIDED=0 7% 15% Higher is better 8-cores per node 12

13 OpenAtom 性能测试结果 - ibverbs 直接使用 ibverbs 能更好提供运行速度及可扩展性 Up 29% higher performance versus tuned Open MPI Performance advantage increases as cluster size scales 29% Higher is better 8-cores per node 13

14 OpenAtom 性能测试结果 - Interconnect InfiniBand 提供了更快的系统性能比万兆以太网快 1.74 倍, 比千兆以太网快 5.71 倍随着系统规模的增加,InfiniBand 能保持程序的可扩展性 571% 174% Higher is better 8-cores per node 14

NAMD A parallel molecular dynamics code that received the 2002 Gordon Bell

systems Scales to hundreds of processors and millions of atoms Developed by

Group (TCB) and the Parallel Programming Laboratory (PPL) at the University

15 NAMD A parallel molecular dynamics code that received the 2002 Gordon Bell Award Designed for high-performance simulation of large biomolecular systems Scales to hundreds of processors and millions of atoms Developed by the joint collaboration of the Theoretical and Computational Biophysics Group (TCB) and the Parallel Programming Laboratory (PPL) at the University of Illinois at Urbana-Champaign NAMD is distributed free of charge with source code 15

16 NAMD Performance Result Interconnect InfiniBand enables higher scalability Up to 312% higher performance than Ethernet at 14 nodes Four InfiniBand connected servers deliver higher performance vs 14 Ethernet connected servers 312% Higher is better 12 Cores/Node 16

17 NAMD Performance Result Interconnect I/F NAMD can use MPI or the native InfiniBand interface (verbs) as the interface to the interconnect (InfiniBand) IB verbs version provides better productivity versus MPI 10% improvement over 14 nodes IB verbs provides a lower level interface to the interconnect versus MPI lower overhead 10% Higher is better 12 Cores/Node 17

PARATEC PARATEC stands for PARAllel Total Energy Code Performs ab-initio quantum-mechanical total energy calculations using pseudopotentials and a plane wave basis set Designed to run on

18 PARATEC PARATEC stands for PARAllel Total Energy Code Performs ab-initio quantum-mechanical total energy calculations using pseudopotentials and a plane wave basis set Designed to run on massively parallel computing platforms and clusters Developed through a joint collaboration between LBNL Université Pierre et Marie CURIE University of Montreal University of Cambridge 18

19 PARATEC 性能分析运行时间分布随着系统大小的增加, 通信时间所占的比例持续增加 19

20 PARATEC 性能分析 MPI Functions Mostly used MPI functions MPI_Wait, MPI_Allreduce, and MPI_Bcast are the mostly used MPI functions MPI_Allreduce 负载比例持续增加 32P 64P 128P 20

21 PARATEC 性能分析消息大小 Messages with big communication overhead are Large messages >1MB Small message <256Bytes 128 Processes 21

22 PARATEC 性能测试结果优化的 MPI 运行参数提供更高程序性能 Up to 33% higher performance with customized MPI_Gather, barrier, and XRC parameter --mca btl_openib_receive_queues X,9216,256,128,32:X,65536,256,128,32 --mca coll_tuned_use_dynamic_rules 1 --mca coll_tuned_gather_algorithm 1 --mca coll_tuned_barrier_algorithm 3 33% Higher is better 8-cores per node 22

23 PARATEC 性能测试结果 Open MPI with optimization enables higher performance Up to 22% higher performance than Intel MPI 22% Higher is better 8-cores per node 23

24 PARATEC 性能测试结果 InfiniBand enables better application performance and scalability Up to 69% higher performance than 10GigE and 100% than GigE 16-node cluster Application performance over InfiniBand scales as cluster size increases 100% 69% Higher is better 8-cores per node 24

assembling larger genomes Implemented using MPI ABySS

25 ABySS ABySS is a de novo, parallel, paired-end sequence assembler designed for short reads Capable of assembling larger genomes Implemented using MPI ABySS was developed at Canada's Michael Smith Genome Sciences Centre 25

26 ABySS Benchmark Results - Compilers Two different compilers were used to compile ABySS Open64 provides better performance than GCC at 128 cores 13% Higher is better 8-cores per node 26

27 ABySS Benchmark Results - Interconnect InfiniBand enables higher performance and scalability Up to 88% higher performance than GigE and 81% higher than 10GigE Both GigE and 10GigE don t scale well beyond 8 nodes 88% 81% Higher is better 8-cores per node 27

for 16-node cluster As cluster size increases, more power can be saved $5700 $/KWh = KWh *

28 Power Cost Savings with Different Interconnect To achieve same number of ABySS jobs over GigE InfiniBand saves power up to $5700 versus GigE and $5300 versus 10GigE Yearly based for 16-node cluster As cluster size increases, more power can be saved $5700 $/KWh = KWh * $0.20 For more information

29 ABySS 性能测试总结 Interconnect comparison shows InfiniBand delivers superior performance in every cluster size versus GigE and 10GigE Performance advantage extends as cluster size increases Open64 compiler delivers higher performance InfiniBand enables power saving Up to $5700/year power savings versus GigE and $5300 versus 10GigE on16 node cluster Maximum return on investment through efficiency and utilization 29

30 谢谢国际高性能计算咨询委员会

VASP应用运行优化

VASP应用运行优化 1 VASP wszhang@ustc.edu.cn April 8, 2018 Contents 1 2 2 2 3 2 4 2 4.1........................................................ 2 4.2..................................................... 3 5 4 5.1..........................................................