CASAPerformance.pptx

Size: px

Start display at page:

Download "CASAPerformance.pptx"

失沈
4 years ago
Views:

1 Performance and Status Jeff Kern Group Lead James Robnett Science Computing Infrastructure Lead Atacama Large Millimeter/submillimeter Array Expanded Very Large Array Robert C. Byrd Green Bank Telescope Very Long Baseline Array

2 Performance Realized Performance is a function of several factors: Usage Patterns Data Access Rate (Disk I/O) Memory Utilization Processor Speed team is addressing bottlenecks: Repeated disk access (Scratchless Operation) Unified access patterns We have optimized for modern data from VLA and ALMA e.g. We assume the visibility data does not fit in system memory

3 Memory assumes that the visibility data is too large to hold in memory. True for modern ALMA and VLA data sets Implies that imaging is the dominant stage in terms of memory usage attempts to hold only the necessary information consistent with efficient operation in memory. For basic imaging (no A- or W-terms, single scale, single frequency term) the peak memory usage is approximately 24 bytes / pixel 1 Double Precision Complex Grid 1 Single Precision Complex Grid For spectral line this is per output channel Image Size 3k x 3k 6k x 6k 10k x 10k Memory Required 206 MB 864 MB 2.4 GB 3

4 Memory MS-MFS can be the limiting step in memory usage: Convolution Functions (in Images): ½(N S2 N T2 )(20S Max / N pix ) 2 Residual Images: N S N T Model Images: N T Cache Functions: Approximately 30% N S N T S Max / N pix Images 3k x 3k 10k x 10k % GB 18 GB % GB 20.8 GB 5 2 5% GB 35.6 GB 4

5 Data Access (Disk I/O) Although some stages of processing access only a fraction of the data, many operations require accessing the full visibility data For many operations the I/O access either in terms of absolute data rate or number of transactions is the limiting factor in performance. is working on improvements which decrease the data access requirements, and ensuring that reads are as efficient as possible. Peak Data Rate OS Realized Data Rate Time for 500 GB file SATA Disk 115 MB/s 60 MB/s 2h 22 m Raid MB/s 375 MB/s 22 m Lustre (10 GB) 1.2 GB/s 900 MB/s 9.5 m 5

6 Data Access (Disk I/O) Unlike many other aspects of the processing equation, time alone will not solve the data access problem Processing Characteristics vs Time EVLA Data Size System Memory Disk Volume CPU Performance 10 Disk I/O rate

7 Processor Speed Processor clock speeds are no longer dramatically increasing Industry has gone to more cores, rather than faster clocks Successive generations of processors will show modest increases in performance due to improved instruction pipelines etc. Only weakly affects observed performance of serial Currently is more limited on data access and memory footprint than on processor speed. We are currently trading increased computation load for decreased memory and data access. 7

8 Processor Speed Continuum and Cube Imaging time vs. Processor model Runtime Minutes E5645 (6) 2.4Ghz E (6) 2.4Ghz E (6) Supermicro Continuum Imagine time Procesor Model E (8) 2.3Ghz E (6) 2.5Ghz E (8) 2.4Ghz E (6) 2.9Ghz Cube Imaging Time Continuum Imaging Cost effeciency Cube Imaging Cost efficiency E (8) 2.6Ghz E (8) 2.9Ghz Cost Efficiency 8

9 So What Should I Buy? Well it depends: What type of data do you expect? How will you process it? How long are you willing to wait for processes to complete? How much do you want to spend? How much IT support do you have? will continue to evolve, we are continually working to bring algorithms to the community which optimize our hardware utilization. This means the correct answer will change over time as we respond to changes in algorithms and hardware ecosystem. 9

10 That doesn t help! Update pending Coming in 2015 General Guidelines Do not neglect I/O Multi Disk RAID is probably the best option for small groups. Large Groups w/ IT support might consider High Performance File Systems Consider your memory requirements. 16 cores doesn t help if you can only run 1 engine because of memory. Where we put our Money Dual Intel E (2.6GHz) 8 core processors 64GB of memory 40Gbit/s Infiniband NIC 600 TB Lustre System 12 GB/s 336 disks $200K Total Cost per Node: $6K Not Including Storage 10

11 Parallel Many stages of process are embarrassingly parallel. has the tools to support parallel processing GB 3c147 EVLA B-Array flagging/calibration runtime vs parallelization breadth Partition Manual Flag M inu 1500 Auto Flag t e s Applycal Amp+Phase Gaincal BandPass Phase Gaincal 0 1 engine 2 engines 4 engines 8 engines 12 engines 2 nodes x 8 engines 2 nodes x 12 engines Setjy 11

12 Parallel The technique is to create a Multi-MS (MMS): Eventually this will be completely transparent to Users Sub MS 1 Original MS MS Selection Sub MS 2 Sub MS 3 Reference MS MS Reference Sub MS 4 Sub MS 5 Supported by mstransform and mstransform based tasks 12

13 Parallel Most time consuming tasks are now MMS aware and will automatically parallelize where possible. By default we use 80% of the available cores on the current host Usage can be configured using cluster spec file. Beginning to parallelize the pipelines Ensure standard path works Has been work in progress for several years now. Our focus is on delivering this capability to end users, that is making it transparent enough for general use. 13

14 Parallel : Imaging Parallel Imaging is implemented in the task tclean Massive re-architecting of Clean task to support uniform treatment of parallel and non-parallel tasks. Continuum Technique is to segment the data, gridding each portion separately and then combine for the minor cycle. Recall that this is the memory intensive portion If you don t have enough memory, you can thread the gridding up to 4 ways (1,2, or 4 threads are supported) Not quite as efficient, but less memory intensive Spectral Line Segment data on output channel, fully parallel imaging Stitch cube back together at the end (can be expensive) 14

15 Current Testing Testing by team 106 Pointing Mosaic AW-Projection MS-MFS 2 Terms 3 Scales 64 Cores 4 Nodes 16 cores Image Courtesy Rao and Bhatnagar Major Cycle 5 hours Opposed to 10 days 15

16 Parallel : Continuum Imaging Continuum Imaging, 3K x 3k Image, 5K iterations Minutes Run time engine 2engines 4 engines 8 engines 12 engines 2 nodes x 12 engine 4 nodes x 12 engine 4 nodes x 12 engine 1K iter 4 nodes x 12 engine 10K iter 16

17 Parallel : Spectral Imaging Spectral Line Imaging, 3K x 3K image x 1K channels, 5000 iterations Minutes Effeciency Imaging Cube creation Effeciency node x 1 engine 1 node x 2 engines 1 node x 4 engines 2 nodes x 4 engines 4 nodes x 4 engines 0 17

18 The National Radio Astronomy Observatory is a facility of the National Science Foundation operated under cooperative agreement by Associated Universities, Inc. science.nrao.edu Review March

VASP应用运行优化

VASP应用运行优化 1 VASP wszhang@ustc.edu.cn April 8, 2018 Contents 1 2 2 2 3 2 4 2 4.1........................................................ 2 4.2..................................................... 3 5 4 5.1..........................................................