Analysis of Topology-Dependent MPI Performance on Gemini Networks

Analysis of Topology-Dependent MPI Performance on Gemini Networks

Analysis of Topology-Dependent MPI Performance on Gemini Networks Antonio J. Pea, Ralf G. Correa Carvalho, James Dinan, Pavan Balaji, Rajeev Thakur, and William Gropp Motivation Network properties can have a significant impact on application performance BW uses a 3-dimensional Cray Gemini torus featuring anisotropic properties Twice the Y-dimension bandwidth in the X and Z dimensions A Gemini ASIC is shared by two nodes

Task placement considering these properties is highly beneficial EuroMPI 2013 - Madrid (Spain) - 15-18 September 2013 2 Outline Background Contributions System: Job Placement and Rank Ordering in BW Network Layout Experimental Evaluation

Basic Micro-benchmarks Collective Communications Stencil Communications Conclusions EuroMPI 2013 - Madrid (Spain) - 15-18 September 2013 3 Contributions EuroMPI 2013 - Madrid (Spain) - 15-18 September 2013 4 Contributions Characterization of the Gemini anisotropic behavior based in

point-to-point micro-benchmarks Prove Y-wise placement of the dual nodes per network Cartesian point to be highly beneficial Demonstrate potential gains of MPI-network topology matching versus the available node placement EuroMPI 2013 - Madrid (Spain) - 15-18 September 2013 5 System: Job Placement / Rank Ordering Network Layout EuroMPI 2013 - Madrid (Spain) - 15-18 September 2013 6 Job Placement and Rank Ordering in BW

Cray MPICH follows the node ordering assigned by the job scheduler Ranks are ordered in a zigzag fashion First and last ranks are adjacent Decrease hop count Increase bisection bandwidth Given that: XE6 routers contain two nodes Z links are faster than X links

Every 5th link is crossing a cabinet (slower) 4 x 2 x 8 building blocks Carl Albing, Norm Troullier, Stephen Whalen, Ryan Olson, Joe Glenski, Howard Pritchard, and Hugo Mills. Scalable node allocation for improved performance in regular and anisotropic 3D torus supercomputers. In Recent Advances in the Message Passing Interface, volume 6960 of LNCS, 2011. EuroMPI 2013 - Madrid (Spain) - 15-18 September 2013 7 Blue Waters Network Layout X dimension im d Z

n io s en EuroMPI 2013 - Madrid (Spain) - 15-18 September 2013 Y dimension Blue Waters Network Layout X dimension Y dimension

EuroMPI 2013 - Madrid (Spain) - 15-18 September 2013 Basic Micro-benchmarks Experimental Evaluation EuroMPI 2013 - Madrid (Spain) - 15-18 September 2013 10 Point-to-Point Communication (single process) Point-to-point benchmarking

Anisotropic behavior illustrated Communications in the Y direction perform signifficantly lower: links Unidirectional Z links offer much higher TR than Y X and Z: largely different behaviors Latency per hop: ~0.1s Bidirectional

EuroMPI 2013 - Madrid (Spain) - 15-18 September 2013 11 Point-to-Point Communication (multiple processes) Internode aggregate transfer rate 2 parallel paths transfer concurrently Optimal node ordering and matching

between MPI ranks and network topo. Collectives saturating links greatly improve performance on Y direction Contiguous nodes in these experiments Double X and Z links become shared Aggregate TR increases for Y Placement of dual nodes/ASIC along Y Extra performance improvement EuroMPI 2013 - Madrid (Spain) - 15-18 September 2013 12 Blue Waters Network Layout

X dimension im d Z n io s en EuroMPI 2013 - Madrid (Spain) - 15-18 September 2013 Y dimension Collective Communications

Topology matching exploited by row-wise and plane-wise collectives Y direction faster! Row-wise: up to 74% (alltoall) and 54% (allgather) Row-wise MPI_Alltoall Row-wise MPI_Allgather EuroMPI 2013 - Madrid (Spain) - 15-18 September 2013 14 Collective Communications Topology matching exploited by row-wise and plane-wise collectives Y direction faster!

Row-wise: up to 74% (alltoall) and 54% (allgather) Plane-wise: up to 59% (alltoall) and 53% (allgather) Plane-wise MPI_Alltoall Plane-wise MPI_Allgather EuroMPI 2013 - Madrid (Spain) - 15-18 September 2013 15 Stencil Communications Cray MPICH ignores the reorder

parameter in MPI_Cart_create MPI topo. not matching network 2D & 3D halo exchange (contig. nodes): Plain: Manual ordering X-Y-Z Cart_create: Y-major / Z-Y-X Custom: MPI-network matching 2D

2D: Cart_create worst performance Plain up to 1.4%; Cart_create 4% 3D: Topology matching outperforms MPI-assisted sorting up to 5% Topology matching favors scalability 3D EuroMPI 2013 - Madrid (Spain) - 15-18 September 2013 16 Conclusions EuroMPI 2013 - Madrid (Spain) - 15-18 September 2013

17 Conclusions Studied the anisotropic implications of Cray Gemini networks on MPI comms. Characterized this network be means of point-to-point micro-benchmarks Studied the behavior of MPI collectives along the different dimensions / planes

Considering the nodes sharing a network Cartesian coordinate along the Y dimension is highly beneficial, maximizing the use of the network resources Including awareness of the network topology in the MPI library outperforms the available heuristic-based rank ordering Future work: non-contiguous allocations employing existing mapping libraries EuroMPI 2013 - Madrid (Spain) - 15-18 September 2013 18

Thank you! EuroMPI 2013 - Madrid (Spain) - 15-18 September 2013 19

Recently Viewed Presentations

  • Transplantation and Rejection - kau

    Transplantation and Rejection - kau

    Transplantation and Rejection Studying the immunology of transplantation and rejection is important because: Its impact on understanding of immunological practice Its applications in the development of clinical transplantation IT has led to: Discovery of MHC molecules Better understanding of T...
  • Michigan Health Information Network

    Michigan Health Information Network

    Sparrow Care Network. The Physician Alliance. United Physicians. ... Invitation to Tim Pletcher, Executive Director of MiHIN to March 2015 meeting. Discussed several areas for physician influence. Formed Physician-Payer Quality Collaborative (PPQC)
  • 90 Day Maintenance Drug Retail Option CURRENT 2015

    90 Day Maintenance Drug Retail Option CURRENT 2015

    Pay Bands are adjusted annually, based on the change in California consumer price index from February of prior year to February of current year (determined by the California Department of Finance). The index used is that for urban wage earners...
  • Intro to Aural Rehabilitation - Augustana University

    Intro to Aural Rehabilitation - Augustana University

    Mild HL Linked to Brain Atrophy in Older Adults Early intervention could prevent slide toward speech comprehension difficulties Research suggests that hearing sensitivity has cascading consequences for the neural processes supporting both perception and cognition Definitions Auditory Training Aural Rehabilitation...
  • Bergamot Citrus bergamia dTERRA University dTERRA Product Tools

    Bergamot Citrus bergamia dTERRA University dTERRA Product Tools

    dōTERRA® Product Tools. dōTERRA® Product Tools. dōTERRA® University. Can be applied undiluted . Internal Usage Suggestions - Change regular tea to Earl Grey with the addition of Bergamot
  • Valuegenesis2 Ten Years Later: A Study of Two

    Valuegenesis2 Ten Years Later: A Study of Two

    Hamlet Canosa Vice President for Education Columbia Union Conference ... 22 saw their school as dull. In a similar "bipolar" question, 64% versus 16% saw schools as warm rather than cold. What about our Schools? In a query of Christian...
  • ES&PC Plan Review Checklist - gaswcc.georgia.gov

    ES&PC Plan Review Checklist - gaswcc.georgia.gov

    "Non-exempt activities shall not be conducted within the 25 or 50-foot undisturbed stream buffers as measured from the point of wrested vegetation without first acquiring the necessary variances and permits." See Part IV.
  • "Jesus Walking On The Water" and "The Rich Fool"

    "Jesus Walking On The Water" and "The Rich Fool"

    Contributed in Love by Brent and Di Kington, Becky Gross, Jane Evers, Earline Elkins, Kathy Whitaker, Glenda Knight, Connie Beisterfield, Lois Harker, Delta McArthy and more! PowerPoint Creation: Brad Taylor, Terry Taylor, James & Stacy Stapleton. Visit eBibleTeacher.com For more...