Basic Communication Operations 2-Parallel Processing-Lecture Slides, Slides of Parallel Computing and Programming

Download Basic Communication Operations 2-Parallel Processing-Lecture Slides and more Slides Parallel Computing and Programming in PDF only on Docsity! Basic Communication Operations docsity.com Communication Patterns in Different Topologies • One-to-All Broadcast and All-to-One Reduction • All-to-All Broadcast and Reduction & All-Reduce Operations • Scatter and Gather (one to all & all to one personalized comm) docsity.com All-to-All Broadcast & Reduction on a Ring • Simplest approach: perform p one-to-all broadcasts This is not the most efficient way, though. • Each node first sends to one of its neighbors the data it needs to broadcast. • In subsequent steps, it forwards the data received from one of its neighbors to its other neighbor. • The algorithm terminates in p-1 steps. docsity.com All-to-all Broadcast on a Torus • Performed in two phases - in the first phase, each row of the mesh performs an all-to-all broadcast using the procedure for the linear array. • In this phase, all nodes collect √p messages corresponding to the √p nodes of their respective rows. Each node consolidates this information into a single message of size m√p. • The second communication phase is a columnwise all-to-all broadcast of the consolidated messages. docsity.com All-to-all broadcast on a Hypercube • Generalization of the mesh algorithm to log p dimensions. • Message size doubles at each of the log p steps. docsity.com Scatter and Gather Operations M p-| Scatter M | 80-8 <M 00-8 Scatter and gather operations. ® docsity.com Scatter and Gather • In the scatter operation, a single node sends a unique message of size m to every other node (also called a one-to-all personalized communication). • In the gather operation, a single node collects a unique message from each node. • While the scatter operation is fundamentally different from broadcast, the algorithmic structure is similar, except for differences in message sizes (messages get smaller in scatter and stay constant in broadcast). • The gather operation is exactly the inverse of the scatter operation and can be executed as such. docsity.com Scatter/Gather comm time analysis on different topologies • Scatter – Linear array – 2-D Mesh – Hypercube – Tree • Gather – Linear Array – 2-D Mesh – Hypercube – Tree docsity.com