Parallel I/O and Disk Striping in Computer Science - Prof. Edgar Gabriel, Study notes of Computer Science

Download Parallel I/O and Disk Striping in Computer Science - Prof. Edgar Gabriel and more Study notes Computer Science in PDF only on Docsity! 1 Edgar Gabriel COSC 6374 Parallel Computation Parallel I/O (I) – I/O basics Edgar Gabriel Spring 2009 COSC 6374 – Parallel Computation Edgar Gabriel Concept of a clusters Compute node message passing network administrative network M e m o ry Processor 1 Processor 2 N e tw o rk c a rd 1 N e tw o rk c a rd 2 local disks 2 COSC 6374 – Parallel Computation Edgar Gabriel I/O Problem (I) • Every node has its own local disk • Most applications require data and executable to be locally available – e.g. an MPI application using multiple nodes requires • executable to be available on all nodes • in the same directory • using the same name • Multiple processes need to access the same file – potentially different portions – efficiency COSC 6374 – Parallel Computation Edgar Gabriel Basic characteristics of storage devices • Capacity: amount of data a device can store • Transfer rate or bandwidth: amount of data at which a device can read/write in a certain amount of time • Access time or latency: delay before the first byte is moved Prefix Abbreviation Base ten Base two kilo, kibi K, Ki 10^3 2^10=1024 Mega, mebi M, Mi 10^6 2^20 Giga, gibi G, Gi 10^9 2^30 Tera, tebi T, Ti 10^12 2^40 Peta, pebi P, Pi 10^15 2^50 5 COSC 6374 – Parallel Computation Edgar Gabriel Improving Disk Bandwidth: disk striping • Utilize multiple hard drives • Split a file into constant chunks and distribute them across all disks • Three relevant parameters: – Stripe factor: number of disks – Stripe depth: size of each block – Which disk contains the first block of the file Disk 1 Disk 2 Disk 3 Disk 4 … Block 1 Block 2 Block 3 … Block n COSC 6374 – Parallel Computation Edgar Gabriel Disk striping • Ideal assumption b(N, p) = p * b(N, 1) with N: number of bytes to be written b: bandwidth p: number of disks • Realistically: b(N,p) < p * b(N,1) since – N is often not large enough to fully utilize p hard drives – networking overhead 6 COSC 6374 – Parallel Computation Edgar Gabriel Two levels of disk striping (I) • Using a RAID controller – Hardware – typically a ‘single box’ – number of disks: 3…n COSC 6374 – Parallel Computation Edgar Gabriel Redundant arrays of independent disks (RAID) • Goals: improve reliability and performance of an I/O system – improve performance of an I/O system • Several RAID levels defined • RAID 0: disk striping without redundant storage (“JBOD”= just a bunch of disks) – No fault tolerance – Good for high transfer rates • i.e. read/write bandwidth of a single large file – Good for high request rates • i.e. access time to many (small) files • RAID 1: mirroring – All data is replicated on two or more disks – Does not improve write performance and just moderately the read performance 7 COSC 6374 – Parallel Computation Edgar Gabriel RAID level 2 • RAID 2: Hamming codes – Each group of data bits has several check bits appended to it forming Hamming code words – Each bit of a Hamming code word is stored on a separate disk – Very high additional costs: e.g. up to 50% additional capacity required • Hardly used today since parity based codes faster and easier COSC 6374 – Parallel Computation Edgar Gabriel RAID level 3 • Parity based protection: – Based on exclusive OR (XOR) – Reversible – Example 01101010 (data byte 1) XOR 11001001 (data byte 2) -------------------------------------- 10100011 (parity byte) – Recovery 11001001 (data byte 2) XOR 10100011 (parity byte) --------------------------------------- 01101010 (recovered data byte 1) 10 COSC 6374 – Parallel Computation Edgar Gabriel RAID level 10 • Is RAID level 1 + RAID level 0 RAID 1 mirroring RAID 0 striping • Also available: RAID 53 (RAID 0 + RAID 3) COSC 6374 – Parallel Computation Edgar Gabriel Comparing RAID levels RAID level Protection Space usage Good at.. Poor at.. 0 None N Performance Data protect. 1 Mirroring 2N Data protect. Space effic. 2 Hamming codes ~1.5N Transfer rate Request rate 3 Parity N+1 Transfer rate Request rate 4 Parity N+1 Read req. rate Write perf. 5 Parity N+1 Request rate Transfer rate 6 P+Q or 2-D (N+2) or (MN+M+N) Data protect. Write perf. 10 Mirroring 2N Performance Space effic. 53 parity N+striping factor Performance Space effic. 11 COSC 6374 – Parallel Computation Edgar Gabriel Two levels of disk striping (II) • Using a parallel file system – exposes the individual units capable of handling data • often called storage servers, I/O nodes, etc. – each storage server might use multiple hard drives underneath the hood to increase its read/write bandwidth – Metadata server which keeps track of which parts of a file are on which storage server – Single disk failure less of a problem, if each server uses underneath the hood a RAID 5 storage system COSC 6374 – Parallel Computation Edgar Gabriel Compute nodes Meta-data server storage server 0 storage server 1 storage server 2 storage server 3 Parallel File Systems: Conceptual overview 12 COSC 6374 – Parallel Computation Edgar Gabriel File access on a parallel file system Compute node Metadata server Application calls write() OS requests list of relevant I/O nodes for this write operation MD server sends storage IDs, offsets etc. OS sends data to storage servers COSC 6374 – Parallel Computation Edgar Gabriel Disk striping • Requirements to improve performance of I/O operations using disk striping: – Multiple physical disks – Have to balance network bandwidth and I/O bandwidth • Problem of simple disk striping: – for a fixed file size, the number of disks which can be used in parallel is limited • Prominent parallel file systems – PVFS2 – Lustre – GPFS – NFS v4.2 (new standard currently being ratified)