Parallelism on Rik Kisnah - Bloghttps://www.rik-kisnah.ai/tags/parallelism/Recent content in Parallelism on Rik Kisnah - BlogHugoenTue, 15 Sep 2020 09:00:00 -0700Data, Model, and Pipeline Parallelismhttps://www.rik-kisnah.ai/teach/gpu-ai/data-model-and-pipeline-parallelism/Tue, 15 Sep 2020 09:00:00 -0700https://www.rik-kisnah.ai/teach/gpu-ai/data-model-and-pipeline-parallelism/Three ways to split a job that is too big for one GPU. Split the data, split the model’s layers across machines, or split each layer across machines. Each one puts a different kind of traffic on the network, and that decides your cluster design.