{"type": "excalidraw", "version": 2, "source": "generated", "elements": [{"id": "laggnktbk19x4b11vs8d", "type": "frame", "x": 390, "y": 10, "width": 2470, "height": 1543.75, "angle": 0, "strokeColor": "#bbb", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 1, "strokeStyle": "solid", "roughness": 0, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0000", "roundness": null, "seed": 2017504764, "version": 1, "versionNonce": 1238785980, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false, "name": "System diagram"}, {"id": "ujOeHwdFcAefAZhnM6Jy", "type": "text", "x": -900, "y": 40, "width": 278.85, "height": 97.5, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0001", "roundness": null, "seed": 949539217, "version": 1, "versionNonce": 200071089, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false, "text": "REQUIREMENTS (ask 5, then move)\nPretrain, fine-tune or RL \u2014 one fleet?\nTotal GPUs, and the largest single job?\nOne region, or may a job span sites?\nWho submits, and the priority order?\nDone = a registered model version?", "fontSize": 13, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "REQUIREMENTS (ask 5, then move)\nPretrain, fine-tune or RL \u2014 one fleet?\nTotal GPUs, and the largest single job?\nOne region, or may a job span sites?\nWho submits, and the priority order?\nDone = a registered model version?", "autoResize": true, "lineHeight": 1.25}, {"id": "IizHJIQgJlgSIMEGWC5w", "type": "text", "x": -900, "y": 171.5, "width": 286.0, "height": 113.75, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0002", "roundness": null, "seed": 1066984056, "version": 1, "versionNonce": 772092315, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false, "text": "A USER CAN\nSubmit a job: base model, GPU count,\ndataset, priority.\nStart only when every GPU is ready.\nWatch step, loss, throughput, failures.\nGet the trained model back, versioned.\nCancel and resume. Run many in parallel.", "fontSize": 13, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "A USER CAN\nSubmit a job: base model, GPU count,\ndataset, priority.\nStart only when every GPU is ready.\nWatch step, loss, throughput, failures.\nGet the trained model back, versioned.\nCancel and resume. Run many in parallel.", "autoResize": true, "lineHeight": 1.25}, {"id": "RpJG2Tr8hzUjEcPVJ2tR", "type": "text", "x": -900, "y": 319.25, "width": 314.6, "height": 130.0, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0003", "roundness": null, "seed": 2133201996, "version": 1, "versionNonce": 1959386987, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false, "text": "TECHNICAL CHALLENGES\nA job is a GANG: partial start is worthless.\nOne dead GPU kills the job, so effective\nMTBF falls linearly with job size.\nCheckpoint needs a barrier across ranks.\nFabric decides placement: cross-island\nall-reduce is taxed every step.\nIdle GPU = money burned.", "fontSize": 13, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "TECHNICAL CHALLENGES\nA job is a GANG: partial start is worthless.\nOne dead GPU kills the job, so effective\nMTBF falls linearly with job size.\nCheckpoint needs a barrier across ranks.\nFabric decides placement: cross-island\nall-reduce is taxed every step.\nIdle GPU = money burned.", "autoResize": true, "lineHeight": 1.25}, {"id": "efqReTtJQBS2v6wLEnrT", "type": "text", "x": -900, "y": 483.25, "width": 336.05, "height": 130.0, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0004", "roundness": null, "seed": 1708957521, "version": 1, "versionNonce": 1679116189, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false, "text": "CORE ENTITIES\nTeam \u00b7 Job(gpus, dataset, priority, generation)\nAllocation(nodes, ranks, island, generation)\nNode(8 GPUs, pool, resident base models)\nCheckpoint(epoch, per-rank manifest, committed)\nDataset(shards, seed, cursor) \u00b7 Quota\nHealthEvent(node, gpu, class, ts)\nNOT entities: GPU alone, epoch, the weights", "fontSize": 13, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "CORE ENTITIES\nTeam \u00b7 Job(gpus, dataset, priority, generation)\nAllocation(nodes, ranks, island, generation)\nNode(8 GPUs, pool, resident base models)\nCheckpoint(epoch, per-rank manifest, committed)\nDataset(shards, seed, cursor) \u00b7 Quota\nHealthEvent(node, gpu, class, ts)\nNOT entities: GPU alone, epoch, the weights", "autoResize": true, "lineHeight": 1.25}, {"id": "4EkyriAIR9QxofjOaZlr", "type": "text", "x": -900, "y": 647.25, "width": 321.75, "height": 162.5, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0005", "roundness": null, "seed": 625675343, "version": 1, "versionNonce": 1799361520, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false, "text": "API\nPOST /jobs {base_model, gpus, dataset,\n  epochs, priority} -> job_id   (idempotent)\nGET /jobs/{id} -> state, epoch, step,\n  last committed checkpoint, restarts\nPOST /jobs/{id}/cancel | /resume\nGET /models/{name}/versions  <- deliverable\nInternal: /nodes/{id}/health \u00b7\n  /allocations/{id}/report {rank, generation}\n  POST /checkpoints  (commits the manifest)", "fontSize": 13, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "API\nPOST /jobs {base_model, gpus, dataset,\n  epochs, priority} -> job_id   (idempotent)\nGET /jobs/{id} -> state, epoch, step,\n  last committed checkpoint, restarts\nPOST /jobs/{id}/cancel | /resume\nGET /models/{name}/versions  <- deliverable\nInternal: /nodes/{id}/health \u00b7\n  /allocations/{id}/report {rank, generation}\n  POST /checkpoints  (commits the manifest)", "autoResize": true, "lineHeight": 1.25}, {"id": "HLth16OTC27QIyyDyl9B", "type": "text", "x": -900, "y": 843.75, "width": 278.85, "height": 130.0, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0006", "roundness": null, "seed": 472138490, "version": 1, "versionNonce": 1460519318, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false, "text": "WHAT BREAKS FIRST\nA 4,000-GPU job on 500 nodes at 6-month\nnode MTBF fails every ~9 HOURS.\nA 3-week run is interrupted ~50 times.\nRestart cost, not peak throughput,\nis the design driver.\nFail-slow is worse than fail-stop:\none straggler slows all 4,000 ranks.", "fontSize": 13, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "WHAT BREAKS FIRST\nA 4,000-GPU job on 500 nodes at 6-month\nnode MTBF fails every ~9 HOURS.\nA 3-week run is interrupted ~50 times.\nRestart cost, not peak throughput,\nis the design driver.\nFail-slow is worse than fail-stop:\none straggler slows all 4,000 ranks.", "autoResize": true, "lineHeight": 1.25}, {"id": "LgJH6MemxN7LDhE8DtiU", "type": "rectangle", "x": 420, "y": 40, "width": 240, "height": 84, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "#f3f4f6", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0007", "roundness": {"type": 3}, "seed": 1137122203, "version": 1, "versionNonce": 2055665571, "isDeleted": false, "boundElements": [{"type": "text", "id": "Zkb6GjHbG81RqwkVHWun"}, {"type": "arrow", "id": "R9zAwfwuCRxGs7g4o2fq"}], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false}, {"id": "Zkb6GjHbG81RqwkVHWun", "type": "text", "x": 430, "y": 63.25, "width": 220, "height": 37.5, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0008", "roundness": null, "seed": 838145800, "version": 1, "versionNonce": 1028162214, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false, "text": "Researcher\nsubmits a job", "fontSize": 15, "baseFontSize": null, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "LgJH6MemxN7LDhE8DtiU", "originalText": "Researcher\nsubmits a job", "autoResize": true, "lineHeight": 1.25}, {"id": "YToGwbbrqQ7B697wnomm", "type": "rectangle", "x": 900, "y": 40, "width": 240, "height": 84, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "#dbeafe", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0009", "roundness": {"type": 3}, "seed": 8196149, "version": 1, "versionNonce": 2059322682, "isDeleted": false, "boundElements": [{"type": "text", "id": "4vNZhySm3ANf6SCUfj9b"}, {"type": "arrow", "id": "R9zAwfwuCRxGs7g4o2fq"}, {"type": "arrow", "id": "CvI5qiGogkdmtsVrFlvb"}], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false}, {"id": "4vNZhySm3ANf6SCUfj9b", "type": "text", "x": 910, "y": 63.25, "width": 220, "height": 37.5, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0010", "roundness": null, "seed": 1998679711, "version": 1, "versionNonce": 627813882, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false, "text": "API gateway\nauth \u00b7 idempotency key", "fontSize": 15, "baseFontSize": null, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "YToGwbbrqQ7B697wnomm", "originalText": "API gateway\nauth \u00b7 idempotency key", "autoResize": true, "lineHeight": 1.25}, {"id": "LKD6jHbXTgUi9m2bnFVu", "type": "rectangle", "x": 1380, "y": 40, "width": 300, "height": 84, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "#dbeafe", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0011", "roundness": {"type": 3}, "seed": 1799645191, "version": 1, "versionNonce": 562957180, "isDeleted": false, "boundElements": [{"type": "text", "id": "dT3PYGZ2ijF2WLWjiMhd"}, {"type": "arrow", "id": "CvI5qiGogkdmtsVrFlvb"}, {"type": "arrow", "id": "ZaM2AdPxF8LQclqawu9u"}, {"type": "arrow", "id": "TmT8ExDQVMNejpUsJadq"}], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false}, {"id": "dT3PYGZ2ijF2WLWjiMhd", "type": "text", "x": 1390, "y": 53.875, "width": 280, "height": 56.25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0012", "roundness": null, "seed": 2072265938, "version": 1, "versionNonce": 455736474, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false, "text": "Job service\n+ quota service\nadmit or reject by team", "fontSize": 15, "baseFontSize": null, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "LKD6jHbXTgUi9m2bnFVu", "originalText": "Job service\n+ quota service\nadmit or reject by team", "autoResize": true, "lineHeight": 1.25}, {"id": "2dlcgCb3du8LmrFYFpG2", "type": "rectangle", "x": 1860, "y": 40, "width": 300, "height": 110, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "#fef9c3", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0013", "roundness": {"type": 3}, "seed": 1114940396, "version": 1, "versionNonce": 870111104, "isDeleted": false, "boundElements": [{"type": "text", "id": "0ihBeoePW3jSOi27n7yE"}, {"type": "arrow", "id": "ZaM2AdPxF8LQclqawu9u"}, {"type": "arrow", "id": "6maTpey1e5U0rdPNj8Bt"}], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false}, {"id": "0ihBeoePW3jSOi27n7yE", "type": "text", "x": 1870, "y": 60.0, "width": 280, "height": 70.0, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0014", "roundness": null, "seed": 960836460, "version": 1, "versionNonce": 693491441, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false, "text": "Priority queue\nabsorbs bursts;\nages so big jobs\nnever starve", "fontSize": 14, "baseFontSize": null, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "2dlcgCb3du8LmrFYFpG2", "originalText": "Priority queue\nabsorbs bursts;\nages so big jobs\nnever starve", "autoResize": true, "lineHeight": 1.25}, {"id": "R9zAwfwuCRxGs7g4o2fq", "type": "arrow", "x": 660, "y": 82.0, "width": 240, "height": 1, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0015", "roundness": null, "seed": 779757290, "version": 1, "versionNonce": 1161557138, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false, "points": [[0, 0], [240, 0.0]], "startBinding": {"elementId": "LgJH6MemxN7LDhE8DtiU", "mode": "orbit", "fixedPoint": [1, 0.5]}, "endBinding": {"elementId": "YToGwbbrqQ7B697wnomm", "mode": "orbit", "fixedPoint": [0, 0.5]}, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false}, {"id": "UY0P6zHFEurXl3qbfqLn", "type": "text", "x": 680, "y": 56.0, "width": 85.80000000000001, "height": 16.25, "angle": 0, "strokeColor": "#374151", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0016", "roundness": null, "seed": 1135808359, "version": 1, "versionNonce": 522601131, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false, "text": "1 POST /jobs", "fontSize": 13, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "1 POST /jobs", "autoResize": true, "lineHeight": 1.25}, {"id": "CvI5qiGogkdmtsVrFlvb", "type": "arrow", "x": 1140, "y": 82.0, "width": 240, "height": 1, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0017", "roundness": null, "seed": 1075669244, "version": 1, "versionNonce": 158696257, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false, "points": [[0, 0], [240, 0.0]], "startBinding": {"elementId": "YToGwbbrqQ7B697wnomm", "mode": "orbit", "fixedPoint": [1, 0.5]}, "endBinding": {"elementId": "LKD6jHbXTgUi9m2bnFVu", "mode": "orbit", "fixedPoint": [0, 0.5]}, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false}, {"id": "aTIlD5gYAEZyFQ8vZRNz", "type": "text", "x": 1160, "y": 56.0, "width": 50.050000000000004, "height": 16.25, "angle": 0, "strokeColor": "#374151", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0018", "roundness": null, "seed": 1492725227, "version": 1, "versionNonce": 233596964, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false, "text": "2 quota", "fontSize": 13, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "2 quota", "autoResize": true, "lineHeight": 1.25}, {"id": "ZaM2AdPxF8LQclqawu9u", "type": "arrow", "x": 1680, "y": 82.0, "width": 180, "height": 13.0, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0019", "roundness": null, "seed": 147947538, "version": 1, "versionNonce": 1329498207, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false, "points": [[0, 0], [180, 13.0]], "startBinding": {"elementId": "LKD6jHbXTgUi9m2bnFVu", "mode": "orbit", "fixedPoint": [1, 0.5]}, "endBinding": {"elementId": "2dlcgCb3du8LmrFYFpG2", "mode": "orbit", "fixedPoint": [0, 0.5]}, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false}, {"id": "nlufrOpWfYiKysNf70j3", "type": "text", "x": 1700, "y": 56.0, "width": 64.35000000000001, "height": 16.25, "angle": 0, "strokeColor": "#374151", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0020", "roundness": null, "seed": 1672960502, "version": 1, "versionNonce": 1400738089, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false, "text": "3 enqueue", "fontSize": 13, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "3 enqueue", "autoResize": true, "lineHeight": 1.25}, {"id": "SErMiZSFARF4UJYaQXS7", "type": "ellipse", "x": 1380, "y": 200, "width": 300, "height": 94, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "#dcfce7", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0021", "roundness": {"type": 2}, "seed": 987587880, "version": 1, "versionNonce": 365466112, "isDeleted": false, "boundElements": [{"type": "text", "id": "biwgZIMMQEaXUFHOeTpe"}, {"type": "arrow", "id": "TmT8ExDQVMNejpUsJadq"}], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false}, {"id": "biwgZIMMQEaXUFHOeTpe", "type": "text", "x": 1390, "y": 228.25, "width": 280, "height": 37.5, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0022", "roundness": null, "seed": 1140563901, "version": 1, "versionNonce": 1008396568, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false, "text": "Job store\n(jobs, allocations)", "fontSize": 15, "baseFontSize": null, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "SErMiZSFARF4UJYaQXS7", "originalText": "Job store\n(jobs, allocations)", "autoResize": true, "lineHeight": 1.25}, {"id": "TmT8ExDQVMNejpUsJadq", "type": "arrow", "x": 1530.0, "y": 124, "width": 1, "height": 76, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0023", "roundness": null, "seed": 427431139, "version": 1, "versionNonce": 934986292, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false, "points": [[0, 0], [0.0, 76]], "startBinding": {"elementId": "LKD6jHbXTgUi9m2bnFVu", "mode": "orbit", "fixedPoint": [0.5, 1]}, "endBinding": {"elementId": "SErMiZSFARF4UJYaQXS7", "mode": "orbit", "fixedPoint": [0.5, 0]}, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false}, {"id": "PsGCCh3m86bCY8Bqn6nK", "type": "rectangle", "x": 1860, "y": 260, "width": 300, "height": 140, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "#ffedd5", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0024", "roundness": {"type": 3}, "seed": 608771575, "version": 1, "versionNonce": 1124460935, "isDeleted": false, "boundElements": [{"type": "text", "id": "7iYF2Ro3EbaEBsivt0aU"}, {"type": "arrow", "id": "6maTpey1e5U0rdPNj8Bt"}, {"type": "arrow", "id": "xCXdmdLwu7cUQ5sSKX6e"}], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false}, {"id": "7iYF2Ro3EbaEBsivt0aU", "type": "text", "x": 1870, "y": 286.25, "width": 280, "height": 87.5, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0025", "roundness": null, "seed": 1710511787, "version": 1, "versionNonce": 515573653, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false, "text": "GANG SCHEDULER\nall N nodes or none \u00b7\ntopology-aware (one island) \u00b7\nprefers nodes holding the\nbase model \u00b7 generation fence", "fontSize": 14, "baseFontSize": null, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "PsGCCh3m86bCY8Bqn6nK", "originalText": "GANG SCHEDULER\nall N nodes or none \u00b7\ntopology-aware (one island) \u00b7\nprefers nodes holding the\nbase model \u00b7 generation fence", "autoResize": true, "lineHeight": 1.25}, {"id": "6maTpey1e5U0rdPNj8Bt", "type": "arrow", "x": 2010.0, "y": 150, "width": 1, "height": 110, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0026", "roundness": null, "seed": 1603522945, "version": 1, "versionNonce": 1837162109, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false, "points": [[0, 0], [0.0, 110]], "startBinding": {"elementId": "2dlcgCb3du8LmrFYFpG2", "mode": "orbit", "fixedPoint": [0.5, 1]}, "endBinding": {"elementId": "PsGCCh3m86bCY8Bqn6nK", "mode": "orbit", "fixedPoint": [0.5, 0]}, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false}, {"id": "2YN46ISdTBUNrd5hDvsT", "type": "text", "x": 2022.0, "y": 172, "width": 100.10000000000001, "height": 32.5, "angle": 0, "strokeColor": "#374151", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0027", "roundness": null, "seed": 1117417594, "version": 1, "versionNonce": 1744655514, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false, "text": "4 dequeue by\npriority + age", "fontSize": 13, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "4 dequeue by\npriority + age", "autoResize": true, "lineHeight": 1.25}, {"id": "OsIykkm4En49Bilfvfoq", "type": "rectangle", "x": 1380, "y": 470, "width": 300, "height": 110, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "#fee2e2", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0028", "roundness": {"type": 3}, "seed": 868192600, "version": 1, "versionNonce": 86250368, "isDeleted": false, "boundElements": [{"type": "text", "id": "UzzGxudr7hFN1nqpyB70"}, {"type": "arrow", "id": "zODHjAgeMmA98kiC3oHV"}], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false}, {"id": "UzzGxudr7hFN1nqpyB70", "type": "text", "x": 1390, "y": 490.0, "width": 280, "height": 70.0, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0029", "roundness": null, "seed": 93675450, "version": 1, "versionNonce": 546521803, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false, "text": "HOT pool  70%\nbase models resident\non NVMe -> start in\nminutes", "fontSize": 14, "baseFontSize": null, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "OsIykkm4En49Bilfvfoq", "originalText": "HOT pool  70%\nbase models resident\non NVMe -> start in\nminutes", "autoResize": true, "lineHeight": 1.25}, {"id": "bR3DKay5Z18pgjGQ6SOV", "type": "rectangle", "x": 1380, "y": 640, "width": 300, "height": 110, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "#fee2e2", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0030", "roundness": {"type": 3}, "seed": 1964196104, "version": 1, "versionNonce": 365086747, "isDeleted": false, "boundElements": [{"type": "text", "id": "IcWo5OshpNRgeGKynLas"}, {"type": "arrow", "id": "4cshl8K5x1BqW6gKMnwi"}], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false}, {"id": "IcWo5OshpNRgeGKynLas", "type": "text", "x": 1390, "y": 668.75, "width": 280, "height": 52.5, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0031", "roundness": null, "seed": 1978651188, "version": 1, "versionNonce": 1196593557, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false, "text": "COLD pool  30%\nun-configured; new\nbase models", "fontSize": 14, "baseFontSize": null, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "bR3DKay5Z18pgjGQ6SOV", "originalText": "COLD pool  30%\nun-configured; new\nbase models", "autoResize": true, "lineHeight": 1.25}, {"id": "7N2Dop7RtbEPApP5ocuA", "type": "rectangle", "x": 1380, "y": 810, "width": 300, "height": 110, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "#fee2e2", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0032", "roundness": {"type": 3}, "seed": 1702345557, "version": 1, "versionNonce": 850745598, "isDeleted": false, "boundElements": [{"type": "text", "id": "as0eEmVmCq3gML3E4dK5"}, {"type": "arrow", "id": "zdU2zbVXNyz6AjhfJwVi"}], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false}, {"id": "as0eEmVmCq3gML3E4dK5", "type": "text", "x": 1390, "y": 838.75, "width": 280, "height": 52.5, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0033", "roundness": null, "seed": 233467471, "version": 1, "versionNonce": 914609341, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false, "text": "SPARE pool\ncovers GPU infant\nmortality", "fontSize": 14, "baseFontSize": null, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "7N2Dop7RtbEPApP5ocuA", "originalText": "SPARE pool\ncovers GPU infant\nmortality", "autoResize": true, "lineHeight": 1.25}, {"id": "bKzSlBStheklOGCtS09B", "type": "rectangle", "x": 1860, "y": 470, "width": 360, "height": 140, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "#ede9fe", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0034", "roundness": {"type": 3}, "seed": 467969500, "version": 1, "versionNonce": 12329676, "isDeleted": false, "boundElements": [{"type": "text", "id": "efA2IVxVtXfRmHBuTDNp"}, {"type": "arrow", "id": "xCXdmdLwu7cUQ5sSKX6e"}, {"type": "arrow", "id": "zODHjAgeMmA98kiC3oHV"}, {"type": "arrow", "id": "4cshl8K5x1BqW6gKMnwi"}, {"type": "arrow", "id": "zdU2zbVXNyz6AjhfJwVi"}, {"type": "arrow", "id": "hAfBFcNfTWFdFx6i19TY"}, {"type": "arrow", "id": "3ioObawgwnKKmMDi6pjf"}, {"type": "arrow", "id": "SFNZGz6nQyVh9wdryazA"}], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false}, {"id": "efA2IVxVtXfRmHBuTDNp", "type": "text", "x": 1870, "y": 505.0, "width": 340, "height": 70.0, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0035", "roundness": null, "seed": 1738485151, "version": 1, "versionNonce": 174588222, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false, "text": "TRAINING NODES\n8 GPUs each, one fabric island\nrank per GPU \u00b7 all-reduce each step\nagent per node", "fontSize": 14, "baseFontSize": null, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "bKzSlBStheklOGCtS09B", "originalText": "TRAINING NODES\n8 GPUs each, one fabric island\nrank per GPU \u00b7 all-reduce each step\nagent per node", "autoResize": true, "lineHeight": 1.25}, {"id": "xCXdmdLwu7cUQ5sSKX6e", "type": "arrow", "x": 2010.0, "y": 400, "width": 30.0, "height": 70, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0036", "roundness": null, "seed": 1004447940, "version": 1, "versionNonce": 460684065, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false, "points": [[0, 0], [30.0, 70]], "startBinding": {"elementId": "PsGCCh3m86bCY8Bqn6nK", "mode": "orbit", "fixedPoint": [0.5, 1]}, "endBinding": {"elementId": "bKzSlBStheklOGCtS09B", "mode": "orbit", "fixedPoint": [0.5, 0]}, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false}, {"id": "D77xpAE5lXTZVLutwWem", "type": "text", "x": 2022.0, "y": 422, "width": 143.0, "height": 32.5, "angle": 0, "strokeColor": "#374151", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0037", "roundness": null, "seed": 686925852, "version": 1, "versionNonce": 1062177694, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false, "text": "5 reserve atomically\n(generation G)", "fontSize": 13, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "5 reserve atomically\n(generation G)", "autoResize": true, "lineHeight": 1.25}, {"id": "zODHjAgeMmA98kiC3oHV", "type": "arrow", "x": 1680, "y": 525.0, "width": 180, "height": 20.0, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0038", "roundness": null, "seed": 520398592, "version": 1, "versionNonce": 1262391480, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false, "points": [[0, 0], [180, -20.0]], "startBinding": {"elementId": "OsIykkm4En49Bilfvfoq", "mode": "orbit", "fixedPoint": [1, 0.5]}, "endBinding": {"elementId": "bKzSlBStheklOGCtS09B", "mode": "orbit", "fixedPoint": [0, 0.25]}, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false}, {"id": "sJxTmppj2Juy9FoYO9gD", "type": "text", "x": 1700, "y": 495.0, "width": 85.80000000000001, "height": 16.25, "angle": 0, "strokeColor": "#374151", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0039", "roundness": null, "seed": 992625179, "version": 1, "versionNonce": 1925421609, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false, "text": "6 pick nodes", "fontSize": 13, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "6 pick nodes", "autoResize": true, "lineHeight": 1.25}, {"id": "4cshl8K5x1BqW6gKMnwi", "type": "arrow", "x": 1680, "y": 695.0, "width": 180, "height": 150.0, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "dashed", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0040", "roundness": null, "seed": 876072107, "version": 1, "versionNonce": 1094845895, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false, "points": [[0, 0], [180, -150.0]], "startBinding": {"elementId": "bR3DKay5Z18pgjGQ6SOV", "mode": "orbit", "fixedPoint": [1, 0.5]}, "endBinding": {"elementId": "bKzSlBStheklOGCtS09B", "mode": "orbit", "fixedPoint": [0, 0.5357142857142857]}, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false}, {"id": "cT4YYzxMebEDzXPjHOyq", "type": "text", "x": 1700, "y": 649.0, "width": 107.25000000000001, "height": 32.5, "angle": 0, "strokeColor": "#374151", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0041", "roundness": null, "seed": 1216769353, "version": 1, "versionNonce": 1321113729, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false, "text": "cold start:\npull base model", "fontSize": 13, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "cold start:\npull base model", "autoResize": true, "lineHeight": 1.25}, {"id": "zdU2zbVXNyz6AjhfJwVi", "type": "arrow", "x": 1680, "y": 865.0, "width": 180, "height": 280.0, "angle": 0, "strokeColor": "#b23a28", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "dashed", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0042", "roundness": null, "seed": 222030063, "version": 1, "versionNonce": 612019126, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false, "points": [[0, 0], [180, -280.0]], "startBinding": {"elementId": "7N2Dop7RtbEPApP5ocuA", "mode": "orbit", "fixedPoint": [1, 0.5]}, "endBinding": {"elementId": "bKzSlBStheklOGCtS09B", "mode": "orbit", "fixedPoint": [0, 0.8214285714285714]}, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false}, {"id": "N4fMwFjrG5gEX7mh68Dd", "type": "text", "x": 1700, "y": 835.0, "width": 114.4, "height": 16.25, "angle": 0, "strokeColor": "#374151", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0043", "roundness": null, "seed": 1665997139, "version": 1, "versionNonce": 370631740, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false, "text": "replace on fault", "fontSize": 13, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "replace on fault", "autoResize": true, "lineHeight": 1.25}, {"id": "4MZjWnz0ZlnyGxhpS3cI", "type": "ellipse", "x": 2380, "y": 470, "width": 300, "height": 120, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "#dcfce7", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0044", "roundness": {"type": 2}, "seed": 163761524, "version": 1, "versionNonce": 1392440441, "isDeleted": false, "boundElements": [{"type": "text", "id": "hLIMsAKAOBBbMEoVWClD"}, {"type": "arrow", "id": "hAfBFcNfTWFdFx6i19TY"}], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false}, {"id": "hLIMsAKAOBBbMEoVWClD", "type": "text", "x": 2390, "y": 492.5, "width": 280, "height": 75.0, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0045", "roundness": null, "seed": 459888255, "version": 1, "versionNonce": 288281056, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false, "text": "Dataset store\nsharded \u00b7 NVMe\nread-through \u00b7\nseeded order", "fontSize": 15, "baseFontSize": null, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "4MZjWnz0ZlnyGxhpS3cI", "originalText": "Dataset store\nsharded \u00b7 NVMe\nread-through \u00b7\nseeded order", "autoResize": true, "lineHeight": 1.25}, {"id": "hAfBFcNfTWFdFx6i19TY", "type": "arrow", "x": 2220, "y": 540.0, "width": 160, "height": 10.0, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0046", "roundness": null, "seed": 831966533, "version": 1, "versionNonce": 565292755, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false, "points": [[0, 0], [160, -10.0]], "startBinding": {"elementId": "bKzSlBStheklOGCtS09B", "mode": "orbit", "fixedPoint": [1, 0.5]}, "endBinding": {"elementId": "4MZjWnz0ZlnyGxhpS3cI", "mode": "orbit", "fixedPoint": [0, 0.5]}, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false}, {"id": "8E74kW5evUj3rYiF5mqF", "type": "text", "x": 2240, "y": 498.0, "width": 114.4, "height": 32.5, "angle": 0, "strokeColor": "#374151", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0047", "roundness": null, "seed": 1370427509, "version": 1, "versionNonce": 1598874371, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false, "text": "7 read shard\n(cursor resumes)", "fontSize": 13, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "7 read shard\n(cursor resumes)", "autoResize": true, "lineHeight": 1.25}, {"id": "clj6Q3kWhGNw2IJ2g9Ny", "type": "rectangle", "x": 1860, "y": 700, "width": 360, "height": 110, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "#dcfce7", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0048", "roundness": {"type": 3}, "seed": 1595435269, "version": 1, "versionNonce": 1137110303, "isDeleted": false, "boundElements": [{"type": "text", "id": "xwjufoM7sFt7153TUnsM"}, {"type": "arrow", "id": "3ioObawgwnKKmMDi6pjf"}, {"type": "arrow", "id": "Uu2uo4NQP8D0R1BSJtnM"}], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false}, {"id": "xwjufoM7sFt7153TUnsM", "type": "text", "x": 1870, "y": 728.75, "width": 340, "height": 52.5, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0049", "roundness": null, "seed": 1793951859, "version": 1, "versionNonce": 1563793272, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false, "text": "NVMe tier\nper-rank shard snapshot\nat the BARRIER (<1 s)", "fontSize": 14, "baseFontSize": null, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "clj6Q3kWhGNw2IJ2g9Ny", "originalText": "NVMe tier\nper-rank shard snapshot\nat the BARRIER (<1 s)", "autoResize": true, "lineHeight": 1.25}, {"id": "3ioObawgwnKKmMDi6pjf", "type": "arrow", "x": 2040.0, "y": 610, "width": 1, "height": 90, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0050", "roundness": null, "seed": 621445795, "version": 1, "versionNonce": 1158610987, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false, "points": [[0, 0], [0.0, 90]], "startBinding": {"elementId": "bKzSlBStheklOGCtS09B", "mode": "orbit", "fixedPoint": [0.5, 1]}, "endBinding": {"elementId": "clj6Q3kWhGNw2IJ2g9Ny", "mode": "orbit", "fixedPoint": [0.5, 0]}, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false}, {"id": "yqaNIvOB6Tp4cGzodWaI", "type": "text", "x": 2052.0, "y": 632, "width": 128.70000000000002, "height": 48.75, "angle": 0, "strokeColor": "#374151", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0051", "roundness": null, "seed": 847217401, "version": 1, "versionNonce": 611030145, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false, "text": "8 checkpoint:\nbarrier, snapshot,\nrelease", "fontSize": 13, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "8 checkpoint:\nbarrier, snapshot,\nrelease", "autoResize": true, "lineHeight": 1.25}, {"id": "zGNOzMFdM9SDHxB4eOk9", "type": "ellipse", "x": 2380, "y": 700, "width": 300, "height": 120, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "#dcfce7", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0052", "roundness": {"type": 2}, "seed": 1122819322, "version": 1, "versionNonce": 997706677, "isDeleted": false, "boundElements": [{"type": "text", "id": "Nh356qdNQQ48s53fFk40"}, {"type": "arrow", "id": "Uu2uo4NQP8D0R1BSJtnM"}, {"type": "arrow", "id": "7K7FJjY6ok6VE9Iguf53"}], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false}, {"id": "Nh356qdNQQ48s53fFk40", "type": "text", "x": 2390, "y": 731.875, "width": 280, "height": 56.25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0053", "roundness": null, "seed": 870917924, "version": 1, "versionNonce": 683700006, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false, "text": "Object store\nasync chunked upload;\nmanifest commits LAST", "fontSize": 15, "baseFontSize": null, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "zGNOzMFdM9SDHxB4eOk9", "originalText": "Object store\nasync chunked upload;\nmanifest commits LAST", "autoResize": true, "lineHeight": 1.25}, {"id": "Uu2uo4NQP8D0R1BSJtnM", "type": "arrow", "x": 2220, "y": 755.0, "width": 160, "height": 5.0, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0054", "roundness": null, "seed": 334158532, "version": 1, "versionNonce": 736784718, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false, "points": [[0, 0], [160, 5.0]], "startBinding": {"elementId": "clj6Q3kWhGNw2IJ2g9Ny", "mode": "orbit", "fixedPoint": [1, 0.5]}, "endBinding": {"elementId": "zGNOzMFdM9SDHxB4eOk9", "mode": "orbit", "fixedPoint": [0, 0.5]}, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false}, {"id": "ibg5vibcQNRTeKwYY3e1", "type": "text", "x": 2240, "y": 729.0, "width": 50.050000000000004, "height": 16.25, "angle": 0, "strokeColor": "#374151", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0055", "roundness": null, "seed": 1648591715, "version": 1, "versionNonce": 460059252, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false, "text": "9 async", "fontSize": 13, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "9 async", "autoResize": true, "lineHeight": 1.25}, {"id": "pmc64UfUNDiXOsuqv5dV", "type": "ellipse", "x": 2380, "y": 900, "width": 300, "height": 110, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "#dcfce7", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0056", "roundness": {"type": 2}, "seed": 1377946587, "version": 1, "versionNonce": 2044759943, "isDeleted": false, "boundElements": [{"type": "text", "id": "0MbzBVvRHn1fYkamV8aE"}, {"type": "arrow", "id": "7K7FJjY6ok6VE9Iguf53"}], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false}, {"id": "0MbzBVvRHn1fYkamV8aE", "type": "text", "x": 2390, "y": 936.25, "width": 280, "height": 37.5, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0057", "roundness": null, "seed": 2110908104, "version": 1, "versionNonce": 792503906, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false, "text": "Model registry\nTHE DELIVERABLE", "fontSize": 15, "baseFontSize": null, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "pmc64UfUNDiXOsuqv5dV", "originalText": "Model registry\nTHE DELIVERABLE", "autoResize": true, "lineHeight": 1.25}, {"id": "7K7FJjY6ok6VE9Iguf53", "type": "arrow", "x": 2530.0, "y": 900, "width": 1, "height": 80, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0058", "roundness": null, "seed": 370109807, "version": 1, "versionNonce": 1813034498, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false, "points": [[0, 0], [0.0, -80]], "startBinding": {"elementId": "pmc64UfUNDiXOsuqv5dV", "mode": "orbit", "fixedPoint": [0.5, 0]}, "endBinding": {"elementId": "zGNOzMFdM9SDHxB4eOk9", "mode": "orbit", "fixedPoint": [0.5, 1]}, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false}, {"id": "3bmq3Fx2oCGUUOvuj0PU", "type": "text", "x": 2542.0, "y": 860, "width": 114.4, "height": 32.5, "angle": 0, "strokeColor": "#374151", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0059", "roundness": null, "seed": 728214014, "version": 1, "versionNonce": 1989244755, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false, "text": "10 promote\nfinal checkpoint", "fontSize": 13, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "10 promote\nfinal checkpoint", "autoResize": true, "lineHeight": 1.25}, {"id": "BVJhC2olsR0jjpuGjul8", "type": "rectangle", "x": 900, "y": 700, "width": 300, "height": 110, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "#fef9c3", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0060", "roundness": {"type": 3}, "seed": 437255894, "version": 1, "versionNonce": 706958155, "isDeleted": false, "boundElements": [{"type": "text", "id": "7gx8XTAmNgmycyXQFNCi"}, {"type": "arrow", "id": "SFNZGz6nQyVh9wdryazA"}, {"type": "arrow", "id": "9VOWDWoRQ8QDXXwOtEMf"}], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false}, {"id": "7gx8XTAmNgmycyXQFNCi", "type": "text", "x": 910, "y": 737.5, "width": 280, "height": 35.0, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0061", "roundness": null, "seed": 1738230715, "version": 1, "versionNonce": 23698169, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false, "text": "Event bus (Kafka)\nfaults + progress", "fontSize": 14, "baseFontSize": null, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "BVJhC2olsR0jjpuGjul8", "originalText": "Event bus (Kafka)\nfaults + progress", "autoResize": true, "lineHeight": 1.25}, {"id": "T4AJUAoS2VR0QNCtMgAW", "type": "rectangle", "x": 900, "y": 880, "width": 300, "height": 110, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "#fef9c3", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0062", "roundness": {"type": 3}, "seed": 671983700, "version": 1, "versionNonce": 1074003764, "isDeleted": false, "boundElements": [{"type": "text", "id": "0Db1GO23uaZ4gpnS6mv0"}, {"type": "arrow", "id": "9VOWDWoRQ8QDXXwOtEMf"}], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false}, {"id": "0Db1GO23uaZ4gpnS6mv0", "type": "text", "x": 910, "y": 917.5, "width": 280, "height": 35.0, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0063", "roundness": null, "seed": 1961875301, "version": 1, "versionNonce": 880390641, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false, "text": "Health monitor\n+ metrics \u00b7 GOODPUT", "fontSize": 14, "baseFontSize": null, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "T4AJUAoS2VR0QNCtMgAW", "originalText": "Health monitor\n+ metrics \u00b7 GOODPUT", "autoResize": true, "lineHeight": 1.25}, {"id": "SFNZGz6nQyVh9wdryazA", "type": "arrow", "x": 1940, "y": 610, "width": 890, "height": 200, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "dashed", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0064", "roundness": null, "seed": 1512365367, "version": 1, "versionNonce": 1138842710, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false, "points": [[0, 0], [0, 365], [-890, 365], [-890, 200]], "startBinding": {"elementId": "bKzSlBStheklOGCtS09B", "mode": "orbit", "fixedPoint": [0.2222222222222222, 1]}, "endBinding": {"elementId": "BVJhC2olsR0jjpuGjul8", "mode": "orbit", "fixedPoint": [0.5, 1]}, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false}, {"id": "gsy7897Ck5eXlNSYjPZX", "type": "text", "x": 1180, "y": 985, "width": 386.1, "height": 16.25, "angle": 0, "strokeColor": "#374151", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0065", "roundness": null, "seed": 1775051908, "version": 1, "versionNonce": 2010477982, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false, "text": "agents emit  XID \u00b7 ECC \u00b7 link down \u00b7 step-time outlier", "fontSize": 13, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "agents emit  XID \u00b7 ECC \u00b7 link down \u00b7 step-time outlier", "autoResize": true, "lineHeight": 1.25}, {"id": "9VOWDWoRQ8QDXXwOtEMf", "type": "arrow", "x": 1050.0, "y": 810, "width": 1, "height": 70, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0066", "roundness": null, "seed": 1556631855, "version": 1, "versionNonce": 656069825, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false, "points": [[0, 0], [0.0, 70]], "startBinding": {"elementId": "BVJhC2olsR0jjpuGjul8", "mode": "orbit", "fixedPoint": [0.5, 1]}, "endBinding": {"elementId": "T4AJUAoS2VR0QNCtMgAW", "mode": "orbit", "fixedPoint": [0.5, 0]}, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false}, {"id": "50dZ4WiZNaa7OpgioWvj", "type": "arrow", "x": 1050.0, "y": 880, "width": 840.0, "height": 480, "angle": 0, "strokeColor": "#b23a28", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "dashed", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0067", "roundness": null, "seed": 1728657016, "version": 1, "versionNonce": 721179825, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false, "points": [[0, 0], [0.0, 160], [690.0, 160], [840.0, -480]], "startBinding": null, "endBinding": null, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false}, {"id": "LQ8f3INsEne0PhhzZDIE", "type": "text", "x": 960, "y": 1050, "width": 307.45000000000005, "height": 32.5, "angle": 0, "strokeColor": "#374151", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0068", "roundness": null, "seed": 620282758, "version": 1, "versionNonce": 2110427651, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false, "text": "11 drain node, bump generation,\nrestart gang from last committed checkpoint", "fontSize": 13, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "11 drain node, bump generation,\nrestart gang from last committed checkpoint", "autoResize": true, "lineHeight": 1.25}, {"id": "pkLTjtREsCA9PlwObcT9", "type": "text", "x": 420, "y": 1120, "width": 52.800000000000004, "height": 20.0, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0069", "roundness": null, "seed": 403622670, "version": 1, "versionNonce": 2079500278, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false, "text": "Legend", "fontSize": 16, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "Legend", "autoResize": true, "lineHeight": 1.25}, {"id": "E3cSMu1wDGVnAAIZsZzF", "type": "rectangle", "x": 420, "y": 1150, "width": 22, "height": 22, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "#dbeafe", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0070", "roundness": {"type": 3}, "seed": 1166933806, "version": 1, "versionNonce": 1480948539, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false}, {"id": "8OXutsKNWcS2Hdsal5LO", "type": "text", "x": 450, "y": 1151, "width": 92.95, "height": 16.25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0071", "roundness": null, "seed": 1615097591, "version": 1, "versionNonce": 631582010, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false, "text": "control plane", "fontSize": 13, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "control plane", "autoResize": true, "lineHeight": 1.25}, {"id": "MRKQnPCVglcW4OwZWIps", "type": "rectangle", "x": 650, "y": 1150, "width": 22, "height": 22, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "#ffedd5", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0072", "roundness": {"type": 3}, "seed": 1811551542, "version": 1, "versionNonce": 147063202, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false}, {"id": "tAN5dJchYJ4BayK68Dzg", "type": "text", "x": 680, "y": 1151, "width": 71.5, "height": 16.25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0073", "roundness": null, "seed": 2028051431, "version": 1, "versionNonce": 911722428, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false, "text": "scheduling", "fontSize": 13, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "scheduling", "autoResize": true, "lineHeight": 1.25}, {"id": "3MAaP91nhDrJBU5wURiV", "type": "rectangle", "x": 880, "y": 1150, "width": 22, "height": 22, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "#ede9fe", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0074", "roundness": {"type": 3}, "seed": 1259040910, "version": 1, "versionNonce": 2139309364, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false}, {"id": "C5p7SaaOYetTk1EdwJBP", "type": "text", "x": 910, "y": 1151, "width": 71.5, "height": 16.25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0075", "roundness": null, "seed": 622368776, "version": 1, "versionNonce": 501245905, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false, "text": "data plane", "fontSize": 13, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "data plane", "autoResize": true, "lineHeight": 1.25}, {"id": "wNNzxW6WJsd8RZuL8ZLt", "type": "rectangle", "x": 1110, "y": 1150, "width": 22, "height": 22, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "#dcfce7", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0076", "roundness": {"type": 3}, "seed": 1840659182, "version": 1, "versionNonce": 1057038836, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false}, {"id": "xQL3YraqAKY3cZY1Jq0Y", "type": "text", "x": 1140, "y": 1151, "width": 50.050000000000004, "height": 16.25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0077", "roundness": null, "seed": 2147380350, "version": 1, "versionNonce": 1489723359, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false, "text": "storage", "fontSize": 13, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "storage", "autoResize": true, "lineHeight": 1.25}, {"id": "HHExWS8tdyR5KaxHHwdy", "type": "rectangle", "x": 1340, "y": 1150, "width": 22, "height": 22, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "#fef9c3", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0078", "roundness": {"type": 3}, "seed": 1114722375, "version": 1, "versionNonce": 1378653572, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false}, {"id": "DKlllRwIyGjc9xgNWjLv", "type": "text", "x": 1370, "y": 1151, "width": 150.15, "height": 16.25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0079", "roundness": null, "seed": 88347952, "version": 1, "versionNonce": 404091006, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false, "text": "queue / observability", "fontSize": 13, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "queue / observability", "autoResize": true, "lineHeight": 1.25}, {"id": "c91EJqWA6VY7pcmlfdI1", "type": "rectangle", "x": 1570, "y": 1150, "width": 22, "height": 22, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "#fee2e2", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0080", "roundness": {"type": 3}, "seed": 1968301625, "version": 1, "versionNonce": 2090942405, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false}, {"id": "64dLyh7pIN7PyBj79ncp", "type": "text", "x": 1600, "y": 1151, "width": 71.5, "height": 16.25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0081", "roundness": null, "seed": 1511875402, "version": 1, "versionNonce": 254591815, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false, "text": "node pools", "fontSize": 13, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "node pools", "autoResize": true, "lineHeight": 1.25}, {"id": "34ZcWRO9diU6PsKUguph", "type": "text", "x": 420, "y": 1210, "width": 783.7500000000001, "height": 168.75, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0082", "roundness": null, "seed": 2067224217, "version": 1, "versionNonce": 1630567430, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false, "text": "NUMBERS TO SAY OUT LOUD\n20,000 GPUs / 8 per node = 2,500 nodes.   A job is 64 to 4,096 GPUs.\n4,000-GPU job = 500 nodes; node MTBF 6 months -> a failure every ~9 hours.\n100B params x 16 bytes (weights + optimizer moments) = ~1.6 TB of state;\n   sharded 4,000 ways = ~400 MB per rank; VRAM -> NVMe at 5 GB/s is <1 s.\n   The BARRIER is the cost, not the bytes. The upload is off the critical path.\nCheckpoint overhead: 60 s every 20 min = 5% of the fleet. Epoch-aligned (~1 h) < 2%.\nExpected lost work on a crash = half the checkpoint interval.\nMetric that exposes everything: GOODPUT = GPU-hours that committed steps / GPU-hours allocated.", "fontSize": 15, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "NUMBERS TO SAY OUT LOUD\n20,000 GPUs / 8 per node = 2,500 nodes.   A job is 64 to 4,096 GPUs.\n4,000-GPU job = 500 nodes; node MTBF 6 months -> a failure every ~9 hours.\n100B params x 16 bytes (weights + optimizer moments) = ~1.6 TB of state;\n   sharded 4,000 ways = ~400 MB per rank; VRAM -> NVMe at 5 GB/s is <1 s.\n   The BARRIER is the cost, not the bytes. The upload is off the critical path.\nCheckpoint overhead: 60 s every 20 min = 5% of the fleet. Epoch-aligned (~1 h) < 2%.\nExpected lost work on a crash = half the checkpoint interval.\nMetric that exposes everything: GOODPUT = GPU-hours that committed steps / GPU-hours allocated.", "autoResize": true, "lineHeight": 1.25}, {"id": "koiQaSmc5n513iBg50MC", "type": "text", "x": 420, "y": 1430, "width": 866.2500000000001, "height": 93.75, "angle": 0, "strokeColor": "#b23a28", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0083", "roundness": null, "seed": 1459340018, "version": 1, "versionNonce": 1385408520, "isDeleted": false, "boundElements": [], "updated": 1790867244655, "created": 1790867244655, "link": null, "locked": false, "text": "RED: what fails and what you say\nFail-stop (GPU off the bus, ECC, link down): agent detects, drain, swap a spare, restart from checkpoint.\nFail-slow (straggler): detect by step-time outlier; one slow rank slows all 4,000.\nSilent corruption: per-rank gradient-norm outlier + shadow recompute; answer was_it_hardware(job, steps).\nCheckpoint on EPOCH boundaries, not a timer: researchers compare quality per epoch and restart from one.", "fontSize": 15, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "RED: what fails and what you say\nFail-stop (GPU off the bus, ECC, link down): agent detects, drain, swap a spare, restart from checkpoint.\nFail-slow (straggler): detect by step-time outlier; one slow rank slows all 4,000.\nSilent corruption: per-rank gradient-norm outlier + shadow recompute; answer was_it_hardware(job, steps).\nCheckpoint on EPOCH boundaries, not a timer: researchers compare quality per epoch and restart from one.", "autoResize": true, "lineHeight": 1.25}], "appState": {"viewBackgroundColor": "#ffffff", "gridSize": null}, "files": {}}