{"type": "excalidraw", "version": 2, "source": "generated", "elements": [{"id": "IJcWGxMlUkoLZ7k06nxw", "type": "frame", "x": 390, "y": -90, "width": 1998.6000000000004, "height": 1187.5, "angle": 0, "strokeColor": "#bbb", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 1, "strokeStyle": "solid", "roughness": 0, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0000", "roundness": null, "seed": 12345, "version": 1, "versionNonce": 54321, "isDeleted": false, "boundElements": [], "updated": 1790457285619, "created": 1790457285619, "link": null, "locked": false, "name": "System diagram"}, {"id": "8dSD6z4j9zlvNcyaekor", "type": "text", "x": 420, "y": -60, "width": 831.6, "height": 35.0, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0001", "roundness": null, "seed": 2143598409, "version": 1, "versionNonce": 1786709833, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false, "text": "LLM inference API with batched GPU serving, the answer", "fontSize": 28, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "LLM inference API with batched GPU serving, the answer", "autoResize": true, "lineHeight": 1.25}, {"id": "hcMxfy3wiyPRrtWAISLm", "type": "text", "x": 420, "y": -20, "width": 1370.6000000000001, "height": 17.5, "angle": 0, "strokeColor": "#4b5563", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0002", "roundness": null, "seed": 877663843, "version": 1, "versionNonce": 2069665228, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false, "text": "Route by model+version to a replica pool, batch per replica by size-or-time, stream finished sequences back early. Solid = request path, dashed = control, cold start and failure.", "fontSize": 14, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "Route by model+version to a replica pool, batch per replica by size-or-time, stream finished sequences back early. Solid = request path, dashed = control, cold start and failure.", "autoResize": true, "lineHeight": 1.25}, {"id": "CkhlhLtIAHUWj70YC7Yo", "type": "rectangle", "x": 420, "y": 60, "width": 240, "height": 84, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "#f3f4f6", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0003", "roundness": {"type": 3}, "seed": 1676953455, "version": 1, "versionNonce": 864715398, "isDeleted": false, "boundElements": [{"type": "text", "id": "ePjS3xecGjTLFPjCcHXj"}, {"type": "arrow", "id": "bou7E4WX4xWBAPOb2oQE"}, {"type": "arrow", "id": "zGZqg9kzx0WeDcwuCO76"}], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false}, {"id": "ePjS3xecGjTLFPjCcHXj", "type": "text", "x": 430, "y": 90.0, "width": 220, "height": 37.5, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0004", "roundness": null, "seed": 1059784310, "version": 1, "versionNonce": 1389768816, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false, "text": "Clients\nsync or streaming \u00b7 idempotency key", "fontSize": 15, "baseFontSize": null, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "CkhlhLtIAHUWj70YC7Yo", "originalText": "Clients\nsync or streaming \u00b7 idempotency key", "autoResize": true, "lineHeight": 1.25}, {"id": "OwmiU6dfQWouwwaQqDYc", "type": "rectangle", "x": 900, "y": 60, "width": 240, "height": 84, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "#dbeafe", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0005", "roundness": {"type": 3}, "seed": 793178988, "version": 1, "versionNonce": 1284595704, "isDeleted": false, "boundElements": [{"type": "text", "id": "qXQIOK0RqOF5hC7eT3Ld"}, {"type": "arrow", "id": "bou7E4WX4xWBAPOb2oQE"}, {"type": "arrow", "id": "bTvC0k4Uq7Dccv95uV5T"}, {"type": "arrow", "id": "iyDHIM5RPUJmFuNkmizh"}], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false}, {"id": "qXQIOK0RqOF5hC7eT3Ld", "type": "text", "x": 910, "y": 90.0, "width": 220, "height": 37.5, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0006", "roundness": null, "seed": 1131065133, "version": 1, "versionNonce": 736327932, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false, "text": "API gateway\nauth \u00b7 per-tenant limits \u00b7 429", "fontSize": 15, "baseFontSize": null, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "OwmiU6dfQWouwwaQqDYc", "originalText": "API gateway\nauth \u00b7 per-tenant limits \u00b7 429", "autoResize": true, "lineHeight": 1.25}, {"id": "YtphvgRZdGXWqGQG947h", "type": "rectangle", "x": 1380, "y": 60, "width": 240, "height": 84, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "#ffedd5", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0007", "roundness": {"type": 3}, "seed": 106451209, "version": 1, "versionNonce": 1498827587, "isDeleted": false, "boundElements": [{"type": "text", "id": "KJqpWXahSAj6uezzOVfD"}, {"type": "arrow", "id": "bTvC0k4Uq7Dccv95uV5T"}, {"type": "arrow", "id": "i8gJOVcN1jlDq7VIZvWv"}, {"type": "arrow", "id": "3jDdO74wEXnTl9iK87L4"}], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false}, {"id": "KJqpWXahSAj6uezzOVfD", "type": "text", "x": 1390, "y": 90.0, "width": 220, "height": 37.5, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0008", "roundness": null, "seed": 2088747626, "version": 1, "versionNonce": 1662874959, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false, "text": "Router\nmodel+version \u2192 replica pool", "fontSize": 15, "baseFontSize": null, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "YtphvgRZdGXWqGQG947h", "originalText": "Router\nmodel+version \u2192 replica pool", "autoResize": true, "lineHeight": 1.25}, {"id": "XOg8ssFHVH5WOrNFVrXS", "type": "ellipse", "x": 1860, "y": 60, "width": 240, "height": 94, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "#dcfce7", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0009", "roundness": {"type": 2}, "seed": 767695033, "version": 1, "versionNonce": 2083537830, "isDeleted": false, "boundElements": [{"type": "text", "id": "2km0PMIqfXuAJyhBMQg5"}, {"type": "arrow", "id": "i8gJOVcN1jlDq7VIZvWv"}], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false}, {"id": "2km0PMIqfXuAJyhBMQg5", "type": "text", "x": 1870, "y": 95.0, "width": 220, "height": 37.5, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0010", "roundness": null, "seed": 1811560025, "version": 1, "versionNonce": 43800895, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false, "text": "Model registry\nmodel, version, min GPUs, state", "fontSize": 15, "baseFontSize": null, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "XOg8ssFHVH5WOrNFVrXS", "originalText": "Model registry\nmodel, version, min GPUs, state", "autoResize": true, "lineHeight": 1.25}, {"id": "Dhpmc1mKQlgTSLMizFUk", "type": "rectangle", "x": 420, "y": 240, "width": 240, "height": 84, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "#ede9fe", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0011", "roundness": {"type": 3}, "seed": 639316925, "version": 1, "versionNonce": 1643713742, "isDeleted": false, "boundElements": [{"type": "text", "id": "qsuOK6Q7JtH6SWdvHGsy"}, {"type": "arrow", "id": "IbuCNDsoeUv86BTfm4A3"}], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false}, {"id": "qsuOK6Q7JtH6SWdvHGsy", "type": "text", "x": 430, "y": 270.0, "width": 220, "height": 37.5, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0012", "roundness": null, "seed": 1627500048, "version": 1, "versionNonce": 53516006, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false, "text": "Queue-depth monitor\ntighten limits \u00b7 shed free tier", "fontSize": 15, "baseFontSize": null, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "Dhpmc1mKQlgTSLMizFUk", "originalText": "Queue-depth monitor\ntighten limits \u00b7 shed free tier", "autoResize": true, "lineHeight": 1.25}, {"id": "IotiRxFckxTyhkktyVmL", "type": "rectangle", "x": 900, "y": 240, "width": 240, "height": 84, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "#ffedd5", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0013", "roundness": {"type": 3}, "seed": 1375778783, "version": 1, "versionNonce": 1608604753, "isDeleted": false, "boundElements": [{"type": "text", "id": "VuCCR8TaUGNcaahhSVzn"}, {"type": "arrow", "id": "iyDHIM5RPUJmFuNkmizh"}, {"type": "arrow", "id": "nYPANLrhWm4q0IpzN1vD"}, {"type": "arrow", "id": "IbuCNDsoeUv86BTfm4A3"}], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false}, {"id": "VuCCR8TaUGNcaahhSVzn", "type": "text", "x": 910, "y": 270.0, "width": 220, "height": 37.5, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0014", "roundness": null, "seed": 1883995348, "version": 1, "versionNonce": 1016954682, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false, "text": "Per-tenant priority queues\npaid ahead of free \u00b7 isolation", "fontSize": 15, "baseFontSize": null, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "IotiRxFckxTyhkktyVmL", "originalText": "Per-tenant priority queues\npaid ahead of free \u00b7 isolation", "autoResize": true, "lineHeight": 1.25}, {"id": "hpB3ThX83kK6pJN7ZDKm", "type": "rectangle", "x": 1380, "y": 240, "width": 240, "height": 84, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "#ffedd5", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0015", "roundness": {"type": 3}, "seed": 715038348, "version": 1, "versionNonce": 1044507554, "isDeleted": false, "boundElements": [{"type": "text", "id": "AnbrnDv0ruS8oCBXOQ0S"}, {"type": "arrow", "id": "3jDdO74wEXnTl9iK87L4"}, {"type": "arrow", "id": "nYPANLrhWm4q0IpzN1vD"}, {"type": "arrow", "id": "gaWxhz7j3jhgm9UwndbG"}, {"type": "arrow", "id": "uVSFQMScrYfqEvOG41nF"}, {"type": "arrow", "id": "Yv3p4ciuu13HVTzzk9Wq"}, {"type": "arrow", "id": "2Djo126jbKPZUqsHoqWE"}], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false}, {"id": "AnbrnDv0ruS8oCBXOQ0S", "type": "text", "x": 1390, "y": 270.0, "width": 220, "height": 37.5, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0016", "roundness": null, "seed": 1962383060, "version": 1, "versionNonce": 339034098, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false, "text": "Batcher, per replica\nflush at N or at 50 ms", "fontSize": 15, "baseFontSize": null, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "hpB3ThX83kK6pJN7ZDKm", "originalText": "Batcher, per replica\nflush at N or at 50 ms", "autoResize": true, "lineHeight": 1.25}, {"id": "GqqAE4x1qOPzCPpH5kRX", "type": "rectangle", "x": 1860, "y": 240, "width": 240, "height": 84, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "#ede9fe", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0017", "roundness": {"type": 3}, "seed": 845602248, "version": 1, "versionNonce": 347578710, "isDeleted": false, "boundElements": [{"type": "text", "id": "BjcmHOUyP5cIo7juwUPd"}, {"type": "arrow", "id": "2Djo126jbKPZUqsHoqWE"}, {"type": "arrow", "id": "lgRS7qUM0oxbIJQPQsdF"}], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false}, {"id": "BjcmHOUyP5cIo7juwUPd", "type": "text", "x": 1870, "y": 270.0, "width": 220, "height": 37.5, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0018", "roundness": null, "seed": 2040353315, "version": 1, "versionNonce": 1496557061, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false, "text": "Scheduler / autoscaler\nqueue depth \u2192 warm or cold start", "fontSize": 15, "baseFontSize": null, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "GqqAE4x1qOPzCPpH5kRX", "originalText": "Scheduler / autoscaler\nqueue depth \u2192 warm or cold start", "autoResize": true, "lineHeight": 1.25}, {"id": "gopxl6ZNmYmgN9esaQgR", "type": "rectangle", "x": 420, "y": 440, "width": 240, "height": 84, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "#dbeafe", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0019", "roundness": {"type": 3}, "seed": 1844812234, "version": 1, "versionNonce": 400738407, "isDeleted": false, "boundElements": [{"type": "text", "id": "apQ1A7vCJWhnKVMx1TPE"}, {"type": "arrow", "id": "xem2I0V5tZZZQCHcUdD4"}, {"type": "arrow", "id": "zGZqg9kzx0WeDcwuCO76"}, {"type": "arrow", "id": "bhS67hJeFlY37PSa7iDN"}], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false}, {"id": "apQ1A7vCJWhnKVMx1TPE", "type": "text", "x": 430, "y": 470.0, "width": 220, "height": 37.5, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0020", "roundness": null, "seed": 1984494701, "version": 1, "versionNonce": 1823813070, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false, "text": "Response service\nfinished sequences leave early \u00b7 stream", "fontSize": 15, "baseFontSize": null, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "gopxl6ZNmYmgN9esaQgR", "originalText": "Response service\nfinished sequences leave early \u00b7 stream", "autoResize": true, "lineHeight": 1.25}, {"id": "ymNWPddOOfc292MetsB4", "type": "rectangle", "x": 900, "y": 440, "width": 240, "height": 84, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "#fee2e2", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0021", "roundness": {"type": 3}, "seed": 1494272279, "version": 1, "versionNonce": 728100120, "isDeleted": false, "boundElements": [{"type": "text", "id": "xKS9agPPtzavwB4yg1yr"}, {"type": "arrow", "id": "gaWxhz7j3jhgm9UwndbG"}, {"type": "arrow", "id": "xem2I0V5tZZZQCHcUdD4"}], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false}, {"id": "xKS9agPPtzavwB4yg1yr", "type": "text", "x": 910, "y": 470.0, "width": 220, "height": 37.5, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0022", "roundness": null, "seed": 1474298518, "version": 1, "versionNonce": 388332307, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false, "text": "Replica A1 \u00b7 Opus\n4 GPUs \u00b7 weights loaded \u00b7 KV cache", "fontSize": 15, "baseFontSize": null, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "ymNWPddOOfc292MetsB4", "originalText": "Replica A1 \u00b7 Opus\n4 GPUs \u00b7 weights loaded \u00b7 KV cache", "autoResize": true, "lineHeight": 1.25}, {"id": "hQF4t6KbjGK4E1Jz2Mnz", "type": "rectangle", "x": 1380, "y": 440, "width": 240, "height": 84, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "#fee2e2", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0023", "roundness": {"type": 3}, "seed": 528359556, "version": 1, "versionNonce": 1573433564, "isDeleted": false, "boundElements": [{"type": "text", "id": "059FpPM8x1CITEblPPww"}, {"type": "arrow", "id": "uVSFQMScrYfqEvOG41nF"}, {"type": "arrow", "id": "AscoTU8GBq690DtsPAGI"}], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false}, {"id": "059FpPM8x1CITEblPPww", "type": "text", "x": 1390, "y": 470.0, "width": 220, "height": 37.5, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0024", "roundness": null, "seed": 1498431284, "version": 1, "versionNonce": 1167433978, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false, "text": "Replica A2 \u00b7 Opus\n4 GPUs \u00b7 weights loaded \u00b7 KV cache", "fontSize": 15, "baseFontSize": null, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "hQF4t6KbjGK4E1Jz2Mnz", "originalText": "Replica A2 \u00b7 Opus\n4 GPUs \u00b7 weights loaded \u00b7 KV cache", "autoResize": true, "lineHeight": 1.25}, {"id": "nH2LM0ZsRAsh0hhWaaxj", "type": "rectangle", "x": 1860, "y": 440, "width": 240, "height": 84, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "#fee2e2", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0025", "roundness": {"type": 3}, "seed": 1590122804, "version": 1, "versionNonce": 961990708, "isDeleted": false, "boundElements": [{"type": "text", "id": "7FIfc6aRrwupRagQ8jPp"}, {"type": "arrow", "id": "Yv3p4ciuu13HVTzzk9Wq"}, {"type": "arrow", "id": "lgRS7qUM0oxbIJQPQsdF"}], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false}, {"id": "7FIfc6aRrwupRagQ8jPp", "type": "text", "x": 1870, "y": 470.0, "width": 220, "height": 37.5, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0026", "roundness": null, "seed": 1625762155, "version": 1, "versionNonce": 299141902, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false, "text": "Replica B1 \u00b7 Haiku\n1 GPU \u00b7 weights loaded", "fontSize": 15, "baseFontSize": null, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "nH2LM0ZsRAsh0hhWaaxj", "originalText": "Replica B1 \u00b7 Haiku\n1 GPU \u00b7 weights loaded", "autoResize": true, "lineHeight": 1.25}, {"id": "nF1D0XfHxScS718iyuRh", "type": "ellipse", "x": 420, "y": 640, "width": 240, "height": 94, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "#dcfce7", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0027", "roundness": {"type": 2}, "seed": 1764782677, "version": 1, "versionNonce": 1227781242, "isDeleted": false, "boundElements": [{"type": "text", "id": "hwQzC6k5ifWQf2dJHhIZ"}, {"type": "arrow", "id": "bhS67hJeFlY37PSa7iDN"}], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false}, {"id": "hwQzC6k5ifWQf2dJHhIZ", "type": "text", "x": 430, "y": 675.0, "width": 220, "height": 37.5, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0028", "roundness": null, "seed": 1723528687, "version": 1, "versionNonce": 597592542, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false, "text": "Metrics\nqueue depth \u00b7 batch fill \u00b7 TTFT \u00b7 P99 \u00b7 GPU util", "fontSize": 15, "baseFontSize": null, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "nF1D0XfHxScS718iyuRh", "originalText": "Metrics\nqueue depth \u00b7 batch fill \u00b7 TTFT \u00b7 P99 \u00b7 GPU util", "autoResize": true, "lineHeight": 1.25}, {"id": "cfvIBOFO4VjiCkzgCpzZ", "type": "rectangle", "x": 1380, "y": 640, "width": 240, "height": 84, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "#fef9c3", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0029", "roundness": {"type": 3}, "seed": 417055889, "version": 1, "versionNonce": 163090152, "isDeleted": false, "boundElements": [{"type": "text", "id": "KYp3w8wKQy0iUW5KXkWg"}, {"type": "arrow", "id": "AscoTU8GBq690DtsPAGI"}, {"type": "arrow", "id": "JGpO8IegGABMPrOPjaLR"}], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false}, {"id": "KYp3w8wKQy0iUW5KXkWg", "type": "text", "x": 1390, "y": 670.0, "width": 220, "height": 37.5, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0030", "roundness": null, "seed": 72188123, "version": 1, "versionNonce": 140713940, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false, "text": "Warm pool, hardware only\nGPUs up, no weights loaded", "fontSize": 15, "baseFontSize": null, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "cfvIBOFO4VjiCkzgCpzZ", "originalText": "Warm pool, hardware only\nGPUs up, no weights loaded", "autoResize": true, "lineHeight": 1.25}, {"id": "22xJIyYPYxi1RPbz1qRc", "type": "ellipse", "x": 1860, "y": 640, "width": 240, "height": 94, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "#dcfce7", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0031", "roundness": {"type": 2}, "seed": 1376189094, "version": 1, "versionNonce": 708604657, "isDeleted": false, "boundElements": [{"type": "text", "id": "h5qdb0KeqGQ9pUmeJIvq"}, {"type": "arrow", "id": "JGpO8IegGABMPrOPjaLR"}], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false}, {"id": "h5qdb0KeqGQ9pUmeJIvq", "type": "text", "x": 1870, "y": 675.0, "width": 220, "height": 37.5, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0032", "roundness": null, "seed": 998757552, "version": 1, "versionNonce": 1725769547, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false, "text": "Weights store\nNVMe / object \u00b7 ~1 TB per model", "fontSize": 15, "baseFontSize": null, "fontFamily": 5, "textAlign": "center", "verticalAlign": "middle", "containerId": "22xJIyYPYxi1RPbz1qRc", "originalText": "Weights store\nNVMe / object \u00b7 ~1 TB per model", "autoResize": true, "lineHeight": 1.25}, {"id": "bou7E4WX4xWBAPOb2oQE", "type": "arrow", "x": 660, "y": 102.0, "width": 240, "height": 1, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0033", "roundness": null, "seed": 1573209351, "version": 1, "versionNonce": 793441995, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false, "points": [[0, 0], [240, 0.0]], "startBinding": {"elementId": "CkhlhLtIAHUWj70YC7Yo", "mode": "orbit", "fixedPoint": [1, 0.5]}, "endBinding": {"elementId": "OwmiU6dfQWouwwaQqDYc", "mode": "orbit", "fixedPoint": [0, 0.5]}, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false}, {"id": "M4T9szqw18nlcrwaaO8x", "type": "text", "x": 680, "y": 76, "width": 243.10000000000002, "height": 16.25, "angle": 0, "strokeColor": "#374151", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0034", "roundness": null, "seed": 318812426, "version": 1, "versionNonce": 1451937938, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false, "text": "1  request(model, version, stream)", "fontSize": 13, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "1  request(model, version, stream)", "autoResize": true, "lineHeight": 1.25}, {"id": "bTvC0k4Uq7Dccv95uV5T", "type": "arrow", "x": 1140, "y": 102.0, "width": 240, "height": 1, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0035", "roundness": null, "seed": 195953177, "version": 1, "versionNonce": 2103482792, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false, "points": [[0, 0], [240, 0.0]], "startBinding": {"elementId": "OwmiU6dfQWouwwaQqDYc", "mode": "orbit", "fixedPoint": [1, 0.5]}, "endBinding": {"elementId": "YtphvgRZdGXWqGQG947h", "mode": "orbit", "fixedPoint": [0, 0.5]}, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false}, {"id": "VlMc3bkX7025uU3qLwiJ", "type": "text", "x": 1160, "y": 76, "width": 78.65, "height": 16.25, "angle": 0, "strokeColor": "#374151", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0036", "roundness": null, "seed": 639621779, "version": 1, "versionNonce": 982416129, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false, "text": "2  admitted", "fontSize": 13, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "2  admitted", "autoResize": true, "lineHeight": 1.25}, {"id": "i8gJOVcN1jlDq7VIZvWv", "type": "arrow", "x": 1620, "y": 102.0, "width": 240, "height": 5.0, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "dashed", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0037", "roundness": null, "seed": 1279659908, "version": 1, "versionNonce": 875874931, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false, "points": [[0, 0], [240, 5.0]], "startBinding": {"elementId": "YtphvgRZdGXWqGQG947h", "mode": "orbit", "fixedPoint": [1, 0.5]}, "endBinding": {"elementId": "XOg8ssFHVH5WOrNFVrXS", "mode": "orbit", "fixedPoint": [0, 0.5]}, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false}, {"id": "PFsh4OcZlzAcV4Esr4LO", "type": "text", "x": 1640, "y": 76, "width": 178.75000000000003, "height": 16.25, "angle": 0, "strokeColor": "#374151", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0038", "roundness": null, "seed": 1652268586, "version": 1, "versionNonce": 67150073, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false, "text": "lookup replicas, min GPUs", "fontSize": 13, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "lookup replicas, min GPUs", "autoResize": true, "lineHeight": 1.25}, {"id": "3jDdO74wEXnTl9iK87L4", "type": "arrow", "x": 1500.0, "y": 144, "width": 1, "height": 96, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0039", "roundness": null, "seed": 1507487144, "version": 1, "versionNonce": 165031895, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false, "points": [[0, 0], [0.0, 96]], "startBinding": {"elementId": "YtphvgRZdGXWqGQG947h", "mode": "orbit", "fixedPoint": [0.5, 1]}, "endBinding": {"elementId": "hpB3ThX83kK6pJN7ZDKm", "mode": "orbit", "fixedPoint": [0.5, 0]}, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false}, {"id": "oCqFkWWxPAQwH7S0cX9i", "type": "text", "x": 1512, "y": 166, "width": 171.60000000000002, "height": 16.25, "angle": 0, "strokeColor": "#374151", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0040", "roundness": null, "seed": 1637841282, "version": 1, "versionNonce": 1927519905, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false, "text": "3  to the pool's batcher", "fontSize": 13, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "3  to the pool's batcher", "autoResize": true, "lineHeight": 1.25}, {"id": "iyDHIM5RPUJmFuNkmizh", "type": "arrow", "x": 1020.0, "y": 144, "width": 1, "height": 96, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0041", "roundness": null, "seed": 493013237, "version": 1, "versionNonce": 71906036, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false, "points": [[0, 0], [0.0, 96]], "startBinding": {"elementId": "OwmiU6dfQWouwwaQqDYc", "mode": "orbit", "fixedPoint": [0.5, 1]}, "endBinding": {"elementId": "IotiRxFckxTyhkktyVmL", "mode": "orbit", "fixedPoint": [0.5, 0]}, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false}, {"id": "SvCE9IdnxyYp9Rbsmorx", "type": "text", "x": 1032, "y": 166, "width": 157.3, "height": 16.25, "angle": 0, "strokeColor": "#374151", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0042", "roundness": null, "seed": 2115675255, "version": 1, "versionNonce": 1932683274, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false, "text": "tenant queue: fairness", "fontSize": 13, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "tenant queue: fairness", "autoResize": true, "lineHeight": 1.25}, {"id": "nYPANLrhWm4q0IpzN1vD", "type": "arrow", "x": 1140, "y": 282.0, "width": 240, "height": 1, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0043", "roundness": null, "seed": 967411683, "version": 1, "versionNonce": 605194492, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false, "points": [[0, 0], [240, 0.0]], "startBinding": {"elementId": "IotiRxFckxTyhkktyVmL", "mode": "orbit", "fixedPoint": [1, 0.5]}, "endBinding": {"elementId": "hpB3ThX83kK6pJN7ZDKm", "mode": "orbit", "fixedPoint": [0, 0.5]}, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false}, {"id": "NSTo0wZcoN46VBwdXGFR", "type": "text", "x": 1160, "y": 256, "width": 200.20000000000002, "height": 16.25, "angle": 0, "strokeColor": "#374151", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0044", "roundness": null, "seed": 27746335, "version": 1, "versionNonce": 502512896, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false, "text": "fill paid first, top up free", "fontSize": 13, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "fill paid first, top up free", "autoResize": true, "lineHeight": 1.25}, {"id": "IbuCNDsoeUv86BTfm4A3", "type": "arrow", "x": 660, "y": 282.0, "width": 240, "height": 1, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "dashed", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0045", "roundness": null, "seed": 2059033033, "version": 1, "versionNonce": 33411486, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false, "points": [[0, 0], [240, 0.0]], "startBinding": {"elementId": "Dhpmc1mKQlgTSLMizFUk", "mode": "orbit", "fixedPoint": [1, 0.5]}, "endBinding": {"elementId": "IotiRxFckxTyhkktyVmL", "mode": "orbit", "fixedPoint": [0, 0.5]}, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false}, {"id": "VdNPaUjP1T7ds42LKd2L", "type": "text", "x": 680, "y": 256, "width": 135.85000000000002, "height": 16.25, "angle": 0, "strokeColor": "#374151", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0046", "roundness": null, "seed": 309863765, "version": 1, "versionNonce": 2092813664, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false, "text": "depth \u2192 limits, 429", "fontSize": 13, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "depth \u2192 limits, 429", "autoResize": true, "lineHeight": 1.25}, {"id": "gaWxhz7j3jhgm9UwndbG", "type": "arrow", "x": 1440.0, "y": 324, "width": 420.0, "height": 116, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0047", "roundness": null, "seed": 807980037, "version": 1, "versionNonce": 1770406758, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false, "points": [[0, 0], [0.0, 76], [-420.0, 76], [-420.0, 116]], "startBinding": {"elementId": "hpB3ThX83kK6pJN7ZDKm", "mode": "orbit", "fixedPoint": [0.25, 1]}, "endBinding": {"elementId": "ymNWPddOOfc292MetsB4", "mode": "orbit", "fixedPoint": [0.5, 0]}, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false}, {"id": "s6cfZJJsbHOKruA63gmO", "type": "text", "x": 1032, "y": 402, "width": 85.80000000000001, "height": 16.25, "angle": 0, "strokeColor": "#374151", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0048", "roundness": null, "seed": 379574110, "version": 1, "versionNonce": 1348023705, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false, "text": "4  batch \u2264 N", "fontSize": 13, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "4  batch \u2264 N", "autoResize": true, "lineHeight": 1.25}, {"id": "uVSFQMScrYfqEvOG41nF", "type": "arrow", "x": 1500.0, "y": 324, "width": 1, "height": 116, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0049", "roundness": null, "seed": 926843152, "version": 1, "versionNonce": 306371468, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false, "points": [[0, 0], [0.0, 116]], "startBinding": {"elementId": "hpB3ThX83kK6pJN7ZDKm", "mode": "orbit", "fixedPoint": [0.5, 1]}, "endBinding": {"elementId": "hQF4t6KbjGK4E1Jz2Mnz", "mode": "orbit", "fixedPoint": [0.5, 0]}, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false}, {"id": "9JKQQ8ZgNtb8YbfEV4nj", "type": "text", "x": 1512, "y": 346, "width": 143.0, "height": 16.25, "angle": 0, "strokeColor": "#374151", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0050", "roundness": null, "seed": 567264918, "version": 1, "versionNonce": 746125687, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false, "text": "least-loaded replica", "fontSize": 13, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "least-loaded replica", "autoResize": true, "lineHeight": 1.25}, {"id": "Yv3p4ciuu13HVTzzk9Wq", "type": "arrow", "x": 1560.0, "y": 324, "width": 380.0, "height": 116, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0051", "roundness": null, "seed": 1997957568, "version": 1, "versionNonce": 268677786, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false, "points": [[0, 0], [0.0, 76], [380.0, 76], [380.0, 116]], "startBinding": {"elementId": "hpB3ThX83kK6pJN7ZDKm", "mode": "orbit", "fixedPoint": [0.75, 1]}, "endBinding": {"elementId": "nH2LM0ZsRAsh0hhWaaxj", "mode": "orbit", "fixedPoint": [0.3333333333333333, 0]}, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false}, {"id": "xem2I0V5tZZZQCHcUdD4", "type": "arrow", "x": 900, "y": 482.0, "width": 240, "height": 1, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0052", "roundness": null, "seed": 525916986, "version": 1, "versionNonce": 339912868, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false, "points": [[0, 0], [-240, 0.0]], "startBinding": {"elementId": "ymNWPddOOfc292MetsB4", "mode": "orbit", "fixedPoint": [0, 0.5]}, "endBinding": {"elementId": "gopxl6ZNmYmgN9esaQgR", "mode": "orbit", "fixedPoint": [1, 0.5]}, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false}, {"id": "MOxpnmOZ9Fembm6zGDE5", "type": "text", "x": 680, "y": 456, "width": 214.50000000000003, "height": 16.25, "angle": 0, "strokeColor": "#374151", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0053", "roundness": null, "seed": 713023220, "version": 1, "versionNonce": 965847803, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false, "text": "5  outputs as sequences finish", "fontSize": 13, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "5  outputs as sequences finish", "autoResize": true, "lineHeight": 1.25}, {"id": "zGZqg9kzx0WeDcwuCO76", "type": "arrow", "x": 540.0, "y": 440, "width": 1, "height": 296, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "dashed", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0054", "roundness": null, "seed": 243875916, "version": 1, "versionNonce": 1455259526, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false, "points": [[0, 0], [0.0, -296]], "startBinding": {"elementId": "gopxl6ZNmYmgN9esaQgR", "mode": "orbit", "fixedPoint": [0.5, 0]}, "endBinding": {"elementId": "CkhlhLtIAHUWj70YC7Yo", "mode": "orbit", "fixedPoint": [0.5, 1]}, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false}, {"id": "AxQtKksuhVZkINuSZVdW", "type": "text", "x": 552, "y": 166, "width": 100.10000000000001, "height": 16.25, "angle": 0, "strokeColor": "#374151", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0055", "roundness": null, "seed": 1400347481, "version": 1, "versionNonce": 592088847, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false, "text": "6  stream back", "fontSize": 13, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "6  stream back", "autoResize": true, "lineHeight": 1.25}, {"id": "bhS67hJeFlY37PSa7iDN", "type": "arrow", "x": 540.0, "y": 524, "width": 1, "height": 116, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "dashed", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0056", "roundness": null, "seed": 733142318, "version": 1, "versionNonce": 543675145, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false, "points": [[0, 0], [0.0, 116]], "startBinding": {"elementId": "gopxl6ZNmYmgN9esaQgR", "mode": "orbit", "fixedPoint": [0.5, 1]}, "endBinding": {"elementId": "nF1D0XfHxScS718iyuRh", "mode": "orbit", "fixedPoint": [0.5, 0]}, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false}, {"id": "4jnz0OnStOgIBvYXFaiC", "type": "text", "x": 552, "y": 546, "width": 28.6, "height": 16.25, "angle": 0, "strokeColor": "#374151", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0057", "roundness": null, "seed": 1529433571, "version": 1, "versionNonce": 1073434660, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false, "text": "emit", "fontSize": 13, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "emit", "autoResize": true, "lineHeight": 1.25}, {"id": "2Djo126jbKPZUqsHoqWE", "type": "arrow", "x": 1860, "y": 302.0, "width": 240, "height": 1, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "dashed", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0058", "roundness": null, "seed": 1814500823, "version": 1, "versionNonce": 8810207, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false, "points": [[0, 0], [-240, 0.0]], "startBinding": {"elementId": "GqqAE4x1qOPzCPpH5kRX", "mode": "orbit", "fixedPoint": [0, 0.7380952380952381]}, "endBinding": {"elementId": "hpB3ThX83kK6pJN7ZDKm", "mode": "orbit", "fixedPoint": [1, 0.7380952380952381]}, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false}, {"id": "uDpH02YJpGJnd7YAnGrP", "type": "text", "x": 1640, "y": 276, "width": 143.0, "height": 16.25, "angle": 0, "strokeColor": "#374151", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0059", "roundness": null, "seed": 608458756, "version": 1, "versionNonce": 764865691, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false, "text": "scale by queue depth", "fontSize": 13, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "scale by queue depth", "autoResize": true, "lineHeight": 1.25}, {"id": "lgRS7qUM0oxbIJQPQsdF", "type": "arrow", "x": 1980.0, "y": 324, "width": 1, "height": 116, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "dashed", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0060", "roundness": null, "seed": 476474826, "version": 1, "versionNonce": 1448067716, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false, "points": [[0, 0], [0.0, 116]], "startBinding": {"elementId": "GqqAE4x1qOPzCPpH5kRX", "mode": "orbit", "fixedPoint": [0.5, 1]}, "endBinding": {"elementId": "nH2LM0ZsRAsh0hhWaaxj", "mode": "orbit", "fixedPoint": [0.5, 0]}, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false}, {"id": "uu4glHRaCq21qeLQqSI5", "type": "text", "x": 1992, "y": 346, "width": 228.8, "height": 16.25, "angle": 0, "strokeColor": "#374151", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0061", "roundness": null, "seed": 1746588246, "version": 1, "versionNonce": 1197563182, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false, "text": "cold start: GPUs, weights, ready", "fontSize": 13, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "cold start: GPUs, weights, ready", "autoResize": true, "lineHeight": 1.25}, {"id": "AscoTU8GBq690DtsPAGI", "type": "arrow", "x": 1500.0, "y": 640, "width": 1, "height": 116, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "dashed", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0062", "roundness": null, "seed": 508560792, "version": 1, "versionNonce": 1732709278, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false, "points": [[0, 0], [0.0, -116]], "startBinding": {"elementId": "cfvIBOFO4VjiCkzgCpzZ", "mode": "orbit", "fixedPoint": [0.5, 0]}, "endBinding": {"elementId": "hQF4t6KbjGK4E1Jz2Mnz", "mode": "orbit", "fixedPoint": [0.5, 1]}, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false}, {"id": "yqoqzp14VefMoak8lsNn", "type": "text", "x": 1512, "y": 600, "width": 150.15, "height": 16.25, "angle": 0, "strokeColor": "#374151", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0063", "roundness": null, "seed": 1183203249, "version": 1, "versionNonce": 2001855727, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false, "text": "promote warm hardware", "fontSize": 13, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "promote warm hardware", "autoResize": true, "lineHeight": 1.25}, {"id": "JGpO8IegGABMPrOPjaLR", "type": "arrow", "x": 1860, "y": 687.0, "width": 240, "height": 5.0, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "dashed", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0064", "roundness": null, "seed": 199338102, "version": 1, "versionNonce": 101211377, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false, "points": [[0, 0], [-240, -5.0]], "startBinding": {"elementId": "22xJIyYPYxi1RPbz1qRc", "mode": "orbit", "fixedPoint": [0, 0.5]}, "endBinding": {"elementId": "cfvIBOFO4VjiCkzgCpzZ", "mode": "orbit", "fixedPoint": [1, 0.5]}, "startArrowhead": null, "endArrowhead": "arrow", "elbowed": false}, {"id": "dezuFFIpYs8sMwBAkwuE", "type": "text", "x": 1640, "y": 656, "width": 157.3, "height": 16.25, "angle": 0, "strokeColor": "#374151", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0065", "roundness": null, "seed": 110600682, "version": 1, "versionNonce": 780277446, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false, "text": "load weights on demand", "fontSize": 13, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "load weights on demand", "autoResize": true, "lineHeight": 1.25}, {"id": "ZP4kkuKlm5dAFJGzuW3t", "type": "text", "x": 900, "y": 780, "width": 1458.6000000000001, "height": 48.75, "angle": 0, "strokeColor": "#b91c1c", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0066", "roundness": null, "seed": 2070209849, "version": 1, "versionNonce": 69636070, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false, "text": "Failure: a replica dies mid-batch \u2192 the request\u2192batch map re-enqueues those requests at the head of another replica's queue; the client's idempotency key prevents a double answer.\nCold start: a request for a model with no live replica \u2192 scheduler takes warm hardware (up, empty), streams weights from the weights store (~1 TB, minutes), marks the replica ready, router starts sending.\nKeep hardware warm, never the weights of unused models: pinned idle weights block another model from those GPUs.", "fontSize": 13, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "Failure: a replica dies mid-batch \u2192 the request\u2192batch map re-enqueues those requests at the head of another replica's queue; the client's idempotency key prevents a double answer.\nCold start: a request for a model with no live replica \u2192 scheduler takes warm hardware (up, empty), streams weights from the weights store (~1 TB, minutes), marks the replica ready, router starts sending.\nKeep hardware warm, never the weights of unused models: pinned idle weights block another model from those GPUs.", "autoResize": true, "lineHeight": 1.25}, {"id": "ztJGIIP5nQyXLzyMPHbu", "type": "text", "x": 420, "y": 900, "width": 52.800000000000004, "height": 20.0, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0067", "roundness": null, "seed": 2113479152, "version": 1, "versionNonce": 1565864877, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false, "text": "Legend", "fontSize": 16, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "Legend", "autoResize": true, "lineHeight": 1.25}, {"id": "3WX7Lr8LjxcCts0VtkgL", "type": "rectangle", "x": 420, "y": 930, "width": 22, "height": 22, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "#f3f4f6", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0068", "roundness": {"type": 3}, "seed": 137986159, "version": 1, "versionNonce": 106640302, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false}, {"id": "PzwElEVVUJqST4p0Ia4R", "type": "text", "x": 450, "y": 931, "width": 42.900000000000006, "height": 16.25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0069", "roundness": null, "seed": 1146050644, "version": 1, "versionNonce": 434432368, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false, "text": "client", "fontSize": 13, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "client", "autoResize": true, "lineHeight": 1.25}, {"id": "dumcHJQty6Yp3f05zKQq", "type": "rectangle", "x": 650, "y": 930, "width": 22, "height": 22, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "#dbeafe", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0070", "roundness": {"type": 3}, "seed": 1014279157, "version": 1, "versionNonce": 1751726847, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false}, {"id": "d1RQODH1CAoYNyFr3nZG", "type": "text", "x": 680, "y": 931, "width": 71.5, "height": 16.25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0071", "roundness": null, "seed": 1437827814, "version": 1, "versionNonce": 219113284, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false, "text": "front door", "fontSize": 13, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "front door", "autoResize": true, "lineHeight": 1.25}, {"id": "ce7upIXO9jAgMfSv5VQw", "type": "rectangle", "x": 880, "y": 930, "width": 22, "height": 22, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "#ffedd5", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0072", "roundness": {"type": 3}, "seed": 786221928, "version": 1, "versionNonce": 729212370, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false}, {"id": "pIyazWODmKOCIRDnmLVv", "type": "text", "x": 910, "y": 931, "width": 171.60000000000002, "height": 16.25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0073", "roundness": null, "seed": 202637284, "version": 1, "versionNonce": 614814520, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false, "text": "the service under design", "fontSize": 13, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "the service under design", "autoResize": true, "lineHeight": 1.25}, {"id": "jlLVEprzwY9rJfP5AFD8", "type": "rectangle", "x": 1110, "y": 930, "width": 22, "height": 22, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "#fee2e2", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0074", "roundness": {"type": 3}, "seed": 649511027, "version": 1, "versionNonce": 856647729, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false}, {"id": "OyJq7ya5xRh6NElO4NrO", "type": "text", "x": 1140, "y": 931, "width": 171.60000000000002, "height": 16.25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0075", "roundness": null, "seed": 58176518, "version": 1, "versionNonce": 1688675063, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false, "text": "replicas: GPUs + weights", "fontSize": 13, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "replicas: GPUs + weights", "autoResize": true, "lineHeight": 1.25}, {"id": "sVtrWV0WXGse1NZNs7LX", "type": "rectangle", "x": 1340, "y": 930, "width": 22, "height": 22, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "#dcfce7", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0076", "roundness": {"type": 3}, "seed": 514109624, "version": 1, "versionNonce": 2057866541, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false}, {"id": "XomA7N11kJ6z0BJk27Zm", "type": "text", "x": 1370, "y": 931, "width": 185.9, "height": 16.25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0077", "roundness": null, "seed": 1719644660, "version": 1, "versionNonce": 43981234, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false, "text": "storage, registry, metrics", "fontSize": 13, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "storage, registry, metrics", "autoResize": true, "lineHeight": 1.25}, {"id": "IgSoL0il4WsekrG3YQuj", "type": "rectangle", "x": 1570, "y": 930, "width": 22, "height": 22, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "#ede9fe", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0078", "roundness": {"type": 3}, "seed": 1348877894, "version": 1, "versionNonce": 1157677239, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false}, {"id": "25AVlIAOoXwdLoaeuPjv", "type": "text", "x": 1600, "y": 931, "width": 92.95, "height": 16.25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0079", "roundness": null, "seed": 259568492, "version": 1, "versionNonce": 1075982659, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false, "text": "control plane", "fontSize": 13, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "control plane", "autoResize": true, "lineHeight": 1.25}, {"id": "wcoN8zDnlj0EP8jzW4aj", "type": "rectangle", "x": 1800, "y": 930, "width": 22, "height": 22, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "#fef9c3", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0080", "roundness": {"type": 3}, "seed": 820042334, "version": 1, "versionNonce": 1644334855, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false}, {"id": "tr7mUEmJC4eHYkryhvCR", "type": "text", "x": 1830, "y": 931, "width": 64.35000000000001, "height": 16.25, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0081", "roundness": null, "seed": 555661297, "version": 1, "versionNonce": 1999656683, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false, "text": "warm pool", "fontSize": 13, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "warm pool", "autoResize": true, "lineHeight": 1.25}, {"id": "ZuaZAQvMdZqBrWT3o7L5", "type": "text", "x": 420, "y": 980, "width": 1393.7, "height": 87.5, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": "IJcWGxMlUkoLZ7k06nxw", "index": "a0082", "roundness": null, "seed": 1172151286, "version": 1, "versionNonce": 2098879876, "isDeleted": false, "boundElements": [], "updated": 1790457285618, "created": 1790457285618, "link": null, "locked": false, "text": "Numbers to say out loud\nPer replica: batch N \u00f7 batch time. Fixed-batch contract 100 per 100 ms \u2192 1,000 req/s per GPU \u2192 10 GPUs floor at 10k RPS; run at ~65% + spares \u2192 ~18.\nBudget 500 ms P95: \u226450 ms in the batcher + 100 ms on the GPU + tens of ms network, keep ~200 ms for one retry. At 100 RPS a 50 ms window holds five requests: the timer is the lever.\nWeights \u2248 1 TB per frontier model, loaded into GPU memory: serving is memory-bound, a model needs a minimum GPU count, and unused weights pinned in memory block another model.\nEight GPUs, two models: reserve all eight atomically for the big one, drain small batches, age so neither starves, say the idle cost.", "fontSize": 14, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "Numbers to say out loud\nPer replica: batch N \u00f7 batch time. Fixed-batch contract 100 per 100 ms \u2192 1,000 req/s per GPU \u2192 10 GPUs floor at 10k RPS; run at ~65% + spares \u2192 ~18.\nBudget 500 ms P95: \u226450 ms in the batcher + 100 ms on the GPU + tens of ms network, keep ~200 ms for one retry. At 100 RPS a 50 ms window holds five requests: the timer is the lever.\nWeights \u2248 1 TB per frontier model, loaded into GPU memory: serving is memory-bound, a model needs a minimum GPU count, and unused weights pinned in memory block another model.\nEight GPUs, two models: reserve all eight atomically for the big one, drain small batches, age so neither starves, say the idle cost.", "autoResize": true, "lineHeight": 1.25}, {"id": "l9c16hTrF2ExuH3QDViX", "type": "text", "x": -760, "y": -60, "width": 485.1, "height": 87.5, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0083", "roundness": null, "seed": 1195962320, "version": 1, "versionNonce": 569138677, "isDeleted": false, "boundElements": [], "updated": 1790457079551, "created": 1790457079551, "link": null, "locked": false, "text": "Requirements (what the user can do)\nSend a prompt and get tokens back, streaming or in one response\nChoose a model and a version, including older versions\nPaid and free tiers, paid served first\nOne API surface across all models and replicas", "fontSize": 14, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "Requirements (what the user can do)\nSend a prompt and get tokens back, streaming or in one response\nChoose a model and a version, including older versions\nPaid and free tiers, paid served first\nOne API surface across all models and replicas", "autoResize": true, "lineHeight": 1.25}, {"id": "G0VbL86zWJ90vM947QV0", "type": "text", "x": -760, "y": 55.5, "width": 569.8000000000001, "height": 105.0, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0084", "roundness": null, "seed": 1551636961, "version": 1, "versionNonce": 781889195, "isDeleted": false, "boundElements": [], "updated": 1790457079551, "created": 1790457079551, "link": null, "locked": false, "text": "Technical challenges (not 'non-functional')\nKeep GPUs near 100% busy: batch at the replica, size-or-time\nBounded tail latency: P95 500 ms, finished sequences leave the batch early\nCold start for inactive models and versions; minimum GPUs per model\nIsolation: one tenant's huge request must not stall the others\nAvailability over consistency; metrics on queue depth, fill, TTFT, P99", "fontSize": 14, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "Technical challenges (not 'non-functional')\nKeep GPUs near 100% busy: batch at the replica, size-or-time\nBounded tail latency: P95 500 ms, finished sequences leave the batch early\nCold start for inactive models and versions; minimum GPUs per model\nIsolation: one tenant's huge request must not stall the others\nAvailability over consistency; metrics on queue depth, fill, TTFT, P99", "autoResize": true, "lineHeight": 1.25}, {"id": "GoklkWUMwUCAsRVEzxRW", "type": "text", "x": -760, "y": 188.5, "width": 693.0, "height": 105.0, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0085", "roundness": null, "seed": 1438031355, "version": 1, "versionNonce": 2033109964, "isDeleted": false, "boundElements": [], "updated": 1790457079551, "created": 1790457079551, "link": null, "locked": false, "text": "Core entities\nRequest(id, tenant_id, model, version, prompt, max_tokens, stream, idempotency_key, state)\nModel(name); ModelVersion(model, version, min_gpus, weights_ref, state: active|inactive)\nReplica(id, model_version, gpu_ids, state: warming|ready|draining, queue)\nTenant/Limit(tenant_id, tier, rpm, tpm)\nBatch is a property of the replica's queue, not an entity", "fontSize": 14, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "Core entities\nRequest(id, tenant_id, model, version, prompt, max_tokens, stream, idempotency_key, state)\nModel(name); ModelVersion(model, version, min_gpus, weights_ref, state: active|inactive)\nReplica(id, model_version, gpu_ids, state: warming|ready|draining, queue)\nTenant/Limit(tenant_id, tier, rpm, tpm)\nBatch is a property of the replica's queue, not an entity", "autoResize": true, "lineHeight": 1.25}, {"id": "LZLCJRWorFyx8kC5HInK", "type": "text", "x": -760, "y": 321.5, "width": 785.4000000000001, "height": 87.5, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0086", "roundness": null, "seed": 1105175912, "version": 1, "versionNonce": 1037514232, "isDeleted": false, "boundElements": [], "updated": 1790457079551, "created": 1790457079551, "link": null, "locked": false, "text": "API\nPOST /v1/messages {model, version?, prompt, max_tokens, stream} \u2192 200 or SSE stream; 429 + retry_after\nGET  /v1/requests/{id} \u2192 state, tokens so far\nGET  /v1/models \u2192 versions, state, warm replicas\nPOST /internal/replicas/{id}/drain \u00b7 /warm \u00b7 /load {model_version}", "fontSize": 14, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "API\nPOST /v1/messages {model, version?, prompt, max_tokens, stream} \u2192 200 or SSE stream; 429 + retry_after\nGET  /v1/requests/{id} \u2192 state, tokens so far\nGET  /v1/models \u2192 versions, state, warm replicas\nPOST /internal/replicas/{id}/drain \u00b7 /warm \u00b7 /load {model_version}", "autoResize": true, "lineHeight": 1.25}, {"id": "gla2fJylnV6WK0loET1y", "type": "text", "x": -760, "y": 437.0, "width": 677.6, "height": 70.0, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0087", "roundness": null, "seed": 1403747371, "version": 1, "versionNonce": 2103705182, "isDeleted": false, "boundElements": [], "updated": 1790457079551, "created": 1790457079551, "link": null, "locked": false, "text": "Data flow\nauth + tenant limit \u2192 router picks pool by model+version (cold start if none) \u2192\nper-tenant queue \u2192 per-replica batcher (N or 50 ms) \u2192 replica runs prefill then decode \u2192\nfinished sequences return early \u2192 tokens streamed \u2192 metrics emitted", "fontSize": 14, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "Data flow\nauth + tenant limit \u2192 router picks pool by model+version (cold start if none) \u2192\nper-tenant queue \u2192 per-replica batcher (N or 50 ms) \u2192 replica runs prefill then decode \u2192\nfinished sequences return early \u2192 tokens streamed \u2192 metrics emitted", "autoResize": true, "lineHeight": 1.25}, {"id": "iO53KJOOIM1yLNj3t7Z1", "type": "text", "x": -760, "y": 535.0, "width": 654.5, "height": 70.0, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "index": "a0088", "roundness": null, "seed": 471625561, "version": 1, "versionNonce": 49001392, "isDeleted": false, "boundElements": [], "updated": 1790457079551, "created": 1790457079551, "link": null, "locked": false, "text": "Deep dives to offer\nbatcher constants from the budget \u00b7 capacity maths \u00b7 cold start and warm pool \u00b7\nisolation / noisy neighbour \u00b7 a replica dies mid-batch \u00b7 8-GPU two-model scheduling \u00b7\nKV cache and prefix cache: only if the hit rate pays", "fontSize": 14, "baseFontSize": null, "fontFamily": 5, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "Deep dives to offer\nbatcher constants from the budget \u00b7 capacity maths \u00b7 cold start and warm pool \u00b7\nisolation / noisy neighbour \u00b7 a replica dies mid-batch \u00b7 8-GPU two-model scheduling \u00b7\nKV cache and prefix cache: only if the hit rate pays", "autoResize": true, "lineHeight": 1.25}], "appState": {"viewBackgroundColor": "#ffffff", "gridSize": null}, "files": {}}