[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"summary-9c16c4c155dcf489-scaling-tpus-on-gke-for-massive-ai-workloads-summary":3,"summaries-facets-categories":224,"summary-related-9c16c4c155dcf489-scaling-tpus-on-gke-for-massive-ai-workloads-summary":3793},{"id":4,"title":5,"ai":6,"body":13,"categories":198,"created_at":200,"date_modified":200,"description":201,"extension":202,"faq":200,"featured":203,"kicker_label":200,"meta":204,"navigation":205,"path":206,"published_at":207,"question":200,"scraped_at":208,"seo":209,"sitemap":210,"source_id":211,"source_name":212,"source_type":213,"source_url":214,"stem":215,"tags":216,"thumbnail_url":200,"tldr":221,"tweet":200,"unknown_tags":222,"__hash__":223},"summaries\u002Fsummaries\u002F9c16c4c155dcf489-scaling-tpus-on-gke-for-massive-ai-workloads-summary.md","Scaling TPUs on GKE for Massive AI Workloads",{"provider":7,"model":8,"input_tokens":9,"output_tokens":10,"processing_time_ms":11,"cost_usd":12},"openrouter","x-ai\u002Fgrok-4.1-fast",8516,2468,54357,0.0029147,{"type":14,"value":15,"toc":188},"minimark",[16,21,25,28,31,34,38,41,44,67,70,73,76,80,83,97,100,103,106,110,113,116,119,123,126,146,149,152,155,158,162],[17,18,20],"h2",{"id":19},"tpu-power-specialized-hardware-for-ai-matrix-crunching","TPU Power: Specialized Hardware for AI Matrix Crunching",[22,23,24],"p",{},"Kavitha Gowda, product manager for TPUs on GKE, describes TPUs as Google's custom ASICs optimized for machine learning, particularly heavy matrix multiplications in LLMs and recommendation models. The core is the Matrix Multiply Unit (MXU), a \"dedicated matrix math wizard\" that processes billions of operations per image in recognition tasks thousands of times faster than general-purpose chips.",[22,26,27],{},"TPUs feature high-bandwidth memory (HBM) to handle large models and batches on-chip, minimizing data transfer bottlenecks. They interconnect from one chip to thousands via high-speed ICI links and optical circuit switching, enabling massive-scale training and inference. The seventh-generation Ironwood TPU pod supports 9,216 chips, with peak BF16 TFLOPS jumping dramatically—numbers Yufeng Guo initially mistook for typos due to the leap from prior generations like Trillium and v5e.",[22,29,30],{},"\"MXU is the hardware that makes TPUs so powerful. It's dedicated matrix math wizard that can perform this massive calculation in a single step, making the entire process thousands times faster and more efficient than a general-purpose chip,\" Gowda explains, highlighting the specialized architecture.",[22,32,33],{},"Frameworks like JAX, TensorFlow, and PyTorch are fully supported, integrating seamlessly with GKE, Vertex AI, and Cloud TPU APIs.",[17,35,37],{"id":36},"gkes-atomic-slicing-hiding-complexity-for-exponential-scale","GKE's Atomic Slicing: Hiding Complexity for Exponential Scale",[22,39,40],{},"GKE abstracts TPU chip intricacies, exposing them as containerized workloads while preserving Kubernetes advantages. It treats TPU 'slices'—from single chips to 9,216-chip pods—as atomic units for provisioning, scheduling, failover, and resilience, maximizing interconnect performance.",[22,42,43],{},"Slice types scale progressively:",[45,46,47,55,61],"ul",{},[48,49,50,54],"li",{},[51,52,53],"strong",{},"Single-host TPU",": One VM with 1-8 chips at zero network latency, ideal for fine-tuning, interactive dev, or small inference. Scales like CPU VMs via horizontal pod autoscaling.",[48,56,57,60],{},[51,58,59],{},"Multi-host TPU",": Multiple VMs (e.g., 16 VMs with 4 chips each for 64 chips) in one node pool, interconnected via ICI for larger training\u002Finference.",[48,62,63,66],{},[51,64,65],{},"Multi-slice TPU",": Spans node pools (e.g., 50k-100k chips), with intra-pool ICI links and inter-pool data center networking. Developers must align workloads to high-speed (ICI) vs. slower (DCN) paths.",[22,68,69],{},"GKE supports 130k nodes, enabling thousands of TPUs as one unit for frontier models. JobSets and multi-slice networking provide atomic failover: if one VM fails in a 50k-chip slice, GKE auto-repairs the unit and resumes training, boosting 'goodput' (effective throughput) over raw throughput.",[22,71,72],{},"\"GKE hides the underlying complexity of the chip architecture and relays the TPU chip power to the container-based workloads,\" Gowda notes, emphasizing ecosystem perks like storage, load balancers, and observability.",[22,74,75],{},"Yufeng Guo stresses software-hardware co-design: \"We're really seeing this combination of having to have knowledge of the software as well as the hardware in order to be able to take full advantage of these systems.\"",[17,77,79],{"id":78},"capacity-flexibility-dws-cuds-and-spot-for-cost-control","Capacity Flexibility: DWS, CUDs, and Spot for Cost Control",[22,81,82],{},"TPU availability spans options for reliability and economy:",[45,84,85,91],{},[48,86,87,90],{},[51,88,89],{},"Committed Use Discounts (CUDs)",": Reserved capacity for enterprise needs, from massive training to online inference.",[48,92,93,96],{},[51,94,95],{},"Dynamic Workload Scheduler (DWS)",": New in 2025, with Flex (pay-as-you-go, up to 7 days for bursty POCs\u002Fexperiments) and Calendar (1-3 month reservations for guaranteed, uninterrupted runs).",[22,98,99],{},"GKE autoscales DWS Flex node pools only when workloads deploy, billing solely during execution—scale down post-job for zero idle costs. Calendar ensures dedicated, compact placement without maintenance interruptions, vital for month-long fine-tuning where failures would be \"crippling,\" as Guo observes.",[22,101,102],{},"Combine modes: Reserve Calendar for critical jobs, burst to Flex. All backed by on-demand and spot.",[22,104,105],{},"\"DWS Flex is like an on-demand elasticity... Mostly used for bursty workloads, for experimentation, for POCs... you just pay for what you're running,\" Gowda clarifies.",[17,107,109],{"id":108},"custom-compute-classes-automated-fallbacks-across-tiers","Custom Compute Classes: Automated Fallbacks Across Tiers",[22,111,112],{},"Custom compute classes define prioritized hierarchies (e.g., Trillium reservation > spot > DWS Flex > on-demand). GKE automatically falls back if primary capacity lacks, promoting to higher tiers when available—optimizing for power, cost, or availability.",[22,114,115],{},"Users previously scripted this; now it's native, with GCP optimizing efficiency. Supports 3+ layers (latency trade-offs apply) and even GPU\u002FTPU fallback via vLLM for serving. Example: Start TPU reservations, scale to GPUs.",[22,117,118],{},"\"With custom compute classes, you can define prioritized hierarchy of TPU configuration... GKE can automatically fall back,\" Gowda says, noting use for low-priority jobs starting on spot then escalating.",[17,120,122],{"id":121},"storage-and-ecosystem-fueling-data-intensive-workloads","Storage and Ecosystem: Fueling Data-Intensive Workloads",[22,124,125],{},"GKE optimizes AI I\u002FO:",[45,127,128,134,140],{},[48,129,130,133],{},[51,131,132],{},"Secondary boot disks",": Preload data\u002Fimages per node for faster pod startup.",[48,135,136,139],{},[51,137,138],{},"GCS Fuse + CSI driver",": Caches\u002Fparallel-downloads from object storage, yielding 9x faster model loads via PersistentVolumeClaims.",[48,141,142,145],{},[51,143,144],{},"Managed Lustre",": Parallel filesystem for high-concurrency IO in training\u002Fcheckpointing.",[22,147,148],{},"Integrates open-source like Kubray (orchestrator) and vLLM (serving), plus dashboards.",[22,150,151],{},"Companies like Anthropic, Moloco, and Light Tricks already use Kubernetes+TPUs.",[22,153,154],{},"Resources: Google AI Hypercomputer, GKE for AI\u002FML inference docs, TPU-on-GKE LLM fine-tuning tutorial.",[22,156,157],{},"\"By leveraging GKE's job set and multi-slice networking, you gain an atomic failover model... helps you resume your training if one infrastructure fails,\" Gowda adds on maximizing expensive TPU utilization.",[17,159,161],{"id":160},"key-takeaways","Key Takeaways",[45,163,164,167,170,173,176,179,182,185],{},[48,165,166],{},"Treat TPU slices as atomic units in GKE for provisioning up to 9k+ interconnected chips, aligning workloads to ICI (intra-pool) vs. DCN (inter-pool) speeds.",[48,168,169],{},"Use DWS Flex for bursty experiments (pay-as-you-go, autoscaling) and Calendar for 1-3 month guaranteed reservations to avoid crippling mid-training failures.",[48,171,172],{},"Implement custom compute classes for automatic fallbacks (e.g., reservation > spot > Flex) to optimize cost\u002Favailability without custom scripts.",[48,174,175],{},"Accelerate startup with secondary boot disks, GCS Fuse (9x model load speedup), and Managed Lustre for high-IO training.",[48,177,178],{},"Co-design software for TPU hardware: Leverage MXU\u002FHBM for matrix-heavy LLMs, scale via single\u002Fmulti-host\u002Fslices.",[48,180,181],{},"Combine CUDs for steady-state with DWS\u002Fspot for bursts; fallback to GPUs via vLLM for serving resilience.",[48,183,184],{},"Maximize goodput with GKE JobSets' atomic failover and auto-resume on VM failures.",[48,186,187],{},"Start with Ironwood\u002FTrillium pods on GKE for JAX\u002FTF\u002FPyTorch; reference tutorials for LLM fine-tuning.",{"title":189,"searchDepth":190,"depth":190,"links":191},"",2,[192,193,194,195,196,197],{"id":19,"depth":190,"text":20},{"id":36,"depth":190,"text":37},{"id":78,"depth":190,"text":79},{"id":108,"depth":190,"text":109},{"id":121,"depth":190,"text":122},{"id":160,"depth":190,"text":161},[199],"DevOps & Cloud",null,"Google AI Hypercomputer → https:\u002F\u002Fgoo.gle\u002F3ObrQLK  \nGKE for AI\u002FML inference → https:\u002F\u002Fgoo.gle\u002F4cg4k8y  \n[Tutorial] Fine tune a LLM using TPUs on GKE → https:\u002F\u002Fgoo.gle\u002F48hT4Hu\n\nTensor Processing Units (TPUs) are now in their 7th generation. They allow machine learning workloads to reach massive scale, especially when running on Google Kubernetes Engine (GKE). But how does that work, and what do you need to know in order to run TPUs on GKE successfully? \n\nJoin Yufeng Guo as he sits down with Kavitha Gowda, the product manager of TPUs on GKE, to get into the details of how to scale TPU workloads on GKE.\n\nSpeakers: Yufeng Guo, Kavitha Gowda\nProducts Mentioned: Google Kubernetes Engine, Cloud Tensor Processing Units, AI Hypercomputer","md",false,{},true,"\u002Fsummaries\u002F9c16c4c155dcf489-scaling-tpus-on-gke-for-massive-ai-workloads-summary","2026-04-09 19:00:41","2026-04-10 03:09:44",{"title":5,"description":201},{"loc":206},"9c16c4c155dcf489","Google Cloud Tech","video","https:\u002F\u002Fwww.youtube.com\u002Fwatch?v=coP5_SmE4AI","summaries\u002F9c16c4c155dcf489-scaling-tpus-on-gke-for-massive-ai-workloads-summary",[217,218,219,220],"machine-learning","devops","cloud","kubernetes","GKE treats TPU slices as atomic units for seamless scaling up to 9k+ chips, with flexible capacity like DWS Flex\u002FCalendar and custom fallbacks for cost-efficient ML training\u002Finference.",[220],"a3lw8W4rx5X2n-REtU3W0luTS1Z52UcKBmv5OJIrpAI",[225,228,231,234,237,240,242,244,246,248,250,252,255,257,259,261,263,265,267,269,271,273,276,279,281,283,286,288,290,293,295,297,299,301,303,305,307,309,311,313,315,317,319,321,323,325,327,329,331,333,335,337,339,341,343,345,347,349,351,353,355,357,359,361,363,365,367,369,371,373,375,377,379,381,383,385,387,389,391,393,395,397,399,401,403,405,407,409,411,413,415,417,419,421,423,425,427,429,431,433,435,437,439,441,443,445,447,449,451,453,455,457,459,461,463,465,467,469,471,473,475,477,479,481,483,485,487,489,491,493,495,497,499,501,503,505,507,509,511,513,515,517,519,521,523,525,527,529,531,533,535,537,539,541,543,545,547,549,551,553,555,557,559,561,563,565,567,569,571,573,575,577,579,581,583,585,587,589,591,593,595,597,599,601,603,605,607,609,611,613,615,617,619,621,623,625,627,629,631,633,635,637,639,641,643,645,647,649,651,653,655,657,659,661,663,665,667,669,671,673,675,677,679,681,683,685,687,689,691,693,695,697,699,701,703,705,707,709,711,713,715,717,719,721,723,725,727,729,731,733,735,737,739,741,743,745,747,749,751,753,755,757,759,761,763,765,767,769,771,773,775,777,779,781,783,785,787,789,791,793,795,797,799,801,803,805,807,809,811,813,815,817,819,821,823,825,827,829,831,833,835,837,839,841,843,845,847,849,851,853,855,857,859,861,863,865,867,869,871,873,875,877,879,881,883,885,887,889,891,893,895,897,899,901,903,905,907,909,911,913,915,917,919,921,923,925,927,929,931,933,935,937,939,941,943,945,947,949,951,953,955,957,959,961,963,965,967,969,971,973,975,977,979,981,983,985,987,989,991,993,995,997,999,1001,1003,1005,1007,1009,1011,1013,1015,1017,1019,1021,1023,1025,1027,1029,1031,1033,1035,1037,1039,1041,1043,1045,1047,1049,1051,1053,1055,1057,1059,1061,1063,1065,1067,1069,1071,1073,1075,1077,1079,1081,1083,1085,1087,1089,1091,1093,1095,1097,1099,1101,1103,1105,1107,1109,1111,1113,1115,1117,1119,1121,1123,1125,1127,1129,1131,1133,1135,1137,1139,1141,1143,1145,1147,1149,1151,1153,1155,1157,1159,1161,1163,1165,1167,1169,1171,1173,1175,1177,1179,1181,1183,1185,1187,1189,1191,1193,1195,1197,1199,1201,1203,1205,1207,1209,1211,1213,1215,1217,1219,1221,1223,1225,1227,1229,1231,1233,1235,1237,1239,1241,1243,1245,1247,1249,1251,1253,1255,1257,1259,1261,1263,1265,1267,1269,1271,1273,1275,1277,1279,1281,1283,1285,1287,1289,1291,1293,1295,1297,1299,1301,1303,1305,1307,1309,1311,1313,1315,1317,1319,1321,1323,1325,1327,1329,1331,1333,1335,1337,1339,1341,1343,1345,1347,1349,1351,1353,1355,1357,1359,1361,1363,1365,1367,1369,1371,1373,1375,1377,1379,1381,1383,1385,1387,1389,1391,1393,1395,1397,1399,1401,1403,1405,1407,1409,1411,1413,1415,1417,1419,1421,1423,1425,1427,1429,1431,1433,1435,1437,1439,1441,1443,1445,1447,1449,1451,1453,1455,1457,1459,1461,1463,1465,1467,1469,1471,1473,1475,1477,1479,1481,1483,1485,1487,1489,1491,1493,1495,1497,1499,1501,1503,1505,1507,1509,1511,1513,1515,1517,1519,1521,1523,1525,1527,1529,1531,1533,1535,1537,1539,1541,1543,1545,1547,1549,1551,1553,1555,1557,1559,1561,1563,1565,1567,1569,1571,1573,1575,1577,1579,1581,1583,1585,1587,1589,1591,1593,1595,1597,1599,1601,1603,1605,1607,1609,1611,1613,1615,1617,1619,1621,1623,1625,1627,1629,1631,1633,1635,1637,1639,1641,1643,1645,1647,1649,1651,1653,1655,1657,1659,1661,1663,1665,1667,1669,1671,1673,1675,1677,1679,1681,1683,1685,1687,1689,1691,1693,1695,1697,1699,1701,1703,1705,1707,1709,1711,1713,1715,1717,1719,1721,1723,1725,1727,1729,1731,1733,1735,1737,1739,1741,1743,1745,1747,1749,1751,1753,1755,1757,1759,1761,1763,1765,1767,1769,1771,1773,1775,1777,1779,1781,1783,1785,1787,1789,1791,1793,1795,1797,1799,1801,1803,1805,1807,1809,1811,1813,1815,1817,1819,1821,1823,1825,1827,1829,1831,1833,1835,1837,1839,1841,1843,1845,1847,1849,1851,1853,1855,1857,1859,1861,1863,1865,1867,1869,1871,1873,1875,1877,1879,1881,1883,1885,1887,1889,1891,1893,1895,1897,1899,1901,1903,1905,1907,1909,1911,1913,1915,1917,1919,1921,1923,1925,1927,1929,1931,1933,1935,1937,1939,1941,1943,1945,1947,1949,1951,1953,1955,1957,1959,1961,1963,1965,1967,1969,1971,1973,1975,1977,1979,1981,1983,1985,1987,1989,1991,1993,1995,1997,1999,2001,2003,2005,2007,2009,2011,2013,2015,2017,2019,2021,2023,2025,2027,2029,2031,2033,2035,2037,2039,2041,2043,2045,2047,2049,2051,2053,2055,2057,2059,2061,2063,2065,2067,2069,2071,2073,2075,2077,2079,2081,2083,2085,2087,2089,2091,2093,2095,2097,2099,2101,2103,2105,2107,2109,2111,2113,2115,2117,2119,2121,2123,2125,2127,2129,2131,2133,2135,2137,2139,2141,2143,2145,2147,2149,2151,2153,2155,2157,2159,2161,2163,2165,2167,2169,2171,2173,2175,2177,2179,2181,2183,2185,2187,2189,2191,2193,2195,2197,2199,2201,2203,2205,2207,2209,2211,2213,2215,2217,2219,2221,2223,2225,2227,2229,2231,2233,2235,2237,2239,2241,2243,2245,2247,2249,2251,2253,2255,2257,2259,2261,2263,2265,2267,2269,2271,2273,2275,2277,2279,2281,2283,2285,2287,2289,2291,2293,2295,2297,2299,2301,2303,2305,2307,2309,2311,2313,2315,2317,2319,2321,2323,2325,2327,2329,2331,2333,2335,2337,2339,2341,2343,2345,2347,2349,2351,2353,2355,2357,2359,2361,2363,2365,2367,2369,2371,2373,2375,2377,2379,2381,2383,2385,2387,2389,2391,2393,2395,2397,2399,2401,2403,2405,2407,2409,2411,2413,2415,2417,2419,2421,2423,2425,2427,2429,2431,2433,2435,2437,2439,2441,2443,2445,2447,2449,2451,2453,2455,2457,2459,2461,2463,2465,2467,2469,2471,2473,2475,2477,2479,2481,2483,2485,2487,2489,2491,2493,2495,2497,2499,2501,2503,2505,2507,2509,2511,2513,2515,2517,2519,2521,2523,2525,2527,2529,2531,2533,2535,2537,2539,2541,2543,2545,2547,2549,2551,2553,2555,2557,2559,2561,2563,2565,2567,2569,2571,2573,2575,2577,2579,2581,2583,2585,2587,2589,2591,2593,2595,2597,2599,2601,2603,2605,2607,2609,2611,2613,2615,2617,2619,2621,2623,2625,2627,2629,2631,2633,2635,2637,2639,2641,2643,2645,2647,2649,2651,2653,2655,2657,2659,2661,2663,2665,2667,2669,2671,2673,2675,2677,2679,2681,2683,2685,2687,2689,2691,2693,2695,2697,2699,2701,2703,2705,2707,2709,2711,2713,2715,2717,2719,2721,2723,2725,2727,2729,2731,2733,2735,2737,2739,2741,2743,2745,2747,2749,2751,2753,2755,2757,2759,2761,2763,2765,2767,2769,2771,2773,2775,2777,2779,2781,2783,2785,2787,2789,2791,2793,2795,2797,2799,2801,2803,2805,2807,2809,2811,2813,2815,2817,2819,2821,2823,2825,2827,2829,2831,2833,2835,2837,2839,2841,2843,2845,2847,2849,2851,2853,2855,2857,2859,2861,2863,2865,2867,2869,2871,2873,2875,2877,2879,2881,2883,2885,2887,2889,2891,2893,2895,2897,2899,2901,2903,2905,2907,2909,2911,2913,2915,2917,2919,2921,2923,2925,2927,2929,2931,2933,2935,2937,2939,2941,2943,2945,2947,2949,2951,2953,2955,2957,2959,2961,2963,2965,2967,2969,2971,2973,2975,2977,2979,2981,2983,2985,2987,2989,2991,2993,2995,2997,2999,3001,3003,3005,3007,3009,3011,3013,3015,3017,3019,3021,3023,3025,3027,3029,3031,3033,3035,3037,3039,3041,3043,3045,3047,3049,3051,3053,3055,3057,3059,3061,3063,3065,3067,3069,3071,3073,3075,3077,3079,3081,3083,3085,3087,3089,3091,3093,3095,3097,3099,3101,3103,3105,3107,3109,3111,3113,3115,3117,3119,3121,3123,3125,3127,3129,3131,3133,3135,3137,3139,3141,3143,3145,3147,3149,3151,3153,3155,3157,3159,3161,3163,3165,3167,3169,3171,3173,3175,3177,3179,3181,3183,3185,3187,3189,3191,3193,3195,3197,3199,3201,3203,3205,3207,3209,3211,3213,3215,3217,3219,3221,3223,3225,3227,3229,3231,3233,3235,3237,3239,3241,3243,3245,3247,3249,3251,3253,3255,3257,3259,3261,3263,3265,3267,3269,3271,3273,3275,3277,3279,3281,3283,3285,3287,3289,3291,3293,3295,3297,3299,3301,3303,3305,3307,3309,3311,3313,3315,3317,3319,3321,3323,3325,3327,3329,3331,3333,3335,3337,3339,3341,3343,3345,3347,3349,3351,3353,3355,3357,3359,3361,3363,3365,3367,3369,3371,3373,3375,3377,3379,3381,3383,3385,3387,3389,3391,3393,3395,3397,3399,3401,3403,3405,3407,3409,3411,3413,3415,3417,3419,3421,3423,3425,3427,3429,3431,3433,3435,3437,3439,3441,3443,3445,3447,3449,3451,3453,3455,3457,3459,3461,3463,3465,3467,3469,3471,3473,3475,3477,3479,3481,3483,3485,3487,3489,3491,3493,3495,3497,3499,3501,3503,3505,3507,3509,3511,3513,3515,3517,3519,3521,3523,3525,3527,3529,3531,3533,3535,3537,3539,3541,3543,3545,3547,3549,3551,3553,3555,3557,3559,3561,3563,3565,3567,3569,3571,3573,3575,3577,3579,3581,3583,3585,3587,3589,3591,3593,3595,3597,3599,3601,3603,3605,3607,3609,3611,3613,3615,3617,3619,3621,3623,3625,3627,3629,3631,3633,3635,3637,3639,3641,3643,3645,3647,3649,3651,3653,3655,3657,3659,3661,3663,3665,3667,3669,3671,3673,3675,3677,3679,3681,3683,3685,3687,3689,3691,3693,3695,3697,3699,3701,3703,3705,3707,3709,3711,3713,3715,3717,3719,3721,3723,3725,3727,3729,3731,3733,3735,3737,3739,3741,3743,3745,3747,3749,3751,3753,3755,3757,3759,3761,3763,3765,3767,3769,3771,3773,3775,3777,3779,3781,3783,3785,3787,3789,3791],{"categories":226},[227],"Developer Productivity",{"categories":229},[230],"Business & SaaS",{"categories":232},[233],"AI & LLMs",{"categories":235},[236],"AI Automation",{"categories":238},[239],"Product Strategy",{"categories":241},[233],{"categories":243},[227],{"categories":245},[230],{"categories":247},[],{"categories":249},[233],{"categories":251},[],{"categories":253},[254],"AI News & Trends",{"categories":256},[236],{"categories":258},[254],{"categories":260},[236],{"categories":262},[236],{"categories":264},[233],{"categories":266},[233],{"categories":268},[254],{"categories":270},[233],{"categories":272},[],{"categories":274},[275],"Design & Frontend",{"categories":277},[278],"Data Science & Visualization",{"categories":280},[254],{"categories":282},[],{"categories":284},[285],"Software Engineering",{"categories":287},[233],{"categories":289},[236],{"categories":291},[292],"Marketing & Growth",{"categories":294},[233],{"categories":296},[236],{"categories":298},[],{"categories":300},[],{"categories":302},[275],{"categories":304},[236],{"categories":306},[227],{"categories":308},[275],{"categories":310},[233],{"categories":312},[236],{"categories":314},[254],{"categories":316},[],{"categories":318},[],{"categories":320},[236],{"categories":322},[285],{"categories":324},[],{"categories":326},[230],{"categories":328},[],{"categories":330},[],{"categories":332},[236],{"categories":334},[236],{"categories":336},[233],{"categories":338},[],{"categories":340},[285],{"categories":342},[],{"categories":344},[],{"categories":346},[],{"categories":348},[233],{"categories":350},[292],{"categories":352},[275],{"categories":354},[275],{"categories":356},[233],{"categories":358},[236],{"categories":360},[233],{"categories":362},[233],{"categories":364},[236],{"categories":366},[236],{"categories":368},[278],{"categories":370},[254],{"categories":372},[236],{"categories":374},[292],{"categories":376},[236],{"categories":378},[239],{"categories":380},[],{"categories":382},[236],{"categories":384},[],{"categories":386},[236],{"categories":388},[285],{"categories":390},[275],{"categories":392},[233],{"categories":394},[],{"categories":396},[],{"categories":398},[236],{"categories":400},[],{"categories":402},[233],{"categories":404},[],{"categories":406},[227],{"categories":408},[285],{"categories":410},[230],{"categories":412},[254],{"categories":414},[233],{"categories":416},[],{"categories":418},[233],{"categories":420},[],{"categories":422},[285],{"categories":424},[278],{"categories":426},[],{"categories":428},[233],{"categories":430},[275],{"categories":432},[],{"categories":434},[275],{"categories":436},[236],{"categories":438},[],{"categories":440},[236],{"categories":442},[254],{"categories":444},[233],{"categories":446},[],{"categories":448},[236],{"categories":450},[233],{"categories":452},[239],{"categories":454},[],{"categories":456},[233],{"categories":458},[236],{"categories":460},[236],{"categories":462},[],{"categories":464},[278],{"categories":466},[233],{"categories":468},[],{"categories":470},[227],{"categories":472},[230],{"categories":474},[233],{"categories":476},[236],{"categories":478},[285],{"categories":480},[233],{"categories":482},[],{"categories":484},[],{"categories":486},[233],{"categories":488},[],{"categories":490},[275],{"categories":492},[],{"categories":494},[233],{"categories":496},[],{"categories":498},[236],{"categories":500},[233],{"categories":502},[275],{"categories":504},[],{"categories":506},[233],{"categories":508},[233],{"categories":510},[230],{"categories":512},[236],{"categories":514},[233],{"categories":516},[275],{"categories":518},[236],{"categories":520},[],{"categories":522},[],{"categories":524},[254],{"categories":526},[],{"categories":528},[233],{"categories":530},[230,292],{"categories":532},[],{"categories":534},[233],{"categories":536},[],{"categories":538},[],{"categories":540},[233],{"categories":542},[],{"categories":544},[233],{"categories":546},[199],{"categories":548},[],{"categories":550},[254],{"categories":552},[275],{"categories":554},[],{"categories":556},[254],{"categories":558},[254],{"categories":560},[233],{"categories":562},[292],{"categories":564},[],{"categories":566},[230],{"categories":568},[],{"categories":570},[233,199],{"categories":572},[233],{"categories":574},[233],{"categories":576},[236],{"categories":578},[233,285],{"categories":580},[278],{"categories":582},[233],{"categories":584},[292],{"categories":586},[236],{"categories":588},[236],{"categories":590},[],{"categories":592},[236],{"categories":594},[233,230],{"categories":596},[],{"categories":598},[275],{"categories":600},[275],{"categories":602},[],{"categories":604},[],{"categories":606},[254],{"categories":608},[],{"categories":610},[227],{"categories":612},[285],{"categories":614},[233],{"categories":616},[275],{"categories":618},[236],{"categories":620},[285],{"categories":622},[254],{"categories":624},[275],{"categories":626},[],{"categories":628},[233],{"categories":630},[233],{"categories":632},[233],{"categories":634},[254],{"categories":636},[227],{"categories":638},[233],{"categories":640},[236],{"categories":642},[199],{"categories":644},[275],{"categories":646},[236],{"categories":648},[],{"categories":650},[],{"categories":652},[275],{"categories":654},[254],{"categories":656},[278],{"categories":658},[],{"categories":660},[233],{"categories":662},[233],{"categories":664},[230],{"categories":666},[233],{"categories":668},[233],{"categories":670},[254],{"categories":672},[],{"categories":674},[236],{"categories":676},[285],{"categories":678},[],{"categories":680},[233],{"categories":682},[233],{"categories":684},[236],{"categories":686},[],{"categories":688},[],{"categories":690},[233],{"categories":692},[],{"categories":694},[230],{"categories":696},[236],{"categories":698},[],{"categories":700},[227],{"categories":702},[233],{"categories":704},[230],{"categories":706},[254],{"categories":708},[],{"categories":710},[],{"categories":712},[],{"categories":714},[254],{"categories":716},[254],{"categories":718},[],{"categories":720},[],{"categories":722},[230],{"categories":724},[],{"categories":726},[],{"categories":728},[227],{"categories":730},[],{"categories":732},[292],{"categories":734},[236],{"categories":736},[230],{"categories":738},[236],{"categories":740},[],{"categories":742},[239],{"categories":744},[275],{"categories":746},[285],{"categories":748},[233],{"categories":750},[236],{"categories":752},[230],{"categories":754},[233],{"categories":756},[],{"categories":758},[],{"categories":760},[285],{"categories":762},[278],{"categories":764},[239],{"categories":766},[236],{"categories":768},[233],{"categories":770},[],{"categories":772},[199],{"categories":774},[],{"categories":776},[236],{"categories":778},[],{"categories":780},[],{"categories":782},[233],{"categories":784},[275],{"categories":786},[292],{"categories":788},[236],{"categories":790},[],{"categories":792},[227],{"categories":794},[],{"categories":796},[254],{"categories":798},[233,199],{"categories":800},[254],{"categories":802},[233],{"categories":804},[230],{"categories":806},[233],{"categories":808},[],{"categories":810},[230],{"categories":812},[],{"categories":814},[285],{"categories":816},[275],{"categories":818},[254],{"categories":820},[278],{"categories":822},[227],{"categories":824},[233],{"categories":826},[285],{"categories":828},[],{"categories":830},[],{"categories":832},[239],{"categories":834},[],{"categories":836},[233],{"categories":838},[],{"categories":840},[275],{"categories":842},[275],{"categories":844},[275],{"categories":846},[],{"categories":848},[],{"categories":850},[254],{"categories":852},[236],{"categories":854},[233],{"categories":856},[233],{"categories":858},[233],{"categories":860},[230],{"categories":862},[233],{"categories":864},[],{"categories":866},[285],{"categories":868},[285],{"categories":870},[230],{"categories":872},[],{"categories":874},[233],{"categories":876},[233],{"categories":878},[230],{"categories":880},[254],{"categories":882},[292],{"categories":884},[236],{"categories":886},[],{"categories":888},[275],{"categories":890},[],{"categories":892},[233],{"categories":894},[],{"categories":896},[230],{"categories":898},[236],{"categories":900},[],{"categories":902},[199],{"categories":904},[278],{"categories":906},[285],{"categories":908},[292],{"categories":910},[285],{"categories":912},[236],{"categories":914},[],{"categories":916},[],{"categories":918},[236],{"categories":920},[227],{"categories":922},[236],{"categories":924},[239],{"categories":926},[230],{"categories":928},[],{"categories":930},[233],{"categories":932},[239],{"categories":934},[233],{"categories":936},[233],{"categories":938},[292],{"categories":940},[275],{"categories":942},[236],{"categories":944},[],{"categories":946},[],{"categories":948},[199],{"categories":950},[285],{"categories":952},[],{"categories":954},[236],{"categories":956},[233],{"categories":958},[275,233],{"categories":960},[227],{"categories":962},[],{"categories":964},[233],{"categories":966},[227],{"categories":968},[275],{"categories":970},[236],{"categories":972},[285],{"categories":974},[],{"categories":976},[233],{"categories":978},[],{"categories":980},[227],{"categories":982},[],{"categories":984},[236],{"categories":986},[239],{"categories":988},[233],{"categories":990},[233],{"categories":992},[275],{"categories":994},[236],{"categories":996},[199],{"categories":998},[275],{"categories":1000},[236],{"categories":1002},[233],{"categories":1004},[233],{"categories":1006},[233],{"categories":1008},[254],{"categories":1010},[],{"categories":1012},[239],{"categories":1014},[236],{"categories":1016},[275],{"categories":1018},[236],{"categories":1020},[285],{"categories":1022},[275],{"categories":1024},[236],{"categories":1026},[254],{"categories":1028},[],{"categories":1030},[233],{"categories":1032},[275],{"categories":1034},[233],{"categories":1036},[227],{"categories":1038},[254],{"categories":1040},[233],{"categories":1042},[292],{"categories":1044},[233],{"categories":1046},[233],{"categories":1048},[236],{"categories":1050},[236],{"categories":1052},[233],{"categories":1054},[236],{"categories":1056},[275],{"categories":1058},[233],{"categories":1060},[],{"categories":1062},[],{"categories":1064},[285],{"categories":1066},[],{"categories":1068},[227],{"categories":1070},[199],{"categories":1072},[],{"categories":1074},[227],{"categories":1076},[230],{"categories":1078},[292],{"categories":1080},[],{"categories":1082},[230],{"categories":1084},[],{"categories":1086},[],{"categories":1088},[],{"categories":1090},[],{"categories":1092},[],{"categories":1094},[233],{"categories":1096},[236],{"categories":1098},[199],{"categories":1100},[227],{"categories":1102},[233],{"categories":1104},[285],{"categories":1106},[239],{"categories":1108},[233],{"categories":1110},[292],{"categories":1112},[233],{"categories":1114},[233],{"categories":1116},[233],{"categories":1118},[233,227],{"categories":1120},[285],{"categories":1122},[285],{"categories":1124},[275],{"categories":1126},[233],{"categories":1128},[],{"categories":1130},[],{"categories":1132},[],{"categories":1134},[285],{"categories":1136},[278],{"categories":1138},[254],{"categories":1140},[275],{"categories":1142},[],{"categories":1144},[233],{"categories":1146},[233],{"categories":1148},[],{"categories":1150},[],{"categories":1152},[236],{"categories":1154},[233],{"categories":1156},[230],{"categories":1158},[],{"categories":1160},[227],{"categories":1162},[233],{"categories":1164},[227],{"categories":1166},[233],{"categories":1168},[285],{"categories":1170},[292],{"categories":1172},[233,275],{"categories":1174},[254],{"categories":1176},[275],{"categories":1178},[],{"categories":1180},[199],{"categories":1182},[275],{"categories":1184},[236],{"categories":1186},[],{"categories":1188},[],{"categories":1190},[],{"categories":1192},[],{"categories":1194},[285],{"categories":1196},[236],{"categories":1198},[236],{"categories":1200},[233],{"categories":1202},[233],{"categories":1204},[],{"categories":1206},[275],{"categories":1208},[],{"categories":1210},[],{"categories":1212},[236],{"categories":1214},[],{"categories":1216},[],{"categories":1218},[292],{"categories":1220},[292],{"categories":1222},[236],{"categories":1224},[],{"categories":1226},[233],{"categories":1228},[233],{"categories":1230},[285],{"categories":1232},[275],{"categories":1234},[275],{"categories":1236},[236],{"categories":1238},[227],{"categories":1240},[233],{"categories":1242},[275],{"categories":1244},[275],{"categories":1246},[236],{"categories":1248},[236],{"categories":1250},[233],{"categories":1252},[],{"categories":1254},[],{"categories":1256},[233],{"categories":1258},[236],{"categories":1260},[254],{"categories":1262},[285],{"categories":1264},[227],{"categories":1266},[233],{"categories":1268},[],{"categories":1270},[236],{"categories":1272},[236],{"categories":1274},[],{"categories":1276},[227],{"categories":1278},[233],{"categories":1280},[227],{"categories":1282},[227],{"categories":1284},[],{"categories":1286},[],{"categories":1288},[236],{"categories":1290},[236],{"categories":1292},[233],{"categories":1294},[233],{"categories":1296},[254],{"categories":1298},[278],{"categories":1300},[239],{"categories":1302},[254],{"categories":1304},[275],{"categories":1306},[],{"categories":1308},[254],{"categories":1310},[],{"categories":1312},[],{"categories":1314},[],{"categories":1316},[],{"categories":1318},[285],{"categories":1320},[278],{"categories":1322},[],{"categories":1324},[233],{"categories":1326},[233],{"categories":1328},[278],{"categories":1330},[285],{"categories":1332},[],{"categories":1334},[],{"categories":1336},[236],{"categories":1338},[254],{"categories":1340},[254],{"categories":1342},[236],{"categories":1344},[227],{"categories":1346},[233,199],{"categories":1348},[],{"categories":1350},[275],{"categories":1352},[227],{"categories":1354},[236],{"categories":1356},[275],{"categories":1358},[],{"categories":1360},[236],{"categories":1362},[236],{"categories":1364},[233],{"categories":1366},[292],{"categories":1368},[285],{"categories":1370},[275],{"categories":1372},[],{"categories":1374},[236],{"categories":1376},[233],{"categories":1378},[236],{"categories":1380},[236],{"categories":1382},[236],{"categories":1384},[292],{"categories":1386},[236],{"categories":1388},[233],{"categories":1390},[],{"categories":1392},[292],{"categories":1394},[254],{"categories":1396},[236],{"categories":1398},[],{"categories":1400},[],{"categories":1402},[233],{"categories":1404},[236],{"categories":1406},[254],{"categories":1408},[236],{"categories":1410},[],{"categories":1412},[],{"categories":1414},[],{"categories":1416},[236],{"categories":1418},[],{"categories":1420},[],{"categories":1422},[278],{"categories":1424},[233],{"categories":1426},[278],{"categories":1428},[254],{"categories":1430},[233],{"categories":1432},[233],{"categories":1434},[236],{"categories":1436},[233],{"categories":1438},[],{"categories":1440},[],{"categories":1442},[199],{"categories":1444},[],{"categories":1446},[],{"categories":1448},[227],{"categories":1450},[],{"categories":1452},[],{"categories":1454},[],{"categories":1456},[],{"categories":1458},[285],{"categories":1460},[254],{"categories":1462},[292],{"categories":1464},[230],{"categories":1466},[233],{"categories":1468},[233],{"categories":1470},[230],{"categories":1472},[],{"categories":1474},[275],{"categories":1476},[236],{"categories":1478},[230],{"categories":1480},[233],{"categories":1482},[233],{"categories":1484},[227],{"categories":1486},[],{"categories":1488},[227],{"categories":1490},[233],{"categories":1492},[292],{"categories":1494},[236],{"categories":1496},[254],{"categories":1498},[230],{"categories":1500},[233],{"categories":1502},[236],{"categories":1504},[],{"categories":1506},[233],{"categories":1508},[227],{"categories":1510},[233],{"categories":1512},[],{"categories":1514},[254],{"categories":1516},[233],{"categories":1518},[],{"categories":1520},[230],{"categories":1522},[233],{"categories":1524},[],{"categories":1526},[],{"categories":1528},[],{"categories":1530},[233],{"categories":1532},[],{"categories":1534},[199],{"categories":1536},[233],{"categories":1538},[],{"categories":1540},[233],{"categories":1542},[233],{"categories":1544},[233],{"categories":1546},[233,199],{"categories":1548},[233],{"categories":1550},[233],{"categories":1552},[275],{"categories":1554},[236],{"categories":1556},[],{"categories":1558},[236],{"categories":1560},[233],{"categories":1562},[233],{"categories":1564},[233],{"categories":1566},[227],{"categories":1568},[227],{"categories":1570},[285],{"categories":1572},[275],{"categories":1574},[236],{"categories":1576},[],{"categories":1578},[233],{"categories":1580},[254],{"categories":1582},[233],{"categories":1584},[230],{"categories":1586},[],{"categories":1588},[199],{"categories":1590},[275],{"categories":1592},[275],{"categories":1594},[236],{"categories":1596},[254],{"categories":1598},[236],{"categories":1600},[233],{"categories":1602},[],{"categories":1604},[233],{"categories":1606},[],{"categories":1608},[],{"categories":1610},[233],{"categories":1612},[233],{"categories":1614},[233],{"categories":1616},[236],{"categories":1618},[233],{"categories":1620},[],{"categories":1622},[278],{"categories":1624},[236],{"categories":1626},[],{"categories":1628},[233],{"categories":1630},[254],{"categories":1632},[],{"categories":1634},[275],{"categories":1636},[199],{"categories":1638},[254],{"categories":1640},[285],{"categories":1642},[285],{"categories":1644},[254],{"categories":1646},[254],{"categories":1648},[199],{"categories":1650},[],{"categories":1652},[254],{"categories":1654},[233],{"categories":1656},[227],{"categories":1658},[254],{"categories":1660},[],{"categories":1662},[278],{"categories":1664},[254],{"categories":1666},[285],{"categories":1668},[254],{"categories":1670},[199],{"categories":1672},[233],{"categories":1674},[233],{"categories":1676},[],{"categories":1678},[230],{"categories":1680},[],{"categories":1682},[],{"categories":1684},[233],{"categories":1686},[233],{"categories":1688},[233],{"categories":1690},[233],{"categories":1692},[],{"categories":1694},[278],{"categories":1696},[227],{"categories":1698},[],{"categories":1700},[233],{"categories":1702},[233],{"categories":1704},[199],{"categories":1706},[199],{"categories":1708},[],{"categories":1710},[236],{"categories":1712},[254],{"categories":1714},[254],{"categories":1716},[233],{"categories":1718},[236],{"categories":1720},[],{"categories":1722},[275],{"categories":1724},[233],{"categories":1726},[233],{"categories":1728},[],{"categories":1730},[],{"categories":1732},[199],{"categories":1734},[233],{"categories":1736},[285],{"categories":1738},[230],{"categories":1740},[233],{"categories":1742},[],{"categories":1744},[236],{"categories":1746},[227],{"categories":1748},[227],{"categories":1750},[],{"categories":1752},[233],{"categories":1754},[275],{"categories":1756},[236],{"categories":1758},[],{"categories":1760},[233],{"categories":1762},[233],{"categories":1764},[236],{"categories":1766},[],{"categories":1768},[236],{"categories":1770},[285],{"categories":1772},[],{"categories":1774},[233],{"categories":1776},[],{"categories":1778},[233],{"categories":1780},[],{"categories":1782},[233],{"categories":1784},[233],{"categories":1786},[],{"categories":1788},[233],{"categories":1790},[254],{"categories":1792},[233],{"categories":1794},[233],{"categories":1796},[227],{"categories":1798},[233],{"categories":1800},[254],{"categories":1802},[236],{"categories":1804},[],{"categories":1806},[233],{"categories":1808},[292],{"categories":1810},[],{"categories":1812},[],{"categories":1814},[],{"categories":1816},[227],{"categories":1818},[254],{"categories":1820},[236],{"categories":1822},[233],{"categories":1824},[275],{"categories":1826},[236],{"categories":1828},[],{"categories":1830},[236],{"categories":1832},[],{"categories":1834},[233],{"categories":1836},[236],{"categories":1838},[233],{"categories":1840},[],{"categories":1842},[233],{"categories":1844},[233],{"categories":1846},[254],{"categories":1848},[275],{"categories":1850},[236],{"categories":1852},[275],{"categories":1854},[230],{"categories":1856},[],{"categories":1858},[],{"categories":1860},[233],{"categories":1862},[227],{"categories":1864},[254],{"categories":1866},[],{"categories":1868},[],{"categories":1870},[285],{"categories":1872},[275],{"categories":1874},[],{"categories":1876},[233],{"categories":1878},[],{"categories":1880},[292],{"categories":1882},[233],{"categories":1884},[199],{"categories":1886},[285],{"categories":1888},[],{"categories":1890},[236],{"categories":1892},[233],{"categories":1894},[236],{"categories":1896},[236],{"categories":1898},[233],{"categories":1900},[],{"categories":1902},[227],{"categories":1904},[233],{"categories":1906},[230],{"categories":1908},[285],{"categories":1910},[275],{"categories":1912},[],{"categories":1914},[],{"categories":1916},[],{"categories":1918},[236],{"categories":1920},[275],{"categories":1922},[254],{"categories":1924},[233],{"categories":1926},[254],{"categories":1928},[275],{"categories":1930},[],{"categories":1932},[275],{"categories":1934},[254],{"categories":1936},[230],{"categories":1938},[233],{"categories":1940},[254],{"categories":1942},[292],{"categories":1944},[],{"categories":1946},[],{"categories":1948},[278],{"categories":1950},[233,285],{"categories":1952},[254],{"categories":1954},[233],{"categories":1956},[236],{"categories":1958},[236],{"categories":1960},[233],{"categories":1962},[],{"categories":1964},[285],{"categories":1966},[233],{"categories":1968},[278],{"categories":1970},[236],{"categories":1972},[292],{"categories":1974},[199],{"categories":1976},[],{"categories":1978},[227],{"categories":1980},[236],{"categories":1982},[236],{"categories":1984},[285],{"categories":1986},[233],{"categories":1988},[233],{"categories":1990},[],{"categories":1992},[],{"categories":1994},[],{"categories":1996},[199],{"categories":1998},[254],{"categories":2000},[233],{"categories":2002},[233],{"categories":2004},[233],{"categories":2006},[],{"categories":2008},[278],{"categories":2010},[230],{"categories":2012},[],{"categories":2014},[236],{"categories":2016},[199],{"categories":2018},[],{"categories":2020},[275],{"categories":2022},[275],{"categories":2024},[],{"categories":2026},[285],{"categories":2028},[275],{"categories":2030},[233],{"categories":2032},[],{"categories":2034},[254],{"categories":2036},[233],{"categories":2038},[275],{"categories":2040},[236],{"categories":2042},[254],{"categories":2044},[],{"categories":2046},[236],{"categories":2048},[275],{"categories":2050},[233],{"categories":2052},[],{"categories":2054},[233],{"categories":2056},[233],{"categories":2058},[199],{"categories":2060},[254],{"categories":2062},[278],{"categories":2064},[278],{"categories":2066},[],{"categories":2068},[],{"categories":2070},[],{"categories":2072},[236],{"categories":2074},[285],{"categories":2076},[285],{"categories":2078},[],{"categories":2080},[],{"categories":2082},[233],{"categories":2084},[],{"categories":2086},[236],{"categories":2088},[233],{"categories":2090},[],{"categories":2092},[233],{"categories":2094},[230],{"categories":2096},[233],{"categories":2098},[292],{"categories":2100},[236],{"categories":2102},[233],{"categories":2104},[285],{"categories":2106},[254],{"categories":2108},[236],{"categories":2110},[],{"categories":2112},[254],{"categories":2114},[236],{"categories":2116},[236],{"categories":2118},[],{"categories":2120},[230],{"categories":2122},[236],{"categories":2124},[],{"categories":2126},[233],{"categories":2128},[227],{"categories":2130},[254],{"categories":2132},[199],{"categories":2134},[236],{"categories":2136},[236],{"categories":2138},[227],{"categories":2140},[233],{"categories":2142},[],{"categories":2144},[],{"categories":2146},[275],{"categories":2148},[233,230],{"categories":2150},[],{"categories":2152},[227],{"categories":2154},[278],{"categories":2156},[233],{"categories":2158},[285],{"categories":2160},[233],{"categories":2162},[236],{"categories":2164},[233],{"categories":2166},[233],{"categories":2168},[254],{"categories":2170},[236],{"categories":2172},[],{"categories":2174},[],{"categories":2176},[236],{"categories":2178},[233],{"categories":2180},[199],{"categories":2182},[],{"categories":2184},[233],{"categories":2186},[236],{"categories":2188},[],{"categories":2190},[233],{"categories":2192},[292],{"categories":2194},[278],{"categories":2196},[236],{"categories":2198},[233],{"categories":2200},[199],{"categories":2202},[],{"categories":2204},[233],{"categories":2206},[292],{"categories":2208},[275],{"categories":2210},[233],{"categories":2212},[],{"categories":2214},[292],{"categories":2216},[254],{"categories":2218},[233],{"categories":2220},[233],{"categories":2222},[227],{"categories":2224},[],{"categories":2226},[],{"categories":2228},[275],{"categories":2230},[233],{"categories":2232},[278],{"categories":2234},[292],{"categories":2236},[292],{"categories":2238},[254],{"categories":2240},[],{"categories":2242},[],{"categories":2244},[233],{"categories":2246},[],{"categories":2248},[233,285],{"categories":2250},[254],{"categories":2252},[236],{"categories":2254},[285],{"categories":2256},[233],{"categories":2258},[227],{"categories":2260},[],{"categories":2262},[],{"categories":2264},[227],{"categories":2266},[292],{"categories":2268},[233],{"categories":2270},[],{"categories":2272},[275,233],{"categories":2274},[199],{"categories":2276},[227],{"categories":2278},[],{"categories":2280},[230],{"categories":2282},[230],{"categories":2284},[233],{"categories":2286},[285],{"categories":2288},[236],{"categories":2290},[254],{"categories":2292},[292],{"categories":2294},[275],{"categories":2296},[233],{"categories":2298},[233],{"categories":2300},[233],{"categories":2302},[227],{"categories":2304},[233],{"categories":2306},[236],{"categories":2308},[254],{"categories":2310},[],{"categories":2312},[],{"categories":2314},[278],{"categories":2316},[285],{"categories":2318},[233],{"categories":2320},[275],{"categories":2322},[278],{"categories":2324},[233],{"categories":2326},[233],{"categories":2328},[236],{"categories":2330},[236],{"categories":2332},[233,230],{"categories":2334},[],{"categories":2336},[275],{"categories":2338},[],{"categories":2340},[233],{"categories":2342},[254],{"categories":2344},[227],{"categories":2346},[227],{"categories":2348},[236],{"categories":2350},[233],{"categories":2352},[230],{"categories":2354},[285],{"categories":2356},[292],{"categories":2358},[],{"categories":2360},[254],{"categories":2362},[233],{"categories":2364},[233],{"categories":2366},[254],{"categories":2368},[285],{"categories":2370},[233],{"categories":2372},[236],{"categories":2374},[254],{"categories":2376},[233],{"categories":2378},[275],{"categories":2380},[233],{"categories":2382},[233],{"categories":2384},[199],{"categories":2386},[239],{"categories":2388},[236],{"categories":2390},[233],{"categories":2392},[254],{"categories":2394},[236],{"categories":2396},[292],{"categories":2398},[233],{"categories":2400},[],{"categories":2402},[233],{"categories":2404},[],{"categories":2406},[],{"categories":2408},[],{"categories":2410},[230],{"categories":2412},[233],{"categories":2414},[236],{"categories":2416},[254],{"categories":2418},[254],{"categories":2420},[254],{"categories":2422},[254],{"categories":2424},[],{"categories":2426},[227],{"categories":2428},[236],{"categories":2430},[254],{"categories":2432},[227],{"categories":2434},[236],{"categories":2436},[233],{"categories":2438},[233,236],{"categories":2440},[236],{"categories":2442},[199],{"categories":2444},[254],{"categories":2446},[254],{"categories":2448},[236],{"categories":2450},[233],{"categories":2452},[],{"categories":2454},[254],{"categories":2456},[292],{"categories":2458},[227],{"categories":2460},[233],{"categories":2462},[233],{"categories":2464},[],{"categories":2466},[285],{"categories":2468},[],{"categories":2470},[227],{"categories":2472},[236],{"categories":2474},[254],{"categories":2476},[233],{"categories":2478},[254],{"categories":2480},[227],{"categories":2482},[254],{"categories":2484},[254],{"categories":2486},[],{"categories":2488},[230],{"categories":2490},[236],{"categories":2492},[254],{"categories":2494},[254],{"categories":2496},[254],{"categories":2498},[254],{"categories":2500},[254],{"categories":2502},[254],{"categories":2504},[254],{"categories":2506},[254],{"categories":2508},[254],{"categories":2510},[254],{"categories":2512},[278],{"categories":2514},[227],{"categories":2516},[233],{"categories":2518},[233],{"categories":2520},[],{"categories":2522},[233,227],{"categories":2524},[],{"categories":2526},[236],{"categories":2528},[254],{"categories":2530},[236],{"categories":2532},[233],{"categories":2534},[233],{"categories":2536},[233],{"categories":2538},[233],{"categories":2540},[233],{"categories":2542},[236],{"categories":2544},[230],{"categories":2546},[275],{"categories":2548},[254],{"categories":2550},[233],{"categories":2552},[],{"categories":2554},[],{"categories":2556},[236],{"categories":2558},[275],{"categories":2560},[233],{"categories":2562},[],{"categories":2564},[],{"categories":2566},[292],{"categories":2568},[233],{"categories":2570},[],{"categories":2572},[],{"categories":2574},[227],{"categories":2576},[230],{"categories":2578},[233],{"categories":2580},[230],{"categories":2582},[275],{"categories":2584},[],{"categories":2586},[254],{"categories":2588},[],{"categories":2590},[275],{"categories":2592},[233],{"categories":2594},[292],{"categories":2596},[],{"categories":2598},[292],{"categories":2600},[],{"categories":2602},[],{"categories":2604},[236],{"categories":2606},[],{"categories":2608},[230],{"categories":2610},[227],{"categories":2612},[275],{"categories":2614},[285],{"categories":2616},[],{"categories":2618},[],{"categories":2620},[233],{"categories":2622},[227],{"categories":2624},[292],{"categories":2626},[],{"categories":2628},[236],{"categories":2630},[236],{"categories":2632},[254],{"categories":2634},[233],{"categories":2636},[236],{"categories":2638},[233],{"categories":2640},[236],{"categories":2642},[233],{"categories":2644},[239],{"categories":2646},[254],{"categories":2648},[],{"categories":2650},[292],{"categories":2652},[285],{"categories":2654},[236],{"categories":2656},[],{"categories":2658},[233],{"categories":2660},[236],{"categories":2662},[230],{"categories":2664},[227],{"categories":2666},[233],{"categories":2668},[275],{"categories":2670},[285],{"categories":2672},[285],{"categories":2674},[233],{"categories":2676},[278],{"categories":2678},[233],{"categories":2680},[236],{"categories":2682},[230],{"categories":2684},[236],{"categories":2686},[233],{"categories":2688},[233],{"categories":2690},[236],{"categories":2692},[254],{"categories":2694},[],{"categories":2696},[227],{"categories":2698},[233],{"categories":2700},[236],{"categories":2702},[233],{"categories":2704},[233],{"categories":2706},[],{"categories":2708},[275],{"categories":2710},[230],{"categories":2712},[254],{"categories":2714},[233],{"categories":2716},[233],{"categories":2718},[275],{"categories":2720},[292],{"categories":2722},[278],{"categories":2724},[233],{"categories":2726},[254],{"categories":2728},[233],{"categories":2730},[236],{"categories":2732},[199],{"categories":2734},[233],{"categories":2736},[236],{"categories":2738},[278],{"categories":2740},[],{"categories":2742},[236],{"categories":2744},[285],{"categories":2746},[275],{"categories":2748},[233],{"categories":2750},[227],{"categories":2752},[230],{"categories":2754},[285],{"categories":2756},[],{"categories":2758},[236],{"categories":2760},[233],{"categories":2762},[],{"categories":2764},[254],{"categories":2766},[],{"categories":2768},[254],{"categories":2770},[233],{"categories":2772},[236],{"categories":2774},[236],{"categories":2776},[236],{"categories":2778},[],{"categories":2780},[],{"categories":2782},[233],{"categories":2784},[233],{"categories":2786},[],{"categories":2788},[275],{"categories":2790},[236],{"categories":2792},[292],{"categories":2794},[227],{"categories":2796},[],{"categories":2798},[],{"categories":2800},[254],{"categories":2802},[285],{"categories":2804},[233],{"categories":2806},[233],{"categories":2808},[233],{"categories":2810},[285],{"categories":2812},[254],{"categories":2814},[275],{"categories":2816},[233],{"categories":2818},[233],{"categories":2820},[233],{"categories":2822},[254],{"categories":2824},[233],{"categories":2826},[254],{"categories":2828},[236],{"categories":2830},[236],{"categories":2832},[285],{"categories":2834},[236],{"categories":2836},[233],{"categories":2838},[285],{"categories":2840},[275],{"categories":2842},[],{"categories":2844},[236],{"categories":2846},[],{"categories":2848},[],{"categories":2850},[230],{"categories":2852},[233],{"categories":2854},[236],{"categories":2856},[227],{"categories":2858},[236],{"categories":2860},[292],{"categories":2862},[],{"categories":2864},[236],{"categories":2866},[],{"categories":2868},[227],{"categories":2870},[236],{"categories":2872},[],{"categories":2874},[236],{"categories":2876},[233],{"categories":2878},[254],{"categories":2880},[233],{"categories":2882},[236],{"categories":2884},[254],{"categories":2886},[236],{"categories":2888},[285],{"categories":2890},[275],{"categories":2892},[227],{"categories":2894},[],{"categories":2896},[236],{"categories":2898},[275],{"categories":2900},[254],{"categories":2902},[233],{"categories":2904},[275],{"categories":2906},[227],{"categories":2908},[],{"categories":2910},[236],{"categories":2912},[236],{"categories":2914},[233],{"categories":2916},[],{"categories":2918},[236],{"categories":2920},[239],{"categories":2922},[254],{"categories":2924},[236],{"categories":2926},[230],{"categories":2928},[],{"categories":2930},[233],{"categories":2932},[239],{"categories":2934},[233],{"categories":2936},[236],{"categories":2938},[254],{"categories":2940},[227],{"categories":2942},[199],{"categories":2944},[233],{"categories":2946},[233],{"categories":2948},[233],{"categories":2950},[254],{"categories":2952},[230],{"categories":2954},[233],{"categories":2956},[275],{"categories":2958},[254],{"categories":2960},[199],{"categories":2962},[233],{"categories":2964},[],{"categories":2966},[],{"categories":2968},[199],{"categories":2970},[278],{"categories":2972},[236],{"categories":2974},[236],{"categories":2976},[254],{"categories":2978},[233],{"categories":2980},[227],{"categories":2982},[275],{"categories":2984},[236],{"categories":2986},[233],{"categories":2988},[292],{"categories":2990},[233],{"categories":2992},[236],{"categories":2994},[],{"categories":2996},[233],{"categories":2998},[233],{"categories":3000},[254],{"categories":3002},[227],{"categories":3004},[],{"categories":3006},[233],{"categories":3008},[233],{"categories":3010},[285],{"categories":3012},[275],{"categories":3014},[233,236],{"categories":3016},[292,230],{"categories":3018},[233],{"categories":3020},[],{"categories":3022},[236],{"categories":3024},[],{"categories":3026},[285],{"categories":3028},[233],{"categories":3030},[254],{"categories":3032},[],{"categories":3034},[236],{"categories":3036},[],{"categories":3038},[236],{"categories":3040},[227],{"categories":3042},[236],{"categories":3044},[233],{"categories":3046},[199],{"categories":3048},[292],{"categories":3050},[230],{"categories":3052},[230],{"categories":3054},[227],{"categories":3056},[227],{"categories":3058},[233],{"categories":3060},[236],{"categories":3062},[233],{"categories":3064},[233],{"categories":3066},[227],{"categories":3068},[233],{"categories":3070},[292],{"categories":3072},[254],{"categories":3074},[233],{"categories":3076},[236],{"categories":3078},[233],{"categories":3080},[],{"categories":3082},[285],{"categories":3084},[],{"categories":3086},[236],{"categories":3088},[227],{"categories":3090},[],{"categories":3092},[199],{"categories":3094},[233],{"categories":3096},[],{"categories":3098},[254],{"categories":3100},[236],{"categories":3102},[285],{"categories":3104},[233],{"categories":3106},[236],{"categories":3108},[285],{"categories":3110},[236],{"categories":3112},[254],{"categories":3114},[227],{"categories":3116},[254],{"categories":3118},[285],{"categories":3120},[233],{"categories":3122},[275],{"categories":3124},[233],{"categories":3126},[233],{"categories":3128},[233],{"categories":3130},[233],{"categories":3132},[236],{"categories":3134},[233],{"categories":3136},[236],{"categories":3138},[233],{"categories":3140},[227],{"categories":3142},[233],{"categories":3144},[236],{"categories":3146},[275],{"categories":3148},[227],{"categories":3150},[236],{"categories":3152},[275],{"categories":3154},[],{"categories":3156},[233],{"categories":3158},[233],{"categories":3160},[285],{"categories":3162},[],{"categories":3164},[236],{"categories":3166},[292],{"categories":3168},[233],{"categories":3170},[254],{"categories":3172},[292],{"categories":3174},[236],{"categories":3176},[230],{"categories":3178},[230],{"categories":3180},[233],{"categories":3182},[227],{"categories":3184},[],{"categories":3186},[233],{"categories":3188},[],{"categories":3190},[227],{"categories":3192},[233],{"categories":3194},[236],{"categories":3196},[236],{"categories":3198},[],{"categories":3200},[285],{"categories":3202},[285],{"categories":3204},[292],{"categories":3206},[275],{"categories":3208},[],{"categories":3210},[233],{"categories":3212},[227],{"categories":3214},[233],{"categories":3216},[285],{"categories":3218},[227],{"categories":3220},[254],{"categories":3222},[254],{"categories":3224},[],{"categories":3226},[254],{"categories":3228},[236],{"categories":3230},[275],{"categories":3232},[278],{"categories":3234},[233],{"categories":3236},[],{"categories":3238},[254],{"categories":3240},[285],{"categories":3242},[230],{"categories":3244},[233],{"categories":3246},[227],{"categories":3248},[199],{"categories":3250},[227],{"categories":3252},[],{"categories":3254},[],{"categories":3256},[254],{"categories":3258},[],{"categories":3260},[236],{"categories":3262},[236],{"categories":3264},[236],{"categories":3266},[],{"categories":3268},[233],{"categories":3270},[],{"categories":3272},[254],{"categories":3274},[227],{"categories":3276},[275],{"categories":3278},[233],{"categories":3280},[254],{"categories":3282},[254],{"categories":3284},[],{"categories":3286},[254],{"categories":3288},[227],{"categories":3290},[233],{"categories":3292},[],{"categories":3294},[236],{"categories":3296},[236],{"categories":3298},[227],{"categories":3300},[],{"categories":3302},[],{"categories":3304},[],{"categories":3306},[275],{"categories":3308},[236],{"categories":3310},[233],{"categories":3312},[],{"categories":3314},[],{"categories":3316},[],{"categories":3318},[275],{"categories":3320},[],{"categories":3322},[227],{"categories":3324},[],{"categories":3326},[],{"categories":3328},[275],{"categories":3330},[233],{"categories":3332},[254],{"categories":3334},[],{"categories":3336},[292],{"categories":3338},[254],{"categories":3340},[292],{"categories":3342},[233],{"categories":3344},[],{"categories":3346},[],{"categories":3348},[236],{"categories":3350},[],{"categories":3352},[],{"categories":3354},[236],{"categories":3356},[233],{"categories":3358},[],{"categories":3360},[236],{"categories":3362},[254],{"categories":3364},[292],{"categories":3366},[278],{"categories":3368},[236],{"categories":3370},[236],{"categories":3372},[],{"categories":3374},[],{"categories":3376},[],{"categories":3378},[254],{"categories":3380},[],{"categories":3382},[],{"categories":3384},[275],{"categories":3386},[227],{"categories":3388},[],{"categories":3390},[230],{"categories":3392},[292],{"categories":3394},[233],{"categories":3396},[285],{"categories":3398},[227],{"categories":3400},[278],{"categories":3402},[230],{"categories":3404},[285],{"categories":3406},[],{"categories":3408},[],{"categories":3410},[236],{"categories":3412},[227],{"categories":3414},[275],{"categories":3416},[227],{"categories":3418},[236],{"categories":3420},[199],{"categories":3422},[236],{"categories":3424},[],{"categories":3426},[233],{"categories":3428},[254],{"categories":3430},[285],{"categories":3432},[],{"categories":3434},[275],{"categories":3436},[254],{"categories":3438},[227],{"categories":3440},[236],{"categories":3442},[233],{"categories":3444},[230],{"categories":3446},[236,199],{"categories":3448},[236],{"categories":3450},[285],{"categories":3452},[233],{"categories":3454},[278],{"categories":3456},[292],{"categories":3458},[236],{"categories":3460},[],{"categories":3462},[236],{"categories":3464},[233],{"categories":3466},[230],{"categories":3468},[],{"categories":3470},[],{"categories":3472},[233],{"categories":3474},[278],{"categories":3476},[233],{"categories":3478},[],{"categories":3480},[254],{"categories":3482},[],{"categories":3484},[254],{"categories":3486},[285],{"categories":3488},[236],{"categories":3490},[233],{"categories":3492},[292],{"categories":3494},[285],{"categories":3496},[],{"categories":3498},[254],{"categories":3500},[233],{"categories":3502},[],{"categories":3504},[233],{"categories":3506},[236],{"categories":3508},[233],{"categories":3510},[236],{"categories":3512},[233],{"categories":3514},[233],{"categories":3516},[233],{"categories":3518},[233],{"categories":3520},[230],{"categories":3522},[],{"categories":3524},[239],{"categories":3526},[254],{"categories":3528},[233],{"categories":3530},[],{"categories":3532},[285],{"categories":3534},[233],{"categories":3536},[233],{"categories":3538},[236],{"categories":3540},[254],{"categories":3542},[233],{"categories":3544},[233],{"categories":3546},[230],{"categories":3548},[236],{"categories":3550},[275],{"categories":3552},[],{"categories":3554},[278],{"categories":3556},[233],{"categories":3558},[],{"categories":3560},[254],{"categories":3562},[292],{"categories":3564},[],{"categories":3566},[],{"categories":3568},[254],{"categories":3570},[254],{"categories":3572},[292],{"categories":3574},[227],{"categories":3576},[236],{"categories":3578},[236],{"categories":3580},[233],{"categories":3582},[230],{"categories":3584},[],{"categories":3586},[],{"categories":3588},[254],{"categories":3590},[278],{"categories":3592},[285],{"categories":3594},[236],{"categories":3596},[275],{"categories":3598},[278],{"categories":3600},[278],{"categories":3602},[],{"categories":3604},[254],{"categories":3606},[233],{"categories":3608},[233],{"categories":3610},[285],{"categories":3612},[],{"categories":3614},[254],{"categories":3616},[254],{"categories":3618},[254],{"categories":3620},[],{"categories":3622},[236],{"categories":3624},[233],{"categories":3626},[],{"categories":3628},[227],{"categories":3630},[230],{"categories":3632},[],{"categories":3634},[233],{"categories":3636},[233],{"categories":3638},[],{"categories":3640},[285],{"categories":3642},[],{"categories":3644},[],{"categories":3646},[],{"categories":3648},[],{"categories":3650},[233],{"categories":3652},[254],{"categories":3654},[],{"categories":3656},[],{"categories":3658},[233],{"categories":3660},[233],{"categories":3662},[233],{"categories":3664},[278],{"categories":3666},[233],{"categories":3668},[278],{"categories":3670},[],{"categories":3672},[278],{"categories":3674},[278],{"categories":3676},[199],{"categories":3678},[236],{"categories":3680},[285],{"categories":3682},[],{"categories":3684},[],{"categories":3686},[278],{"categories":3688},[285],{"categories":3690},[285],{"categories":3692},[285],{"categories":3694},[],{"categories":3696},[227],{"categories":3698},[285],{"categories":3700},[285],{"categories":3702},[227],{"categories":3704},[285],{"categories":3706},[230],{"categories":3708},[285],{"categories":3710},[285],{"categories":3712},[285],{"categories":3714},[278],{"categories":3716},[254],{"categories":3718},[254],{"categories":3720},[233],{"categories":3722},[285],{"categories":3724},[278],{"categories":3726},[199],{"categories":3728},[278],{"categories":3730},[278],{"categories":3732},[278],{"categories":3734},[],{"categories":3736},[230],{"categories":3738},[],{"categories":3740},[199],{"categories":3742},[285],{"categories":3744},[285],{"categories":3746},[285],{"categories":3748},[236],{"categories":3750},[254,230],{"categories":3752},[278],{"categories":3754},[],{"categories":3756},[],{"categories":3758},[278],{"categories":3760},[],{"categories":3762},[278],{"categories":3764},[254],{"categories":3766},[236],{"categories":3768},[],{"categories":3770},[285],{"categories":3772},[233],{"categories":3774},[275],{"categories":3776},[],{"categories":3778},[233],{"categories":3780},[],{"categories":3782},[254],{"categories":3784},[227],{"categories":3786},[278],{"categories":3788},[],{"categories":3790},[285],{"categories":3792},[254],[3794,3868,3937,3998],{"id":3795,"title":3796,"ai":3797,"body":3802,"categories":3838,"created_at":200,"date_modified":200,"description":189,"extension":202,"faq":200,"featured":203,"kicker_label":200,"meta":3839,"navigation":205,"path":3855,"published_at":3856,"question":200,"scraped_at":3856,"seo":3857,"sitemap":3858,"source_id":3859,"source_name":3860,"source_type":3861,"source_url":3862,"stem":3863,"tags":3864,"thumbnail_url":200,"tldr":3865,"tweet":200,"unknown_tags":3866,"__hash__":3867},"summaries\u002Fsummaries\u002Fa2a811b50a4c64f5-mrc-resilient-networking-for-100k-gpu-ai-training-summary.md","MRC: Resilient Networking for 100K+ GPU AI Training",{"provider":7,"model":8,"input_tokens":3798,"output_tokens":3799,"processing_time_ms":3800,"cost_usd":3801},9014,2044,25377,0.0028023,{"type":14,"value":3803,"toc":3832},[3804,3808,3811,3815,3818,3822,3825,3829],[17,3805,3807],{"id":3806},"multi-plane-topologies-slash-switch-tiers-and-power-for-massive-clusters","Multi-Plane Topologies Slash Switch Tiers and Power for Massive Clusters",[22,3809,3810],{},"Traditional 800Gb\u002Fs networks require three or four tiers of switches to connect over 100,000 GPUs, increasing power use, failure points, and cost. MRC splits each 800Gb\u002Fs interface into eight 100Gb\u002Fs links, creating eight parallel 'planes' that connect to separate switches. A 64-port 800Gb\u002Fs switch now handles 512 ports at 100Gb\u002Fs, enabling full connectivity for 131,000 GPUs using only two tiers. This design boosts path diversity—keeping more traffic local to Tier 0 switches—while cutting components, power, and cost compared to single-plane setups. Without changes, single-path flows (like classic RoCE) still congest links as flows collide, especially in AI's collective communications where worst-case latency stalls synchronous training.",[17,3812,3814],{"id":3813},"packet-spraying-and-srv6-eliminate-congestion-and-dynamic-routing","Packet Spraying and SRv6 Eliminate Congestion and Dynamic Routing",[22,3816,3817],{},"MRC sprays packets from a single transfer across hundreds of paths spanning all planes, using final memory addresses for out-of-order reassembly at the destination. Adaptive load-balancing monitors paths: congestion triggers path swaps, packet loss retires the path (with probes for recovery), and 'packet trimming' at switches forwards headers only during destination congestion to prompt retransmits without false failure alarms. This achieves microsecond failure detection and rerouting, versus seconds for traditional fabrics. MRC replaces BGP dynamic routing with static SRv6 source routing: senders embed full switch ID sequences in IPv6 addresses. Switches shift addresses and follow pre-configured static tables, blindly forwarding without recomputing routes. Failures simply retire paths at endpoints, simplifying control planes and eliminating routing bugs from switch software.",[17,3819,3821],{"id":3820},"production-impact-zero-measurable-downtime-amid-constant-failures","Production Impact: Zero-Measurable Downtime Amid Constant Failures",[22,3823,3824],{},"In OpenAI's NVIDIA GB200 supercomputers (including OCI's Abilene Stargate site and Microsoft's Fairwater), MRC handles millions of links with frequent flaps—multiple per minute between tiers—yet synchronous pretraining jobs show no measurable impact, allowing deferred repairs. Rebooting four Tier-1 switches or repairing links during jobs requires no coordination; MRC avoids bad paths automatically. Real training data shows quick recovery from full T1 switch loss with temporary slowdowns far less than physical capacity loss (e.g., one failed port on an 8-port interface reduces max rate by 1\u002F8th but sustains better effective throughput via path recalculation). Multi-job clusters avoid inter-job interference due to core-wide congestion elimination, maximizing GPU utilization for frontier models like those powering ChatGPT (900M weekly users).",[17,3826,3828],{"id":3827},"strategic-wins-simpler-stacks-for-stargate-scale-compute","Strategic Wins: Simpler Stacks for Stargate-Scale Compute",[22,3830,3831],{},"MRC delivers three edges: two-tier multi-plane redundancy with lower power; zero core congestion for consistent flow throughput in sync training; and SRv6 for instant failure bypass via static planes. Deployed with AMD, Broadcom, Intel, Microsoft, NVIDIA hardware, it's released via Open Compute Project for industry adoption, supporting OpenAI's compute strategy of shared standards to scale AI infrastructure efficiently.",{"title":189,"searchDepth":190,"depth":190,"links":3833},[3834,3835,3836,3837],{"id":3806,"depth":190,"text":3807},{"id":3813,"depth":190,"text":3814},{"id":3820,"depth":190,"text":3821},{"id":3827,"depth":190,"text":3828},[199],{"content_references":3840,"triage":3850},[3841,3846],{"type":3842,"title":3843,"url":3844,"context":3845},"other","OCP MRC 1.0","https:\u002F\u002Fwww.opencompute.org\u002Fdocuments\u002Focp-mrc-1-0-pdf","mentioned",{"type":3847,"title":3848,"url":3849,"context":3845},"paper","Resilient AI Supercomputer Networking using MRC and SRv6","https:\u002F\u002Fcdn.openai.com\u002Fpdf\u002Fresilient-ai-supercomputer-networking-using-mrc-and-srv6.pdf",{"relevance":3851,"novelty":3851,"quality":3852,"actionability":190,"composite":3853,"reasoning":3854},3,4,3.05,"Category: DevOps & Cloud. The article discusses the MRC protocol's innovative networking solutions for AI training, which could be relevant for those building AI-powered products. However, it lacks direct actionable insights for the audience, focusing more on technical specifications than practical applications.","\u002Fsummaries\u002Fa2a811b50a4c64f5-mrc-resilient-networking-for-100k-gpu-ai-training-summary","2026-05-11 15:04:27",{"title":3796,"description":189},{"loc":3855},"a2a811b50a4c64f5","OpenAI News","article","https:\u002F\u002Fopenai.com\u002Findex\u002Fmrc-supercomputer-networking","summaries\u002Fa2a811b50a4c64f5-mrc-resilient-networking-for-100k-gpu-ai-training-summary",[217,218,219],"OpenAI's MRC protocol uses multi-plane topologies and packet spraying across hundreds of paths with SRv6 source routing to eliminate congestion, route around failures in microseconds, and connect 131k GPUs with just two switch tiers, enabling non-stop frontier model training.",[],"BYXvfLzxxajQIir95xuUTVdTfvID4wPt3TOVHNxrCSU",{"id":3869,"title":3870,"ai":3871,"body":3876,"categories":3913,"created_at":200,"date_modified":200,"description":189,"extension":202,"faq":200,"featured":203,"kicker_label":200,"meta":3914,"navigation":205,"path":3924,"published_at":3925,"question":200,"scraped_at":3926,"seo":3927,"sitemap":3928,"source_id":3929,"source_name":3930,"source_type":3861,"source_url":3931,"stem":3932,"tags":3933,"thumbnail_url":200,"tldr":3934,"tweet":200,"unknown_tags":3935,"__hash__":3936},"summaries\u002Fsummaries\u002F30072e6e8b386729-mrc-openai-s-protocol-for-resilient-ai-training-ne-summary.md","MRC: OpenAI's Protocol for Resilient AI Training Networks",{"provider":7,"model":8,"input_tokens":3872,"output_tokens":3873,"processing_time_ms":3874,"cost_usd":3875},8465,1915,20569,0.00214365,{"type":14,"value":3877,"toc":3908},[3878,3882,3885,3888,3891,3895,3898,3901,3905],[17,3879,3881],{"id":3880},"multipath-mechanisms-eliminate-congestion-and-enable-fast-recovery","Multipath Mechanisms Eliminate Congestion and Enable Fast Recovery",[22,3883,3884],{},"In large AI training clusters, network congestion, link failures, and jitter cause GPU idle time, amplifying costs as clusters scale to millions of data transfers per step. MRC builds on RoCEv2 for hardware-accelerated RDMA over Ethernet and SRv6 for static source routing, shifting intelligence to NICs while switches follow pre-configured paths blindly. This avoids interference from dynamic routing.",[22,3886,3887],{},"Adaptive packet spraying distributes packets across hundreds of paths at the NIC level, achieving higher bandwidth, reduced tail latency, and packet-level load balancing—unlike single-path RoCEv2. For failures, MRC detects issues in microseconds and reroutes: if an 8-port 800Gb\u002Fs NIC loses one port, it drops to 7\u002F8 capacity but recalculates paths instantly, notifies peers to avoid the failed plane, and restores it within a minute upon recovery. Conventional fabrics take seconds to tens of seconds, often crashing jobs; MRC keeps training alive with minimal performance hit.",[22,3889,3890],{},"AMD's NSCC congestion control integrates via UEC specs, preserving RDMA semantics while adding multipath support.",[17,3892,3894],{"id":3893},"multi-plane-architecture-cuts-tiers-costs-and-latency","Multi-Plane Architecture Cuts Tiers, Costs, and Latency",[22,3896,3897],{},"MRC reimagines NICs as multiple smaller links (e.g., one 800Gb\u002Fs interface split into eight 100Gb\u002Fs to eight switches), enabling a two-tier Clos network for 131,000 GPUs versus three-to-four tiers in 800Gb\u002Fs designs. Longest paths cross three switches instead of five-to-seven, slashing latency.",[22,3899,3900],{},"For full bisection bandwidth, this uses 2\u002F3 the optics and 3\u002F5 the switches of three-tier networks, reducing power, cost, and failure blast radius. A tier-1 switch failure (e.g., rebooting four during training) no longer halts jobs.",[17,3902,3904],{"id":3903},"production-on-named-hardware-across-openai-clusters","Production on Named Hardware Across OpenAI Clusters",[22,3906,3907],{},"Deployed on 400\u002F800Gb\u002Fs RDMA NICs like NVIDIA ConnectX-8, AMD Pollara\u002FVulcano, Broadcom Thor Ultra; SRv6 switches include NVIDIA Spectrum-4\u002F5 (Cumulus\u002FSONiC) and Broadcom Tomahawk 5 (Arista EOS). Powers NVIDIA GB200 supercomputers in OpenAI's Stargate (OCI Abilene, TX) and Microsoft's Fairwater (Atlanta\u002FWisconsin), training ChatGPT and Codex models without job interruptions from failures.",{"title":189,"searchDepth":190,"depth":190,"links":3909},[3910,3911,3912],{"id":3880,"depth":190,"text":3881},{"id":3893,"depth":190,"text":3894},{"id":3903,"depth":190,"text":3904},[199],{"content_references":3915,"triage":3922},[3916,3918],{"type":3847,"title":3848,"url":3849,"context":3917},"cited",{"type":3842,"title":3919,"url":3920,"context":3921},"MRC Supercomputer Networking Technical Details","https:\u002F\u002Fopenai.com\u002Findex\u002Fmrc-supercomputer-networking\u002F","recommended",{"relevance":3851,"novelty":3851,"quality":3852,"actionability":190,"composite":3853,"reasoning":3923},"Category: AI & LLMs. The article discusses OpenAI's MRC protocol, which is relevant to AI infrastructure but lacks direct applicability for product builders looking for actionable insights. While it presents some new technical details about network optimization for AI training, it does not provide practical steps or frameworks that the audience can implement.","\u002Fsummaries\u002F30072e6e8b386729-mrc-openai-s-protocol-for-resilient-ai-training-ne-summary","2026-05-07 07:50:02","2026-05-07 11:24:11",{"title":3870,"description":189},{"loc":3924},"30072e6e8b386729","MarkTechPost","https:\u002F\u002Fwww.marktechpost.com\u002F2026\u002F05\u002F07\u002Fopenai-introduces-mrc-multipath-reliable-connection-a-new-open-networking-protocol-for-large-scale-ai-supercomputer-training-clusters\u002F","summaries\u002F30072e6e8b386729-mrc-openai-s-protocol-for-resilient-ai-training-ne-summary",[217,218,219],"OpenAI's MRC extends RoCE with multipath spraying, microsecond failure recovery via SRv6, and multi-plane designs to deliver predictable performance in 131k-GPU clusters, using 2\u002F3 fewer optics and 3\u002F5 fewer switches than traditional setups.",[],"XbDsma4E_5cuB3WLtPi6GgqSNlQtb2CdSK-eHkIrlrc",{"id":3938,"title":3939,"ai":3940,"body":3945,"categories":3973,"created_at":200,"date_modified":200,"description":189,"extension":202,"faq":200,"featured":203,"kicker_label":200,"meta":3974,"navigation":205,"path":3985,"published_at":3986,"question":200,"scraped_at":3987,"seo":3988,"sitemap":3989,"source_id":3990,"source_name":3991,"source_type":3861,"source_url":3992,"stem":3993,"tags":3994,"thumbnail_url":200,"tldr":3995,"tweet":200,"unknown_tags":3996,"__hash__":3997},"summaries\u002Fsummaries\u002Ff78d6045a31221d2-mrc-enables-100k-gpu-clusters-with-resilient-multi-summary.md","MRC Enables 100k+ GPU Clusters with Resilient Multipath Networking",{"provider":7,"model":8,"input_tokens":3941,"output_tokens":3942,"processing_time_ms":3943,"cost_usd":3944},4244,1621,21683,0.00163665,{"type":14,"value":3946,"toc":3968},[3947,3951,3954,3958,3961,3965],[17,3948,3950],{"id":3949},"multipath-routing-fixes-core-bottlenecks-in-ai-training","Multipath Routing Fixes Core Bottlenecks in AI Training",[22,3952,3953],{},"MRC (Multipath Reliable Connection) eliminates congestion in AI supercomputers by distributing packets across hundreds of network paths simultaneously, rather than single paths. This delivers faster, more predictable GPU-to-GPU data transfers critical for training massive models. On failures—links, switches, or paths—MRC reroutes in microseconds, versus seconds or tens of seconds for standard 800 Gb\u002Fs fabrics. Result: Training jobs survive reboots and maintenance without stalls. OpenAI's multi-plane design connects over 100,000 GPUs using only two Ethernet switch tiers, slashing component count, power use, and costs compared to conventional three- or four-tier setups.",[17,3955,3957],{"id":3956},"proven-at-scale-on-frontier-supercomputers","Proven at Scale on Frontier Supercomputers",[22,3959,3960],{},"Deployed across OpenAI's largest NVIDIA GB200 clusters—including Oracle Cloud in Abilene, Texas, and Microsoft's Fairwater—MRC handled a real-world test during frontier model training for ChatGPT and Codex. Four tier-1 switches rebooted without coordinating with running jobs, proving zero-disruption resilience. This lets operators maintain networks mid-training, boosting uptime for trillion-parameter models where network stalls previously cost hours or days.",[17,3962,3964],{"id":3963},"open-standards-accelerate-adoption","Open Standards Accelerate Adoption",[22,3966,3967],{},"Specification released via Open Compute Project (OCP MRC 1.0), with contributions from AMD, Broadcom, Intel, Microsoft, and NVIDIA. Builders can implement now for Ethernet-based AI fabrics, avoiding proprietary lock-in while hitting supercomputer-scale performance.",{"title":189,"searchDepth":190,"depth":190,"links":3969},[3970,3971,3972],{"id":3949,"depth":190,"text":3950},{"id":3956,"depth":190,"text":3957},{"id":3963,"depth":190,"text":3964},[254],{"content_references":3975,"triage":3983},[3976,3978,3980],{"type":3847,"title":3977,"url":3849,"context":3845},"Resilient AI Supercomputer Networking Using MRC and SRv6",{"type":3842,"title":3843,"publisher":3979,"url":3844,"context":3845},"Open Compute Project",{"type":3842,"title":3981,"author":3982,"url":3920,"context":3917},"MRC Supercomputer Networking","OpenAI",{"relevance":3851,"novelty":3851,"quality":3852,"actionability":190,"composite":3853,"reasoning":3984},"Category: AI & LLMs. The article discusses a new networking protocol that addresses bottlenecks in AI supercomputing, which is relevant to AI engineering. However, it lacks direct actionable insights for product builders on how to implement or leverage this technology in their own projects.","\u002Fsummaries\u002Ff78d6045a31221d2-mrc-enables-100k-gpu-clusters-with-resilient-multi-summary","2026-05-06 19:13:21","2026-05-07 11:24:04",{"title":3939,"description":189},{"loc":3985},"f78d6045a31221d2","The Decoder","https:\u002F\u002Fthe-decoder.com\u002Fopenai-built-a-networking-protocol-with-amd-broadcom-intel-microsoft-and-nvidia-to-fix-ai-supercomputer-bottlenecks\u002F","summaries\u002Ff78d6045a31221d2-mrc-enables-100k-gpu-clusters-with-resilient-multi-summary",[218,219,217],"OpenAI's MRC protocol spreads packets across hundreds of paths for microsecond failure recovery, connecting 100,000+ GPUs via just 2 switch tiers—cutting power, cost, and downtime in AI training supercomputers.",[],"LvMASfYTesYX0l3RENkA3FOBQpD3T6H-0KnDqYX6HvU",{"id":3999,"title":4000,"ai":4001,"body":4006,"categories":4040,"created_at":200,"date_modified":200,"description":189,"extension":202,"faq":200,"featured":203,"kicker_label":200,"meta":4041,"navigation":205,"path":4056,"published_at":4057,"question":200,"scraped_at":4058,"seo":4059,"sitemap":4060,"source_id":4061,"source_name":4062,"source_type":3861,"source_url":4063,"stem":4064,"tags":4065,"thumbnail_url":200,"tldr":4066,"tweet":200,"unknown_tags":4067,"__hash__":4068},"summaries\u002Fsummaries\u002F6ee9b4f709da3a06-tpus-dominate-at-infrastructure-scale-over-per-chi-summary.md","TPUs Dominate at Infrastructure Scale Over Per-Chip GPU Wins",{"provider":7,"model":8,"input_tokens":4002,"output_tokens":4003,"processing_time_ms":4004,"cost_usd":4005},5399,1852,23082,0.00198315,{"type":14,"value":4007,"toc":4035},[4008,4012,4015,4018,4022,4025,4028,4032],[17,4009,4011],{"id":4010},"infrastructure-scaling-trumps-per-chip-performance","Infrastructure Scaling Trumps Per-Chip Performance",[22,4013,4014],{},"Google's TPU v8t for training and v8i for inference trail Nvidia's Rubin and AMD GPUs in raw per-chip compute and memory. However, evaluating at infrastructure level reveals TPUs' edge: Nvidia's NVL72 scales 72 Rubin GPUs per rack, while Google's 4x4x4 cube interconnects up to 9600 TPUs into a superpod delivering 121 exaFLOPS in FP4—surpassing Nvidia's 1152-GPU Rubin pod at 60 exaFLOPS FP4. Google's Virgo network further scales out to 134,000 chips, potentially reaching 1 million, minimizing network overhead via ICI and optical interconnects. This Lego-like modularity avoids the scaling cliffs Nvidia faces when stacking GPUs, where interconnect overhead erodes per-chip advantages.",[22,4016,4017],{},"Nvidia balances scale-out with InfiniBand for diverse customers (neo-clouds like CoreWeave, labs like OpenAI\u002FMeta, hyperscalers like Microsoft\u002FAmazon), prioritizing broad demand profiles. Google, serving internal apps like Gemini and Vertex AI plus external deals (Anthropic's $1B TPU commitment: 40% owned, 60% rented; Meta's multi-billion rental), optimizes purely for its high-volume needs without market fragmentation risks.",[17,4019,4021],{"id":4020},"workload-profiles-dictate-hardware-choices","Workload Profiles Dictate Hardware Choices",[22,4023,4024],{},"AI tasks bifurcate demands: training prioritizes network bandwidth over compute\u002Fmemory, benefiting TPU's topology. Inference splits further—prefill (pink line in SemiAnalysis chart) is compute\u002Fmemory-bound for KV cache parallelization; decode (white line) is bandwidth\u002Flatency-bound for autoregressive token streaming. TPU v8t\u002F8i bifurcation matches this: v8t for training's network focus, v8i for inference's varied needs. Virgo flattens network bottlenecks, challenging Nvidia's inference dominance.",[22,4026,4027],{},"Replicating Google's scaling on Nvidia chips risks inefficiency for its varied clientele, locking into a 'balanced diet' pod architecture over specialized superpods.",[17,4029,4031],{"id":4030},"explosive-demand-drives-economics","Explosive Demand Drives Economics",[22,4033,4034],{},"Epoch AI projects 450+ new pre-trained models by 2030, many exceeding GPT-5's ~66 septillion FLOPs (total math ops for weights). A 9600-TPU superpod could theoretically pretrain GPT-5-scale models in under 7 days at FP4 (realistically 3-4 weeks), but efficiency cliffs emerge from memory, bandwidth, or latency based on scale-up\u002Fout choices. Rising inference\u002Ftraining demand amplifies TPU economics: internal fab control ensures supply for massive token serving, positioning Google against Nvidia as workloads evolve toward bandwidth constraints.",{"title":189,"searchDepth":190,"depth":190,"links":4036},[4037,4038,4039],{"id":4010,"depth":190,"text":4011},{"id":4020,"depth":190,"text":4021},{"id":4030,"depth":190,"text":4031},[254],{"content_references":4042,"triage":4054},[4043,4047,4051],{"type":4044,"title":4045,"url":4046,"context":3921},"tool","Mammoth AI","http:\u002F\u002Fmammouth.ai",{"type":4048,"title":4049,"author":4050,"context":3917},"report","SemiAnalysis AI Demand Profiles Diagram","SemiAnalysis",{"type":4048,"title":4052,"author":4053,"context":3917},"Epoch AI Pre-Trained Models Projection","Epoch AI",{"relevance":3851,"novelty":3851,"quality":3852,"actionability":190,"composite":3853,"reasoning":4055},"Category: AI & LLMs. The article discusses the performance of Google's TPUs compared to Nvidia GPUs, which is relevant to AI infrastructure but lacks direct actionable insights for product builders. While it provides some new perspectives on scaling AI workloads, it does not offer specific frameworks or techniques that the audience can implement.","\u002Fsummaries\u002F6ee9b4f709da3a06-tpus-dominate-at-infrastructure-scale-over-per-chi-summary","2026-04-30 02:16:18","2026-05-03 16:52:02",{"title":4000,"description":189},{"loc":4056},"a42442ea33b32f06","Caleb Writes Code","https:\u002F\u002Fwww.youtube.com\u002Fwatch?v=b_KxiTPBIb0","summaries\u002F6ee9b4f709da3a06-tpus-dominate-at-infrastructure-scale-over-per-chi-summary",[217,219,218],"Google's TPU v8t (training) and v8i (inference) lag Nvidia GPUs per chip but deliver superior performance at scale—9600-chip superpods hit 121 exaFLOPS FP4—via cube topology and Virgo networking, optimizing for AI's bandwidth-heavy workloads.",[],"EDdnhIFUjAM7yxc1JlkafqMae9PUuMTF3sbgBpRfDo4"]