[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"summary-daadf93b5b409781-rainbow-deploys-git-sha-kubernetes-for-stateful-dr-summary":3,"summaries-facets-categories":137,"summary-related-daadf93b5b409781-rainbow-deploys-git-sha-kubernetes-for-stateful-dr-summary":3706},{"id":4,"title":5,"ai":6,"body":13,"categories":90,"created_at":92,"date_modified":92,"description":84,"extension":93,"faq":92,"featured":94,"kicker_label":92,"meta":95,"navigation":119,"path":120,"published_at":121,"question":92,"scraped_at":122,"seo":123,"sitemap":124,"source_id":125,"source_name":126,"source_type":127,"source_url":128,"stem":129,"tags":130,"thumbnail_url":92,"tldr":134,"tweet":92,"unknown_tags":135,"__hash__":136},"summaries\u002Fsummaries\u002Fdaadf93b5b409781-rainbow-deploys-git-sha-kubernetes-for-stateful-dr-summary.md","Rainbow Deploys: Git SHA Kubernetes for Stateful Drains",{"provider":7,"model":8,"input_tokens":9,"output_tokens":10,"processing_time_ms":11,"cost_usd":12},"openrouter","x-ai\u002Fgrok-4.1-fast",4988,1999,16938,0.0019746,{"type":14,"value":15,"toc":83},"minimark",[16,21,25,28,32,40,66,76,80],[17,18,20],"h2",{"id":19},"zero-downtime-deploys-for-stateful-services","Zero-Downtime Deploys for Stateful Services",[22,23,24],"p",{},"Stateful services like Olark's chat backend hold websocket-to-XMPP connections per pod; sudden pod restarts force all users to reconnect, spiking load. Traditional Kubernetes rolling deploys kill pods immediately after new ones start, disrupting everyone. Instead, run multiple parallel Deployments indefinitely until connections drain naturally over 24-48 hours. Each Deployment needs 16 pods (2GB RAM, 1 CPU each). This preserves user sessions without hacks like hot code reloading, which containers avoid.",[22,26,27],{},"Previous approaches failed: Porting 'up' tool (forks new workers, drains old over days) required unreliable session stickiness via service-loadbalancer and hours-long terminationGracePeriodSeconds, but dropped connections prematurely. Blue\u002FGreen with two Deployments limited deploys to once daily due to drain times; scaling to 8 colors for 4x\u002Fday deploys idled 128 pods constantly, wasting resources.",[17,29,31],{"id":30},"git-sha-rainbow-deployments","Git SHA Rainbow Deployments",[22,33,34,35,39],{},"Name Deployments after git commit SHAs (first 6 chars double as hex colors): ",[36,37,38],"code",{},"chat-olark-com-\u003CSHA>",". Deploy process:",[41,42,43,51,54,57,63],"ol",{},[44,45,46,47,50],"li",{},"Create new Deployment ",[36,48,49],{},"chat-olark-com-\u003CNEW_SHA>",".",[44,52,53],{},"Scale to 16 ready pods.",[44,55,56],{},"Update Service selector to match new Deployment's labels, routing all traffic there instantly.",[44,58,59,60,50],{},"Rollback by switching Service back to ",[36,61,62],{},"\u003COLD_SHA>",[44,64,65],{},"After 24-48 hours, when connections burn down (few users left reconnect to newer), delete old Deployment.",[22,67,68,69,75],{},"Demo at ",[70,71,72],"a",{"href":72,"rel":73},"https:\u002F\u002Fgithub.com\u002Fbdimcheff\u002Frainbow-deploys",[74],"nofollow"," shows YAML and GitLab CI pipelines used since June 2017—simpler and more reliable than alternatives. No production downtime, deploys as frequent as needed without fixed color limits.",[17,77,79],{"id":78},"cleanup-challenges-and-future-ideas","Cleanup Challenges and Future Ideas",[22,81,82],{},"Manual cleanup inspects connection counts to avoid disruption; automation hard since detecting low traffic reliably eludes simple metrics. Ideal Kubernetes evolution: Native 'Immutable' strategy creates new pods without auto-killing old, plus lifecycle hooks signaling pods to self-shutdown when deselected from Service. Until then, rainbow SHAs scale indefinitely without resource bloat of pre-provisioned colors.",{"title":84,"searchDepth":85,"depth":85,"links":86},"",2,[87,88,89],{"id":19,"depth":85,"text":20},{"id":30,"depth":85,"text":31},{"id":78,"depth":85,"text":79},[91],"DevOps & Cloud",null,"md",false,{"content_references":96,"triage":114},[97,101,105,108],{"type":98,"title":99,"url":72,"context":100},"tool","rainbow-deploys","recommended",{"type":98,"title":102,"url":103,"context":104},"up","https:\u002F\u002Fgithub.com\u002Folark\u002Fup","mentioned",{"type":98,"title":106,"url":107,"context":104},"service-loadbalancer","https:\u002F\u002Fgithub.com\u002Fkubernetes\u002Fcontrib\u002Ftree\u002Fmaster\u002Fservice-loadbalancer",{"type":109,"title":110,"author":111,"url":112,"context":113},"other","BlueGreenDeployment","Martin Fowler","https:\u002F\u002Fmartinfowler.com\u002Fbliki\u002FBlueGreenDeployment.html","cited",{"relevance":115,"novelty":116,"quality":115,"actionability":115,"composite":117,"reasoning":118},4,3,3.8,"Category: DevOps & Cloud. The article provides a practical solution for deploying stateful services in Kubernetes, addressing a specific pain point of maintaining user sessions during updates. It outlines a clear deployment process with actionable steps, making it relevant for developers looking to implement zero-downtime deployments.",true,"\u002Fsummaries\u002Fdaadf93b5b409781-rainbow-deploys-git-sha-kubernetes-for-stateful-dr-summary","2018-02-13 13:50:50","2026-04-15 15:32:58",{"title":5,"description":84},{"loc":120},"daadf93b5b409781","__oneoff__","article","https:\u002F\u002Fbrandon.dimcheff.com\u002F2018\u002F02\u002Frainbow-deploys-with-kubernetes\u002F","summaries\u002Fdaadf93b5b409781-rainbow-deploys-git-sha-kubernetes-for-stateful-dr-summary",[131,132,133],"devops","cloud","deployment","For stateful services like websocket backends needing hours to drain connections, deploy Kubernetes with git SHA-named Deployments, switch Service selectors to new ones, and manually delete old after traffic burns down—avoids mass reconnects unlike rolling updates.",[133],"zTB73bpZbO7DiZLIYEnoi9klozkV2t8vUqed4scZd1c",[138,141,144,147,150,153,155,157,159,161,163,165,168,170,172,174,176,178,180,182,184,186,189,192,194,196,199,201,203,206,208,210,212,214,216,218,220,222,224,226,228,230,232,234,236,238,240,242,244,246,248,250,252,254,256,258,260,262,264,266,268,270,272,274,276,278,280,282,284,286,288,290,292,294,296,298,300,302,304,306,308,310,312,314,316,318,320,322,324,326,328,330,332,334,336,338,340,342,344,346,348,350,352,354,356,358,360,362,364,366,368,370,372,374,376,378,380,382,384,386,388,390,392,394,396,398,400,402,404,406,408,410,412,414,416,418,420,422,424,426,428,430,432,434,436,438,440,442,444,446,448,450,452,454,456,458,460,462,464,466,468,470,472,474,476,478,480,482,484,486,488,490,492,494,496,498,500,502,504,506,508,510,512,514,516,518,520,522,524,526,528,530,532,534,536,538,540,542,544,546,548,550,552,554,556,558,560,562,564,566,568,570,572,574,576,578,580,582,584,586,588,590,592,594,596,598,600,602,604,606,608,610,612,614,616,618,620,622,624,626,628,630,632,634,636,638,640,642,644,646,648,650,652,654,656,658,660,662,664,666,668,670,672,674,676,678,680,682,684,686,688,690,692,694,696,698,700,702,704,706,708,710,712,714,716,718,720,722,724,726,728,730,732,734,736,738,740,742,744,746,748,750,752,754,756,758,760,762,764,766,768,770,772,774,776,778,780,782,784,786,788,790,792,794,796,798,800,802,804,806,808,810,812,814,816,818,820,822,824,826,828,830,832,834,836,838,840,842,844,846,848,850,852,854,856,858,860,862,864,866,868,870,872,874,876,878,880,882,884,886,888,890,892,894,896,898,900,902,904,906,908,910,912,914,916,918,920,922,924,926,928,930,932,934,936,938,940,942,944,946,948,950,952,954,956,958,960,962,964,966,968,970,972,974,976,978,980,982,984,986,988,990,992,994,996,998,1000,1002,1004,1006,1008,1010,1012,1014,1016,1018,1020,1022,1024,1026,1028,1030,1032,1034,1036,1038,1040,1042,1044,1046,1048,1050,1052,1054,1056,1058,1060,1062,1064,1066,1068,1070,1072,1074,1076,1078,1080,1082,1084,1086,1088,1090,1092,1094,1096,1098,1100,1102,1104,1106,1108,1110,1112,1114,1116,1118,1120,1122,1124,1126,1128,1130,1132,1134,1136,1138,1140,1142,1144,1146,1148,1150,1152,1154,1156,1158,1160,1162,1164,1166,1168,1170,1172,1174,1176,1178,1180,1182,1184,1186,1188,1190,1192,1194,1196,1198,1200,1202,1204,1206,1208,1210,1212,1214,1216,1218,1220,1222,1224,1226,1228,1230,1232,1234,1236,1238,1240,1242,1244,1246,1248,1250,1252,1254,1256,1258,1260,1262,1264,1266,1268,1270,1272,1274,1276,1278,1280,1282,1284,1286,1288,1290,1292,1294,1296,1298,1300,1302,1304,1306,1308,1310,1312,1314,1316,1318,1320,1322,1324,1326,1328,1330,1332,1334,1336,1338,1340,1342,1344,1346,1348,1350,1352,1354,1356,1358,1360,1362,1364,1366,1368,1370,1372,1374,1376,1378,1380,1382,1384,1386,1388,1390,1392,1394,1396,1398,1400,1402,1404,1406,1408,1410,1412,1414,1416,1418,1420,1422,1424,1426,1428,1430,1432,1434,1436,1438,1440,1442,1444,1446,1448,1450,1452,1454,1456,1458,1460,1462,1464,1466,1468,1470,1472,1474,1476,1478,1480,1482,1484,1486,1488,1490,1492,1494,1496,1498,1500,1502,1504,1506,1508,1510,1512,1514,1516,1518,1520,1522,1524,1526,1528,1530,1532,1534,1536,1538,1540,1542,1544,1546,1548,1550,1552,1554,1556,1558,1560,1562,1564,1566,1568,1570,1572,1574,1576,1578,1580,1582,1584,1586,1588,1590,1592,1594,1596,1598,1600,1602,1604,1606,1608,1610,1612,1614,1616,1618,1620,1622,1624,1626,1628,1630,1632,1634,1636,1638,1640,1642,1644,1646,1648,1650,1652,1654,1656,1658,1660,1662,1664,1666,1668,1670,1672,1674,1676,1678,1680,1682,1684,1686,1688,1690,1692,1694,1696,1698,1700,1702,1704,1706,1708,1710,1712,1714,1716,1718,1720,1722,1724,1726,1728,1730,1732,1734,1736,1738,1740,1742,1744,1746,1748,1750,1752,1754,1756,1758,1760,1762,1764,1766,1768,1770,1772,1774,1776,1778,1780,1782,1784,1786,1788,1790,1792,1794,1796,1798,1800,1802,1804,1806,1808,1810,1812,1814,1816,1818,1820,1822,1824,1826,1828,1830,1832,1834,1836,1838,1840,1842,1844,1846,1848,1850,1852,1854,1856,1858,1860,1862,1864,1866,1868,1870,1872,1874,1876,1878,1880,1882,1884,1886,1888,1890,1892,1894,1896,1898,1900,1902,1904,1906,1908,1910,1912,1914,1916,1918,1920,1922,1924,1926,1928,1930,1932,1934,1936,1938,1940,1942,1944,1946,1948,1950,1952,1954,1956,1958,1960,1962,1964,1966,1968,1970,1972,1974,1976,1978,1980,1982,1984,1986,1988,1990,1992,1994,1996,1998,2000,2002,2004,2006,2008,2010,2012,2014,2016,2018,2020,2022,2024,2026,2028,2030,2032,2034,2036,2038,2040,2042,2044,2046,2048,2050,2052,2054,2056,2058,2060,2062,2064,2066,2068,2070,2072,2074,2076,2078,2080,2082,2084,2086,2088,2090,2092,2094,2096,2098,2100,2102,2104,2106,2108,2110,2112,2114,2116,2118,2120,2122,2124,2126,2128,2130,2132,2134,2136,2138,2140,2142,2144,2146,2148,2150,2152,2154,2156,2158,2160,2162,2164,2166,2168,2170,2172,2174,2176,2178,2180,2182,2184,2186,2188,2190,2192,2194,2196,2198,2200,2202,2204,2206,2208,2210,2212,2214,2216,2218,2220,2222,2224,2226,2228,2230,2232,2234,2236,2238,2240,2242,2244,2246,2248,2250,2252,2254,2256,2258,2260,2262,2264,2266,2268,2270,2272,2274,2276,2278,2280,2282,2284,2286,2288,2290,2292,2294,2296,2298,2300,2302,2304,2306,2308,2310,2312,2314,2316,2318,2320,2322,2324,2326,2328,2330,2332,2334,2336,2338,2340,2342,2344,2346,2348,2350,2352,2354,2356,2358,2360,2362,2364,2366,2368,2370,2372,2374,2376,2378,2380,2382,2384,2386,2388,2390,2392,2394,2396,2398,2400,2402,2404,2406,2408,2410,2412,2414,2416,2418,2420,2422,2424,2426,2428,2430,2432,2434,2436,2438,2440,2442,2444,2446,2448,2450,2452,2454,2456,2458,2460,2462,2464,2466,2468,2470,2472,2474,2476,2478,2480,2482,2484,2486,2488,2490,2492,2494,2496,2498,2500,2502,2504,2506,2508,2510,2512,2514,2516,2518,2520,2522,2524,2526,2528,2530,2532,2534,2536,2538,2540,2542,2544,2546,2548,2550,2552,2554,2556,2558,2560,2562,2564,2566,2568,2570,2572,2574,2576,2578,2580,2582,2584,2586,2588,2590,2592,2594,2596,2598,2600,2602,2604,2606,2608,2610,2612,2614,2616,2618,2620,2622,2624,2626,2628,2630,2632,2634,2636,2638,2640,2642,2644,2646,2648,2650,2652,2654,2656,2658,2660,2662,2664,2666,2668,2670,2672,2674,2676,2678,2680,2682,2684,2686,2688,2690,2692,2694,2696,2698,2700,2702,2704,2706,2708,2710,2712,2714,2716,2718,2720,2722,2724,2726,2728,2730,2732,2734,2736,2738,2740,2742,2744,2746,2748,2750,2752,2754,2756,2758,2760,2762,2764,2766,2768,2770,2772,2774,2776,2778,2780,2782,2784,2786,2788,2790,2792,2794,2796,2798,2800,2802,2804,2806,2808,2810,2812,2814,2816,2818,2820,2822,2824,2826,2828,2830,2832,2834,2836,2838,2840,2842,2844,2846,2848,2850,2852,2854,2856,2858,2860,2862,2864,2866,2868,2870,2872,2874,2876,2878,2880,2882,2884,2886,2888,2890,2892,2894,2896,2898,2900,2902,2904,2906,2908,2910,2912,2914,2916,2918,2920,2922,2924,2926,2928,2930,2932,2934,2936,2938,2940,2942,2944,2946,2948,2950,2952,2954,2956,2958,2960,2962,2964,2966,2968,2970,2972,2974,2976,2978,2980,2982,2984,2986,2988,2990,2992,2994,2996,2998,3000,3002,3004,3006,3008,3010,3012,3014,3016,3018,3020,3022,3024,3026,3028,3030,3032,3034,3036,3038,3040,3042,3044,3046,3048,3050,3052,3054,3056,3058,3060,3062,3064,3066,3068,3070,3072,3074,3076,3078,3080,3082,3084,3086,3088,3090,3092,3094,3096,3098,3100,3102,3104,3106,3108,3110,3112,3114,3116,3118,3120,3122,3124,3126,3128,3130,3132,3134,3136,3138,3140,3142,3144,3146,3148,3150,3152,3154,3156,3158,3160,3162,3164,3166,3168,3170,3172,3174,3176,3178,3180,3182,3184,3186,3188,3190,3192,3194,3196,3198,3200,3202,3204,3206,3208,3210,3212,3214,3216,3218,3220,3222,3224,3226,3228,3230,3232,3234,3236,3238,3240,3242,3244,3246,3248,3250,3252,3254,3256,3258,3260,3262,3264,3266,3268,3270,3272,3274,3276,3278,3280,3282,3284,3286,3288,3290,3292,3294,3296,3298,3300,3302,3304,3306,3308,3310,3312,3314,3316,3318,3320,3322,3324,3326,3328,3330,3332,3334,3336,3338,3340,3342,3344,3346,3348,3350,3352,3354,3356,3358,3360,3362,3364,3366,3368,3370,3372,3374,3376,3378,3380,3382,3384,3386,3388,3390,3392,3394,3396,3398,3400,3402,3404,3406,3408,3410,3412,3414,3416,3418,3420,3422,3424,3426,3428,3430,3432,3434,3436,3438,3440,3442,3444,3446,3448,3450,3452,3454,3456,3458,3460,3462,3464,3466,3468,3470,3472,3474,3476,3478,3480,3482,3484,3486,3488,3490,3492,3494,3496,3498,3500,3502,3504,3506,3508,3510,3512,3514,3516,3518,3520,3522,3524,3526,3528,3530,3532,3534,3536,3538,3540,3542,3544,3546,3548,3550,3552,3554,3556,3558,3560,3562,3564,3566,3568,3570,3572,3574,3576,3578,3580,3582,3584,3586,3588,3590,3592,3594,3596,3598,3600,3602,3604,3606,3608,3610,3612,3614,3616,3618,3620,3622,3624,3626,3628,3630,3632,3634,3636,3638,3640,3642,3644,3646,3648,3650,3652,3654,3656,3658,3660,3662,3664,3666,3668,3670,3672,3674,3676,3678,3680,3682,3684,3686,3688,3690,3692,3694,3696,3698,3700,3702,3704],{"categories":139},[140],"Developer Productivity",{"categories":142},[143],"Business & SaaS",{"categories":145},[146],"AI & LLMs",{"categories":148},[149],"AI Automation",{"categories":151},[152],"Product Strategy",{"categories":154},[146],{"categories":156},[140],{"categories":158},[143],{"categories":160},[],{"categories":162},[146],{"categories":164},[],{"categories":166},[167],"AI News & Trends",{"categories":169},[149],{"categories":171},[167],{"categories":173},[149],{"categories":175},[149],{"categories":177},[146],{"categories":179},[146],{"categories":181},[167],{"categories":183},[146],{"categories":185},[],{"categories":187},[188],"Design & Frontend",{"categories":190},[191],"Data Science & Visualization",{"categories":193},[167],{"categories":195},[],{"categories":197},[198],"Software Engineering",{"categories":200},[146],{"categories":202},[149],{"categories":204},[205],"Marketing & Growth",{"categories":207},[146],{"categories":209},[149],{"categories":211},[],{"categories":213},[],{"categories":215},[188],{"categories":217},[149],{"categories":219},[140],{"categories":221},[188],{"categories":223},[146],{"categories":225},[149],{"categories":227},[167],{"categories":229},[],{"categories":231},[],{"categories":233},[149],{"categories":235},[198],{"categories":237},[],{"categories":239},[143],{"categories":241},[],{"categories":243},[],{"categories":245},[149],{"categories":247},[149],{"categories":249},[146],{"categories":251},[],{"categories":253},[198],{"categories":255},[],{"categories":257},[],{"categories":259},[],{"categories":261},[146],{"categories":263},[205],{"categories":265},[188],{"categories":267},[188],{"categories":269},[146],{"categories":271},[149],{"categories":273},[146],{"categories":275},[146],{"categories":277},[149],{"categories":279},[149],{"categories":281},[191],{"categories":283},[167],{"categories":285},[149],{"categories":287},[205],{"categories":289},[149],{"categories":291},[152],{"categories":293},[],{"categories":295},[149],{"categories":297},[],{"categories":299},[149],{"categories":301},[198],{"categories":303},[188],{"categories":305},[146],{"categories":307},[],{"categories":309},[],{"categories":311},[149],{"categories":313},[],{"categories":315},[146],{"categories":317},[],{"categories":319},[140],{"categories":321},[198],{"categories":323},[143],{"categories":325},[167],{"categories":327},[146],{"categories":329},[],{"categories":331},[146],{"categories":333},[],{"categories":335},[198],{"categories":337},[191],{"categories":339},[],{"categories":341},[146],{"categories":343},[188],{"categories":345},[],{"categories":347},[188],{"categories":349},[149],{"categories":351},[],{"categories":353},[149],{"categories":355},[167],{"categories":357},[146],{"categories":359},[],{"categories":361},[149],{"categories":363},[146],{"categories":365},[152],{"categories":367},[],{"categories":369},[146],{"categories":371},[149],{"categories":373},[149],{"categories":375},[],{"categories":377},[191],{"categories":379},[146],{"categories":381},[],{"categories":383},[140],{"categories":385},[143],{"categories":387},[146],{"categories":389},[149],{"categories":391},[198],{"categories":393},[146],{"categories":395},[],{"categories":397},[],{"categories":399},[146],{"categories":401},[],{"categories":403},[188],{"categories":405},[],{"categories":407},[146],{"categories":409},[],{"categories":411},[149],{"categories":413},[146],{"categories":415},[188],{"categories":417},[],{"categories":419},[146],{"categories":421},[146],{"categories":423},[143],{"categories":425},[149],{"categories":427},[146],{"categories":429},[188],{"categories":431},[149],{"categories":433},[],{"categories":435},[],{"categories":437},[167],{"categories":439},[],{"categories":441},[146],{"categories":443},[143,205],{"categories":445},[],{"categories":447},[146],{"categories":449},[],{"categories":451},[],{"categories":453},[146],{"categories":455},[],{"categories":457},[146],{"categories":459},[91],{"categories":461},[],{"categories":463},[167],{"categories":465},[188],{"categories":467},[],{"categories":469},[167],{"categories":471},[167],{"categories":473},[146],{"categories":475},[205],{"categories":477},[],{"categories":479},[143],{"categories":481},[],{"categories":483},[146,91],{"categories":485},[146],{"categories":487},[146],{"categories":489},[149],{"categories":491},[146,198],{"categories":493},[191],{"categories":495},[146],{"categories":497},[205],{"categories":499},[149],{"categories":501},[149],{"categories":503},[],{"categories":505},[149],{"categories":507},[146,143],{"categories":509},[],{"categories":511},[188],{"categories":513},[188],{"categories":515},[],{"categories":517},[],{"categories":519},[167],{"categories":521},[],{"categories":523},[140],{"categories":525},[198],{"categories":527},[146],{"categories":529},[188],{"categories":531},[149],{"categories":533},[198],{"categories":535},[167],{"categories":537},[188],{"categories":539},[],{"categories":541},[146],{"categories":543},[146],{"categories":545},[146],{"categories":547},[167],{"categories":549},[140],{"categories":551},[146],{"categories":553},[149],{"categories":555},[91],{"categories":557},[188],{"categories":559},[149],{"categories":561},[],{"categories":563},[],{"categories":565},[188],{"categories":567},[167],{"categories":569},[191],{"categories":571},[],{"categories":573},[146],{"categories":575},[146],{"categories":577},[143],{"categories":579},[146],{"categories":581},[146],{"categories":583},[167],{"categories":585},[],{"categories":587},[149],{"categories":589},[198],{"categories":591},[],{"categories":593},[146],{"categories":595},[146],{"categories":597},[149],{"categories":599},[],{"categories":601},[],{"categories":603},[146],{"categories":605},[],{"categories":607},[143],{"categories":609},[149],{"categories":611},[],{"categories":613},[140],{"categories":615},[146],{"categories":617},[143],{"categories":619},[167],{"categories":621},[],{"categories":623},[],{"categories":625},[],{"categories":627},[167],{"categories":629},[167],{"categories":631},[],{"categories":633},[],{"categories":635},[143],{"categories":637},[],{"categories":639},[],{"categories":641},[140],{"categories":643},[],{"categories":645},[205],{"categories":647},[149],{"categories":649},[143],{"categories":651},[149],{"categories":653},[],{"categories":655},[152],{"categories":657},[188],{"categories":659},[198],{"categories":661},[146],{"categories":663},[149],{"categories":665},[143],{"categories":667},[146],{"categories":669},[],{"categories":671},[],{"categories":673},[198],{"categories":675},[191],{"categories":677},[152],{"categories":679},[149],{"categories":681},[146],{"categories":683},[],{"categories":685},[91],{"categories":687},[],{"categories":689},[149],{"categories":691},[],{"categories":693},[],{"categories":695},[146],{"categories":697},[188],{"categories":699},[205],{"categories":701},[149],{"categories":703},[],{"categories":705},[140],{"categories":707},[],{"categories":709},[167],{"categories":711},[146,91],{"categories":713},[167],{"categories":715},[146],{"categories":717},[143],{"categories":719},[146],{"categories":721},[],{"categories":723},[143],{"categories":725},[],{"categories":727},[198],{"categories":729},[188],{"categories":731},[167],{"categories":733},[191],{"categories":735},[140],{"categories":737},[146],{"categories":739},[198],{"categories":741},[],{"categories":743},[],{"categories":745},[152],{"categories":747},[],{"categories":749},[146],{"categories":751},[],{"categories":753},[188],{"categories":755},[188],{"categories":757},[188],{"categories":759},[],{"categories":761},[],{"categories":763},[167],{"categories":765},[149],{"categories":767},[146],{"categories":769},[146],{"categories":771},[146],{"categories":773},[143],{"categories":775},[146],{"categories":777},[],{"categories":779},[198],{"categories":781},[198],{"categories":783},[143],{"categories":785},[],{"categories":787},[146],{"categories":789},[146],{"categories":791},[143],{"categories":793},[167],{"categories":795},[205],{"categories":797},[149],{"categories":799},[],{"categories":801},[188],{"categories":803},[],{"categories":805},[146],{"categories":807},[],{"categories":809},[143],{"categories":811},[149],{"categories":813},[],{"categories":815},[91],{"categories":817},[191],{"categories":819},[198],{"categories":821},[205],{"categories":823},[198],{"categories":825},[149],{"categories":827},[],{"categories":829},[],{"categories":831},[149],{"categories":833},[140],{"categories":835},[149],{"categories":837},[152],{"categories":839},[143],{"categories":841},[],{"categories":843},[146],{"categories":845},[152],{"categories":847},[146],{"categories":849},[146],{"categories":851},[205],{"categories":853},[188],{"categories":855},[149],{"categories":857},[],{"categories":859},[],{"categories":861},[91],{"categories":863},[198],{"categories":865},[],{"categories":867},[149],{"categories":869},[146],{"categories":871},[188,146],{"categories":873},[140],{"categories":875},[],{"categories":877},[146],{"categories":879},[140],{"categories":881},[188],{"categories":883},[149],{"categories":885},[198],{"categories":887},[],{"categories":889},[146],{"categories":891},[],{"categories":893},[140],{"categories":895},[],{"categories":897},[149],{"categories":899},[152],{"categories":901},[146],{"categories":903},[146],{"categories":905},[188],{"categories":907},[149],{"categories":909},[91],{"categories":911},[188],{"categories":913},[149],{"categories":915},[146],{"categories":917},[146],{"categories":919},[146],{"categories":921},[167],{"categories":923},[],{"categories":925},[152],{"categories":927},[149],{"categories":929},[188],{"categories":931},[149],{"categories":933},[198],{"categories":935},[188],{"categories":937},[149],{"categories":939},[167],{"categories":941},[],{"categories":943},[146],{"categories":945},[188],{"categories":947},[146],{"categories":949},[140],{"categories":951},[167],{"categories":953},[146],{"categories":955},[205],{"categories":957},[146],{"categories":959},[146],{"categories":961},[149],{"categories":963},[149],{"categories":965},[146],{"categories":967},[149],{"categories":969},[188],{"categories":971},[146],{"categories":973},[],{"categories":975},[],{"categories":977},[198],{"categories":979},[],{"categories":981},[140],{"categories":983},[91],{"categories":985},[],{"categories":987},[140],{"categories":989},[143],{"categories":991},[205],{"categories":993},[],{"categories":995},[143],{"categories":997},[],{"categories":999},[],{"categories":1001},[],{"categories":1003},[],{"categories":1005},[],{"categories":1007},[146],{"categories":1009},[149],{"categories":1011},[91],{"categories":1013},[140],{"categories":1015},[146],{"categories":1017},[198],{"categories":1019},[152],{"categories":1021},[146],{"categories":1023},[205],{"categories":1025},[146],{"categories":1027},[146],{"categories":1029},[146],{"categories":1031},[146,140],{"categories":1033},[198],{"categories":1035},[198],{"categories":1037},[188],{"categories":1039},[146],{"categories":1041},[],{"categories":1043},[],{"categories":1045},[],{"categories":1047},[198],{"categories":1049},[191],{"categories":1051},[167],{"categories":1053},[188],{"categories":1055},[],{"categories":1057},[146],{"categories":1059},[146],{"categories":1061},[],{"categories":1063},[],{"categories":1065},[149],{"categories":1067},[146],{"categories":1069},[143],{"categories":1071},[],{"categories":1073},[140],{"categories":1075},[146],{"categories":1077},[140],{"categories":1079},[146],{"categories":1081},[198],{"categories":1083},[205],{"categories":1085},[146,188],{"categories":1087},[167],{"categories":1089},[188],{"categories":1091},[],{"categories":1093},[91],{"categories":1095},[188],{"categories":1097},[149],{"categories":1099},[],{"categories":1101},[],{"categories":1103},[],{"categories":1105},[],{"categories":1107},[198],{"categories":1109},[149],{"categories":1111},[149],{"categories":1113},[146],{"categories":1115},[146],{"categories":1117},[],{"categories":1119},[188],{"categories":1121},[],{"categories":1123},[],{"categories":1125},[149],{"categories":1127},[],{"categories":1129},[],{"categories":1131},[205],{"categories":1133},[205],{"categories":1135},[149],{"categories":1137},[],{"categories":1139},[146],{"categories":1141},[146],{"categories":1143},[198],{"categories":1145},[188],{"categories":1147},[188],{"categories":1149},[149],{"categories":1151},[140],{"categories":1153},[146],{"categories":1155},[188],{"categories":1157},[188],{"categories":1159},[149],{"categories":1161},[149],{"categories":1163},[146],{"categories":1165},[],{"categories":1167},[],{"categories":1169},[146],{"categories":1171},[149],{"categories":1173},[167],{"categories":1175},[198],{"categories":1177},[140],{"categories":1179},[146],{"categories":1181},[],{"categories":1183},[149],{"categories":1185},[149],{"categories":1187},[],{"categories":1189},[140],{"categories":1191},[146],{"categories":1193},[140],{"categories":1195},[140],{"categories":1197},[],{"categories":1199},[],{"categories":1201},[149],{"categories":1203},[149],{"categories":1205},[146],{"categories":1207},[146],{"categories":1209},[167],{"categories":1211},[191],{"categories":1213},[152],{"categories":1215},[167],{"categories":1217},[188],{"categories":1219},[],{"categories":1221},[167],{"categories":1223},[],{"categories":1225},[],{"categories":1227},[],{"categories":1229},[],{"categories":1231},[198],{"categories":1233},[191],{"categories":1235},[],{"categories":1237},[146],{"categories":1239},[146],{"categories":1241},[191],{"categories":1243},[198],{"categories":1245},[],{"categories":1247},[],{"categories":1249},[149],{"categories":1251},[167],{"categories":1253},[167],{"categories":1255},[149],{"categories":1257},[140],{"categories":1259},[146,91],{"categories":1261},[],{"categories":1263},[188],{"categories":1265},[140],{"categories":1267},[149],{"categories":1269},[188],{"categories":1271},[],{"categories":1273},[149],{"categories":1275},[149],{"categories":1277},[146],{"categories":1279},[205],{"categories":1281},[198],{"categories":1283},[188],{"categories":1285},[],{"categories":1287},[149],{"categories":1289},[146],{"categories":1291},[149],{"categories":1293},[149],{"categories":1295},[149],{"categories":1297},[205],{"categories":1299},[149],{"categories":1301},[146],{"categories":1303},[],{"categories":1305},[205],{"categories":1307},[167],{"categories":1309},[149],{"categories":1311},[],{"categories":1313},[],{"categories":1315},[146],{"categories":1317},[149],{"categories":1319},[167],{"categories":1321},[149],{"categories":1323},[],{"categories":1325},[],{"categories":1327},[],{"categories":1329},[149],{"categories":1331},[],{"categories":1333},[],{"categories":1335},[191],{"categories":1337},[146],{"categories":1339},[191],{"categories":1341},[167],{"categories":1343},[146],{"categories":1345},[146],{"categories":1347},[149],{"categories":1349},[146],{"categories":1351},[],{"categories":1353},[],{"categories":1355},[91],{"categories":1357},[],{"categories":1359},[],{"categories":1361},[140],{"categories":1363},[],{"categories":1365},[],{"categories":1367},[],{"categories":1369},[],{"categories":1371},[198],{"categories":1373},[167],{"categories":1375},[205],{"categories":1377},[143],{"categories":1379},[146],{"categories":1381},[146],{"categories":1383},[143],{"categories":1385},[],{"categories":1387},[188],{"categories":1389},[149],{"categories":1391},[143],{"categories":1393},[146],{"categories":1395},[146],{"categories":1397},[140],{"categories":1399},[],{"categories":1401},[140],{"categories":1403},[146],{"categories":1405},[205],{"categories":1407},[149],{"categories":1409},[167],{"categories":1411},[143],{"categories":1413},[146],{"categories":1415},[149],{"categories":1417},[],{"categories":1419},[146],{"categories":1421},[140],{"categories":1423},[146],{"categories":1425},[],{"categories":1427},[167],{"categories":1429},[146],{"categories":1431},[],{"categories":1433},[143],{"categories":1435},[146],{"categories":1437},[],{"categories":1439},[],{"categories":1441},[],{"categories":1443},[146],{"categories":1445},[],{"categories":1447},[91],{"categories":1449},[146],{"categories":1451},[],{"categories":1453},[146],{"categories":1455},[146],{"categories":1457},[146],{"categories":1459},[146,91],{"categories":1461},[146],{"categories":1463},[146],{"categories":1465},[188],{"categories":1467},[149],{"categories":1469},[],{"categories":1471},[149],{"categories":1473},[146],{"categories":1475},[146],{"categories":1477},[146],{"categories":1479},[140],{"categories":1481},[140],{"categories":1483},[198],{"categories":1485},[188],{"categories":1487},[149],{"categories":1489},[],{"categories":1491},[146],{"categories":1493},[167],{"categories":1495},[146],{"categories":1497},[143],{"categories":1499},[],{"categories":1501},[91],{"categories":1503},[188],{"categories":1505},[188],{"categories":1507},[149],{"categories":1509},[167],{"categories":1511},[149],{"categories":1513},[146],{"categories":1515},[],{"categories":1517},[146],{"categories":1519},[],{"categories":1521},[],{"categories":1523},[146],{"categories":1525},[146],{"categories":1527},[146],{"categories":1529},[149],{"categories":1531},[146],{"categories":1533},[],{"categories":1535},[191],{"categories":1537},[149],{"categories":1539},[],{"categories":1541},[146],{"categories":1543},[167],{"categories":1545},[],{"categories":1547},[188],{"categories":1549},[91],{"categories":1551},[167],{"categories":1553},[198],{"categories":1555},[198],{"categories":1557},[167],{"categories":1559},[167],{"categories":1561},[91],{"categories":1563},[],{"categories":1565},[167],{"categories":1567},[146],{"categories":1569},[140],{"categories":1571},[167],{"categories":1573},[],{"categories":1575},[191],{"categories":1577},[167],{"categories":1579},[198],{"categories":1581},[167],{"categories":1583},[91],{"categories":1585},[146],{"categories":1587},[146],{"categories":1589},[],{"categories":1591},[143],{"categories":1593},[],{"categories":1595},[],{"categories":1597},[146],{"categories":1599},[146],{"categories":1601},[146],{"categories":1603},[146],{"categories":1605},[],{"categories":1607},[191],{"categories":1609},[140],{"categories":1611},[],{"categories":1613},[146],{"categories":1615},[146],{"categories":1617},[91],{"categories":1619},[91],{"categories":1621},[],{"categories":1623},[149],{"categories":1625},[167],{"categories":1627},[167],{"categories":1629},[146],{"categories":1631},[149],{"categories":1633},[],{"categories":1635},[188],{"categories":1637},[146],{"categories":1639},[146],{"categories":1641},[],{"categories":1643},[],{"categories":1645},[91],{"categories":1647},[146],{"categories":1649},[198],{"categories":1651},[143],{"categories":1653},[146],{"categories":1655},[],{"categories":1657},[149],{"categories":1659},[140],{"categories":1661},[140],{"categories":1663},[],{"categories":1665},[146],{"categories":1667},[188],{"categories":1669},[149],{"categories":1671},[],{"categories":1673},[146],{"categories":1675},[146],{"categories":1677},[149],{"categories":1679},[],{"categories":1681},[149],{"categories":1683},[198],{"categories":1685},[],{"categories":1687},[146],{"categories":1689},[],{"categories":1691},[146],{"categories":1693},[],{"categories":1695},[146],{"categories":1697},[146],{"categories":1699},[],{"categories":1701},[146],{"categories":1703},[167],{"categories":1705},[146],{"categories":1707},[146],{"categories":1709},[140],{"categories":1711},[146],{"categories":1713},[167],{"categories":1715},[149],{"categories":1717},[],{"categories":1719},[146],{"categories":1721},[205],{"categories":1723},[],{"categories":1725},[],{"categories":1727},[],{"categories":1729},[140],{"categories":1731},[167],{"categories":1733},[149],{"categories":1735},[146],{"categories":1737},[188],{"categories":1739},[149],{"categories":1741},[],{"categories":1743},[149],{"categories":1745},[],{"categories":1747},[146],{"categories":1749},[149],{"categories":1751},[146],{"categories":1753},[],{"categories":1755},[146],{"categories":1757},[146],{"categories":1759},[167],{"categories":1761},[188],{"categories":1763},[149],{"categories":1765},[188],{"categories":1767},[143],{"categories":1769},[],{"categories":1771},[],{"categories":1773},[146],{"categories":1775},[140],{"categories":1777},[167],{"categories":1779},[],{"categories":1781},[],{"categories":1783},[198],{"categories":1785},[188],{"categories":1787},[],{"categories":1789},[146],{"categories":1791},[],{"categories":1793},[205],{"categories":1795},[146],{"categories":1797},[91],{"categories":1799},[198],{"categories":1801},[],{"categories":1803},[149],{"categories":1805},[146],{"categories":1807},[149],{"categories":1809},[149],{"categories":1811},[146],{"categories":1813},[],{"categories":1815},[140],{"categories":1817},[146],{"categories":1819},[143],{"categories":1821},[198],{"categories":1823},[188],{"categories":1825},[],{"categories":1827},[],{"categories":1829},[],{"categories":1831},[149],{"categories":1833},[188],{"categories":1835},[167],{"categories":1837},[146],{"categories":1839},[167],{"categories":1841},[188],{"categories":1843},[],{"categories":1845},[188],{"categories":1847},[167],{"categories":1849},[143],{"categories":1851},[146],{"categories":1853},[167],{"categories":1855},[205],{"categories":1857},[],{"categories":1859},[],{"categories":1861},[191],{"categories":1863},[146,198],{"categories":1865},[167],{"categories":1867},[146],{"categories":1869},[149],{"categories":1871},[149],{"categories":1873},[146],{"categories":1875},[],{"categories":1877},[198],{"categories":1879},[146],{"categories":1881},[191],{"categories":1883},[149],{"categories":1885},[205],{"categories":1887},[91],{"categories":1889},[],{"categories":1891},[140],{"categories":1893},[149],{"categories":1895},[149],{"categories":1897},[198],{"categories":1899},[146],{"categories":1901},[146],{"categories":1903},[],{"categories":1905},[],{"categories":1907},[],{"categories":1909},[91],{"categories":1911},[167],{"categories":1913},[146],{"categories":1915},[146],{"categories":1917},[146],{"categories":1919},[],{"categories":1921},[191],{"categories":1923},[143],{"categories":1925},[],{"categories":1927},[149],{"categories":1929},[91],{"categories":1931},[],{"categories":1933},[188],{"categories":1935},[188],{"categories":1937},[],{"categories":1939},[198],{"categories":1941},[188],{"categories":1943},[146],{"categories":1945},[],{"categories":1947},[167],{"categories":1949},[146],{"categories":1951},[188],{"categories":1953},[149],{"categories":1955},[167],{"categories":1957},[],{"categories":1959},[149],{"categories":1961},[188],{"categories":1963},[146],{"categories":1965},[],{"categories":1967},[146],{"categories":1969},[146],{"categories":1971},[91],{"categories":1973},[167],{"categories":1975},[191],{"categories":1977},[191],{"categories":1979},[],{"categories":1981},[],{"categories":1983},[],{"categories":1985},[149],{"categories":1987},[198],{"categories":1989},[198],{"categories":1991},[],{"categories":1993},[],{"categories":1995},[146],{"categories":1997},[],{"categories":1999},[149],{"categories":2001},[146],{"categories":2003},[],{"categories":2005},[146],{"categories":2007},[143],{"categories":2009},[146],{"categories":2011},[205],{"categories":2013},[149],{"categories":2015},[146],{"categories":2017},[198],{"categories":2019},[167],{"categories":2021},[149],{"categories":2023},[],{"categories":2025},[167],{"categories":2027},[149],{"categories":2029},[149],{"categories":2031},[],{"categories":2033},[143],{"categories":2035},[149],{"categories":2037},[],{"categories":2039},[146],{"categories":2041},[140],{"categories":2043},[167],{"categories":2045},[91],{"categories":2047},[149],{"categories":2049},[149],{"categories":2051},[140],{"categories":2053},[146],{"categories":2055},[],{"categories":2057},[],{"categories":2059},[188],{"categories":2061},[146,143],{"categories":2063},[],{"categories":2065},[140],{"categories":2067},[191],{"categories":2069},[146],{"categories":2071},[198],{"categories":2073},[146],{"categories":2075},[149],{"categories":2077},[146],{"categories":2079},[146],{"categories":2081},[167],{"categories":2083},[149],{"categories":2085},[],{"categories":2087},[],{"categories":2089},[149],{"categories":2091},[146],{"categories":2093},[91],{"categories":2095},[],{"categories":2097},[146],{"categories":2099},[149],{"categories":2101},[],{"categories":2103},[146],{"categories":2105},[205],{"categories":2107},[191],{"categories":2109},[149],{"categories":2111},[146],{"categories":2113},[91],{"categories":2115},[],{"categories":2117},[146],{"categories":2119},[205],{"categories":2121},[188],{"categories":2123},[146],{"categories":2125},[],{"categories":2127},[205],{"categories":2129},[167],{"categories":2131},[146],{"categories":2133},[146],{"categories":2135},[140],{"categories":2137},[],{"categories":2139},[],{"categories":2141},[188],{"categories":2143},[146],{"categories":2145},[191],{"categories":2147},[205],{"categories":2149},[205],{"categories":2151},[167],{"categories":2153},[],{"categories":2155},[],{"categories":2157},[146],{"categories":2159},[],{"categories":2161},[146,198],{"categories":2163},[167],{"categories":2165},[149],{"categories":2167},[198],{"categories":2169},[146],{"categories":2171},[140],{"categories":2173},[],{"categories":2175},[],{"categories":2177},[140],{"categories":2179},[205],{"categories":2181},[146],{"categories":2183},[],{"categories":2185},[188,146],{"categories":2187},[91],{"categories":2189},[140],{"categories":2191},[],{"categories":2193},[143],{"categories":2195},[143],{"categories":2197},[146],{"categories":2199},[198],{"categories":2201},[149],{"categories":2203},[167],{"categories":2205},[205],{"categories":2207},[188],{"categories":2209},[146],{"categories":2211},[146],{"categories":2213},[146],{"categories":2215},[140],{"categories":2217},[146],{"categories":2219},[149],{"categories":2221},[167],{"categories":2223},[],{"categories":2225},[],{"categories":2227},[191],{"categories":2229},[198],{"categories":2231},[146],{"categories":2233},[188],{"categories":2235},[191],{"categories":2237},[146],{"categories":2239},[146],{"categories":2241},[149],{"categories":2243},[149],{"categories":2245},[146,143],{"categories":2247},[],{"categories":2249},[188],{"categories":2251},[],{"categories":2253},[146],{"categories":2255},[167],{"categories":2257},[140],{"categories":2259},[140],{"categories":2261},[149],{"categories":2263},[146],{"categories":2265},[143],{"categories":2267},[198],{"categories":2269},[205],{"categories":2271},[],{"categories":2273},[167],{"categories":2275},[146],{"categories":2277},[146],{"categories":2279},[167],{"categories":2281},[198],{"categories":2283},[146],{"categories":2285},[149],{"categories":2287},[167],{"categories":2289},[146],{"categories":2291},[188],{"categories":2293},[146],{"categories":2295},[146],{"categories":2297},[91],{"categories":2299},[152],{"categories":2301},[149],{"categories":2303},[146],{"categories":2305},[167],{"categories":2307},[149],{"categories":2309},[205],{"categories":2311},[146],{"categories":2313},[],{"categories":2315},[146],{"categories":2317},[],{"categories":2319},[],{"categories":2321},[],{"categories":2323},[143],{"categories":2325},[146],{"categories":2327},[149],{"categories":2329},[167],{"categories":2331},[167],{"categories":2333},[167],{"categories":2335},[167],{"categories":2337},[],{"categories":2339},[140],{"categories":2341},[149],{"categories":2343},[167],{"categories":2345},[140],{"categories":2347},[149],{"categories":2349},[146],{"categories":2351},[146,149],{"categories":2353},[149],{"categories":2355},[91],{"categories":2357},[167],{"categories":2359},[167],{"categories":2361},[149],{"categories":2363},[146],{"categories":2365},[],{"categories":2367},[167],{"categories":2369},[205],{"categories":2371},[140],{"categories":2373},[146],{"categories":2375},[146],{"categories":2377},[],{"categories":2379},[198],{"categories":2381},[],{"categories":2383},[140],{"categories":2385},[149],{"categories":2387},[167],{"categories":2389},[146],{"categories":2391},[167],{"categories":2393},[140],{"categories":2395},[167],{"categories":2397},[167],{"categories":2399},[],{"categories":2401},[143],{"categories":2403},[149],{"categories":2405},[167],{"categories":2407},[167],{"categories":2409},[167],{"categories":2411},[167],{"categories":2413},[167],{"categories":2415},[167],{"categories":2417},[167],{"categories":2419},[167],{"categories":2421},[167],{"categories":2423},[167],{"categories":2425},[191],{"categories":2427},[140],{"categories":2429},[146],{"categories":2431},[146],{"categories":2433},[],{"categories":2435},[146,140],{"categories":2437},[],{"categories":2439},[149],{"categories":2441},[167],{"categories":2443},[149],{"categories":2445},[146],{"categories":2447},[146],{"categories":2449},[146],{"categories":2451},[146],{"categories":2453},[146],{"categories":2455},[149],{"categories":2457},[143],{"categories":2459},[188],{"categories":2461},[167],{"categories":2463},[146],{"categories":2465},[],{"categories":2467},[],{"categories":2469},[149],{"categories":2471},[188],{"categories":2473},[146],{"categories":2475},[],{"categories":2477},[],{"categories":2479},[205],{"categories":2481},[146],{"categories":2483},[],{"categories":2485},[],{"categories":2487},[140],{"categories":2489},[143],{"categories":2491},[146],{"categories":2493},[143],{"categories":2495},[188],{"categories":2497},[],{"categories":2499},[167],{"categories":2501},[],{"categories":2503},[188],{"categories":2505},[146],{"categories":2507},[205],{"categories":2509},[],{"categories":2511},[205],{"categories":2513},[],{"categories":2515},[],{"categories":2517},[149],{"categories":2519},[],{"categories":2521},[143],{"categories":2523},[140],{"categories":2525},[188],{"categories":2527},[198],{"categories":2529},[],{"categories":2531},[],{"categories":2533},[146],{"categories":2535},[140],{"categories":2537},[205],{"categories":2539},[],{"categories":2541},[149],{"categories":2543},[149],{"categories":2545},[167],{"categories":2547},[146],{"categories":2549},[149],{"categories":2551},[146],{"categories":2553},[149],{"categories":2555},[146],{"categories":2557},[152],{"categories":2559},[167],{"categories":2561},[],{"categories":2563},[205],{"categories":2565},[198],{"categories":2567},[149],{"categories":2569},[],{"categories":2571},[146],{"categories":2573},[149],{"categories":2575},[143],{"categories":2577},[140],{"categories":2579},[146],{"categories":2581},[188],{"categories":2583},[198],{"categories":2585},[198],{"categories":2587},[146],{"categories":2589},[191],{"categories":2591},[146],{"categories":2593},[149],{"categories":2595},[143],{"categories":2597},[149],{"categories":2599},[146],{"categories":2601},[146],{"categories":2603},[149],{"categories":2605},[167],{"categories":2607},[],{"categories":2609},[140],{"categories":2611},[146],{"categories":2613},[149],{"categories":2615},[146],{"categories":2617},[146],{"categories":2619},[],{"categories":2621},[188],{"categories":2623},[143],{"categories":2625},[167],{"categories":2627},[146],{"categories":2629},[146],{"categories":2631},[188],{"categories":2633},[205],{"categories":2635},[191],{"categories":2637},[146],{"categories":2639},[167],{"categories":2641},[146],{"categories":2643},[149],{"categories":2645},[91],{"categories":2647},[146],{"categories":2649},[149],{"categories":2651},[191],{"categories":2653},[],{"categories":2655},[149],{"categories":2657},[198],{"categories":2659},[188],{"categories":2661},[146],{"categories":2663},[140],{"categories":2665},[143],{"categories":2667},[198],{"categories":2669},[],{"categories":2671},[149],{"categories":2673},[146],{"categories":2675},[],{"categories":2677},[167],{"categories":2679},[],{"categories":2681},[167],{"categories":2683},[146],{"categories":2685},[149],{"categories":2687},[149],{"categories":2689},[149],{"categories":2691},[],{"categories":2693},[],{"categories":2695},[146],{"categories":2697},[146],{"categories":2699},[],{"categories":2701},[188],{"categories":2703},[149],{"categories":2705},[205],{"categories":2707},[140],{"categories":2709},[],{"categories":2711},[],{"categories":2713},[167],{"categories":2715},[198],{"categories":2717},[146],{"categories":2719},[146],{"categories":2721},[146],{"categories":2723},[198],{"categories":2725},[167],{"categories":2727},[188],{"categories":2729},[146],{"categories":2731},[146],{"categories":2733},[146],{"categories":2735},[167],{"categories":2737},[146],{"categories":2739},[167],{"categories":2741},[149],{"categories":2743},[149],{"categories":2745},[198],{"categories":2747},[149],{"categories":2749},[146],{"categories":2751},[198],{"categories":2753},[188],{"categories":2755},[],{"categories":2757},[149],{"categories":2759},[],{"categories":2761},[],{"categories":2763},[143],{"categories":2765},[146],{"categories":2767},[149],{"categories":2769},[140],{"categories":2771},[149],{"categories":2773},[205],{"categories":2775},[],{"categories":2777},[149],{"categories":2779},[],{"categories":2781},[140],{"categories":2783},[149],{"categories":2785},[],{"categories":2787},[149],{"categories":2789},[146],{"categories":2791},[167],{"categories":2793},[146],{"categories":2795},[149],{"categories":2797},[167],{"categories":2799},[149],{"categories":2801},[198],{"categories":2803},[188],{"categories":2805},[140],{"categories":2807},[],{"categories":2809},[149],{"categories":2811},[188],{"categories":2813},[167],{"categories":2815},[146],{"categories":2817},[188],{"categories":2819},[140],{"categories":2821},[],{"categories":2823},[149],{"categories":2825},[149],{"categories":2827},[146],{"categories":2829},[],{"categories":2831},[149],{"categories":2833},[152],{"categories":2835},[167],{"categories":2837},[149],{"categories":2839},[143],{"categories":2841},[],{"categories":2843},[146],{"categories":2845},[152],{"categories":2847},[146],{"categories":2849},[149],{"categories":2851},[167],{"categories":2853},[140],{"categories":2855},[91],{"categories":2857},[146],{"categories":2859},[146],{"categories":2861},[146],{"categories":2863},[167],{"categories":2865},[143],{"categories":2867},[146],{"categories":2869},[188],{"categories":2871},[167],{"categories":2873},[91],{"categories":2875},[146],{"categories":2877},[],{"categories":2879},[],{"categories":2881},[91],{"categories":2883},[191],{"categories":2885},[149],{"categories":2887},[149],{"categories":2889},[167],{"categories":2891},[146],{"categories":2893},[140],{"categories":2895},[188],{"categories":2897},[149],{"categories":2899},[146],{"categories":2901},[205],{"categories":2903},[146],{"categories":2905},[149],{"categories":2907},[],{"categories":2909},[146],{"categories":2911},[146],{"categories":2913},[167],{"categories":2915},[140],{"categories":2917},[],{"categories":2919},[146],{"categories":2921},[146],{"categories":2923},[198],{"categories":2925},[188],{"categories":2927},[146,149],{"categories":2929},[205,143],{"categories":2931},[146],{"categories":2933},[],{"categories":2935},[149],{"categories":2937},[],{"categories":2939},[198],{"categories":2941},[146],{"categories":2943},[167],{"categories":2945},[],{"categories":2947},[149],{"categories":2949},[],{"categories":2951},[149],{"categories":2953},[140],{"categories":2955},[149],{"categories":2957},[146],{"categories":2959},[91],{"categories":2961},[205],{"categories":2963},[143],{"categories":2965},[143],{"categories":2967},[140],{"categories":2969},[140],{"categories":2971},[146],{"categories":2973},[149],{"categories":2975},[146],{"categories":2977},[146],{"categories":2979},[140],{"categories":2981},[146],{"categories":2983},[205],{"categories":2985},[167],{"categories":2987},[146],{"categories":2989},[149],{"categories":2991},[146],{"categories":2993},[],{"categories":2995},[198],{"categories":2997},[],{"categories":2999},[149],{"categories":3001},[140],{"categories":3003},[],{"categories":3005},[91],{"categories":3007},[146],{"categories":3009},[],{"categories":3011},[167],{"categories":3013},[149],{"categories":3015},[198],{"categories":3017},[146],{"categories":3019},[149],{"categories":3021},[198],{"categories":3023},[149],{"categories":3025},[167],{"categories":3027},[140],{"categories":3029},[167],{"categories":3031},[198],{"categories":3033},[146],{"categories":3035},[188],{"categories":3037},[146],{"categories":3039},[146],{"categories":3041},[146],{"categories":3043},[146],{"categories":3045},[149],{"categories":3047},[146],{"categories":3049},[149],{"categories":3051},[146],{"categories":3053},[140],{"categories":3055},[146],{"categories":3057},[149],{"categories":3059},[188],{"categories":3061},[140],{"categories":3063},[149],{"categories":3065},[188],{"categories":3067},[],{"categories":3069},[146],{"categories":3071},[146],{"categories":3073},[198],{"categories":3075},[],{"categories":3077},[149],{"categories":3079},[205],{"categories":3081},[146],{"categories":3083},[167],{"categories":3085},[205],{"categories":3087},[149],{"categories":3089},[143],{"categories":3091},[143],{"categories":3093},[146],{"categories":3095},[140],{"categories":3097},[],{"categories":3099},[146],{"categories":3101},[],{"categories":3103},[140],{"categories":3105},[146],{"categories":3107},[149],{"categories":3109},[149],{"categories":3111},[],{"categories":3113},[198],{"categories":3115},[198],{"categories":3117},[205],{"categories":3119},[188],{"categories":3121},[],{"categories":3123},[146],{"categories":3125},[140],{"categories":3127},[146],{"categories":3129},[198],{"categories":3131},[140],{"categories":3133},[167],{"categories":3135},[167],{"categories":3137},[],{"categories":3139},[167],{"categories":3141},[149],{"categories":3143},[188],{"categories":3145},[191],{"categories":3147},[146],{"categories":3149},[],{"categories":3151},[167],{"categories":3153},[198],{"categories":3155},[143],{"categories":3157},[146],{"categories":3159},[140],{"categories":3161},[91],{"categories":3163},[140],{"categories":3165},[],{"categories":3167},[],{"categories":3169},[167],{"categories":3171},[],{"categories":3173},[149],{"categories":3175},[149],{"categories":3177},[149],{"categories":3179},[],{"categories":3181},[146],{"categories":3183},[],{"categories":3185},[167],{"categories":3187},[140],{"categories":3189},[188],{"categories":3191},[146],{"categories":3193},[167],{"categories":3195},[167],{"categories":3197},[],{"categories":3199},[167],{"categories":3201},[140],{"categories":3203},[146],{"categories":3205},[],{"categories":3207},[149],{"categories":3209},[149],{"categories":3211},[140],{"categories":3213},[],{"categories":3215},[],{"categories":3217},[],{"categories":3219},[188],{"categories":3221},[149],{"categories":3223},[146],{"categories":3225},[],{"categories":3227},[],{"categories":3229},[],{"categories":3231},[188],{"categories":3233},[],{"categories":3235},[140],{"categories":3237},[],{"categories":3239},[],{"categories":3241},[188],{"categories":3243},[146],{"categories":3245},[167],{"categories":3247},[],{"categories":3249},[205],{"categories":3251},[167],{"categories":3253},[205],{"categories":3255},[146],{"categories":3257},[],{"categories":3259},[],{"categories":3261},[149],{"categories":3263},[],{"categories":3265},[],{"categories":3267},[149],{"categories":3269},[146],{"categories":3271},[],{"categories":3273},[149],{"categories":3275},[167],{"categories":3277},[205],{"categories":3279},[191],{"categories":3281},[149],{"categories":3283},[149],{"categories":3285},[],{"categories":3287},[],{"categories":3289},[],{"categories":3291},[167],{"categories":3293},[],{"categories":3295},[],{"categories":3297},[188],{"categories":3299},[140],{"categories":3301},[],{"categories":3303},[143],{"categories":3305},[205],{"categories":3307},[146],{"categories":3309},[198],{"categories":3311},[140],{"categories":3313},[191],{"categories":3315},[143],{"categories":3317},[198],{"categories":3319},[],{"categories":3321},[],{"categories":3323},[149],{"categories":3325},[140],{"categories":3327},[188],{"categories":3329},[140],{"categories":3331},[149],{"categories":3333},[91],{"categories":3335},[149],{"categories":3337},[],{"categories":3339},[146],{"categories":3341},[167],{"categories":3343},[198],{"categories":3345},[],{"categories":3347},[188],{"categories":3349},[167],{"categories":3351},[140],{"categories":3353},[149],{"categories":3355},[146],{"categories":3357},[143],{"categories":3359},[149,91],{"categories":3361},[149],{"categories":3363},[198],{"categories":3365},[146],{"categories":3367},[191],{"categories":3369},[205],{"categories":3371},[149],{"categories":3373},[],{"categories":3375},[149],{"categories":3377},[146],{"categories":3379},[143],{"categories":3381},[],{"categories":3383},[],{"categories":3385},[146],{"categories":3387},[191],{"categories":3389},[146],{"categories":3391},[],{"categories":3393},[167],{"categories":3395},[],{"categories":3397},[167],{"categories":3399},[198],{"categories":3401},[149],{"categories":3403},[146],{"categories":3405},[205],{"categories":3407},[198],{"categories":3409},[],{"categories":3411},[167],{"categories":3413},[146],{"categories":3415},[],{"categories":3417},[146],{"categories":3419},[149],{"categories":3421},[146],{"categories":3423},[149],{"categories":3425},[146],{"categories":3427},[146],{"categories":3429},[146],{"categories":3431},[146],{"categories":3433},[143],{"categories":3435},[],{"categories":3437},[152],{"categories":3439},[167],{"categories":3441},[146],{"categories":3443},[],{"categories":3445},[198],{"categories":3447},[146],{"categories":3449},[146],{"categories":3451},[149],{"categories":3453},[167],{"categories":3455},[146],{"categories":3457},[146],{"categories":3459},[143],{"categories":3461},[149],{"categories":3463},[188],{"categories":3465},[],{"categories":3467},[191],{"categories":3469},[146],{"categories":3471},[],{"categories":3473},[167],{"categories":3475},[205],{"categories":3477},[],{"categories":3479},[],{"categories":3481},[167],{"categories":3483},[167],{"categories":3485},[205],{"categories":3487},[140],{"categories":3489},[149],{"categories":3491},[149],{"categories":3493},[146],{"categories":3495},[143],{"categories":3497},[],{"categories":3499},[],{"categories":3501},[167],{"categories":3503},[191],{"categories":3505},[198],{"categories":3507},[149],{"categories":3509},[188],{"categories":3511},[191],{"categories":3513},[191],{"categories":3515},[],{"categories":3517},[167],{"categories":3519},[146],{"categories":3521},[146],{"categories":3523},[198],{"categories":3525},[],{"categories":3527},[167],{"categories":3529},[167],{"categories":3531},[167],{"categories":3533},[],{"categories":3535},[149],{"categories":3537},[146],{"categories":3539},[],{"categories":3541},[140],{"categories":3543},[143],{"categories":3545},[],{"categories":3547},[146],{"categories":3549},[146],{"categories":3551},[],{"categories":3553},[198],{"categories":3555},[],{"categories":3557},[],{"categories":3559},[],{"categories":3561},[],{"categories":3563},[146],{"categories":3565},[167],{"categories":3567},[],{"categories":3569},[],{"categories":3571},[146],{"categories":3573},[146],{"categories":3575},[146],{"categories":3577},[191],{"categories":3579},[146],{"categories":3581},[191],{"categories":3583},[],{"categories":3585},[191],{"categories":3587},[191],{"categories":3589},[91],{"categories":3591},[149],{"categories":3593},[198],{"categories":3595},[],{"categories":3597},[],{"categories":3599},[191],{"categories":3601},[198],{"categories":3603},[198],{"categories":3605},[198],{"categories":3607},[],{"categories":3609},[140],{"categories":3611},[198],{"categories":3613},[198],{"categories":3615},[140],{"categories":3617},[198],{"categories":3619},[143],{"categories":3621},[198],{"categories":3623},[198],{"categories":3625},[198],{"categories":3627},[191],{"categories":3629},[167],{"categories":3631},[167],{"categories":3633},[146],{"categories":3635},[198],{"categories":3637},[191],{"categories":3639},[91],{"categories":3641},[191],{"categories":3643},[191],{"categories":3645},[191],{"categories":3647},[],{"categories":3649},[143],{"categories":3651},[],{"categories":3653},[91],{"categories":3655},[198],{"categories":3657},[198],{"categories":3659},[198],{"categories":3661},[149],{"categories":3663},[167,143],{"categories":3665},[191],{"categories":3667},[],{"categories":3669},[],{"categories":3671},[191],{"categories":3673},[],{"categories":3675},[191],{"categories":3677},[167],{"categories":3679},[149],{"categories":3681},[],{"categories":3683},[198],{"categories":3685},[146],{"categories":3687},[188],{"categories":3689},[],{"categories":3691},[146],{"categories":3693},[],{"categories":3695},[167],{"categories":3697},[140],{"categories":3699},[191],{"categories":3701},[],{"categories":3703},[198],{"categories":3705},[167],[3707,3780,3832,3902],{"id":3708,"title":3709,"ai":3710,"body":3715,"categories":3760,"created_at":92,"date_modified":92,"description":84,"extension":93,"faq":92,"featured":94,"kicker_label":92,"meta":3761,"navigation":119,"path":3767,"published_at":3768,"question":92,"scraped_at":3769,"seo":3770,"sitemap":3771,"source_id":3772,"source_name":3773,"source_type":127,"source_url":3774,"stem":3775,"tags":3776,"thumbnail_url":92,"tldr":3777,"tweet":92,"unknown_tags":3778,"__hash__":3779},"summaries\u002Fsummaries\u002F73f55123201134f9-mount-s3-buckets-as-file-systems-with-aws-s3-files-summary.md","Mount S3 Buckets as File Systems with AWS S3 Files",{"provider":7,"model":8,"input_tokens":3711,"output_tokens":3712,"processing_time_ms":3713,"cost_usd":3714},3939,1507,8922,0.00151865,{"type":14,"value":3716,"toc":3755},[3717,3721,3724,3731,3735,3738,3741,3745,3748],[17,3718,3720],{"id":3719},"s3-files-delivers-native-file-system-access-to-s3","S3 Files Delivers Native File System Access to S3",[22,3722,3723],{},"AWS S3 Files transforms object storage into a POSIX-compliant file system mountable on EC2 instances, containers, and Lambda functions. This eliminates custom hacks like FUSE wrappers or periodic sync scripts, providing low-latency read\u002Fwrite access indistinguishable from local disks for AI\u002FML, data engineering, and DevOps workloads. Under the hood, it leverages S3's metadata for directory structures and supports standard file operations without data migration—your existing buckets work immediately.",[22,3725,3726,3727,3730],{},"To implement, grant IAM roles with s3:PutObject, s3:GetObject, etc., permissions scoped to the bucket prefix, then mount via AWS CLI or SDK: ",[36,3728,3729],{},"aws s3files mount s3:\u002F\u002Fyour-bucket \u002Fmnt\u002Fpoint",". This cuts integration time from hours of scripting to minutes, enabling seamless data access in containerized ML training pipelines or serverless inference.",[17,3732,3734],{"id":3733},"realistic-use-cases-in-aiml-and-devops","Realistic Use Cases in AI\u002FML and DevOps",[22,3736,3737],{},"For AI\u002FML teams, mount training datasets directly into Jupyter on EC2 or SageMaker, avoiding costly EBS volumes or data downloads—process petabyte-scale S3 data at near-native speeds. DevOps benefits include containerized ETL jobs reading\u002Fwriting S3 as local files without volume mounts, and Lambda functions handling file I\u002FO for event-driven processing without temporary storage hacks.",[22,3739,3740],{},"Trade-offs: Strong consistency for small files (\u003C100MB), eventual consistency for large ones; throughput caps at S3's request rates (3,500 PUT\u002FGET per prefix\u002Fsec). Ideal for read-heavy ML feature stores or log processing, less so for high-write transactional DBs.",[17,3742,3744],{"id":3743},"avoid-common-pitfalls-security-cost-data-risks","Avoid Common Pitfalls: Security, Cost, Data Risks",[22,3746,3747],{},"Misconfigurations amplify S3's pitfalls: Broad IAM policies expose buckets publicly—use least-privilege with bucket policies denying public access and encrypting at-rest\u002Ftransit. Costs spike from unoptimized PUTs (e.g., frequent small writes); batch operations and use Intelligent-Tiering to mitigate, monitoring via CloudWatch for >$0.023\u002FGB PUT fees.",[22,3749,3750,3751,3754],{},"Data loss hits from concurrent writes without locks—implement app-level semaphores or use S3 atomic operations. Test mounts in staging: unmount with ",[36,3752,3753],{},"aws s3files unmount \u002Fmnt\u002Fpoint"," to verify no corruption. Always enable versioning and MFA-delete on production buckets.",{"title":84,"searchDepth":85,"depth":85,"links":3756},[3757,3758,3759],{"id":3719,"depth":85,"text":3720},{"id":3733,"depth":85,"text":3734},{"id":3743,"depth":85,"text":3744},[91],{"content_references":3762,"triage":3763},[],{"relevance":3764,"novelty":115,"quality":115,"actionability":115,"composite":3765,"reasoning":3766},5,4.35,"Category: DevOps & Cloud. The article provides a detailed explanation of how AWS S3 Files can be used to enhance AI\u002FML workflows by transforming S3 buckets into file systems, addressing a specific pain point for developers looking to streamline data access. It includes practical implementation steps and highlights potential pitfalls, making it actionable for the target audience.","\u002Fsummaries\u002F73f55123201134f9-mount-s3-buckets-as-file-systems-with-aws-s3-files-summary","2026-04-18 18:01:01","2026-04-19 01:22:18",{"title":3709,"description":84},{"loc":3767},"73f55123201134f9","Towards AI","https:\u002F\u002Fpub.towardsai.net\u002Faws-s3-files-explained-the-smarter-way-to-turn-s3-buckets-into-file-systems-3459560f7046?source=rss----98111c9905da---4","summaries\u002F73f55123201134f9-mount-s3-buckets-as-file-systems-with-aws-s3-files-summary",[131,132],"AWS S3 Files mounts buckets directly as file systems on EC2, containers, and Lambda—eliminating FUSE hacks and sync scripts for AI\u002FML workflows, but misconfigurations risk exposing, corrupting, or losing data.",[],"i9cGWnIzxtJTktm1csYQQ2MxZ9wdsHkt8i8DFEsZ0tQ",{"id":3781,"title":3782,"ai":3783,"body":3788,"categories":3814,"created_at":92,"date_modified":92,"description":84,"extension":93,"faq":92,"featured":94,"kicker_label":92,"meta":3815,"navigation":119,"path":3819,"published_at":3820,"question":92,"scraped_at":3821,"seo":3822,"sitemap":3823,"source_id":3824,"source_name":3825,"source_type":127,"source_url":3826,"stem":3827,"tags":3828,"thumbnail_url":92,"tldr":3829,"tweet":92,"unknown_tags":3830,"__hash__":3831},"summaries\u002Fsummaries\u002F71dc58e232e9091c-zero-leak-debt-kill-100-leaked-secrets-platform-wi-summary.md","Zero Leak Debt: Kill 100+ Leaked Secrets Platform-Wide",{"provider":7,"model":8,"input_tokens":3784,"output_tokens":3785,"processing_time_ms":3786,"cost_usd":3787},3867,1345,10526,0.00142325,{"type":14,"value":3789,"toc":3810},[3790,3794,3797,3800,3804,3807],[17,3791,3793],{"id":3792},"leak-debt-persists-for-years-undermining-platforms","Leak Debt Persists for Years, Undermining Platforms",[22,3795,3796],{},"Leaked secrets accumulate as 'leak debt,' remaining active long after exposure—transaction keys from 2022 continued processing payments undetected. Every platform accumulates this debt differently based on stack, but it kills security and reliability. The author shares hands-on experience eliminating 100+ live leaks across local development, CI\u002FCD pipelines, and production environments, revealing a universal pattern: sprawl leads to chaos until teams commit to zero tolerance.",[22,3798,3799],{},"Static secrets create ongoing risks because they expire unexpectedly or demand manual rotation, amplifying vulnerabilities. Platforms suffer uniquely—GitOps teams battle repo exposures, service meshes grapple with identity issues—but all chase the same outcome: secrets that self-manage without human intervention.",[17,3801,3803],{"id":3802},"ruthless-audit-and-prevention-path-to-zero-debt","Ruthless Audit and Prevention Path to Zero Debt",[22,3805,3806],{},"Transition from chaos requires three steps: discover the mess through comprehensive scans, audit ruthlessly to prioritize live threats (e.g., still-valid 2022 keys), and enforce prevention via dynamic tools. Teams adopt stack-specific solutions like HashiCorp Vault for centralized management, AWS or GCP Secrets Manager for cloud-native rotation, Sealed Secrets for GitOps, or SPIFFE for service meshes.",[22,3808,3809],{},"This isn't a generic checklist but proven patterns from production battles: replace static secrets entirely to eliminate leak debt. Outcomes include no leaks, automatic rotation, and zero manual interventions, securing platforms end-to-end. The content cuts off mid-journey but emphasizes sharing these learnings for peer teams facing identical sprawl.",{"title":84,"searchDepth":85,"depth":85,"links":3811},[3812,3813],{"id":3792,"depth":85,"text":3793},{"id":3802,"depth":85,"text":3803},[91],{"content_references":3816,"triage":3817},[],{"relevance":115,"novelty":116,"quality":115,"actionability":115,"composite":117,"reasoning":3818},"Category: DevOps & Cloud. The article addresses a specific pain point regarding the management of leaked secrets, which is crucial for maintaining security in AI-powered products. It provides actionable steps for auditing and preventing leak debt, making it relevant for developers and product builders.","\u002Fsummaries\u002F71dc58e232e9091c-zero-leak-debt-kill-100-leaked-secrets-platform-wi-summary","2026-04-15 14:55:38","2026-04-15 15:38:58",{"title":3782,"description":84},{"loc":3819},"71dc58e232e9091c","Level Up Coding","https:\u002F\u002Flevelup.gitconnected.com\u002Fmost-leaked-secrets-live-for-years-the-hidden-leak-debt-killing-your-platform-47e74da51697?source=rss----5517fd7b58a6---4","summaries\u002F71dc58e232e9091c-zero-leak-debt-kill-100-leaked-secrets-platform-wi-summary",[131,132],"Leaked secrets from 2022 still process payments as 'leak debt'; ruthlessly audit across local dev, CI\u002FCD, and production to reach zero static secrets that never leak, expire unexpectedly, or need manual rotation.",[],"vQXjNu8RKmXyJRZqlJVXzlosWzob2FyNxRtplwh2n3E",{"id":3833,"title":3834,"ai":3835,"body":3840,"categories":3874,"created_at":92,"date_modified":92,"description":84,"extension":93,"faq":92,"featured":94,"kicker_label":92,"meta":3875,"navigation":119,"path":3892,"published_at":92,"question":92,"scraped_at":3893,"seo":3894,"sitemap":3895,"source_id":3896,"source_name":126,"source_type":127,"source_url":112,"stem":3897,"tags":3898,"thumbnail_url":92,"tldr":3899,"tweet":92,"unknown_tags":3900,"__hash__":3901},"summaries\u002Fsummaries\u002F409c09e756b5a198-blue-green-deployment-for-zero-downtime-releases-summary.md","Blue-Green Deployment for Zero-Downtime Releases",{"provider":7,"model":8,"input_tokens":3836,"output_tokens":3837,"processing_time_ms":3838,"cost_usd":3839},4668,1440,8226,0.00114675,{"type":14,"value":3841,"toc":3869},[3842,3846,3849,3852,3856,3859,3862,3866],[17,3843,3845],{"id":3844},"deploy-with-instant-cutover-and-rollback","Deploy with Instant Cutover and Rollback",[22,3847,3848],{},"Run two production environments—blue (live) and green (idle)—kept as identical as possible, using separate hardware, VMs, or partitioned zones with distinct IP addresses. To release, perform final testing on the inactive green environment. Once verified, switch the router to route all traffic to green, making blue idle. This cutover happens in seconds, minimizing downtime during the transition from testing to production.",[22,3850,3851],{},"For rollback, reverse the router switch to blue if problems emerge post-deployment. Mitigate missed transactions by designing dual feeds to both environments or switching to read-only mode pre-cutover, flushing issues before enabling read-write. Cycle environments: after stabilizing green as live, repurpose blue for next release's staging.",[17,3853,3855],{"id":3854},"overcome-database-schema-hurdles","Overcome Database Schema Hurdles",[22,3857,3858],{},"Databases complicate switches due to schema changes for new app versions. Separate schema updates from app deployments: first apply database refactoring to make the schema backward-compatible, deploy and verify, establishing a rollback point. Then deploy the new app version. Once stable, remove old-version schema support. This ensures both environments share the database safely during transitions.",[22,3860,3861],{},"Variations include bouncing web servers instead of routers for switches or applying blue-green only to web\u002Fdomain layers while sharing the database.",[17,3863,3865],{"id":3864},"test-disaster-recovery-routinely","Test Disaster Recovery Routinely",[22,3867,3868],{},"Blue-green mirrors hot-standby setups, so every release tests your disaster-recovery process—switching live traffic proves failover works. Release frequently to practice this more than actual disasters occur. This technique, named by Daniel Terhorst-North and Jez Humble, supports fully automated deployments in continuous delivery pipelines.",{"title":84,"searchDepth":85,"depth":85,"links":3870},[3871,3872,3873],{"id":3844,"depth":85,"text":3845},{"id":3854,"depth":85,"text":3855},{"id":3864,"depth":85,"text":3865},[91],{"content_references":3876,"triage":3889},[3877,3882,3885],{"type":3878,"title":3879,"author":3880,"url":3881,"context":100},"book","Continuous Delivery","Dave Farley and Jez Humble","https:\u002F\u002Fmartinfowler.com\u002Fbooks\u002FcontinuousDelivery.html",{"type":3878,"title":3883,"url":3884,"context":113},"Refactoring Databases","https:\u002F\u002Fmartinfowler.com\u002Fbooks\u002FrefactoringDatabases.html",{"type":109,"title":3886,"author":3887,"url":3888,"context":104},"gitlab snippet on blue-green name origin","Daniel Terhorst-North and Jez Humble","https:\u002F\u002Fgitlab.com\u002Fsnippets\u002F1846041",{"relevance":3764,"novelty":116,"quality":115,"actionability":115,"composite":3890,"reasoning":3891},4.15,"Category: DevOps & Cloud. The article provides a detailed explanation of blue-green deployment, a relevant technique for minimizing downtime during releases, which directly addresses the audience's need for practical deployment strategies. It includes actionable steps for implementing the technique, such as separating schema updates from app deployments, making it applicable for product builders.","\u002Fsummaries\u002F409c09e756b5a198-blue-green-deployment-for-zero-downtime-releases-summary","2026-04-16 03:04:36",{"title":3834,"description":84},{"loc":3892},"409c09e756b5a198","summaries\u002F409c09e756b5a198-blue-green-deployment-for-zero-downtime-releases-summary",[131,133],"Maintain two identical production environments (blue and green): deploy new version to inactive one, switch traffic instantly for minimal downtime, and rollback by switching back if issues arise.",[133],"6vEoNtfqUHX99CsfIqizU70RHdhY9aOUGHv8ZnYKdPg",{"id":3903,"title":3904,"ai":3905,"body":3910,"categories":3946,"created_at":92,"date_modified":92,"description":84,"extension":93,"faq":92,"featured":94,"kicker_label":92,"meta":3947,"navigation":119,"path":3959,"published_at":3960,"question":92,"scraped_at":3960,"seo":3961,"sitemap":3962,"source_id":3963,"source_name":3964,"source_type":127,"source_url":3965,"stem":3966,"tags":3967,"thumbnail_url":92,"tldr":3969,"tweet":92,"unknown_tags":3970,"__hash__":3971},"summaries\u002Fsummaries\u002Fa2a811b50a4c64f5-mrc-resilient-networking-for-100k-gpu-ai-training-summary.md","MRC: Resilient Networking for 100K+ GPU AI Training",{"provider":7,"model":8,"input_tokens":3906,"output_tokens":3907,"processing_time_ms":3908,"cost_usd":3909},9014,2044,25377,0.0028023,{"type":14,"value":3911,"toc":3940},[3912,3916,3919,3923,3926,3930,3933,3937],[17,3913,3915],{"id":3914},"multi-plane-topologies-slash-switch-tiers-and-power-for-massive-clusters","Multi-Plane Topologies Slash Switch Tiers and Power for Massive Clusters",[22,3917,3918],{},"Traditional 800Gb\u002Fs networks require three or four tiers of switches to connect over 100,000 GPUs, increasing power use, failure points, and cost. MRC splits each 800Gb\u002Fs interface into eight 100Gb\u002Fs links, creating eight parallel 'planes' that connect to separate switches. A 64-port 800Gb\u002Fs switch now handles 512 ports at 100Gb\u002Fs, enabling full connectivity for 131,000 GPUs using only two tiers. This design boosts path diversity—keeping more traffic local to Tier 0 switches—while cutting components, power, and cost compared to single-plane setups. Without changes, single-path flows (like classic RoCE) still congest links as flows collide, especially in AI's collective communications where worst-case latency stalls synchronous training.",[17,3920,3922],{"id":3921},"packet-spraying-and-srv6-eliminate-congestion-and-dynamic-routing","Packet Spraying and SRv6 Eliminate Congestion and Dynamic Routing",[22,3924,3925],{},"MRC sprays packets from a single transfer across hundreds of paths spanning all planes, using final memory addresses for out-of-order reassembly at the destination. Adaptive load-balancing monitors paths: congestion triggers path swaps, packet loss retires the path (with probes for recovery), and 'packet trimming' at switches forwards headers only during destination congestion to prompt retransmits without false failure alarms. This achieves microsecond failure detection and rerouting, versus seconds for traditional fabrics. MRC replaces BGP dynamic routing with static SRv6 source routing: senders embed full switch ID sequences in IPv6 addresses. Switches shift addresses and follow pre-configured static tables, blindly forwarding without recomputing routes. Failures simply retire paths at endpoints, simplifying control planes and eliminating routing bugs from switch software.",[17,3927,3929],{"id":3928},"production-impact-zero-measurable-downtime-amid-constant-failures","Production Impact: Zero-Measurable Downtime Amid Constant Failures",[22,3931,3932],{},"In OpenAI's NVIDIA GB200 supercomputers (including OCI's Abilene Stargate site and Microsoft's Fairwater), MRC handles millions of links with frequent flaps—multiple per minute between tiers—yet synchronous pretraining jobs show no measurable impact, allowing deferred repairs. Rebooting four Tier-1 switches or repairing links during jobs requires no coordination; MRC avoids bad paths automatically. Real training data shows quick recovery from full T1 switch loss with temporary slowdowns far less than physical capacity loss (e.g., one failed port on an 8-port interface reduces max rate by 1\u002F8th but sustains better effective throughput via path recalculation). Multi-job clusters avoid inter-job interference due to core-wide congestion elimination, maximizing GPU utilization for frontier models like those powering ChatGPT (900M weekly users).",[17,3934,3936],{"id":3935},"strategic-wins-simpler-stacks-for-stargate-scale-compute","Strategic Wins: Simpler Stacks for Stargate-Scale Compute",[22,3938,3939],{},"MRC delivers three edges: two-tier multi-plane redundancy with lower power; zero core congestion for consistent flow throughput in sync training; and SRv6 for instant failure bypass via static planes. Deployed with AMD, Broadcom, Intel, Microsoft, NVIDIA hardware, it's released via Open Compute Project for industry adoption, supporting OpenAI's compute strategy of shared standards to scale AI infrastructure efficiently.",{"title":84,"searchDepth":85,"depth":85,"links":3941},[3942,3943,3944,3945],{"id":3914,"depth":85,"text":3915},{"id":3921,"depth":85,"text":3922},{"id":3928,"depth":85,"text":3929},{"id":3935,"depth":85,"text":3936},[91],{"content_references":3948,"triage":3956},[3949,3952],{"type":109,"title":3950,"url":3951,"context":104},"OCP MRC 1.0","https:\u002F\u002Fwww.opencompute.org\u002Fdocuments\u002Focp-mrc-1-0-pdf",{"type":3953,"title":3954,"url":3955,"context":104},"paper","Resilient AI Supercomputer Networking using MRC and SRv6","https:\u002F\u002Fcdn.openai.com\u002Fpdf\u002Fresilient-ai-supercomputer-networking-using-mrc-and-srv6.pdf",{"relevance":116,"novelty":116,"quality":115,"actionability":85,"composite":3957,"reasoning":3958},3.05,"Category: DevOps & Cloud. The article discusses the MRC protocol's innovative networking solutions for AI training, which could be relevant for those building AI-powered products. However, it lacks direct actionable insights for the audience, focusing more on technical specifications than practical applications.","\u002Fsummaries\u002Fa2a811b50a4c64f5-mrc-resilient-networking-for-100k-gpu-ai-training-summary","2026-05-11 15:04:27",{"title":3904,"description":84},{"loc":3959},"a2a811b50a4c64f5","OpenAI News","https:\u002F\u002Fopenai.com\u002Findex\u002Fmrc-supercomputer-networking","summaries\u002Fa2a811b50a4c64f5-mrc-resilient-networking-for-100k-gpu-ai-training-summary",[3968,131,132],"machine-learning","OpenAI's MRC protocol uses multi-plane topologies and packet spraying across hundreds of paths with SRv6 source routing to eliminate congestion, route around failures in microseconds, and connect 131k GPUs with just two switch tiers, enabling non-stop frontier model training.",[],"BYXvfLzxxajQIir95xuUTVdTfvID4wPt3TOVHNxrCSU"]