k3s replicaset start flooding backend service
fixing an issue where backend service got flooded by the pods
System got flooded
when I redeploy 200 replica,
200 requests hit the api at the same time
causing high response latency
and then the pod healthcheck failed, the pod got recreated, creating more requests
After some optimize
added index to mongodb
added cache to flask using python LRU cache with timeout
added random start sleep in pod
⚠️
2023-03-03
Time to recover
Second Test
14:32 start, 14:38 finish
6mins
