k3s replicaset start flooding backend service

fixing an issue where backend service got flooded by the pods

System got flooded

when I redeploy 200 replica,

200 requests hit the api at the same time

causing high response latency

and then the pod healthcheck failed, the pod got recreated, creating more requests

After some optimize

added index to mongodb

added cache to flask using python LRU cache with timeout

added random start sleep in pod


Time to recover

Second Test

14:32 start, 14:38 finish