Samir Behara has some tips for designing resilient microservices:
While architecting distributed cloud applications, you should assume that failures will happen and design your applications for resiliency. A Microservice ecosystem is going to fail at some point or the other and hence you need to learn embracing failures. Don’t design systems with the assumption that its going to be sunny throughout the year. Be realistic and account for the chances of having rain, snow, thunderstorms and other adverse conditions. In short, design your microservices with failure in mind. Things don’t go as per plan always and you need to be prepared for the worst case scenario.
If Service A calls Service B which in turn calls Service C, what happens when Service B is down? What is your fallback plan in such a scenario?
Can you return a pre-decided error message to the user?
Can you call another service to fetch the information?
Can you return values from cache instead?
Can you return a default value?
Microservices have their drawbacks, but one big advantage is that they tend to be concise enough that you can reason about them more clearly than kitchen sink applications. Planning ahead on potential failure modalities differentiates flaky services from robust services.