Troubleshooting

You might be wondering why are we even discussing troubleshooting? It is obvious that while working on a new feature, whichever trouble comes your way you will have to shoot it down.

Working on a big platform composed of thousands of services and cooperating with over a hundred engineering teams as well as various other company stakeholders means that you are almost never working in isolation in your solitary cave on the next block of code waiting to be pushed in production, but rather it makes you a part of a larger organism. And that means you will inevitably run across support tickets and random inquiries ranging from "Let me Google that for you" all the way to interesting head-scratching treasure hunts through our codebase and documentation.

This is exactly why troubleshooting role exists in Infobip - usually each team dedicates a single person to handle any inquiries thrown their way regarding the services their own, but depending on the workload more members can be involved in the troubleshooting role. It is also a crucial part of our incident process - sometimes an issue on your service might affect our clients or inversely, an issue on our platform might affect your services. In those critical moments, your input is going to be crucial in making sure issues are resolved timely and effectively.

What's in it for you?

When you are not the troubleshooter, you get to work calmly on your tasks and plan your time effectively unobstructed from external noise.

But the real kicker is what happens when you ARE the troubleshooter - you learn the most while digging through the code trying to figure out how your services are a part of the bigger picture in ways no onboarding can ever prepare you.

At least I am free after working hours

Well, that depends :smile:

Certain teams that are responsible for crucial infrastructure components (network, hardware etc.) or crucial internal services have readiness schedules. If you are a part of one of these teams, it means that you will be in a rotation certain amount of days per month. You will be on call for any critical alerts that indicate client impact or calls for ongoing incidents that require your teams input for which you are compensated even if no actual interventions happen during that time.

In case that assistance is required from a team that does not fall under the readiness umbrella, you are not obliged to answer intervention calls outside of working hours. But if you do, you are of course compensated for your efforts.

And keep in mind that interventions are not the daily expectation - in Infobip we work hard to make sure our platform is robust enough to allow us to sleep and rest peacefully when we are not in the shift. But when something does go wrong, we need to act quickly to make sure our clients experience minimal disruption.

And remember, you are not alone

While this all sounds very important and critical and responsible (well, mainly because it is), don't forget that you are not alone in this.

You can always count on your teams support. Even if you are the designated troubleshooter, if you need a sanity check or help figuring something out you can always consult your team. We used to have calls where we would literally be WTFing together as a team over something that troubleshooter was tackling at the moment trying to figure it out together. And you can rely on this also in your team.

Same goes for off hours interventions and readiness - while you are the person responsible for any critical interventions during off hours, if you came across something very complex you cannot figure out you can always ask your team for help as well as any other teams that might have the information you need.

We nurture a strong spirit of cooperation especially during incidents so you can always count on having the resources you need to tackle the issue at hand.