AI infrastructure

Like many stories about projects at scale and roll-outs of new technology, the story of AI adoption in Infobip comes down to infrastructure. At each steps of our AI journey we try to approach questions pragmatically. Does it make sense to self-host all models used for everyday coding tasks? No. At least not today. Does it make sense to self-host models for AI research and model training? Yeah. So we did that, and along with the hardware capacity we also built the in-house know-how for self hosting LLMs.

So what does our AI infra looks like? And what are the use-cases IA is enabling for us? Let’s dive in.

Where are our LLMs running?

We use AI for both internal and external use-cases. It's a varied set of tasks, from Claude Code helping our engineers develop features and crush bugs, to our customers running their contact centers on our SaaS solution backed by AI agents. On top of that, there are in-house business processes optimized by AI. It is a mix of decision making, reasoning agents and simple text generation use-cases. It is also a mix of pure inference, eval testing, fine-tuning and training.

We are solving a lot of these use-cases with foundation models from major labs running in cloud. But, like I mentioned, we also have locally running AI hardware. Right now, it’s a pair of nVidia DGX B200s and a H200. But these things change fast, and we are procuring new hardware as I’m writing this. We are using vLLM to serve local models, which include selection of Qwen and Deepseek models.

How do we access LLMs?

For both internal and external use-cases we access LLMs over internally deployed proxies. In both cases they afford us flexibility in picking providers (and in some cases even models). They are also a single place where we can implement cross cutting concerns such as monitoring, metrics, authentication, etc. Each of the 2 broad use-cases have their specific requirements, though. That’s why we are using different implementations for each.

For internal usage all LLM calls go through a self-hosted LiteLLM. This allows us to control where those Claude Code requests land, and balance them across providers according to availability, affordability, etc. We integrated our internal tooling with LiteLLM, so employees can manage their API keys. We extract usage metrics and feed them into internal reporting systems. These numbers serve to guide our internal AI enablement team that can focus and optimize what matters. For example, we tweak default Claude Code settings including installing some default MCP servers and skills based on usage and performance metrics from LiteLLM.

On the other hand, for external, client-facing use-cases we are using an in-house developed proxy service. There we implemented all reliability requirements that we have for our production services. We instrumented it with observability that respects sensitivity of our customer’s data. We integrated it with our internal systems such as customer accounts, billing, reporting, etc. Like LiteLLM it also allows us to switch providers on the fly. This time we can even tweak some of the features, like enabling jailbreak protection without changing the applications that are making those API calls towards LLMs.

What do we do with those LLMs?

Let’s talk a bit more about those internal use-cases. Not every interaction with an LLM requires a fully agentic experience. For those random chats, one-off image generations and simple RAG powered answer bots we have a self-hosted Open WebUI instance. It is connected to LiteLLM, and is protected behind our single sign-on. It was one of our first interfaces to LLMs when it was originally deployed, back in the spring of 2024. It remains useful today, albeit in a smaller capacity, having been surpassed in capability by agentic interfaces.

Speaking of agents, we are heavy Claude Code users, and not just in engineering. We adopted CC across the entire organization, including non-technical departments. This allows us to standardize and share both our know-how and technical artifacts such as skills. In fact, we set up an internal library of agentic components. It lists all our in-house developed / deployed MCP servers, and all skills and plugins that we’ve written. Each department has their own marketplace, but anyone can benefit from those skills. Perhaps even more interestingly, anyone can contribute to them. And, speaking of MCPs, we have them all auto-discovered and listed automatically. We have an MCP proxy that handles authentication and integration with the single sign-on system. All you as a dev need to do is deploy your MCP server and users will be able to find it and access it securely, out of the box.

Working with Claude Code from terminal is just a starting point. As engineers we wanted a persistent environment where our agents can execute long running tasks. Non-technical folks wanted to interact with their agents over Teams chat, to hand them files from OneDrive, to schedule some tasks to run automatically. And both groups wanted to be able to save specific agent configurations composed of the harness, skills, MCP servers, instructions, etc. There may exist some off the shelf solutions for this, but none offered us the flexibility and integration with our specific set of workplace tools and services. So we built one ourselves. It’s called Helios, and it is a remote runtime environment for our agents. It helps agents natively integrate with tools that we use in the company, and implements our security policies, making it more practical and more secure to run autonomous agents.

When it comes to vibe coding, we found that limitations can be a good thing. Especially for folks in non-technical roles. It’s empowering for them to be able to create simple apps, interactive dashboards and functional prototypes. Showing up to a sales meeting with working demo of what final solution could look like is a powerful way to share our vision with our clients. But giving Claude Code free rein can and does result in some bad outcomes. Non-technical users can’t maintain production services, patch their dependencies or update OS kernel on VMs. And they shouldn’t be in a position to have Claude Code instruct them to upload a generated HTML page full of sensitive company information to Heroku, open for everyone to access. This is why we invested in building our own vibe coding platform (that, plus it was such a cool thing to build). We call it Atlas, and it allows anybody from the company to chat with an LLM that builds them a web dashboard. Dashboards are automatically hosted, they don’t pull in separate dependencies, they can have backend scripts and frontend components. They connect to internal data sources through tightly controlled interface, with credentials of the user accessing the dashboard. They even have access to a dedicated document storage database. The builder UX is intentionally beginner friendly; half of the screen is a familiar chat window where users talk with AI about the thing they wish to make, the other half is WYSIWYG live view of the dashboard, as it is being built. Atlas enables anyone from the company to create dynamic presentations, functional mock-ups, data visualizations, interactive reports, etc. all without maintenance costs, without the risk of leaking data, and with proper authentication and authorization.