Why We Built Parlant: From Mission-Critical Systems to Conversational AI
Here's something not many know about our team. While we're now leading the development of Parlant—the open-source Conversational AI engine for reliable customer interactions—most of us didn't actually come from the NLP world.
Actually, our primary experience was in building mission-critical, real-time systems. Coming into NLP from that world was a bit of a shock, to be honest. We were used to ultimate control and determinism as a non-negotiable baseline, yet NLP offered very little of that; in fact, too little. And that was exactly our problem with it—and what we set out to solve.
The Three Challenges That Led to Parlant
So, about a year ago, after getting quite frustrated with the available options for building reliable Conversational AI with LLMs, our team set out to build a better solution, what today is Parlant. After extensive research on technology and countless interviews with others building CAI for the past few years, we set out to solve three big challenges that we've identified:
1. Nobody had a good idea of how to get started with building scalable Conversational AI agents with LLMs. Everybody was reinventing the wheel, and most were following development patterns that were doomed to fail from the start, simply because they were accessible and easier to wrap your head around as a newcomer.
2. Agent complexity couldn't scale. Either prompts got so intractable that the LLM would consistently fail to produce the desired results, or, otherwise, you had to divide your agent's knowledge into different "graph nodes" with isolated prompts that were exclusively specialized for a particular topic, paving the way to the agent's too-frequent incoherence, misinterpretation of conversation history, and, consequently, generating misleading, non-aligned outputs to users.
3. Production agents required efficient collaboration between project stakeholders, particularly the business experts and the developers. However, translating changing business requirements into your agent was notoriously difficult. You couldn't even trust the changes you were making, which begged the question: "Why even bother specifying them?" This has frustrated many people with LLM projects in general, which has unfortunately led to many projects being abandoned after several months of development.
"It doesn't have to be like this" was the thought that set us on our path to building Parlant.
Get in touch with us
Our Core Insight: Contextual Guidelines
Before diving into how we built Parlant's architecture, let me share the key innovation that makes it all work—because this was crucial to fixing the feedback cycle between business experts and developers.
Like I said before, one of the major issues in building customer-facing AI agents is the feedback cycle between business experts and developers. Business experts should be empowered to provide feedback clearly, and developers should be able to implement such feedback easily, quickly, and reliably.
It was clear to us that, as this feedback comes from business experts, it cannot be expected to come in the form of intricate flow-charts and things of that nature. Instead, it had to go through as few "artificial transformations" as possible, from the moment the business expert thinks about it to the moment it is turned into a requirement.
We then identified that when something doesn't work the way it should, a business expert looking at it can usually go, "Oh, that's not how it should be. Instead, it should be like that." But then the issue was that sometimes it should be like that, but other times it shouldn't, depending on the context.
For example, if the business expert says, "Oh, here the agent should be more proactive and push for a sale," it doesn't mean that the agent should act that way in all circumstances. It depends on the context.
This led to a key insight and design decision: Contextual Guidelines. Each guideline in Parlant would have a condition, specifying the set of circumstances in which it should be applied, and an action, which is the actual guidance for the agent's behavior.
This meant that business experts only had to do one thing whenever they desired a change in the agent's behavior: to specify that, "When [some condition holds], then [do something]." And this is extremely doable.
Of course, today, Parlant takes it even a step further. It supports things like guideline relationships, which effectively add another layer of control and precision to your agent's behavior. While not strictly needed in many cases, there are times when you really want to improve user engagement by a few extra percents, and Parlant's fine-grained control mechanisms will be there for you.
Get in touch with us
Parlant's Architectural Vision
With our core innovation defined, we had to build an architecture that could support it reliably. The first thing we had to recognize was that LLMs were fundamentally inconsistent. This led to two critical design decisions:
1. We needed to try to simplify the tasks that LLMs had to do. This meant breaking down the conversational processing pipeline into multiple, manageable steps, each with its own single responsibility.
To take just one example, in Parlant, the engine first has to determine what guidelines are immediately relevant for its next response. It then filters out all of the irrelevant ones, and only then does it pass the relevant guidelines to the LLM, which is now focused on a small set of clear instructions.
This way, we could ensure that the LLM's output was more predictable and aligned with the agent's intended behavior.
2. We had to guide every piece of code we wrote by tests, essentially following TDD (Test Driven Development) to ensure that everything is accounted for and that we're able to keep track of the engine's performance and consistency.
Today, Parlant's built-in test suite has hundreds of tests that cover all of the engine's functionality, and we run them every time we make a change to the codebase.
This way, we can be sure that our changes don't break anything and that the engine continues to perform as expected—even as new LLM versions are released.
From our extensive experience in software development, we knew that if we followed these two principles, we would be taking our best chances at taming this seemingly untameable technology.
Get in touch with us
First Steps: Getting the Conversation API Right
Importantly, one of the things we had to implement was a robust conversation API so that we could easily and quickly set up new conversational agents without reinventing the wheel every time.
But we wanted to make sure that this API was not just a simple wrapper around an LLM. Instead, we wanted it to be a full-fledged conversation engine that could handle all of the complexities of real-world conversations.
We thus built Parlant's conversational infrastructure focusing on the following features:
Built-In Conversation Store
A built-in persistent store of conversations, filled with all of the raw data on each conversation—on both the agent and the customer—because we know that data would be extremely valuable in analyzing agent performance.
Native Conversational Status Updates
Support for status updates, such as "Thinking," "Typing," etc. All seamlessly baked right into the API.
Event-Level Debugging with Correlation IDs
Each event in the conversation is associated with a correlation ID associated with the HTTP request that triggered its creation. This was needed to streamline debugging, so you could ask the system, "Give me all of the logs that came from generating this one particular event in this conversation."
Asynchronous, Free-Flowing Conversation API
An asynchronous API so that we could build a GUI where conversations didn't necessarily have to work like ping-pong, where the customer (or the agent) can only send one message at a time. That's just not how real conversations work. Real people often send multiple messages at a time to get their point across.
So we wanted our API to be future-compatible and enable conversations that weren't just fluent in their content, but also in their flow.
Backend-Initiated Agent Responses
Agent conversations needed to be triggerable asynchronously from the backend as well—for example, initiating a conversation with the user if you find some issue in their account, or, alternatively, updating them on the result of a long-running background task using a callback function.
Imagine your agent booking a flight ticket for a customer. This normally takes about 30-60 seconds to get sorted, but your agent can't just get stuck! It should still be responsive to your customer in their conversation. So we designed for that, too.
From Mission-Critical Systems to Conversational AI
Looking back, our journey from building deterministic, mission-critical systems to creating Parlant has been about bringing that same rigor and reliability to the world of conversational AI.
Parlant's whole existence stems from a desire to bring more control, predictability, and quality into generative AI agents. And now, a year later, we know that we had taken the right approach. Parlant agents are famously consistent in their instruction-following ability compared to those built from scratch with lower-level LLM frameworks.
The principles that guided us in our previous work—breaking down complex problems, testing everything, and empowering stakeholders—turned out to be exactly what the conversational AI space lacked.
We've gotten feedback from many people since, saying things like, "After a few months of banging my head against these LLMs, I realized this is the way it should be done... and then I found Parlant has already done it.".
By applying these principles to the inherently unpredictable world of LLMs, we've created something that finally makes reliable customer-facing AI agents achievable. And we're proud to offer it to the community as an open-source solution.
Try It Yourself!
If you haven't tried it yet, I encourage you to go through the official tutorial and see for yourself how seamless it is to create scalable agents in Parlant. It's all about getting the architecture right. When you do that, things naturally fall into place. And that's exactly what we've achieved with Parlant's architecture.
Get in touch with us