2 posts tagged with "customer-facing-ai"

Here's something not many know about our team. While we're now leading the development of Parlant—the open-source Conversational AI engine for reliable customer interactions—most of us didn't actually come from the NLP world.

Actually, our primary experience was in building mission-critical, real-time systems. Coming into NLP from that world was a bit of a shock, to be honest. We were used to ultimate control and determinism as a non-negotiable baseline, yet NLP offered very little of that; in fact, too little. And that was exactly our problem with it—and what we set out to solve.

The Three Challenges That Led to Parlant​

So, about a year ago, after getting quite frustrated with the available options for building reliable Conversational AI with LLMs, our team set out to build a better solution, what today is Parlant. After extensive research on technology and countless interviews with others building CAI for the past few years, we set out to solve three big challenges that we've identified:

1. Nobody had a good idea of how to get started with building scalable Conversational AI agents with LLMs. Everybody was reinventing the wheel, and most were following development patterns that were doomed to fail from the start, simply because they were accessible and easier to wrap your head around as a newcomer.

2. Agent complexity couldn't scale. Either prompts got so intractable that the LLM would consistently fail to produce the desired results, or, otherwise, you had to divide your agent's knowledge into different "graph nodes" with isolated prompts that were exclusively specialized for a particular topic, paving the way to the agent's too-frequent incoherence, misinterpretation of conversation history, and, consequently, generating misleading, non-aligned outputs to users.

3. Production agents required efficient collaboration between project stakeholders, particularly the business experts and the developers. However, translating changing business requirements into your agent was notoriously difficult. You couldn't even trust the changes you were making, which begged the question: "Why even bother specifying them?" This has frustrated many people with LLM projects in general, which has unfortunately led to many projects being abandoned after several months of development.

"It doesn't have to be like this" was the thought that set us on our path to building Parlant.

Get in touch with us

Our Core Insight: Contextual Guidelines​

Before diving into how we built Parlant's architecture, let me share the key innovation that makes it all work—because this was crucial to fixing the feedback cycle between business experts and developers.

Like I said before, one of the major issues in building customer-facing AI agents is the feedback cycle between business experts and developers. Business experts should be empowered to provide feedback clearly, and developers should be able to implement such feedback easily, quickly, and reliably.

It was clear to us that, as this feedback comes from business experts, it cannot be expected to come in the form of intricate flow-charts and things of that nature. Instead, it had to go through as few "artificial transformations" as possible, from the moment the business expert thinks about it to the moment it is turned into a requirement.

We then identified that when something doesn't work the way it should, a business expert looking at it can usually go, "Oh, that's not how it should be. Instead, it should be like that." But then the issue was that sometimes it should be like that, but other times it shouldn't, depending on the context.

For example, if the business expert says, "Oh, here the agent should be more proactive and push for a sale," it doesn't mean that the agent should act that way in all circumstances. It depends on the context.

This led to a key insight and design decision: Contextual Guidelines. Each guideline in Parlant would have a condition, specifying the set of circumstances in which it should be applied, and an action, which is the actual guidance for the agent's behavior.

This meant that business experts only had to do one thing whenever they desired a change in the agent's behavior: to specify that, "When [some condition holds], then [do something]." And this is extremely doable.

Of course, today, Parlant takes it even a step further. It supports things like guideline relationships, which effectively add another layer of control and precision to your agent's behavior. While not strictly needed in many cases, there are times when you really want to improve user engagement by a few extra percents, and Parlant's fine-grained control mechanisms will be there for you.

Get in touch with us

Parlant's Architectural Vision​

With our core innovation defined, we had to build an architecture that could support it reliably. The first thing we had to recognize was that LLMs were fundamentally inconsistent. This led to two critical design decisions:

1. We needed to try to simplify the tasks that LLMs had to do. This meant breaking down the conversational processing pipeline into multiple, manageable steps, each with its own single responsibility.

To take just one example, in Parlant, the engine first has to determine what guidelines are immediately relevant for its next response. It then filters out all of the irrelevant ones, and only then does it pass the relevant guidelines to the LLM, which is now focused on a small set of clear instructions.

This way, we could ensure that the LLM's output was more predictable and aligned with the agent's intended behavior.

2. We had to guide every piece of code we wrote by tests, essentially following TDD (Test Driven Development) to ensure that everything is accounted for and that we're able to keep track of the engine's performance and consistency.

Today, Parlant's built-in test suite has hundreds of tests that cover all of the engine's functionality, and we run them every time we make a change to the codebase.

This way, we can be sure that our changes don't break anything and that the engine continues to perform as expected—even as new LLM versions are released.

From our extensive experience in software development, we knew that if we followed these two principles, we would be taking our best chances at taming this seemingly untameable technology.

Get in touch with us

First Steps: Getting the Conversation API Right​

Importantly, one of the things we had to implement was a robust conversation API so that we could easily and quickly set up new conversational agents without reinventing the wheel every time.

But we wanted to make sure that this API was not just a simple wrapper around an LLM. Instead, we wanted it to be a full-fledged conversation engine that could handle all of the complexities of real-world conversations.

We thus built Parlant's conversational infrastructure focusing on the following features:

Built-In Conversation Store​

A built-in persistent store of conversations, filled with all of the raw data on each conversation—on both the agent and the customer—because we know that data would be extremely valuable in analyzing agent performance.

Native Conversational Status Updates​

Support for status updates, such as "Thinking," "Typing," etc. All seamlessly baked right into the API.

Event-Level Debugging with Correlation IDs​

Each event in the conversation is associated with a correlation ID associated with the HTTP request that triggered its creation. This was needed to streamline debugging, so you could ask the system, "Give me all of the logs that came from generating this one particular event in this conversation."

Asynchronous, Free-Flowing Conversation API​

An asynchronous API so that we could build a GUI where conversations didn't necessarily have to work like ping-pong, where the customer (or the agent) can only send one message at a time. That's just not how real conversations work. Real people often send multiple messages at a time to get their point across.

So we wanted our API to be future-compatible and enable conversations that weren't just fluent in their content, but also in their flow.

Backend-Initiated Agent Responses​

Agent conversations needed to be triggerable asynchronously from the backend as well—for example, initiating a conversation with the user if you find some issue in their account, or, alternatively, updating them on the result of a long-running background task using a callback function.

Imagine your agent booking a flight ticket for a customer. This normally takes about 30-60 seconds to get sorted, but your agent can't just get stuck! It should still be responsive to your customer in their conversation. So we designed for that, too.

From Mission-Critical Systems to Conversational AI​

Looking back, our journey from building deterministic, mission-critical systems to creating Parlant has been about bringing that same rigor and reliability to the world of conversational AI.

Parlant's whole existence stems from a desire to bring more control, predictability, and quality into generative AI agents. And now, a year later, we know that we had taken the right approach. Parlant agents are famously consistent in their instruction-following ability compared to those built from scratch with lower-level LLM frameworks.

The principles that guided us in our previous work—breaking down complex problems, testing everything, and empowering stakeholders—turned out to be exactly what the conversational AI space lacked.

We've gotten feedback from many people since, saying things like, "After a few months of banging my head against these LLMs, I realized this is the way it should be done... and then I found Parlant has already done it.".

By applying these principles to the inherently unpredictable world of LLMs, we've created something that finally makes reliable customer-facing AI agents achievable. And we're proud to offer it to the community as an open-source solution.

Try It Yourself!​

If you haven't tried it yet, I encourage you to go through the official tutorial and see for yourself how seamless it is to create scalable agents in Parlant. It's all about getting the architecture right. When you do that, things naturally fall into place. And that's exactly what we've achieved with Parlant's architecture.

Get in touch with us

TL;DR: We used to have to specify every single step in an AI agent's conversation flow, which was rigid, inflexible, and a lot of work. Then LLMs came along, promising to revolutionize the way we build AI agents by allowing us to generate adaptive responses on the fly, while only needing us to provide some general background context for their operation. But now, in 2025, we're realizing that LLMs do a poor job at filling in the gaps toward an effective customer interaction, and that the best way to build customer-facing AI agents is to combine the power of LLMs with traditional conversation design techniques.

A Bit of Generative AI History​

Back in early 2023, when GPT 4 just came out, I—along with my co-founder Dor Zohar, and thousands of other developers—were excited about the new possibilities of building customer-facing AI agents. These "new agents" could handle complex interactions and use cases, creating a new and better kind of user experience.

But what we (like many others new to NLP) were totally unaware of at the time was that enterprises had already been building complex AI agents for years, using pretty much the same Transformer models invented in 2017. These had roughly the same architecture that powers today's state-of-the-art LLMs. They were just smaller and more limited in functionality and reliability.

The way those traditional agents were built was by defining a flowchart of the conversation, and then writing utterances for each node in the flowchart. The utterances were written by conversation designers, who were trained to write in a way that was compliant with brand voice and legal requirements, and to make the AI engaging and effective and resolving customer requests.

The transition from one node to another was performed using NLP classifiers, which would classify the user's input into one of the predefined intents. The intent would then determine the next node in the flowchart. These classifiers were often based on those early transformer models, which were much smaller and less capable than today's LLMs.

The Limitations of Traditional AI Agents​

While this approach worked well for many years, it had its limitations. The flowcharts were rigid and inflexible, and the utterances were often too generic or too specific. The classifiers were also limited in their ability to understand the nuances of natural language, which led to misclassifications and poor user experiences.

It also meant that the agents were not very adaptive. Users were confined to pre-defined steps and couldn't express themselves freely like in a normal conversation, and were often frustrated by the experience, as the agents were unable to handle unexpected or mistimed inputs or changes in context. This rigidity made it difficult to create engaging and effective AI agents.

Consequently, user adoption was low, and many companies struggled to get their customers to use these agents, as many users preferred to speak with a human agent instead—completely defeating the purpose of building an AI agent in the first place.

Not only did these hard limitations exist, but, worse, the effort in building an maintaining such agents was nonetheless quite considerable, and only the largest companies were normally able to justify the investment. Conversation designers had to write and maintain thousands of utterances, and the flowcharts (and corresponding NLU classifier models) had to be updated whenever a new intent was added or an existing one was modified. This made the process slow and cumbersome, and it often resulted in outdated or inaccurate agents.

But, despite all of that, these traditional AI agents were (and still are) superior to LLMs in many ways. They were more reliable, more predictable, and more compliant with brand voice and legal requirements. They also provided a better user experience in the so-called "happy-paths" of customer interactions, as they were designed to handle specific use cases and interactions with maximum efficiency and grace, driven by the deep domain understanding of their designers.

Get in touch with us

The New Silver-Bullet: Forget the "How", Just Define the "What"​

Then came the LLMs. They promised to revolutionize the way we build AI agents by allowing us to generate responses on the fly, without the need for predefined designs or flowcharts. This was a game-changer, and many companies jumped on the bandwagon, eager to leverage the power of LLMs.

The idea was that if you provide the LLM with some general background context about the agent's purpose and the domain it operates in, it would be able to generate appropriate responses to user inputs, without the need for predefined utterances or flowcharts. This meant that you could build AI agents that were more adaptive, more flexible, and more engaging than ever before—at a fraction of the effort.

Heck, you didn't even need to know what you were doing! You could just throw at the LLM some auto-parsed background material from your company's documentation, and it would generate a response that was "good enough" for most use cases. This seemed like a huge leap forward, and it opened up new possibilities for building AI agents that were more human-like and more engaging.

And it seemed like anyone could build them. It seemed like Conversational AI was finally accessible to everyone, not just the big enterprises with full-time conversation designers and ML engineers on staff.

No Silver Bullet After All​

But, as we all know, the reality was not quite as rosy as the promise. LLMs were great at generating text, but they were not very good at understanding the nuances of customer interactions. They often produced responses that were irrelevant, inaccurate, or even offensive. They also struggled with maintaining context and coherence over long conversations, which led to frustrating user experiences.

Beyond that, these new LLM agents lacked an important, fundamental quality: a sense of competence and design.

Instead of giving customers a feeling of understanding, assurance, and helpfulness, they were passive and reactive, often responding to user inputs in a way that felt disjointed and unplanned.

They were not designed to handle specific use cases or interactions efficiently, and they often produced responses that made customers feel as if they're talking to a disoriented, compulsively people-pleasing service rep on their first day on the job.

Get in touch with us

The Return of the "How"​

There's a hidden lesson here—which seems to have taken a lot of trial and error to uncover.

Just as we wouldn't trust a complete stranger to fill in the blanks in a sensitive customer interaction, we can't expect LLMs to always infer our intentions, preferences, or requirements.

Left to their own devices, LLMs will need to make guesses—but their guesses may often misalign with our brand, our service standards, or even basic user expectations. Sometimes, they'll get it wrong in subtle ways; other times, the results can be blatantly off-mark or even harmful to the business.

This is why the reality isn't as simple as "Let the LLM figure out what to do." While LLMs excel at generating plausible responses, they lack the context, priorities, and guardrails that come from explicit design and domain expertise.

We need to provide a system of guidance—a framework that tells the LLM not just what the goal is, but how we want to achieve it. This means specifying, at least for critical parts of the interaction, the steps, tone, and boundary conditions that matter most.

This is exactly what we at Parlant have been working on, and it's why we're so excited about the future of AI agents in 2025.

Moving away from the rigidity of traditional flowcharts was necessary, but swinging to the opposite extreme—where we only define the "what" and leave the "how" entirely up to the model—creates new problems.

LLMs need us to step in and explain, sometimes in detail, how we want certain conversations to unfold. And like I said, this isn't just about delivering a better user experience; in many industries, especially those with strict service protocols or regulatory requirements, it's essential for compliance and risk management. It's a prerequisite for safe production deployment.

Ultimately, the most effective AI agents in 2025 are those that blend the adaptability of LLMs with the intentionality and structure of conversation design. By guiding LLMs with clear instructions on both the "what" and the "how," we can create agents that are not only flexible and engaging, but also reliable, compliant, and truly aligned with our goals.

Get in touch with us