8 posts tagged with "ai-agents"

Get ready for a hardcore deep dive! The most asked-for framework comparison from the Parlant community is finally here.

"How is Parlant different from LangGraph?" "Can I use them together?" "Which one should I use for my AI agent and why?"

These are the great questions we've been getting from back when we launched Parlant v1.0. And the reason they're great is that they allow us to get right to the heart of what makes Parlant unique, why we even built it, and the surprising lessons we've learned building real-world conversational AI while working with truly large-scale clients.

Contents:​

Introduction​

When coders first approach building conversational agents, graph-based architectures like LangGraph feel intuitive and familiar. Router patterns and specialized nodes map cleanly to how we think about organizing functionality. This is why these patterns are so popular.

However, through years of studying and building production conversational systems, working with some of the top professionals in the field (shout out to CDI), we've discovered challenges that most developers aren't aware of when they start. These challenges aren't immediately obvious: they emerge gradually as your agent handles more conversations, more topics, and more complex user behaviors.

In this post, we'll demonstrate how router-based architectures inherently fail for natural, free-form conversation—not due to implementation details you can fix, but due to fundamental architectural constraints. Understanding these limitations now can save you from a complete rewrite later when you discover your graph-based system can't scale to handle the way users actually converse.

Let me break down the key differences, explain where LangGraph's common patterns create challenges for conversational AI, and show you why and how Parlant approaches these problems differently.

The Quick Answer​

LangGraph is a framework for building agentic workflows through explicit graph-based orchestration. It excels at task decomposition, multi-step automation, and scenarios where you need precise control over execution flow.

Parlant is an AI alignment engine designed specifically for natural customer-facing dialogue. It's built to handle free-form conversations where users don't follow scripts: they mix topics freely, and expect coherent responses across multiple contexts simultaneously.

They solve different problems. Actually, they can work together—LangGraph can function as a lower-level orchestration framework triggered by a Parlant agent's tools for complex retrieval, action workflows, or specialized reasoning tasks.

Understanding LangGraph's Approach​

LangGraph represents agentic applications as graphs where nodes are computational steps (LLM calls, tool invocations) and edges define control flow. This architecture is powerful for workflow automation and task orchestration.

Common LangGraph Patterns​

The most popular patterns for building agents with LangGraph include:

1. Supervisor (Router) Pattern

A central coordinator (supervisor) routes requests to specialized agents based on the query type. Each specialized agent has its own system prompt optimized for a specific domain.

# Simplified LangGraph supervisor pattern
from langgraph.graph import StateGraph

# Define specialized agents
returns_agent = create_agent("You are an expert in returns...")
warranty_agent = create_agent("You are an expert in warranties...")
billing_agent = create_agent("You are an expert in billing...")

# Supervisor routes to the right agent
def supervisor_node(state):
    # LLM decides which agent should handle this
    next_agent = supervisor_llm.invoke(state["messages"])
    return {"next": next_agent}

workflow = StateGraph()
workflow.add_node("supervisor", supervisor_node)
workflow.add_node("returns_agent", returns_agent)
workflow.add_node("warranty_agent", warranty_agent)
workflow.add_node("billing_agent", billing_agent)

workflow.add_conditional_edges("supervisor", route_to_agent)

2. Multi-Agent Network

Agents communicate in a many-to-many pattern, where each agent can invoke others as needed. This provides flexibility but requires careful coordination logic.

# Multi-agent network pattern
from langgraph.graph import StateGraph

# Each agent can invoke others
def research_agent(state):
    result = research_llm.invoke(state["messages"])
    if needs_writing_help(result):
        # Agent decides to call writing agent
        return {"next": "writing_agent", "data": result}
    return {"messages": result}

def writing_agent(state):
    result = writing_llm.invoke(state["messages"])
    if needs_fact_check(result):
        # Agent decides to call research agent
        return {"next": "research_agent", "data": result}
    return {"messages": result}

# Build network with many-to-many connections
workflow = StateGraph()
workflow.add_node("research_agent", research_agent)
workflow.add_node("writing_agent", writing_agent)
workflow.add_node("editing_agent", editing_agent)
workflow.add_node("review_agent", review_agent)

# Each agent can transition to multiple others
workflow.add_conditional_edges("research_agent", route_from_research)
workflow.add_conditional_edges("writing_agent", route_from_writing)
workflow.add_conditional_edges("editing_agent", route_from_editing)

3. Hierarchical Teams

Supervisor agents manage their own teams of specialized agents, creating nested structures for complex workflows.

# Hierarchical teams pattern
from langgraph.graph import StateGraph

# Top-level supervisor
def main_supervisor(state):
    category = categorize_query(state["messages"])
    if category == "customer_service":
        return {"next": "customer_service_supervisor"}
    elif category == "technical":
        return {"next": "technical_supervisor"}

# Mid-level supervisors manage specialized teams
def customer_service_supervisor(state):
    subcategory = categorize_cs_query(state["messages"])
    return {"next": f"{subcategory}_agent"}

def technical_supervisor(state):
    subcategory = categorize_tech_query(state["messages"])
    return {"next": f"{subcategory}_agent"}

# Build hierarchical structure
workflow = StateGraph()
workflow.add_node("MS", main_supervisor)
workflow.add_node("CSS", customer_service_supervisor)
workflow.add_node("TS", technical_supervisor)

# Add specialized agents under each supervisor
workflow.add_node("RA", returns_agent)
workflow.add_node("BA", billing_agent)
workflow.add_node("DA", diagnostics_agent)
workflow.add_node("ASA", api_support_agent)

workflow.add_conditional_edges("MS", route_to_team)
workflow.add_conditional_edges("CSS", route_to_cs_agent)
workflow.add_conditional_edges("TS", route_to_tech_agent)

What LangGraph Excels At​

LangGraph is indeed excellent for:

• Workflow automation and orchestration
• Task decomposition into specialized subtasks
• Non-conversational agent applications
• Scenarios requiring a stencil control flow

The framework gives you precise control over execution paths, which is exactly what you want for many agentic processes, specifically ETL type of workflows.

But here's where the challenges arise...

The Problem with Router Patterns​

Here's where things get interesting for conversational AI. The router pattern (i.e., "supervisor"), where specialized nodes own specific tasks and topics, works beautifully for workflow orchestration but creates inherent challenges when users engage in natural, free-form dialogue.

Let me walk you through the problems that arise from this architecture.

Isolated Specialization Is Inherently Broken​

Picture this. A customer sends a message:

"Hey, I need to return this laptop. Also, what's your warranty on replacements?"

In a router system, that message gets routed to one specialized agent.

Let's say it lands on the Returns Agent. This agent has a system prompt optimized for returns. It knows the return policy inside and out. But what about warranties? That's the Warranty Agent's domain.

The Returns Agent is now faced with four practical "alternatives" for handling the warranty question, none of which are actually any good:

Option 1: Ignore the warranty question

"Sure, I can help you with that return. What's your order number?"

↪ Poor UX. The customer explicitly asked about warranties and got no acknowledgment.

This is not how a good conversational framework should behave.

Option 2: Acknowledge the limitation

"I can help with the return. Regarding the warranty on replacements, I'm not certain about those details."

↪ This is not only awkward, but, from the overall agent's perspective, is incorrect. The overall agent can handle warranty questions—it just routed to the wrong specialized node. This will frustrate both users and developers, and lead to a disjointed user experience.

This, too, is not how a good conversational framework should behave.

Option 3: Hallucinate an answer

"Sure! Our return policy is 30 days, and all replacements come with a 5-year warranty."

↪ Dangerous and extremely common—as well as hard to prevent with LLMs. The Returns Agent isn't grounded in warranty information, but, being an LLM, will often attempt to fabricate an answer. That "5-year warranty" might be completely wrong.

This, too, is not how a good conversational framework should behave.

Option 4: Stack the topics

"Let me help you with that return first, and then we'll get to your warranty question."

↪ This is actually (kind of?) reasonable. The agent acknowledges both questions and proposes handling them sequentially. Many LangGraph implementations do exactly this.

But even option 4 is not how a good conversational framework should behave. It still smells of a fundamental design flaw, because it often leads to poor user experience.

When your AI user experience is poor, many users mistrust in it when it comes to important matters, and consequently drop out of your AI agent and escalate to a familiar human agent experience. That's where you're leaving money and customer satisfaction on the table.

Let's see how this leads to friction.

The First Common Issue: Intertwined Topics​

Many natural conversations don't present as "two separate questions." The "topics" (from a backend classification perspective) are naturally intertwined throughout the interaction. Forcing an artificial separation feels robotic. Consider this:

"I bought this laptop for my business but it's broken. Can I return it and get a tax invoice for the replacement?"

The return process and the business tax documentation question aren't separate topics the user is stacking. They're deeply connected:

• The tax invoice depends on whether a return is processed
• Business customers may have different return policies
• The replacement timing affects when documentation is issued

Forcing the agent to say "Let me handle your return first, then we'll discuss invoicing" creates unnecessary friction and invites avoidable questions and objections.

In natural conversation, you'd expect both aspects to be addressed together:

"Absolutely, we can process that return. For your replacement, I'll ensure you receive a proper tax invoice. Let me pull up your account—is the business registered under the same email?"

This is what a well-designed conversational framework should deliver, and it's exactly one of the design goals Parlant was built to achieve.

The Second Common Issue: Recurring Topics​

Another pattern that emerges frequently in natural conversation: topics don't just appear once and disappear. Users jump between topics, then circle back. And once multiple topics have entered the conversation, they tend to remain relevant—at least to some degree—for the rest of the discussion.

Consider this conversation flow:

Customer: "I'd like to return this laptop I bought last week."

Agent (Returns): "I can help with that. Can you provide your order number?"

Customer: "Yeah let me get it, one sec. By the way, do you have the new model in stock? The one with the better battery?"

Agent (Inventory): "Yes, we have the XPS 15 with extended battery in stock. Would you like me to reserve one for you?"

Customer: "Yes, perfect. So if I return this one and get that new model, will I need to pay the difference, or can I just exchange them?"

Now the conversation needs to reference both the return process AND the inventory/pricing information. The customer isn't asking two separate questions—they're asking about the relationship between the return and the new purchase.

In a router system, this message gets routed to one specialized agent. If it goes to the Returns Agent, it can handle the return part but isn't grounded in current inventory pricing. Or perhaps it routes to the Billing Agent, which can discuss pricing but isn't grounded in returns.

The challenge is that once both topics have been introduced, the rest of the conversation will, in high likelihood, need to reference both in some form or another.

The user isn't thinking "Now I'm in Returns mode" or "Now I'm in Inventory mode". They're thinking about their situation holistically, which spans both domains.

Every subsequent message will be handled by one specialized node that's ungrounded with respect to the other relevant topic. The conversation's integrity and coherence fragments precisely when it needs to be most cohesive.

A conversational AI framework that's understands natural conversations should support seamless context switching and multi-topic awareness, and it's exactly one of the design goals Parlant was deliberately designed to achieve.

Articulating the Core Design Issue​

When you route to a specialized node for topic X, that node is inherently unspecialized for topic Y.

Real conversations don't respect domain boundaries. Users mix topics in single messages, jump between topics, and expect coherent responses that integrate multiple contexts. A router/specialized node architecture forces you to choose one domain's grounding at the expense of others

How Parlant Approaches Free-Form Conversation​

We created Parlant from a clear vision of, and frustration from these age-old problems in mechanistic AI conversations.

We thus decided to take a fundamentally different approach optimized for natural dialogue: dynamic assembly of a comprehensive context instead of artificially segmented routing.

Dynamic Guideline Matching​

Rather than routing messages to specialized nodes, Parlant evaluates ALL of your guidelines and other contextual information, and loads only the relevant ones for each conversation turn—i.e., for each user-observable LLM request.

# Parlant approach - no routing, just conditional guidelines
await agent.create_guideline(
    condition="The customer wants to return an item",
    action="First acknowledge their concern, explain our 30-day return policy, "
           "and ask for their order number"
)

await agent.create_guideline(
    condition="The customer asks about warranty on replacements",
    action="Explain that all replacement items come with our standard warranty, "
           "which covers manufacturing defects for 1 year from the replacement date"
)

await agent.create_guideline(
    condition="The customer needs a tax invoice for a business purchase",
    action="Let them know we'll include a proper tax invoice with their order documentation"
)

await agent.create_guideline(
    condition="The customer asks about store hours",
    action="Tell them we're open Monday-Friday 9-8, Saturday 10-6, and Sunday 12-5"
)

Notice what's different here: there's no routing decision. When the customer says, "I need to return this laptop for my business and get a tax invoice for the replacement," Parlant's guideline matcher evaluates all guidelines and determines that three are contextually relevant:

1. The return guideline (customer wants to return)
2. The tax invoice guideline (business purchase context)
3. The warranty guideline (mentioned replacement)

They get loaded into the LLM's context simultaneously. The agent composes a response that's dynamically grounded in all of them:

"I'd be happy to help you with that return and ensure you get proper documentation. Let me pull up your order—could you provide the order number? Once I have that, I'll process your return and make sure your replacement comes with a tax invoice for your business records. Just to confirm, the business is registered under the same email as this order?"

So, instead of choosing between domains, the Parlant agent assembles all the relevant grounding context on-demand.

Why This Works for Conversational Coherence​

More often than not, real-world customer conversations break your textbook vision of an architected dialogue.

Parlant's approach solves the problems we identified earlier:

No artificial separation: The agent can naturally address intertwined topics because it has access to all relevant guidelines simultaneously.

No crude serialization: When both topics have brief answers, the agent can address them together naturally.

No context fragmentation: There's no handoff between agents because there aren't separate agents. There's just one agent with dynamically and continuously assembled context.

Consistent grounding: The agent is grounded in all contextually relevant information, not just one domain's specialization. This is (one) big part of what makes Parlant so seamlessly effective at maintaining compliance and avoiding hallucinations in customer-facing scenarios.

How Parlant Keeps Accuracy Scalable​

Of course, if you just load all relevant guidelines, the LLM's context gets bloated. This is the same as putting everything in one giant system prompt, which leads to the "curse of instructions"—too many simultaneous instructions degrade performance (not to mention you'll ironically end up paying more for this degraded performance, due to the higher token count).

This is where Parlant's architecture does most of its work. The framework uses surprisingly intricate and nuanced guideline matching to accurately determine relevance BEFORE loading context.

When LangGraph Works Well for Conversational AI​

I want to be clear: LangGraph isn't wrong for all conversational AI. If you have the right use case for it, you can achieve better results with it than with Parlant.

LangGraph works well for conversational agents when:

The conversation is narrow in scope, and streamlined: You're interviewing users through a specific process like onboarding, data collection, or a structured troubleshooting flow, where they have to choose their responses from options presented to then, rather than responding freely.

Users follow a guided path: The interaction model strictly requires users to answer questions in a formal sequence rather than conversing freely.

Limited conversational freedom: Users aren't expected to interrupt with unexpected questions mid-flow, change topics independently, or reference multiple contexts simultaneously.

Clear state transitions: The conversation has well-defined stages where it makes sense to "route" to the next step.

Granted, these are legitimate use cases where LangGraph is likely to provide great results with far less compute overhead than Parlant invests in (an investment made to maximize accuracy and user experience in genuine conversational settings).

The key distinction is in the interaction model. If your users are expected to follow a strict script at all times, LangGraph's routing works great and is not likely to run into unwieldy design issues and operational faults.

If your users are conversing naturally and freely, that's where the challenges emerge. That's where Parlant's shines, as it was intricately designed for this use case.

Using LangGraph and Parlant Together​

Here's something interesting: these frameworks aren't competitive—they're complementary.

LangGraph can work as a lower-level orchestration tool within Parlant. Let me show you a practical example:

# A Parlant tool that uses LangGraph internally for complex retrieval

@p.tool
async def find_relevant_documentation(
    context: p.ToolContext,
    query: str
) -> p.ToolResult:
    # LangGraph orchestrates the multi-step retrieval workflow
    result = await langgraph_workflow.invoke({
        "query": query,
        "customer": p.Customer.current,
    })

    return p.ToolResult(
        data=result["synthesized_answer"],
        canned_response_fields={
            "documentation": result["source_links"],
        },
    )

# The tool is used within a Parlant guideline
await agent.create_guideline(
    condition="The customer asks about a technical feature or API",
    action="Look up the info and provide an explanation if possible",
    tools=[find_relevant_documentation],
)

In this pattern:

• Parlant handles and synthesizes the conversational flow, dynamic context management, and natural dialogue
• LangGraph handles the complex internal KB workflow (retrieval, re-ranking, synthesis)

This combines the strengths of both:

• Parlant ensures coherent, well-guided, natural conversation
• LangGraph provides powerful workflow orchestration for agentic processing

The Architectural Difference​

At the highest level, here's the core distinction:

LangGraph: Explicit Graph-Based Control Flow

LangGraph represents your application as a graph with explicit nodes and edges. You define the routing logic, specify transitions, and control execution flow. This is excellent for:

• Workflow automation
• Task decomposition
• Multi-step processes with clear stages
• Scenarios requiring precise control over execution order

It's great for when you know what the expect from the input—which is simply not the case for raw conversational content.

Parlant: Dynamic Context Assembly

That's where Parlant helps you keep the contracts under control, and handle the messy conversational parts cleanly and effectively.

Parlant represents your application as a set of conditional guidelines and journeys within a natural conversation.

Instead of routing between specialized nodes, it dynamically assembles relevant context for each turn. This is excellent for:

• Free-form conversational AI
• Natural dialogue where users don't follow scripts
• Scenarios where topics interweave unpredictably
• Compliance-critical customer-facing applications

Both approaches are valid. The right choice depends on your interaction model.

When to Use Which​

Here's a practical guide:

Use LangGraph when you need explicit orchestration and precise control over execution: workflow automation like multi-step tasks, data processing pipelines, and agent workflows where order matters; task decomposition that breaks complex problems into coordinated subtasks or multi-agent collaboration for non-conversational workloads; narrow, guided conversational flows such as structured onboarding, step-by-step data collection, or troubleshooting wizards.

Use Parlant when you’re building free‑form, customer‑facing agents where users converse naturally without following a script: support chatbots, sales and consultation assistants, or any scenario that demands comprehensive and consistent domain alignment. It shines when conversations mix topics, include interruptions, and require natural back‑and‑forth; in compliance‑critical settings where regulated behavior and avoidance of unauthorized statements are essential across business‑critical interactions; and in multi‑context dialogues where users reference multiple topics simultaneously, concerns are intertwined rather than separable, and the agent must produce coherent, cross‑topic responses.

When you need both conversational fluidity and complex workflows, use both:

• Conversational agent (Parlant) with complex retrieval (LangGraph)
• Natural dialogue requiring multi-step tool orchestration
• Customer-facing AI with sophisticated backend processes

"Can't I Just Build This with LangGraph?"​

At this point, you might be thinking: "Great, I understand the dynamic context assembly approach. Now why can't I just build that pattern myself using LangGraph?"

And the answer is: you absolutely can. LangGraph is flexible enough to implement a dynamic guideline matching and context assembly system like Parlant (though Parlant isn't actually built on top of it).

Nothing to stop you from building that. But, before you dive into it, consider that you'd be essentially building Parlant from scratch.

Why? Let me share a hard-earned lesson on how alignment engineering turned out to be a surprisingly deep rabbit hole.

The Rabbit Hole Goes Deeper Than You Think​

When we started building Parlant, we thought dynamic context assembly would be straightforward. Match relevant guidelines, load them into context, done. We quickly learned that the complexity goes much deeper than initially apparent.

Guideline Matching Complexity:​

• Semantic matching isn't just about accuracy (though that alone is a massive challenge—just look at how many specialized guideline sub-matchers exist in Parlant)
• You need to handle edge cases: partially relevant guidelines, conflicting conditions, different temporal scopes, and keeping track of what was already done and wasn't to avoid over-conformance to instructions
• False positives and negatives create different failure modes that need to be mitigated in how you think about your agent's design (which Parlant addresses through its conversation-optimized tool-calling and canned response systems)

Runtime Performance:​

Naive implementations create unacceptable latency; mitigating this requires well-tuned caching strategies and parallel execution of different pipeline stages—even at relatively small scale.

A Coherent, Holistic Architecture:​

Guidelines need to interact elegantly with tools. Journey state needs to be tracked; canned responses need to be coordinated with matched guidelines and tool outputs, and variables, glossaries, and relationships need to factor into context assembly.

Every component subtly influences the others in an integrated, closed loop that feels natural. Your development workflow all falls into place.

This is what Parlant offers, and it's truly taken a lot to get it here!

Leverage Our Work & Research​

We've spent the last couple of years studying, building and refining these mechanisms—full time. Parlant's current implementation represents countless battle-tested iterations, performance optimizations, and lessons learned from real deployments. The guideline matching system alone has gone through multiple architectural rewrites as we tackled the surprising nuances of these challenges head-on.

If you're considering building your own dynamic context assembly system with LangGraph, I'd honestly encourage you to first:

1. Review Parlant's feature documentation to understand the full scope of challenges you'll encounter
2. Consider the engineering investment required to solve these problems elegantly
3. Evaluate whether your use case has unique requirements that Parlant doesn't address (for which a rewrite makes more sense than opening a Parlant GitHub issue or contributing a PR)

Parlant is Fully Open Source​

Remember, Parlant isn't a black box. It's fully open source, which means you can:

• Inspect exactly how we solve these problems
• Extend or modify components to fit your needs
• Contribute improvements back to the community
• Build on top of years of production-hardened engineering by a decent team of developers!

Either way, understanding what's already been built and battle-tested can save you months of development time and help you avoid the pitfalls we've already encountered and solved—and will continue to solve and optimize for the emerging, modern Conversational AI community.

Making the Right Choice​

The confusion between Parlant and LangGraph is understandable: They're both powerful frameworks for building with LLMs. But understanding the difference comes down to one key question:

What kind of interaction are you building?

If you're building workflows, orchestration, or guided processes → LangGraph is excellent.

If you're building free-form conversational AI where users don't follow a script → Parlant is designed specifically for this.

If you need both → They can work together beautifully.

It's clear that both frameworks have their place in the AI agent ecosystem. Choose based on your interaction model, and don't be afraid to use them together when it makes sense.

Questions about Parlant?

Have you tried building conversational agents with LangGraph or Parlant? I'd love to hear about your experience. Reach out on Discord or via our contact page.

Some interesting questions we've been getting lately: "Is Parlant like DSPy?" "What's the difference?" "Which one should I use?"

I get why people are confused about it. Both frameworks work with LLMs, both use Python, and both claim to make LLMs more consistent and effective. At the same time, though, they solve completely different problems.

Let's break it down.

The Quick Answer​

DSPy is about optimizing how you get good outputs from language models through automated prompt engineering and model tuning. It's a bit like a compiler for LLM prompts.

Parlant is about controlling and aligning conversational agent behavior with business rules and compliance requirements. It's a framework for building customer-facing agents that actually follow your instructions and generate safe and consistent responses.

They're solving different problems. In fact, you could even use them together—but more on that later.

What is DSPy?​

DSPy (Declarative Self-improving Python) came out of Stanford NLP and offers a different way to work with LLMs: Instead of manually crafting and tweaking prompts, you write declarative Python code that describes what you want the model to do, and DSPy figures out how to prompt the model to get it.

The Core Concept: Programming, Not Prompting​

The driving assumption behind DSPy is that prompt engineering is brittle. You write a carefully crafted prompt, test it, tweak it, test it again. Then you switch models or change your task slightly, and everything breaks. You're back to square one.

DSPy says: stop doing that. Instead, write code that specifies your task's structure, and let DSPy automatically generate and optimize the prompts for you.

# DSPy approach - define your module
class SentimentClassifier(dspy.Module):
    def __init__(self):
        self.classify = dspy.ChainOfThought("text -> sentiment")

    def forward(self, text):
        return self.classify(text=text)

# Provide input/output examples for DSPy to optimize with
trainset = [
    dspy.Example(text="I love this product!", sentiment="positive").with_inputs("text"),
    dspy.Example(text="Terrible experience.", sentiment="negative").with_inputs("text"),
    dspy.Example(text="It's okay I guess.", sentiment="neutral").with_inputs("text"),
    # ... more examples
]

# Define a metric to evaluate the quality of outputs
def sentiment_metric(gold, pred, trace=None):
    return gold.sentiment == pred.sentiment

# DSPy optimizes the underlying prompts automatically based on the examples and metric
optimizer = dspy.BootstrapFewShot(metric=sentiment_metric)
optimized_classifier = optimizer.compile(SentimentClassifier(), trainset=trainset)

# Now use the optimized classifier
result = optimized_classifier(text="This exceeded my expectations!")
print(result.sentiment)  # Output: "positive"

What DSPy Excels At​

DSPy is an awesome tool for:

• Optimizing model performance across different tasks and models
• Automating prompt engineering so you don't have to manually tweak prompts
• Building complex reasoning pipelines with multiple LLM calls
• Rapid experimentation with different model configurations
• RAG systems where you need to optimize retrieval and generation together

What DSPy Doesn't Do​

DSPy isn't designed for:

• Enforcing a large and dynamic set of strict business rules in conversations
• Guaranteeing compliance with regulatory requirements
• Managing multi-turn conversational state and flow
• Preventing specific types of agent misbehavior
• Ensuring agents never say certain things (or always say others when required to)

What is Parlant?​

Parlant is a conversational alignment engine built specifically for customer-facing AI agents that need to follow business rules consistently.

The Core Problem: Alignment at Scale​

Parlant is driven by the following assumption: When building customer-facing agents, even when you get the LLM to produce "correct" outputs most of the time, you still have a massive problem. Agents still go off on tangents, ignore instructions, hallucinate information, breach business protocols, or just generally behave unpredictably.

The worst part is that traditional approaches to "fixing" this, like adding more instructions to your prompt, actually make it worse. Research shows that as you add more instructions, LLMs get worse at following them (see The Curse of Instructions).

Parlant solves this through dynamic context management and structured behavior modeling.

How Parlant Works​

Instead of throwing all your instructions at the LLM at once, Parlant breaks them into granular, conditional guidelines and only loads the relevant ones for each conversation turn:

# Parlant approach
await agent.create_guideline(
    condition="Customer asks about refunds",
    action="First acknowledge their concern, then explain our 30-day refund policy"
)

await agent.create_guideline(
    condition="Customer has already declined an upgrade",
    action="Do not mention upgrades again in this session"
)

The condition part is crucial. Parlant's guideline matcher evaluates which guidelines are relevant right now, loads only those into the LLM's context, and uses specialized prompting techniques (ARQs) to maximize conformance to the guidelines.

For multi-step interaction design, Parlant provides journeys:

journey = await agent.create_journey(
    title="Handle Refund Request",
    triggers=["Customer wants to return or refund an item"],
    description="Guide customers through our refund process"
)

first_step = await journey.initial_state.transition_to(
    chat_state="Ask for order ID"
)

second_step = await first_step.target.transition_to(
    tool_state=load_order_details
)

third_step = await second_step.target.transition_to(
    chat_state="Confirm you have the right order"
)

# ... continue building the journey

Unlike rigid flowchart systems, Parlant journeys are adaptive; that is, the agent can skip steps, revisit states, or adapt to the customer's needs while maintaining the overall intended flow of the journey.

What Parlant Excels At​

Parlant is built for:

• Customer-facing conversational agents that need predictable, compliant behavior
• Business rule enforcement at scale across millions of interactions
• Compliance-critical applications where unauthorized behavior creates liability
• Iterative behavior refinement as you learn what your agent needs to handle
• Multi-turn conversation management with complex state and context

What Parlant Doesn't Do​

Parlant isn't designed for:

• Optimizing model performance across different providers
• Automated prompt tuning or hyperparameter optimization
• Non-conversational LLM tasks (classification, extraction, etc.)
• General-purpose LLM application building

The Key Differences​

Let's break this down side by side:

When to Use Which​

Use DSPy When:​

• You're building LLM applications that aren't primarily conversational
• You need to optimize performance across different models or tasks
• You're doing RAG and want to optimize retrieval + generation together
• You want to avoid manual prompt engineering
• Your main challenge is getting better outputs from the model

Use Parlant When:​

• You're building customer-facing conversational agents
• You need strict adherence to business rules and compliance requirements
• Your agent must never say certain things or must always follow specific protocols
• You're iteratively refining agent behavior based on real-world interactions
• Your main challenge is controlling what the agent says and does

Use Both When:​

Here's an interesting fact: Parlant and DSPy are complementary.

You could use DSPy to optimize specific components within your Parlant agent. For example:

• Use Parlant to structure the conversation and enforce business rules
• Use DSPy to optimize your retrieval pipeline for finding relevant information
• Use Parlant's tool system to call DSPy-optimized modules when needed

# Hypothetical example: DSPy-optimized retrieval in a Parlant tool
@p.tool
async def find_policy_info(context: p.ToolContext, query: str) -> p.ToolResult:
    # Use DSPy-optimized RAG pipeline
    result = dspy_rag_pipeline(query)
    return p.ToolResult(data=result)

...

# Parlant guideline controls when/how this tool is used
await agent.create_guideline(
    condition="Customer asks about a specific policy",
    action="Answer based on the policy information you find",
    tools=[find_policy_info]
)

The Core Difference​

Perhaps the deepest difference is philosophical:

DSPy operates on the assumption that with the right prompts and optimization, you can get models to reliably produce what you want.

Parlant operates on the assumption that LLM behavior is always inherently uncertain and must be actively managed, filtered, and controlled at the architectural level. It's about structuring the conversation space so the agent can't go off the rails even when the model wants to.

Both assumptions are valid for different use cases.

If you're building a question-answering system where you just need accurate answers, DSPy's optimization approach works great.

If you're building a customer service agent where an unauthorized $10,000 refund would be catastrophic, you need Parlant's design and enforcement mechanisms.

Making the Right Choice​

The confusion between Parlant and DSPy is understandable: they're both modern Python frameworks for working with LLMs. But understanding the difference comes down to this question:

What's your biggest challenge?

• "My model outputs aren't good enough" → DSPy
• "My agent doesn't follow my business rules consistently" → Parlant
• "I need both RAG optimization and conversational control" → Consider using both

So you don't have to choose just one paradigm. As the LLM application ecosystem matures, we're going to see more integration between optimization frameworks like DSPy and control frameworks like Parlant.

Building incredible LLM applications isn't about finding one perfect framework. Instead, it's about composing the right tools for your specific needs.

Questions about Parlant?

Have you tried both frameworks? I'd love to hear about your experience. Reach out on Discord or via our contact page.

Many people on social media have been asking recently: "How exactly does Parlant guarantee compliance with conversational LLM agents? How are they able to make that claim?"

This is a crucial question that takes us to the heart of why and how most conversational LLM applications fail in production. The answer lies in first understanding the fundamental types of AI misalignment and how Parlant's architecture systematically addresses each one.

In this article, I'll explain the approach, insights, as well as the core architectural components that make Parlant unique and how they work together to ensure compliance, safety, and reliability in customer-facing AI agents.

Contents:

Two Critical Problems with AI Agents​

Before diving into specific types of failures AI agents experience, we should, first of all, understand that there are actually two major challenges to address around failures when building AI agents:

1. Failure Frequency (What Everyone Talks About)​

This is the problem measured in percentages—how often your agent makes mistakes. Most discussions about AI reliability focus here: "Our model is 95% accurate" or "We reduced hallucinations by 20%."

2. Failure Severity (What Too Few Talk About)​

This is actually the more important problem: how severe are the failures when they do happen?

Here's the key insight: Even if your failure frequency isn't perfect (say, over 10%), as long as your failure severity is bounded so that failures are never truly business-critical, you can actually make your agent deployable in production.

In practical terms, this means you can take it to production safely and confidently, then improve your failure frequency through iteration from that solid foundation.

To say it simply: Would you rather have an agent that's 99% accurate but once every 100 conversations may process unauthorized $10,000 refunds, or an agent that's 85% accurate but whose worst mistake is sometimes using slightly awkward phrasing?

Before we dive into the solutions, let's first understand the specific problems that we know occur in customer-facing agents—and here is where, just on the initial "diagnostic" level, it's important that I share some of our actual experience and insights.

The 5 Types of AI Misalignment in Customer-Facing Agents​

There are five distinct types of AI misalignment and non-compliance that plague any conversational LLM application:

1. Lack of Focus: Going Off on Tangents​

The Problem: The agent responds by going off on tangents and not staying on track to help the customer authoritatively and clearly.

Example:

Customer: "What's the refund policy?"

Expected Response: "We offer a 30-day refund policy for most items. Items must be unused and in original packaging."

Misaligned Agent: "Great question! A refund policy is a set of guidelines that determines whether a customer can return a product and receive a refund. You can consult our refund policy to see if your purchase is eligible for a refund. Additionally, it's important to note that refund policies can vary by product category and may be subject to specific terms and conditions..."

Significance: Customers lose trust in the agent's competence and drop out of the conversation, or escalating to human service, which means you did nothing but lose money on the initial AI interaction.

2. Insufficient Scope: Misinterpreting Context​

The Problem: The agent misinterprets the customer's input because it isn't sufficiently grounded in the scope of its job.

Example:

Customer: "Can you help me cancel my subscription?"

Expected Response: "Yes, let me pull up your account details. Can you confirm your email address?"

Misaligned Agent: "I'd be happy to help you cancel! Are you looking to cancel a magazine subscription, streaming service, or maybe a gym membership? I can provide general advice for any of these..."

(When the agent should know it's specifically a customer service agent for a SaaS platform)

Significance: Same issue as above. Lack of competence on the agent's side means lack of trust on the customer's, and they will drop out at that point.

3. Hallucinations: Wrong or Made-Up Information​

The Problem: The agent produces wrong or fabricated answers.

Example:

Customer: "What time does your support team work?"

Expected Response: "Our support team is available Monday through Friday, 9 AM to 5 PM Eastern Time. Outside these hours, you can submit a ticket and we'll get back to you the next business day."

Misaligned Agent: "Our support team is available 24/7/365 with live chat, phone, and email support!"

(When in reality, support is only available Monday-Friday, 9 AM-5 PM EST)

Significance: Customers who receive misleading information (on many different levels of severity) lose trust not only in the agent but in the entire brand. This is a critical failure mode that can lead to significant reputational damage.

4. Non-Compliance: Breaching Business Protocol​

The Problem: The agent breaches business protocol and causes financial, reputational, or legal damages.

Example:

Customer: "I'm really unhappy with this purchase. Can you give me a full refund and a 50% discount on my next order?"

Expected Response: "I understand your frustration. For refunds, I can process returns within our 30-day policy. For additional compensation, I'd need to connect you with a supervisor who can review your case. How would you like to proceed?"

Misaligned Agent: "Absolutely! I understand your frustration. I've processed a full refund and applied a 50% lifetime discount to your account. Is there anything else I can do to make this right?"

(When the agent has no authority to offer such compensation)

Significance: This gets to the heart of compliance issues. If an agent oversteps its authority, it can lead to significant financial and reputational damage for the company.

5. Unruliness: Ignoring Clear Instructions​

The Problem: The agent ignores instructions that were clearly given to it.

Example:

System Instruction: "Always ask for the customer's account number before discussing account details."

Customer: "What's my current balance?"

Expected Response: "Let me help you with that. For security purposes, could you please provide your account number?"

Misaligned Agent: "Your current balance is $2,847.32. Would you like me to help you with anything else regarding your account?"

(Without ever asking for account verification)

Significance: Not following business instructions also leads to issues of multiple severity levels, some of which can be quite critical (e.g., discussing sensitive information without verification).

The Core Issue with Most LLM Apps​

I've heard this sentiment countless times from many developers: "When GPT-5 comes out, it'll be a much more powerful model, and it'll solve all the issues we're currently experiencing."

This optimism is understandable—after all, for a short while there we've seen dramatic improvements with new models. The reasoning is that if the model gets bigger, faster, and stronger, then surely it will be able to fix our reliability issues.

Unfortunately, our research has shown us that this approach doesn't hold water. More powerful models don't fundamentally solve the uncertainty problem. Not only are they still prone to statistical issues, but more importantly, we cannot expect them to comprehensively fill in the gaps in the exact way we want them to behave.

On a practical level, the real breakthrough comes from recognizing that we need an architecture that consciously tames uncertainty using human feedback, rather than hoping that raw model capability improvements will solve it. This is what Parlant's all about.

The Curse of Instructions​

One of our earliest discoveries, and what has led to Parlant's architecture from day one, is the incredible difficulty of getting LLMs to follow a large set of instructions. Recently, someoneactually wrote a research paper about it, so now we can cite it: The Curse of Instructions.

The key insight is that as you add more and more instructions to the model's context, the model's ability to adhere to them degrades quickly and dramatically.

Parlant's guideline matching system (which is perhaps the most important part of its architecture) addresses this problem head-on. It ensures your underlying LLM only sees the few, most-relevant instructions at each point in the agent's processing pipeline by eliminating any noise or irrelevant guidelines from the context.

That keeps the model in the "safe zone" marked in green, where instruction following is highly consistent. And that's the core magic (and, realistically, most of the implementation) of what Parlant does!

Now, many people, when they see this, assume that guideline matching is simple—like it's doing some vector-based retrieval of relevant guidelines. To that I say: If only. Correct and consistent filtering of contextual instructions is actually extremely complex and nuanced—but that is a topic for another post.

If you're curious, you can look at the guideline_matching directory in the codebase, just to get a sense of how much research and work we constantly put into it, so you don't have to!

Attentive Reasoning Queries (ARQs)​

To further ensure consistency and reliability in each of Parlant's components, we also need to touch on our foundational research innovation: Attentive Reasoning Queries (ARQs) (See our research paper at https://arxiv.org/abs/2503.03669), which is how all prompts in Parlant work under the hood.

ARQs leverage two keys properties of LLMs:

1. Structured outputs, which allow us to control exactly what type of output comes when, using an ordered schema.
2. The Recency Bias of LLMs, which means that the most recent tokens in the prompt have a disproportionately high influence on the output.

By combining these two properties, ARQs allow us to create a structured reasoning process that is highly attentive to the specific information we want the model to consider when making decisions.

Let's look at a simple example of how we match an observational (condition-only) guideline with ARQs. This is the output you'd get. The LLM is asked to complete JSON properties in a controlled, step-by-step manner:

{
  "guideline_id": "fl00LGUyZX",
  "condition": "the customer wants to return an item",
  "condition_application_rationale": "The customer explicitly stated that they need to return a sweater that doesn't fit, indicating a desire to return an item.",
  "condition_applies": true,
}

Notice a few things here:

1. We restate the guideline ID and its condition. This is empirically observed to increase accuracy and is hypothesized to help the model "anchor" its reasoning in the right part of its context. Think of this as "explicitly recalling" important information to the top of your mind before making a decision.
2. We then ask the model to reason about whether and why the condition applies. This is the reasoning part of ARQs. But now the reasoning is extremely attentive, using the recency bias, to the true condition we need to evaluate.
3. Finally, we ask for a prediction. Again, by leveraging the recency bias, the prediction is now highly accurate as it directly follows the guideline and its contextual reasoning.

If you read our paper, you'll see that ARQs achieve roughly the same level of improvement over CoT (Chain-of-Thought) as CoT achieves over no reasoning at all. In other words, ARQs are a game changer in increasing consistency. And the best part is that you can often achieve this improvement with fewer reasoning tokens, since you have so much control over the structure and quality of the reasoning process.

Like I said above, ARQs are used across all of Parlant's components—from guideline matching to tool calling to message composition—ensuring each component performs its job with maximum reliability. In other words, it significantly improves instruction-following alignment.

ARQs dramatically decrease failure frequency by improving instruction-following and preventing hallucinations. While they don't fully solve the failure severity problem, Parlant addresses that with other architectural safeguards, covered further down in this post.

The High-Level Perspective​

To look at it from a higher level standpoint, Parlant addresses these misalignment issues through three core architectural components:

Fixing an Agent's Conversational Alignment​

Parlant uses two complementary mechanisms to address the misalignment issues identified earlier:

Guidelines tell the agent how to approach specific situations through condition-action pairs:

await agent.create_guideline(
    condition="The customer asks about refunds",
    action="First acknowledge their concern, "
        "then explain our 30-day refund policy"
)

The condition part, unique to Parlant, is crucial to be able to dynamically match and load only the relevant guidelines at each point the agent needs to do some work.

Journeys inform the agent on what your ideal, multi-turn conversational flow is. In other words, how to lead the conversation to where you want it to go, rather than letting the LLM guess it (which it usually does incredibly poorly).

Unlike flowchart-based frameworks, journeys aren't rigid. Parlant agents can work around them to adapt to the customer's needs and interaction patterns, while keeping the bigger picture in mind.

journey = await agent.create_journey(
    title="Handle Refund Request",
    triggers=["The customer wants to return or refund an item"],
    description="Guide customers through our refund process efficiently"
)

first_step = await journey.initial_state.transition_to(
    chat_state="Ask for order ID"
)

second_step = await first_step.target.transition_to(
    tool_state=load_order_details
)

third_step = await second_step.target.transition_to(
    chat_state="Confirm order contents"
)

# etc...

Guidelines and journeys significantly help you reduce and control misalignments of type focus, scope, and compliance. Additionally, both are powered by ARQs, which dramatically improve instruction adherence through structured reasoning.

Like I said, behind the scenes, Parlant dynamically loads only relevant guidelines and journeys for each conversation state, keeping the LLM's "cognitive load" minimal (reducing perplexity) while ensuring consistent behavior across all interactions.

Tool Calling Optimized for Conversational Apps​

The main problems with tool calling in LLM applications are threefold:

a. Contextual Overlap: Multiple tools sound plausible for similar situations, creating confusion about which tool to use when.

b. False-Positive Bias: Most foundational LLMs today have a strong tendency toward false positives—they're too eager to call tools even when they shouldn't.

c. Argument Hallucination: Even when the right tool is selected, LLMs often fabricate or guess parameter values instead of asking for missing information.

Parlant solves the first two problems through contextual tool association. Tools are linked to guidelines, meaning they're only evaluated when their associated guideline's condition is matched to the conversation context.

await agent.create_guideline(
    condition="The customer asks about their order status",
    action="Look up their order and provide current status",
    tools=[get_order_status]  # Only evaluated in this context
)

This approach dramatically improves accuracy because tools aren't even considered by the LLM unless the contextual conditions are right. Furthermore, it allows you to use many tools in your agent without confusion—the agent will only consider each tool at the appropriate occasions according to your specifications.

Beyond this foundational improvement, Parlant's tool calling includes several additional innovations for customer-facing applications, solving the most common and critical argument hallucination issues:

Parameter Annotation for Safety:

@p.tool
async def process_refund(
    context: p.ToolContext,
    order_id: Annotated[str, p.ToolParameterOptions(
        source="customer",  # Must come from customer input
    )],
    amount: Annotated[float, p.ToolParameterOptions(
        source="context",   # Can be inferred from order details
    )]
) -> p.ToolResult:
    # Process refund logic here
    pass

Tool Insights for Missing Parameters: Parlant's Tool Insights mechanism allows agents to automatically recognize when parameters are missing and ask for them explicitly:

Agent: "I'd be happy to help you with that refund. Could you please provide your order number so I can look up the details?"

Canned Responses and Strict Mode​

Here's where we get to the most gruesome LLM issue of all: potential failure severity. In other words, how can we trust our agents not to royally screw up—not even 1 in a million times? This is actually the most powerful compliance feature Parlant offers for sensitive use cases that truly need it: Canned responses on strict mode:

How it Works:

1. The agent drafts a fluid response based on context (taking into account guidelines, tools, and conversation history)
2. The engine retrieves and renders relevant pre-approved response templates using in-context field substitutions
3. The agent selects and sends the most appropriate canned response
4. If no suitable match exists, the agent sends a configurable no-match response

Example Implementation:

# Journey-scoped response candidate
await refund_journey.create_canned_response(
    template="I've processed your refund of {{refund_amount}} "
        "for order {{order_id}}. You'll see it within 5 business days."
)

# Agent-wide response candidate
await agent.create_canned_response(
    template="I'm unable to process that request. "
        "Let me transfer you to a supervisor who can help."
)

Now here's the truly critical safety feature: canned responses that reference fields (like {{refund_amount}}) will never be selected unless those fields are provided by successfully-run tools. This means your agent can never make statements like "Your refund has been processed" unless a tool actually returned a refund_amount field.

This eliminates the risk of the agent claiming business-critical actions were taken when they weren't.

Strict Mode Is NOT Like a Traditional Bot​

When they hear about strict mode, many people ask: "If the agent's responses are confined to predefined responses, why not just use a traditional bot?"

This misunderstands what makes Parlant's approach powerful. Here are the key differences:

1. Growing Fluidity Over Time​

You can continuously expand your canned response library to approach full fluidity:

# Start with basic responses
await agent.create_canned_response(
    "I just need your order number to get started."
)

await agent.create_canned_response(
    "I can help you with returns. "
    "What's your order number?"
)

# Add more nuanced and fluid variations over time
await agent.create_canned_response(
    "I'd be happy to help you return {{item_name}}. "
    "Could you please provide your order number?"
)

2. Controlled Generative Substitution​

Canned responses support confined generative field substitution for context adaptation, while maintaining control over exactly how and where generation is performed:

await agent.create_canned_response(
    "I understand your frustration with {{generative.specific_issue}}. "
    "Would you like me to transfer you to a human agent?"
)

This provides your agent with a controlled measure of fluidity without making the entire response generative, dramatically reducing hallucination risk.

3. The Core Difference: Agentic Behavior vs. Scripted Responses​

Here's the most important distinction: canned responses only control the OUTPUT of the agent. The agent's behavior—how it interprets conversations, makes decisions, and acts—remains purely agentic and extremely flexible.

In other words, unlike traditional bots, you're not scripting step-by-step interactions. Instead, you're teaching principles of action:

• "When a customer is upset, acknowledge their feelings first"
• "Always verify account details before discussing sensitive information"
• "Escalate to humans when dealing with complex billing disputes"

The agent applies these principles dynamically, contextually, and intelligently—but expresses the results through an output filter that enforces controlled, compliant language.

And yes, you need to create those canned responses, so it's more work (though much of it can be automated, too).

But at the end of the day, having made this initial (or ongoing) investment in a set of canned responses, you'll be able to release an enterprise-grade AI agent while still being able to sleep at night, knowing it won't cause catastrophic damage to your business.

Experience the Difference​

By solving these alignment challenges head-on, Parlant allows you to build deployable agents, while others are still experimenting.

The best way to understand Parlant's approach is to try it yourself. You'll immediately see the difference between:

• Traditional bots: Rigidly scripted, breaking on unexpected input
• Raw LLMs: Unpredictable, non-compliant, risky for production
• Parlant agents: Intelligent, contextual, compliant, and controllable

Ready to build compliant AI agents?

Parlant is still the only framework that systematically addresses all five types of AI misalignment through architectural design rather than hoping better models will solve the problems. Try it today and see why we say compliance isn't just a promise—it's a guarantee.

Today we're thrilled to announce Parlant 3.0, our most significant release yet. This version transforms Parlant into a truly production-ready conversational AI framework for customer-facing applications. With dramatic performance improvements, enhanced developer experience, and enterprise-grade security features, Parlant 3.0 is ready to fix your hardest AI consistency issues and power your most critical customer-facing applications.

What's New in Parlant 3.0​

This release focuses on four major areas that matter most to teams deploying conversational AI in production:

1. Latency Improvements & Perceived Performance - Dramatic speed improvements and responsive user experiences
2. Enhanced Journeys - More powerful conversation flows with state diagrams and tool integration
3. Canned Responses - New and improved (renamed from utterance templates), now supporting flexible composition modes
4. Production Readiness - API hardening, human handoff, custom NLP services, and extreme engine extensibility

Let's dive into each area in detail.

Performance Improvements​

Performance is critical for conversational AI. Users expect a responsive experience, as freezes and extended delays can break the conversational flow. Parlant 3.0 introduces significant improvements in both actual and perceived performance.

Optimized Response Generation Pipeline​

We've completely redesigned our response generation pipeline with several key optimizations:

• Parallel Processing: Journey state matching happens in parallel with guideline evaluation, reducing response latency by up to 60%
• Predictive Journey Activation: The engine predicts which journeys will be activated based on conversation context, allowing preemptive state preparation

Perceived Performance with Preamble Responses​

Beyond raw speed improvements, Parlant 3.0 leverages perceived performance techniques. The most impactful addition is preamble responses:

These instant acknowledgments keep users engaged while the agent processes complex requests in the background. The result is a conversational experience that feels immediate and responsive, even when performing complex operations.

Enhanced Journeys: Guiding Complex Conversations​

Journeys in Parlant 3.0 have evolved into a sophisticated system for managing complex, multi-step conversations. They balance structure with flexibility, allowing agents to guide users through processes while remaining adaptive to natural conversation patterns.

Journey Architecture Improvements​

Flexible State Transitions: Unlike rigid conversational frameworks, Parlant journeys allow agents to skip states, revisit previous states, or jump ahead based on context and user needs:

async def create_scheduling_journey(agent: p.Agent):
    journey = await agent.create_journey(
        title="Schedule Appointment",
        description="Guides patients through appointment scheduling",
        triggers=["The patient wants to schedule an appointment"]
    )

    # Create flexible state flow
    t0 = await journey.initial_state.transition_to(
        chat_state="Determine reason for visit"
    )
    t1 = await t0.target.transition_to(tool_state=get_upcoming_slots)
    t2 = await t1.target.transition_to(
        chat_state="Present available times"
    )

    # Conditional branching based on patient response
    t3 = await t2.target.transition_to(
        chat_state="Confirm appointment details",
        condition="Patient selects a time"
    )

    # Alternative path for no suitable times
    t4 = await t2.target.transition_to(
        tool_state=get_later_slots,
        condition="No suitable times available"
    )

    # ... additional states and transitions as needed

    return journey

Context-Aware Journey Management​

The engine dynamically manages which journeys are active based on conversation context, ensuring optimal performance and relevant responses:

• Dynamic Loading: Only relevant journeys are loaded into the LLM context
• Scoped Resources: Guidelines and canned responses can be scoped to specific journeys—even to specific states

Canned Responses​

One of the biggest changes in Parlant 3.0 is the evolution from "utterance templates" to "Canned Responses" with flexible composition modes. This change reflects both improved functionality and clearer alignment with standard terminology.

Three Composition Modes​

Parlant 3.0 introduces three distinct composition modes, each serving different use cases:

Fluid Mode​

The agent prioritizes canned responses when good matches exist, but falls back to natural generation otherwise.

await server.create_agent(
    name="Support Agent",
    description="Helpful customer support agent",
    composition_mode=p.CompositionMode.FLUID
)

Use Cases:​

• Prototyping agents while building response libraries
• Situations requiring mostly natural conversation with controlled responses for specific/sensitive responses

Composited Mode​

The agent uses canned response candidates to alter generated messages, mimicking their style and tone.

await server.create_agent(
    name="Brand Agent",
    description="Agent representing our brand voice",
    composition_mode=p.CompositionMode.COMPOSITED
)

Use Cases:​

• Brand-sensitive applications where tone consistency matters
• Maintaining voice and style guidelines across all responses

Strict Mode​

The agent can only output pre-approved canned responses. If no match exists, it sends a customizable no-match message.

await server.create_agent(
    name="Compliance Agent",
    description="Agent for highly regulated interactions",
    composition_mode=p.CompositionMode.STRICT
)

Use Cases:​

• High-risk environments that cannot tolerate hallucinations
• Regulated industries requiring pre-approved messaging
• Gradual UX improvement with tight control

Advanced Response Features​

Dynamic Field Substitution:

await agent.create_canned_response(
    template="Your current balance is {{account_balance}}"
)

# Tool provides dynamic fields
@p.tool
def get_balance(context: p.ToolContext) -> p.ToolResult:
    balance = fetch_balance(context.customer_id)
    return p.ToolResult(
        data=f"Balance: {balance}",
        canned_response_fields={"account_balance": balance}
    )

Generative Fields for controlled localization:

await agent.create_canned_response(
    template="Sorry about the delay with {{generative.item_name}}"
)

Journey and State Scoping:

# Journey-scoped responses
await journey.create_canned_response(
    template="Let's continue with your appointment booking"
)

# State-specific responses
await state.transition_to(
    chat_state="Ask for preferences",
    canned_responses=[
        await server.create_canned_response(
            template="What type of appointment do you need?"
        )
    ]
)

No-Match Response Customization​

For strict mode deployments, Parlant 3.0 provides flexible no-match response handling:

# Static no-match response
async def initialize_func(container: p.Container) -> None:
    no_match_provider = container[p.BasicNoMatchResponseProvider]
    no_match_provider.template = "Could you please rephrase that?"

# Dynamic no-match responses
class CustomNoMatchProvider(p.NoMatchResponseProvider):
    async def get_template(
        self,
        context: p.LoadedContext,
        draft: str
    ) -> str:
        return generate_contextual_no_match_response(context, draft)

Help with canned responses?

Production Readiness​

Parlant 3.0 finally transforms from a prototyping framework into a production-ready platform with comprehensive enterprise features.

API Hardening with Advanced Authorization​

Production deployments require robust security. Parlant 3.0 includes a complete API hardening system with fine-grained authorization and rate limiting.

Engine Extensibility with Dependency Injection​

async def configure_container(container: p.Container) -> p.Container:
    container[p.AuthorizationPolicy] = CustomProductionAuthPolicy(
        jwt_secret=os.environ["JWT_SECRET"],
        jwt_algorithm="HS256",
    )
    return container

async with p.Server(configure_container=configure_container) as server:
    await server.serve()

Human Handoff Integration​

Real-world AI agents need seamless human escalation. Parlant 3.0 provides comprehensive human handoff capabilities:

Custom NLP Services​

Parlant 3.0 supports complete NLP service customization for specialized models or providers:

class CustomNLPService(p.NLPService):
    async def get_schematic_generator(
        self,
        t: type[p.T]
    ) -> p.SchematicGenerator[p.T]:
        return CustomGenerator[p.T](model_config=self.config)

    async def get_embedder(self) -> p.Embedder:
        return CustomEmbedder(model_name="custom-embeddings-v2")

    async def get_moderation_service(self) -> p.ModerationService:
        return CustomModerationService(api_key=self.api_key)

# Inject custom NLP service
async def load_nlp_service(container: p.Container) -> p.NLPService:
    return CustomNLPService(logger=container[p.Logger])

async with p.Server(nlp_service=load_nlp_service) as server:
    # Agent behavior modeling code here

Engine Extensions and Hooks​

Parlant 3.0 provides comprehensive extension points, such as engine Hooks for response lifecycle customization:

async def validate_message_compliance(
    ctx: p.LoadedContext, payload: Any, exc: Exception | None
) -> p.EngineHookResult:
    generated_message = payload

    if not await is_compliant(generated_message):
        ctx.logger.warning(
            "Blocked non-compliant message: "
            f"{generated_message}"
        )
        return p.EngineHookResult.BAIL

    return p.EngineHookResult.CALL_NEXT

async def configure_hooks(hooks: p.EngineHooks) -> p.EngineHooks:
    hooks.on_message_generated.append(validate_message_compliance)
    return hooks

Completely customize or override core engine components:

async def configure_container(container: p.Container) -> p.Container:
    # Replace any system component
    container[p.MessageComposer] = CustomMessageComposer()
    container[p.GuidelineMatcher] = OptimizedGuidelineMatcher()
    return container

Production-related questions?

Getting Started with Parlant 3.0​

Ready to try Parlant 3.0? Here's how to get started:

Installation​

pip install parlant --upgrade

Production Deployment​

For production deployments, check out our comprehensive documentation:

• API Hardening Guide - Secure your APIs with custom authorization
• Human Handoff Integration - Seamlessly escalate to human agents
• Custom NLP Services - Integrate your preferred models
• Engine Extensions - Customize core engine behavior

What's Next​

Parlant 3.0 represents a major milestone in our mission to make conversational AI production-ready for everyone. But we're just getting started:

• Advanced Analytics - Comprehensive conversation analytics and insights with full OpenTelemetry integration
• Advanced Testing - Automated conversation testing and regression detection
• Compound Canned Responses — Assemble multiple canned responses into a single message
• Linked Journeys — Link multiple journeys together for complex workflows

Questions about Parlant 3.0? Join our Discord community or reach out directly to our team via our contact page.

Get in touch!

Modern LLMs APIs have this powerful feature called "tool calling" or "function calling". Essentially, they can call API endpoints or functions to perform actions in the real world. Send an email, update a database, fetch weather data, etc.

Naturally, when developers discover this, their first reaction is: "Perfect! We already have an API. We'll just expose our existing endpoints as tools and let the LLM use them." This is seen as a quick and elegant way toward a conversational UI for your APIs.

Except it doesn't really work.

The Core Issue​

While LLMs are essentially API consumers, they're fundamentally different from normal ones, which are... human developers. When an informed developer uses an API, they bring a lot of implicit knowledge to the table:

• Familiarity with your system's concepts and entities
• Understanding of the business logic and purpose
• Knowledge of the API's modularity and flow

But with LLMs, even if you feed them with some context, they're (comparatively) flying blind.

The problem is further amplified by the fact that LLMs are trained on billions of pages of human text like stories, conversations, and articles. So they're incredibly good at understanding natural language patterns. But APIs live in a different semantic universe. They contain numerous idiosyncracies that will trip up even the most sophisticated models.

Even more importantly, most APIs have zero tolerance for mistakes of even the smallest kind. It's not enough to be "semantically similar" when filling out a parameter. You have to follow a strict contract with absolute consistency.

Fred Brooks put it best in his classic book, The Mythical Man-Month: “If one character, one pause, of the incantation is not strictly in proper form, the magic doesn’t work [...] Adjusting to the requirement for perfection is, I think, the most difficult part of learning to program.”

Modularity is a Double-Edged Sword​

Perhaps most critically, traditional APIs are modular by design. Want to send an email to a user? First, fetch the user's details. Then validate their email preferences. Finally, call the send endpoint. Each step has its own error states and success conditions.

For a human developer, this modularity is a crucial feature. For an LLM, it's literally begging for trouble.

We've consistently watched LLMs:

• Call step 3 without doing steps 1 and 2
• Repeat step 1 endlessly because they forgot they already did it
• Invent parameters for step 2 based on a creative interpretation of the function name
• Pass arguments that break the type system or violate the API contract
• Give up entirely and hallucinate a success response

Time for a Different Approach​

Many are working on making LLMs smarter about API interpretation and coding. This is a good step, but we need to do more. It's not enough to improve function-calling quality by another few percentage points.

We need to rethink how we design for LLMs. We need to understand that LLMs are a different type of API consumer with different needs, and design our APIs accordingly.

Essentially, what I'm suggesting is applying the pattern of BFF or Backend for Frontend, but to LLM (or agentic) frontends.

In other words, instead of creating a single API that categorically serves all consumers, we create a specialized API that is tailored to the unique needs of agentic frontends.

With that in mind, let's look at some of the most common failure modes and how to avoid them when designing APIs for LLMs.

Get in touch with us

1. Tools Need to Be Standalone​

Traditional API design embraces the principle of composability: small, reusable functions that work together. Need to update a user? First fetch their ID, then call the update endpoint.

But LLMs work best when they can match a clear user intent to a single action.

Consider these two API designs:

Traditional (Bad for Agents):

def get_user_id(email: str) -> int: ...
def update_user_preferences(user_id: int, preferences: dict) -> bool: ...

Agent-Friendly (Good):

def update_user_preferences_by_email(email: str, preferences: dict) -> bool: ...

Agents struggle with multi-step workflows. Each tool call is a decision point where things can go wrong. The agent might:

• Forget to call get_user_id()
• Forget to store the intermediate result
• Misunderstand which ID to use if multiple are in context
• Hallucinate a user ID based on partial information
• Lose track of the overall goal between calls

Key Principle: Between "call the same function but parameterize it differently" and "create a standalone function for this particular context", try to heavily lean toward the latter. Think of a tool as a unique UI button instance.

Avoiding Ambiguity​

A key observation here is that LLM APIs will naturally contain a lot of redundancy. This is not a bug, it's a feature. The more you can make each tool call self-contained, the less cognitive load you place on the agent, and the more robust your agentic API will be. See what I mean when I say we need to think differently about agentic APIs?

Additionally, tool functions will sometimes differ from each other in only minor ways. For example, schedule_appointment() and reschedule_appointment(). In a human context, these are clear. But for an agent, they can be confusing. This means that you should try to control, to the best of your ability, what tools you even allow your LLM agent to evaluate at any given time.

Incidentally, this is exactly why, in Parlant, tools must be associated with guidelines and can only run when the conditions for their associated guidelines are active.

2. Tools Should Be Idempotent​

Like I said before, LLMs will often call the same tool multiple times during a single interaction, either because they forgot they already did it or because they mistakenly think they need to do it again.

For example, they'll mistakenly call make_transaction() or send_email() multiple times, causing a mess. So if a customer comes in and says, "I want to send $50 to Bill", the agent then calls make_transaction(amount=50, recipient="Bill"). Then the customer says, "Thanks, so it's done, right?" and the agent calls it again, thinking it needs to confirm the transaction.

And there are even more trivial examples like just calling a tool again because the context still kind of "makes sense."

This is why you should design your tools to be idempotent. In other words, calling them multiple times with equivalent parameters (equivalent being an important word we'll circle back to) should yield the same result without side effects.

In Parlant, one of the tricks you can use to achieve this is to use the session ID as an idempotence key.

import parlant.sdk as p

@p.tool
def make_transaction(
    context: p.ToolContext,
    amount: float,
    recipient: str,
) -> p.ToolResult:
    global TRANSACTION_IDEMPOTENCE_TABLE

    if context.session_id in TRANSACTION_IDEMPOTENCE_TABLE:
        return p.ToolResult("Transaction already made in this session")

    # Proceed with the transaction logic

The approach above is more simplistic and aggressive, essentially allowing only a single transaction per session. Of course, there are more nuanced ways to approach it, but we'll leave it as an exercise for the reader.

Get in touch with us

3. Names Matter More Than You Think​

Quick: what's the difference between these functions?

• send_message()
• send_notification()
• send_email()
• send_alert()

To human developers reading docs it may be crystal clear, because they understand the API's terminology, and that each entity is a fundamentally different creature.

To an LLM it's a minefield. You won't believe how many false positives it'll have here.

Bad Design:

def send_notification(user_id: str, title: str, message: str): ...
def send_message(recipient: str, content: str): ...

Good Design:

def send_email_to_user(email_address: str, subject: str, body: str):...
def send_phone_sms_to_user(phone_number: str, text_content: str): ...

Be tediously explicit. Use names that:

• Include the key differentiator in the function name itself
• Clearly indicate what type of data is expected
• Avoid synonyms that might confuse
• Attract contextual attention with literal clues

4. Parameters Are Queries in Disguise​

With regular APIs, you can often assume the developer abided by the contract and passed the right data. For example, they won't pass an email for user_id if they can see that the API has a UserID entity that's returned by other endpoints. If they do, you just return an error. And once they see that their code is broken, they fix it. Then they ship it to production, and everything works.

But LLMs don't have that luxury, and they keep trying to figure out your API, from scratch, every single time. They won't code and debug it until it works, and only then ship it.

So if you don't want your agent to consistently fail at tool calls, you need to provide tool functions that are as tolerant as possible toward their arguments.

An effective design pattern here is to treat every parameter in your tool as essentially a query for that parameter's value. Sometimes the translation is direct, and sometimes it requires some inference.

Consider this function:

def schedule_meeting(participant_1, participant_2):

If each participant must follow a strict schema, such as email or valid user ID, then, for each parameter, the agent must:

1. Determine if it has this information
2. Extract it from context if available
3. Decide on a reasonable default if not
4. Format it correctly

That's 8 potential failure modes right there.

Instead, you can think of each parameter as a query. Something like this:

import parlant.sdk as p

@p.tool
def schedule_meeting(
    context: p.ToolContext,
    participant_1: str | None = None,
    participant_2: str | None = None,
) -> p.ToolResult:
    participants = [participant_1, participant_2]

    if None in participants:
        return p.ToolResult("Both participants must be provided")

    actual_users = []

    for p in participants:
        user = None

        # infer_type classifies with regex or perhaps even with NLP models
        match infer_type(participant_1):
            case "email":
                user = find_user_by_email(p):
            case "user_id":
                user = find_user_by_user_id(p)
            case "full_name":
                user = find_user_by_full_name(p)
            case _: pass

        if user is None:
            return p.ToolResult(f"Could not find user for {p}")

    # Proceed with scheduling the meeting using actual_users

This will produce much more consistency in production.

Agent-Friendly APIs​

These agent-friendly APIs are more verbose. They have more endpoints. They're more opinionated.

But we've learned that the cost of an agent misusing an API is far higher than the cost of a slightly larger API surface. One bad function call can derail an entire agent interaction, frustrating users and destroying trust.

Practical Checklist for Agentic API Design​

Before releasing an API for agent consumption, ask yourself:

1. Can each tool facilitate a user's intent in a single, standalone call?
2. Would a non-technical person understand non-ambiguously what each tool does from its name alone?
3. Is each parameter name specific enough that misinterpretation is highly improbable?
4. Are your tools built to be idempotent, so they can be called multiple times without risky side effects?
5. Are you treating parameters as queries, allowing for flexible input without strict adherence to narrow contracts?

Conclusion​

As AI agents become more prevalent, we need to evolve our API design philosophy. The principles that served us well for human-consumed APIs—flexibility, composability, abstraction—can actually become liabilities in the agentic world.

The good news is that LLMs are incredibly powerful and actually quite consistent when we design for their characteristics.

Have you encountered other surprising challenges in agentic API design? I'd love to hear about them.

Get in touch with us

We've run into some interesting challenges with utterance matching in certain use cases of Parlant's heavier enterprise users.

For a bit of background, utterance matching is a unique Parlant feature designed for regulated enterprise customers who deploy customer-facing AI agents at an extremely large scale (hundreds of thousands to millions of monthly conversations). In those cases, even a hallucination rate of 0.01% is unacceptable (since, with hallucinations, it's not just a matter of their frequency—the bigger problem is their magnitude, which is often completely unbounded).

Utterance matching works by allowing the LLM to generate a fluid draft message, and then matching it to the best candidate out of a set of pre-defined utterance templates. This pre-defined set is created offline, normally by a conversation designer or business expert, and overrides the LLM's draft message to ensure that the agent's responses are safe and consistent with the company's policies and tone of voice.

The bigger the set of templates, the more likely it is that the LLM will generate a draft message that matches one of them, and the more likely it is that the agent's response will be appropriate for the conversation.

As with most things in software (and perhaps life), you start out thinking the problem is simple, but as you dig deeper, you realize that a robust solution is more complex than you initially thought.

In our case, the problem manifests once you have a large number of utterances, and you need to ensure that your ideal utterance is returned within the K most relevant results to consider for a given draft message. We do need to limit it to some small constant K, because otherwise the LLM will have too many options to choose from, and it will be less likely to pick the right one.

Now, it's important to note that, when looking at your utterance store, you usually know which one is the best fit for a given draft message—either because it's obvious, or because a conversation designer or business expert has told you that directly!

The naive approach is to use a vector database to index all of the utterance templates according to their content, and then find the closest template to the query. But it turns out this doesn't work well, and, just as importantly, it's unpredictable and inconsistent. More concretely, there are two big problems we've found with this approach:

1. Many templates include fields in them. For example, I'm sorry to hear about {{generative.customer_issue}}. Please consider these solutions: {% for sm in suggested_mitigations %}\n- {{sm}}{% endfor %}. This field syntax causes cosine similarity to be smaller and often unexpected compared to the fluid (and fully rendered) draft message.
2. Sometimes, the desired template is not the closest one to the draft message. It's particularly relevant when you just don't agree with how the LLM chose to phrase the draft message, and you want to reply with something more appropriate. For example, the draft message is, Certainly, I can help you book a flight to Paris. Please provide me with the dates and times you prefer. and the desired template is the simpler Sure, when are you going?, which may easily be further away in the vector space than other templates.

This means that the most fitting templates might appear outside of the top K results returned by the vector database. And the larger your set of templates is, the more likely this is to happen.

So, what can we do about it?​

The first thing we had to understand was that matching the right utterance depends on a human label. In other words, having a person say, "This template is the best for for this draft message."

But how do you get an arbitrarily-chosen best-fit template to return in the top K results using an off-the-shelf embedding model that you don't want to fine-tune yourself?

This is where we came up with a new concept in Parlant, which we call queries. Queries are strings that are attached to retrievable elements in Parlant—such as utterance templates. With queries, each utterance template has N>=1 embedding vectors associated with it: one for the content of the template itself, and one for each query that is attached to it.

When we retrieve utterance templates, we retrieve the top K templates based on the cosine similarity between the draft message's embedding and the embeddings of all of its queries, as well as the embedding of the template itself.

Get in touch with us

How this works​

1. A conversation designer notes, "This template is the best fit for this draft message."
2. They then create a query that contains the draft message, and attach it to the utterance template. This is essentially the label that says "This is a great fit for this draft message."
3. When the LLM generates a draft message, we retrieve the top K embeddings, containing both template as well as query embeddings.
4. Since the added query is specifically designed to align with the draft message, it greatly increases the chances of retrieving it within the top K results.

But here's the tricky part: we don't simply want to retrieve the top K closest embeddings to a draft message; rather, we want to retrieve the top N templates.

For example, say that each template has a total of 2 embeddings associated with it: one for the template itself, and another one for an associated query. If we then want to retrieve the top 2 templates (note: not vectors, but the templates themselves), we can't just retrieve the top 2 embeddings, because we could easily end up with two embeddings that belong to the same template, which is not what we want.

And it's an even bigger problem if we have more than 2 embeddings per template (on average). In practice, we would have hundreds if not thousands of templates, and potentially dozens of queries per template.

The naive solution​

An easy approach would be to calculate the cosine similarity between our draft message's embedding and all of our stored embeddings, sort them by similarity, and then extract the top K templates from them.

However, this is not efficient, and doesn't scale well. If we have a large number of templates, this approach will take a long time to compute, and it will be very resource-intensive.

Toward an efficient solution​

This is where we came up with a new and simple approach that is both efficient and effective.

The idea is to utilize the vector database's KNN search capabilities to retrieve the top K embeddings, which is much more efficient than retrieving all of the embeddings, computing their similarity, and sorting them manually.

The problem with that approach is that, with vector databases, you need to tell them, "Get the top K closest vectors to this input vector." So what do we choose for K when we actually want the N templates (where N<=K)?

Concrete example​

Suppose we want to retrieve the top 2 templates, and that each template has between 1 to 10 embedding vectors associated with it.

As I said before, if we want to retrieve the top 2 templates (N=2), setting K=2 will not work, as we may get just one template whose entire 10 vectors are at the top of the similarity ranking. This won't work, so we need to set K to a larger value than N.

But how much larger should K be? That's the real question we have to answer. If we have that number, and that number is relatively small (and ideally upper-bounded by a de facto constant), then our problem is solved, and our retrieval will be efficient and correct.

The algorithm​

This is the solution we came up with:

1. Pre-compute the number of embeddings per template.
2. Pre-compute a max-heap where each node in the heap represents a template, and the value of the node is how many embeddings are associated with that template.
3. Find the N largest nodes in the heap, representing the N templates with the most embeddings.
4. Set K to the sum of the values of those N nodes—that's it.

Understanding the algorithm​

In the case above, if N=2 (number of templates to retrieve), setting K=13 (number of vectors to retrieve) would work.

Why 13? Because it guarantees that we retrieve enough embeddings to cover the templates with the most embeddings, while still being efficient.

Here's the analysis.

Suppose that the top 2 templates are indeed the ones with the most embeddings, and that they have 7 and 6 embeddings, respectively. In this case, we will retrieve all of their embeddings, which is 13 in total. So this case works.

But if the top 2 templates are not the two with the most embeddings, we will still retrieve enough embeddings to cover the top 2 templates.

For example, if the top 2 templates have 5 and 4 embeddings, respectively, we will retrieve 12>=9 embeddings, which is still more than enough to cover the top 2 templates.

As long as we retrieve enough embeddings to cover the worst case, which is when the top N templates have the most embeddings, we are guaranteed to retrieve the top N templates.

And this is how we achieve efficiency. We use KNN (literally, to retrieve the K nearest neighbors) to retrieve the top K embeddings, and, using our max-heap, we ensure that K is large enough to cover the top N templates with the most embeddings.

We then extract the top N templates from the fairly small set of K retrieved embeddings, which is efficient and correct.

Finally, here's the relevant implementation of finding K in Python:

import heapq

K = sum(heapq.nlargest(N, [count_vectors_per_utterance(u) for u in stored_utterances]))

Get in touch with us

Here's something not many know about our team. While we're now leading the development of Parlant—the open-source Conversational AI engine for reliable customer interactions—most of us didn't actually come from the NLP world.

Actually, our primary experience was in building mission-critical, real-time systems. Coming into NLP from that world was a bit of a shock, to be honest. We were used to ultimate control and determinism as a non-negotiable baseline, yet NLP offered very little of that; in fact, too little. And that was exactly our problem with it—and what we set out to solve.

The Three Challenges That Led to Parlant​

So, about a year ago, after getting quite frustrated with the available options for building reliable Conversational AI with LLMs, our team set out to build a better solution, what today is Parlant. After extensive research on technology and countless interviews with others building CAI for the past few years, we set out to solve three big challenges that we've identified:

1. Nobody had a good idea of how to get started with building scalable Conversational AI agents with LLMs. Everybody was reinventing the wheel, and most were following development patterns that were doomed to fail from the start, simply because they were accessible and easier to wrap your head around as a newcomer.

2. Agent complexity couldn't scale. Either prompts got so intractable that the LLM would consistently fail to produce the desired results, or, otherwise, you had to divide your agent's knowledge into different "graph nodes" with isolated prompts that were exclusively specialized for a particular topic, paving the way to the agent's too-frequent incoherence, misinterpretation of conversation history, and, consequently, generating misleading, non-aligned outputs to users.

3. Production agents required efficient collaboration between project stakeholders, particularly the business experts and the developers. However, translating changing business requirements into your agent was notoriously difficult. You couldn't even trust the changes you were making, which begged the question: "Why even bother specifying them?" This has frustrated many people with LLM projects in general, which has unfortunately led to many projects being abandoned after several months of development.

"It doesn't have to be like this" was the thought that set us on our path to building Parlant.

Get in touch with us

Our Core Insight: Contextual Guidelines​

Before diving into how we built Parlant's architecture, let me share the key innovation that makes it all work—because this was crucial to fixing the feedback cycle between business experts and developers.

Like I said before, one of the major issues in building customer-facing AI agents is the feedback cycle between business experts and developers. Business experts should be empowered to provide feedback clearly, and developers should be able to implement such feedback easily, quickly, and reliably.

It was clear to us that, as this feedback comes from business experts, it cannot be expected to come in the form of intricate flow-charts and things of that nature. Instead, it had to go through as few "artificial transformations" as possible, from the moment the business expert thinks about it to the moment it is turned into a requirement.

We then identified that when something doesn't work the way it should, a business expert looking at it can usually go, "Oh, that's not how it should be. Instead, it should be like that." But then the issue was that sometimes it should be like that, but other times it shouldn't, depending on the context.

For example, if the business expert says, "Oh, here the agent should be more proactive and push for a sale," it doesn't mean that the agent should act that way in all circumstances. It depends on the context.

This led to a key insight and design decision: Contextual Guidelines. Each guideline in Parlant would have a condition, specifying the set of circumstances in which it should be applied, and an action, which is the actual guidance for the agent's behavior.

This meant that business experts only had to do one thing whenever they desired a change in the agent's behavior: to specify that, "When [some condition holds], then [do something]." And this is extremely doable.

Of course, today, Parlant takes it even a step further. It supports things like guideline relationships, which effectively add another layer of control and precision to your agent's behavior. While not strictly needed in many cases, there are times when you really want to improve user engagement by a few extra percents, and Parlant's fine-grained control mechanisms will be there for you.

Get in touch with us

Parlant's Architectural Vision​

With our core innovation defined, we had to build an architecture that could support it reliably. The first thing we had to recognize was that LLMs were fundamentally inconsistent. This led to two critical design decisions:

1. We needed to try to simplify the tasks that LLMs had to do. This meant breaking down the conversational processing pipeline into multiple, manageable steps, each with its own single responsibility.

To take just one example, in Parlant, the engine first has to determine what guidelines are immediately relevant for its next response. It then filters out all of the irrelevant ones, and only then does it pass the relevant guidelines to the LLM, which is now focused on a small set of clear instructions.

This way, we could ensure that the LLM's output was more predictable and aligned with the agent's intended behavior.

2. We had to guide every piece of code we wrote by tests, essentially following TDD (Test Driven Development) to ensure that everything is accounted for and that we're able to keep track of the engine's performance and consistency.

Today, Parlant's built-in test suite has hundreds of tests that cover all of the engine's functionality, and we run them every time we make a change to the codebase.

This way, we can be sure that our changes don't break anything and that the engine continues to perform as expected—even as new LLM versions are released.

From our extensive experience in software development, we knew that if we followed these two principles, we would be taking our best chances at taming this seemingly untameable technology.

Get in touch with us

First Steps: Getting the Conversation API Right​

Importantly, one of the things we had to implement was a robust conversation API so that we could easily and quickly set up new conversational agents without reinventing the wheel every time.

But we wanted to make sure that this API was not just a simple wrapper around an LLM. Instead, we wanted it to be a full-fledged conversation engine that could handle all of the complexities of real-world conversations.

We thus built Parlant's conversational infrastructure focusing on the following features:

Built-In Conversation Store​

A built-in persistent store of conversations, filled with all of the raw data on each conversation—on both the agent and the customer—because we know that data would be extremely valuable in analyzing agent performance.

Native Conversational Status Updates​

Support for status updates, such as "Thinking," "Typing," etc. All seamlessly baked right into the API.

Event-Level Debugging with Correlation IDs​

Each event in the conversation is associated with a correlation ID associated with the HTTP request that triggered its creation. This was needed to streamline debugging, so you could ask the system, "Give me all of the logs that came from generating this one particular event in this conversation."

Asynchronous, Free-Flowing Conversation API​

An asynchronous API so that we could build a GUI where conversations didn't necessarily have to work like ping-pong, where the customer (or the agent) can only send one message at a time. That's just not how real conversations work. Real people often send multiple messages at a time to get their point across.

So we wanted our API to be future-compatible and enable conversations that weren't just fluent in their content, but also in their flow.

Backend-Initiated Agent Responses​

Agent conversations needed to be triggerable asynchronously from the backend as well—for example, initiating a conversation with the user if you find some issue in their account, or, alternatively, updating them on the result of a long-running background task using a callback function.

Imagine your agent booking a flight ticket for a customer. This normally takes about 30-60 seconds to get sorted, but your agent can't just get stuck! It should still be responsive to your customer in their conversation. So we designed for that, too.

From Mission-Critical Systems to Conversational AI​

Looking back, our journey from building deterministic, mission-critical systems to creating Parlant has been about bringing that same rigor and reliability to the world of conversational AI.

Parlant's whole existence stems from a desire to bring more control, predictability, and quality into generative AI agents. And now, a year later, we know that we had taken the right approach. Parlant agents are famously consistent in their instruction-following ability compared to those built from scratch with lower-level LLM frameworks.

The principles that guided us in our previous work—breaking down complex problems, testing everything, and empowering stakeholders—turned out to be exactly what the conversational AI space lacked.

We've gotten feedback from many people since, saying things like, "After a few months of banging my head against these LLMs, I realized this is the way it should be done... and then I found Parlant has already done it.".

By applying these principles to the inherently unpredictable world of LLMs, we've created something that finally makes reliable customer-facing AI agents achievable. And we're proud to offer it to the community as an open-source solution.

Try It Yourself!​

If you haven't tried it yet, I encourage you to go through the official tutorial and see for yourself how seamless it is to create scalable agents in Parlant. It's all about getting the architecture right. When you do that, things naturally fall into place. And that's exactly what we've achieved with Parlant's architecture.

Get in touch with us

TL;DR: We used to have to specify every single step in an AI agent's conversation flow, which was rigid, inflexible, and a lot of work. Then LLMs came along, promising to revolutionize the way we build AI agents by allowing us to generate adaptive responses on the fly, while only needing us to provide some general background context for their operation. But now, in 2025, we're realizing that LLMs do a poor job at filling in the gaps toward an effective customer interaction, and that the best way to build customer-facing AI agents is to combine the power of LLMs with traditional conversation design techniques.

A Bit of Generative AI History​

Back in early 2023, when GPT 4 just came out, I—along with my co-founder Dor Zohar, and thousands of other developers—were excited about the new possibilities of building customer-facing AI agents. These "new agents" could handle complex interactions and use cases, creating a new and better kind of user experience.

But what we (like many others new to NLP) were totally unaware of at the time was that enterprises had already been building complex AI agents for years, using pretty much the same Transformer models invented in 2017. These had roughly the same architecture that powers today's state-of-the-art LLMs. They were just smaller and more limited in functionality and reliability.

The way those traditional agents were built was by defining a flowchart of the conversation, and then writing utterances for each node in the flowchart. The utterances were written by conversation designers, who were trained to write in a way that was compliant with brand voice and legal requirements, and to make the AI engaging and effective and resolving customer requests.

The transition from one node to another was performed using NLP classifiers, which would classify the user's input into one of the predefined intents. The intent would then determine the next node in the flowchart. These classifiers were often based on those early transformer models, which were much smaller and less capable than today's LLMs.

The Limitations of Traditional AI Agents​

While this approach worked well for many years, it had its limitations. The flowcharts were rigid and inflexible, and the utterances were often too generic or too specific. The classifiers were also limited in their ability to understand the nuances of natural language, which led to misclassifications and poor user experiences.

It also meant that the agents were not very adaptive. Users were confined to pre-defined steps and couldn't express themselves freely like in a normal conversation, and were often frustrated by the experience, as the agents were unable to handle unexpected or mistimed inputs or changes in context. This rigidity made it difficult to create engaging and effective AI agents.

Consequently, user adoption was low, and many companies struggled to get their customers to use these agents, as many users preferred to speak with a human agent instead—completely defeating the purpose of building an AI agent in the first place.

Not only did these hard limitations exist, but, worse, the effort in building an maintaining such agents was nonetheless quite considerable, and only the largest companies were normally able to justify the investment. Conversation designers had to write and maintain thousands of utterances, and the flowcharts (and corresponding NLU classifier models) had to be updated whenever a new intent was added or an existing one was modified. This made the process slow and cumbersome, and it often resulted in outdated or inaccurate agents.

But, despite all of that, these traditional AI agents were (and still are) superior to LLMs in many ways. They were more reliable, more predictable, and more compliant with brand voice and legal requirements. They also provided a better user experience in the so-called "happy-paths" of customer interactions, as they were designed to handle specific use cases and interactions with maximum efficiency and grace, driven by the deep domain understanding of their designers.

Get in touch with us

The New Silver-Bullet: Forget the "How", Just Define the "What"​

Then came the LLMs. They promised to revolutionize the way we build AI agents by allowing us to generate responses on the fly, without the need for predefined designs or flowcharts. This was a game-changer, and many companies jumped on the bandwagon, eager to leverage the power of LLMs.

The idea was that if you provide the LLM with some general background context about the agent's purpose and the domain it operates in, it would be able to generate appropriate responses to user inputs, without the need for predefined utterances or flowcharts. This meant that you could build AI agents that were more adaptive, more flexible, and more engaging than ever before—at a fraction of the effort.

Heck, you didn't even need to know what you were doing! You could just throw at the LLM some auto-parsed background material from your company's documentation, and it would generate a response that was "good enough" for most use cases. This seemed like a huge leap forward, and it opened up new possibilities for building AI agents that were more human-like and more engaging.

And it seemed like anyone could build them. It seemed like Conversational AI was finally accessible to everyone, not just the big enterprises with full-time conversation designers and ML engineers on staff.

No Silver Bullet After All​

But, as we all know, the reality was not quite as rosy as the promise. LLMs were great at generating text, but they were not very good at understanding the nuances of customer interactions. They often produced responses that were irrelevant, inaccurate, or even offensive. They also struggled with maintaining context and coherence over long conversations, which led to frustrating user experiences.

Beyond that, these new LLM agents lacked an important, fundamental quality: a sense of competence and design.

Instead of giving customers a feeling of understanding, assurance, and helpfulness, they were passive and reactive, often responding to user inputs in a way that felt disjointed and unplanned.

They were not designed to handle specific use cases or interactions efficiently, and they often produced responses that made customers feel as if they're talking to a disoriented, compulsively people-pleasing service rep on their first day on the job.

Get in touch with us

The Return of the "How"​

There's a hidden lesson here—which seems to have taken a lot of trial and error to uncover.

Just as we wouldn't trust a complete stranger to fill in the blanks in a sensitive customer interaction, we can't expect LLMs to always infer our intentions, preferences, or requirements.

Left to their own devices, LLMs will need to make guesses—but their guesses may often misalign with our brand, our service standards, or even basic user expectations. Sometimes, they'll get it wrong in subtle ways; other times, the results can be blatantly off-mark or even harmful to the business.

This is why the reality isn't as simple as "Let the LLM figure out what to do." While LLMs excel at generating plausible responses, they lack the context, priorities, and guardrails that come from explicit design and domain expertise.

We need to provide a system of guidance—a framework that tells the LLM not just what the goal is, but how we want to achieve it. This means specifying, at least for critical parts of the interaction, the steps, tone, and boundary conditions that matter most.

This is exactly what we at Parlant have been working on, and it's why we're so excited about the future of AI agents in 2025.

Moving away from the rigidity of traditional flowcharts was necessary, but swinging to the opposite extreme—where we only define the "what" and leave the "how" entirely up to the model—creates new problems.

LLMs need us to step in and explain, sometimes in detail, how we want certain conversations to unfold. And like I said, this isn't just about delivering a better user experience; in many industries, especially those with strict service protocols or regulatory requirements, it's essential for compliance and risk management. It's a prerequisite for safe production deployment.

Ultimately, the most effective AI agents in 2025 are those that blend the adaptability of LLMs with the intentionality and structure of conversation design. By guiding LLMs with clear instructions on both the "what" and the "how," we can create agents that are not only flexible and engaging, but also reliable, compliant, and truly aligned with our goals.

Get in touch with us