5 posts tagged with "llm"

TL;DR Conversational AI has transformed from ELIZA's simple rule-based systems in the 1960s to today's sophisticated platforms. The journey progressed through scripted bots in the 80s-90s, hybrid ML-rule frameworks like Rasa in the 2010s, and the revolutionary large language models of the 2020s that enabled natural, free-form interactions. Now, cutting-edge conversation modeling platforms like Parlant combine LLMs' generative power with structured guidelines, creating experiences that are both richly interactive and practically deployable—offering developers unprecedented control, iterative flexibility, and real-world scalability.

ELIZA: The Origin of Conversational Agents (1960s)​

The lineage of conversational AI begins with ELIZA, created by Joseph Weizenbaum at MIT in 1966.

ELIZA was a rule-based chatbot that used simple pattern matching and substitution rules to simulate conversation. Weizenbaum’s most famous script for ELIZA, called “DOCTOR,” parroted a Rogerian psychotherapist: it would reflect the user’s inputs back as questions or prompts. For example, if a user said “I feel stressed about work,” ELIZA might reply, “Why do you feel stressed about work?” This gave an illusion of understanding without any real comprehension of meaning.

ELIZA was one of the first programs to attempt the Turing Test (engaging in dialogue indistinguishable from a human). While it was a very simple system, ELIZA proved that humans could be momentarily convinced they were chatting with an understanding entity – a phenomenon later dubbed the “Eliza effect.” This early success sparked widespread interest and laid the foundation for chatbot development, even though ELIZA’s capabilities were rudimentary and entirely scripted.

Scripted Chatbots: Menu-Driven Systems and AIML (1980s–1990s)​

After ELIZA, conversational systems remained largely rule-based but grew more sophisticated.

Many early customer service bots and phone IVR systems in the 1980s and 1990s were essentially menu-driven – they guided users through predefined options (e.g. “Press 1 for account info, 2 for support”) rather than truly “understanding” free text.

Around the same time, more advanced text-based bots used bigger rule sets and pattern libraries to appear conversational. A landmark was A.L.I.C.E. (Artificial Linguistic Internet Computer Entity), introduced in 1995 by Richard Wallace. ALICE employed a specialized scripting language called AIML (Artificial Intelligence Markup Language) to manage conversation rules. Instead of hard-coding every response, AIML let developers define patterns and template replies. As a result, ALICE had an enormous base of about 41,000 predefined templates and pattern-response pairs. This allowed it to engage in more varied, natural-sounding chats than ELIZA’s simple keyword tricks. ALICE was even awarded the Loebner Prize (a conversational AI contest) multiple times in the early 2000s.

Despite these improvements, bots like ALICE and its contemporaries still relied on static scripts. They lacked true understanding and could be easily led off-track by inputs outside their scripted patterns. In practice, developers often had to anticipate countless phrasings or guide users to stay within expected inputs (hence the popularity of menu-driven designs for reliability). By the late 1990s, the paradigm in industry was that chatbots were essentially expert systems: large collections of if-then rules or decision trees. These systems worked for narrowly defined tasks (like tech support FAQs or simple dialog games) but were brittle and labor-intensive to expand. Still, this era demonstrated that with enough rules, a chatbot could handle surprisingly complex dialogues – a stepping stone toward more data-driven approaches.

The Rise of ML and Hybrid NLU Frameworks (2010s)​

The 2010s saw a shift toward machine learning (ML) in conversational AI, aiming to make chatbots less brittle and easier to build. Instead of manually writing thousands of rules, developers began using statistical Natural Language Understanding (NLU) techniques to interpret user input.

Frameworks like Google’s Dialogflow and the open-source Rasa platform (open-sourced in 2017) exemplified this hybrid approach. They let developers define intents (user’s goals) and entities (key information), and then train ML models on example phrases. The ML model generalizes from those examples, so the bot can recognize a user request even if it’s phrased in an unforeseen way. For instance, whether a user says “Book me a flight for tomorrow” or “I need to fly out tomorrow,” an intent classification model can learn to map both to the same “BookFlight” intent. This significantly reduced the need to hand-craft every possible pattern.

Over time, these NLU models incorporated Transformer-based innovations to boost accuracy. For example, Rasa introduced the DIET (Dual Intent and Entity Transformer) architecture, a lightweight transformer network for intent classification and entity extraction. Such models approach the language-understanding performance of large pre-trained transformers like BERT, but are tailored to the specific intents/entities of the chatbot. Meanwhile, the dialogue management in these frameworks was still often rule-based or followed story graphs defined by developers. In Dialogflow, one would design conversational flows with contexts and transitions. In Rasa, one could write stories or rules that specify how the bot should respond or which action to take next given the recognized intent and dialogue state.

This combination of ML + rules was a major step up. It allowed chatbots to handle more natural language variation while maintaining controlled flows for business logic. Many virtual assistants and customer support bots deployed in the late 2010s (on platforms like Facebook Messenger, Slack, or bank websites) were built this way. However, challenges remained. Designing and maintaining the conversation flows could become complex as an assistant’s scope grew. Every new feature or edge case might require adding new intents, more training data, and more dialogue branches – which risked turning into a tangle of states (a “graph-based” framework that can become overwhelmingly complex as the agent grows).

Moreover, while these systems were more flexible than pure rules, they still could fail if users went truly off-script or asked something outside the trained data.

The LLM Era: Prompt-Based Conversations and RAG (2020s)​

A watershed moment came with the advent of Large Language Models (LLMs) in the early 2020s. Models like OpenAI’s GPT-3 (2020) and later ChatGPT (2022) demonstrated that a single, massive neural network trained on internet-scale data could engage in remarkably fluent open-ended conversations.

ChatGPT, for instance, can generate responses that are often difficult to distinguish from human-written text, and it can carry on a dialogue spanning many turns without explicit rules scripted by a developer. Instead of defining intents or writing dialogue trees, developers could now provide a prompt (e.g. a starting instruction like “You are a helpful customer service agent...”) and let the LLM generate the conversation. This approach flips the old paradigm: rather than the developer explicitly mapping out the conversation, the model itself learned conversational patterns from its training data and can dynamically produce answers.

However, using LLMs for reliable conversational agents brought new challenges. Firstly, large models have a fixed knowledge cutoff (ChatGPT’s base knowledge, for example, only went up to 2021 data in its initial release). And they are prone to “hallucinations” – confidently generating incorrect or fabricated information when asked something outside their knowledge.

To tackle this, a technique called Retrieval-Augmented Generation (RAG) became popular. RAG pairs the LLM with an external knowledge source: when a user asks a question, the system first retrieves relevant documents (from a database or search index) and then feeds those into the model’s context so it can base its answer on up-to-date, factual information. This method helps address the knowledge gap and reduces hallucinations by grounding the LLM’s responses in real data. Many modern QA bots and enterprise assistants use RAG – for example, a customer support chatbot might retrieve policy documents or user account info so that the LLM’s answer is accurate and personalized.

Another tool in this era is the use of system prompts and few-shot examples to steer LLM behavior. By providing instructions like “Always respond in a formal tone,” or giving examples of desired Q&A pairs, developers attempt to guide the model’s style and compliance with rules. This is powerful but not foolproof: LLMs often ignore instructions if a conversation is long or if the prompt is complex, as parts fall out of its attention.

Essentially, pure prompting lacks guarantees – it’s still the model’s learned behavior that decides the outcome. And while RAG can inject facts, it “can’t guide behavior” or enforce complex dialogue flows. For instance, RAG will help a bot cite the correct price from a database, but it won’t ensure the bot follows a company’s escalation protocol or keeps a consistent persona beyond what the prompt suggests.

By late 2024, developers had a mix of approaches for conversational AI:

• Fine-tuning an LLM on custom data to specialize it (which can be expensive and inflexible, often requiring re-training the whole model for small changes).
• Prompt engineering and RAG to leverage pre-trained LLMs without full retraining (quick to prototype, but needing careful tweaking and still lacking strong runtime control and consistency).
• Traditional frameworks (intents/flows or graphical dialog builders) which offer deterministic behavior but at the cost of flexibility and significant manual work, especially as complexity grows.

Each approach had trade-offs. Many teams found themselves combining methods and still encountering issues with consistency and maintainability. This set the stage for a new paradigm aiming to capture the best of both worlds – the knowledge and linguistic fluency of LLMs with the control and predictability of rule-based systems. This emerging paradigm is what we refer to as Conversation Modeling.

Conversation Modeling with Parlant: A New Paradigm​

The latest development in conversational AI is the rise of Conversation Modeling platforms, with Parlant as a prime example. Parlant is an open-source Conversation Modeling Engine designed to build user-facing agents that are adaptive, yet predictable and accurate. In essence, it provides a structured way to shape an LLM-driven conversation without reverting to rigid workflows or expensive model retraining. Instead of coding up dialogue flows or endlessly tweaking prompts, a developer using Parlant focuses on writing guidelines that direct the AI’s behavior.

Get in touch with us

Guideline-Driven Conversations​

Guidelines in Parlant are like contextual rules or principles that the AI agent should follow. Each guideline has a condition (when it applies) and an action (what it should make the agent do).

For example, a guideline might be: When the user is asking to book a hotel room and they haven’t specified the number of guests, then ask for the number of guests. This “when X, then Y” format encapsulates business logic or conversation policy in a flexible, declarative way. The crucial difference from old-school rules is that guidelines don’t script out the exact wording of the bot’s response or a fixed path – they simply set expectations that the generative model must adhere to.

Parlant’s engine takes care of enforcing these guidelines during the conversation. It does so by dynamically injecting the relevant guidelines into the LLM’s context at the right time.

In our hotel booking example, if the user says, “I need a hotel in New York this weekend,” Parlant would recognize that the “ask about number of guests” guideline's condition is met. It would then load that guideline into the prompt for the LLM, so the AI’s response would be guided to, say, “Certainly! I can help with that. How many guests will be staying?” instead of the model’s default response which might have omitted the guest count question. If another guideline says the agent should always respond enthusiastically, that guideline would also be activated, ensuring the tone is upbeat. This way, multiple guidelines can shape each response.

Importantly, Parlant keeps the model’s “cognitive load” light by only including guidelines that are contextually relevant, given the current conversation state. An agent could have dozens of guidelines defined, but the user doesn’t get bombarded with irrelevant behavior – the system is smart about which rules apply when.

This dynamic approach allows richer interactions than a static flowchart: the conversation can go in many directions, but whenever a situation arises that has a guideline, the model will consistently follow that instruction. In effect, the LLM becomes more grounded and consistent in its behavior, without losing its natural language flexibility.

Reliability, Enforcement, and Explainability​

A standout feature of Parlant’s conversation modeling is how it checks and explains the agent’s decisions.

Traditional chatbots might log which intent was matched or which rule fired, but Parlant goes further. It actually supervises the AI’s output before it reaches the user to ensure guidelines were followed. One novel technique the Parlant team developed is called Attentive Reasoning Queries (ARQs).

In simplified terms, ARQs are an internal query the system poses (via the LLM’s reasoning capabilities) to double-check that the response satisfies the active guidelines. If something is off – say the model produced an answer that violates a guideline or contradicts a prior instruction – Parlant can catch that and correct course. This might involve instructing the model to try again or adjusting the context. The result is an extra layer of assurance that the agent’s answers are on-policy and safe before the user sees them.

From a developer’s perspective, this yields a high degree of predictability and makes it easier to debug conversations. Parlant provides extensive feedback on the agent’s decisions and interpretations. One can trace which guideline triggered at a given turn, what the model “thought” the user meant, and why it chose a certain reply.

This level of transparency is rarely available in pure LLM solutions (which can feel like a black box) and even in many ML-based frameworks. If a conversation went wrong, you can quickly see if a guideline was missing or mis-specified, or if the AI misunderstood because no guideline covered a scenario – and then adjust accordingly.

Get in touch with us

Faster Iteration and Scalable Testing​

Conversation modeling also dramatically improves the development lifecycle for AI agents. In older approaches, if a business stakeholder said “Our chatbot should change its behavior in X scenario,” implementing that could mean re-writing parts of a flow, collecting new training data, or even fine-tuning a model – and then testing extensively to ensure nothing else broke. With Parlant, that request usually translates to simply adding or editing a guideline.

For instance, if the sales team decides that during holidays the bot should offer a 10% discount, a developer can implement a guideline: When it is a holiday, then the agent should offer a discount. There’s no need to retrain the language model or overhaul the dialog tree; the guideline is a modular addition.

Parlant was built so that developers can iterate quickly in response to business needs, updating the conversational behavior at the pace of changing requirements. This agility is akin to how a human manager might update a customer service script or policies and immediately all agents follow the new policy – here the “policies” are guidelines and the AI agent follows them immediately once updated.

Because guidelines are discrete and declarative, it’s also easier to test and scale conversational agents built this way. Each guideline can be seen as a testable unit: one can devise example dialogues to verify the guideline triggers properly and the agent’s response meets expectations. Parlant’s deterministic injection of guidelines means the agent will behave consistently for a given scenario, which makes automated testing feasible (you won’t get a completely random response every time, as raw LLMs might give).

The platform’s emphasis on explainability also means you can catch regressions or unintended effects early – you’ll see if a new guideline conflicts with an existing one, for example. This approach lends itself to more robust, enterprise-grade deployments where reliability and compliance are crucial.

Integration with Business Logic and Tools​

Another way Parlant stands apart is in how it separates conversational behavior from back-end logic.

Earlier chatbot frameworks sometimes entangled the two – for example, a dialog flow node might both decide what to say and invoke an API call. Parlant encourages a clean separation: use guidelines for conversation design, and use tool functions (external APIs or code) for any business logic or data retrieval.

Guidelines can trigger those tools, but they don’t contain the logic themselves. This means you can have a guideline like “When the customer asks to track an order, then retrieve the order status and communicate it.”

The actual act of looking up the order status is done by a deterministic function (so no uncertainty there), and the guideline ensures the AI knows when to call it and how to incorporate the result into the conversation. By not embedding complex computations or database queries into the AI’s prompt, Parlant avoids the pitfalls of LLMs struggling with multi-step reasoning or math.

The division of labor leads to more maintainable and reliable systems: developers can update business logic in code without touching the conversation scripts, and vice versa. It’s a design paradigm that scales well as projects grow.

Real-World Impact and Use Cases​

All these capabilities make conversation modeling suitable for applications that were previously very challenging for conversational AI.

Parlant emphasizes use cases like regulated industries and high-stakes customer interactions. For example, in financial services or legal assistance, an AI agent must strictly follow compliance guidelines and wording protocols – a single off-script response can have serious consequences. Parlant’s approach ensures the agent reliably follows prescribed protocols in such domains.

In healthcare communications, accuracy and consistency are paramount; an agent should stick to approved responses and escalate when unsure. Guidelines can encode those requirements (e.g. “if user mentions a medical symptom, always provide the disclaimer and suggest scheduling an appointment”).

Brand-sensitive customer service is another area: companies want AI that reflects their brand voice and policies exactly. With conversation modeling, the brand team can literally read the guidelines as if they are a policy document for the AI. This is a big improvement over hoping an ML model “learned” the desired style from training examples.

Teams using Parlant have noted that it enables richer interactions without sacrificing control. Users aren’t forced down rigid conversational menus; instead, they can ask things naturally and the AI can handle it, because the generative model is free to respond creatively as long as it follows the playbook defined by guidelines.

At the same time, the development overhead is lower – you manage a library of guidelines (which are human-readable and modular) instead of a spaghetti of code. And when the AI does something unexpected, you have the tools to diagnose why and fix it systematically.

In short, Parlant’s conversation modeling represents a convergence of the two historical threads in chatbot evolution: the free-form flexibility of advanced AI language models with the governed reliability of rule-based systems. This paradigm is poised to define the next generation of conversational agents that are both intelligent and trustworthy, from virtual customer assistants to automated advisors across industries.

Get in touch with us

TL;DR Eliminate LLM hallucinations and gain full control over chat style and tone of voice with our new Utterance Templates.

At Parlant, we're committed to pushing the boundaries of conversational AI, ensuring that interactions are not only intelligent but also consistent, compliant, and aligned with your brand's voice.

Today, we're thrilled to unveil a game-changing feature: Utterance Templates.

🧠 What Are Utterance Templates?​

Utterance Templates empower you with precise control over your Parlant agent's responses. By restricting outputs to a predefined set of responses, you ensure your agent communicates with a consistent tone, style, and accuracy, aligning perfectly with your brand voice and service protocols while completely eliminating the risk of even subtle unwanted or hallucinated outputs.

Think of it as handing your agent a curated set of response cards. Based on the conversation context, the agent selects the most appropriate "card" to play, ensuring every interaction is both relevant and on-brand.

🛠️ How It Works​

Under the hood, Utterance Templates operate through a three-stage process:

1. Draft Message Generation: The agent drafts a fluid message based on the current situational awareness, including interaction history, guidelines, and tool results.

2. Template Matching: It then matches the closest utterance template from your predefined set.

3. Rendering: The engine renders the matched template using the Jinja2 templating engine, supporting variable substitutions using tool results, and generative substitutions when explicitly allowed.

This process ensures that responses are not only contextually appropriate but also adhere strictly to your specified guidelines.

🧪 Real-World Use Case: Streamlining Customer Support​

Scenario: A customer inquires about product availability.

Without Utterance Templates:

Customer: Do you have it in stock?

Agent: Yes, we've got this item in stock! Let me know if you need any help finding it.

With Utterance Templates:

Customer: Do you have it in stock?

Agent: Yeah bro. Should I add it to your cart?

By utilizing Utterance Templates, the agent's response is not only concise but also aligns with the brand's conversational style, ensuring consistency across interactions.

Get in touch with us

🔒 Why This Matters: Enhancing Compliance and Brand Integrity​

In industries where compliance and brand consistency are paramount—such as finance, healthcare, and legal services—Utterance Templates offer unparalleled advantages:

• Elimination of Hallucinations: By restricting responses to predefined templates, you eliminate the risk of AI-generated inaccuracies.

• Regulatory Compliance: Ensure that all communications adhere to industry regulations and standards.

• Brand Consistency: Maintain a uniform tone and style across all customer interactions, reinforcing brand identity.

🚀 Getting Started​

Ready to take control of your conversational AI? Here's how to enable Utterance Templates:

1. Define Your Templates: Create a set of response templates using the Jinja2 engine, incorporating variable substitutions as needed.

2. Set Composition Mode: Choose the appropriate composition mode for your use case:

parlant agent update \
  --id $AGENT_ID \
  --composition-mode $COMPOSITION_MODE  # [strict-utterance|fluid-utterance]

• Strict Utterance: Agent can only select responses from defined templates.
• Fluid Utterance: Agent prioritizes templates but may fall back to default message generation if no good match is found.

3. Monitor and Refine: Use Parlant's integrated UI to inspect draft messages and refine templates for optimal performance.

For a detailed guide, visit our Utterance Templates Documentation.

💬 Join the Conversation​

We believe Utterance Templates will revolutionize the way businesses approach conversational AI, offering a blend of control, compliance, and customer satisfaction. We'd love to hear how you're planning to implement this feature in your workflows.

Reach out to our team for personalized assistance.

Here's to more meaningful and consistent conversations!

Get in touch with us

In the market for generic RAG frameworks, the different providers are fighting over who can provide 67% accuracy versus 65%. And when you run an off-the-shelf RAG framework on your use case, it will end up closer to 50% accuracy. Is this the best that the industry can do?

Actually, yes, it is. The problem is that these frameworks are meant to be plug-and-play; you are supposed to be able to build the product once and then sell it to thousands of customers in different verticals. But it just doesn’t work well enough, and it’s unclear whether generic RAGs will ever be able to deliver on this promise. Let’s see why.

Do It Yourself​

The problem with generic RAG frameworks is just that—they are generic and don’t incorporate specific know-how about the vertical or the industry from domain experts. When it comes to information retrieval, the more specific and informed the key/query design of your index, the better accuracy you’ll get. To do that, you need domain experts with continuous refinement processes.

While these generic frameworks don’t deliver, RAG is not that hard to build in-house—and custom implementations are easier to optimize and perform much better than their generic counterparts. At the same time, engineers have every incentive to build RAG frameworks in-house: It doesn’t require extraordinary expertise to achieve good results, and it allows them to play around with interesting technology and improve their skills.

Custom RAGs typically perform dramatically better than generic ones because of the reality that the more case-specific your RAG’s design is, the more accurate it will be.

Are We Building Just For Fun?​

If we want to build just for fun, that’s fine! But if we’re looking for production in real-world cases, accuracy must go higher.

For example, if you were to build a generic RAG framework for the financial vertical, you might get to 90% accuracy based on this narrowing-down of the use ase. But is that good enough for a big bank serving millions of customers—can they afford to get 1 million answers wrong out of every 10? Wouldn’t it make more sense for them to build something in-house quickly and with more control, that could get close to 100% accuracy? In fact, they’d be crazy not to.

What It Takes to Reach 99.999% Accuracy​

It’s possible to build a 100% accurate RAG if you’ve got the right approach and are willing to put in the effort rather than trying to avoid it. We’ve actually done so (and open-sourced it) with parlant-qna. Doing so, for us, has involved making some strategic sacrifices we need to elaborate on:

No Chunked Upstream Information First, unlike generic plug-and-play RAG frameworks, we don't chunk upstream information at all; documents are sent whole to the LLM. This already prevents many lost-in-transition errors.

Independent Q&A Our second strategic choice was to have the questions and answers managed independently rather than heuristically parsing upstream knowledge-bases (KBs). Think about it: You don’t write KB docs the same way you do when you have a conversation. If you want your RAG framework to be more accurate, or if you’d at least like to adjust conversational responses without conflicting with upstream KBs, you have to provide information with the same conversational character that you expect your AI agent to respond in. We just break from convention in having you roll up your sleeves and do that as the fundamental methodology. It’ll be better this way in the long run.

Dynamic Manual Tagging The next decision we made was to support dynamic, manual tagging for every question. One of the big problems with RAG frameworks is what to do when they get answers wrong. With plug-and-play RAG frameworks, in order to update the framework, you usually have to rerun the entire system to re-parse your KBs, and often you’ll end up breaking something else. So we wanted to add a marker that would allow you to always return a particular question response to specific queries. It helps to quickly deploy a fix to production while buying time to iterate on the results of that specific question using your automatic, predictive retrieval mechanisms.

Knowledge-Base Optimization (rather than a perfected RAG) Lastly, we advise our users to spend time curating and optimizing their knowledge-base. When discussing creating RAG frameworks and AI agents in general, we talk a lot—perhaps too much—about creating the perfect algorithms that make them work.

But the key to creating RAG that actually returns highly accurate answers lies in curating the knowledge base, not in creating a perfect generic algorithm. Engineering teams will often spend months trying to create the ideal RAG framework when, in reality, dedicating a tenth of the time to manual knowledge base curation would get them better, more easily maintainable results—much faster. Alas, while some engineers may find this approach uninspiring, from a resource management and time-to-market perspective it’s often the smarter choice.

Disclaimer: It Does Come At A Cost Some of the practices we’ve mentioned above do make RAG more accurate, but they can also increase latency and price. That said, new hardware such as Cerebras, and newer LLMs such as Gemini 2.0 Flash, are now making it practical.

Here’s a framework for assessing whether an approach like Parlant QnA’s is right for your use case.​

1. Risk assessment: How severe are the consequences of an incorrect response?
2. Volume considerations: How many queries will your system handle daily? (As said above, recent hardware and newer LLMs are now making this approach more practical anyway)
3. Expertise availability: Do you have domain experts ready to curate and edit the QnAs properly?
4. Time constraints: How quickly do you need to deploy?

Is 70% 'Good Enough'?​

Perhaps the most frustrating aspect of the conversation about RAG frameworks is that the industry's expectation that 70% accuracy is 'good enough' is flat-out absurd.

I've heard many say, "70% is great because human representatives get even less than that." While that could well be true for some use cases (note that the industry average CSAT and FCR is around 80%), there are two important points to consider for GenAI agents.

First of all, if a human agent makes an error, various types of insurance can cover the business's losses. Second, even when a human agent is only 70% accurate, the remaining 30% of errors are typically limited in impact. It's not often that a human representative will agree to sell a customer a truck for $1, but low-accuracy AI agents are known to do so. Businesses are therefore much more apprehensive about GenAI agents getting it wrong.

For real-life use cases, we need to start talking about AI accuracy the same way we talk about most service-level agreements (SLA): in terms of the number of ninths that come after the decimal point. Instead of 67% versus 65%, we should start building systems that compete between 99.99% and 99.999%, and be willing to make the relative sacrifices in the meantime—such as in response latency—while AI hardware is catching up to match the accuracy with the desired response speed.

As I illustrated before, a large bank running 100 million conversations every month can’t afford 10 million faulty responses every month. Given that (especially in regulated industries where each faulty response could cause reputation and legal risks), such low-accuracy approaches are simply not good enough for real-world applications.

Retrieving RAG to its full potential​

If accuracy is mission-critical, enterprises can't afford to gamble on generic, plug-and-play RAG to power their customer-facing AI agents. A best-effort attempt at retrieval doesn’t cut it—especially when subtle errors can mean financial, legal, or reputational risk.

Yann LeCun, Chief AI Scientist at Meta and respected pioneer in AI research, recently stated that autoregressive LLMs (Large Language Models) are doomed because the probability of generating a sequence of tokens that represents a satisfying answer decreases exponentially by the token. While I hold LeCun in especially high regard, and resonate with many of the insights he shared in the summit, I disagree with him on this particular point.

Although he qualified his statement with "assuming independence of errors" (in each token generation) this, precisely, was the wrong turn in his analysis. Autoregressive LLMs do not actually diverge in the way he implied there, and we can demonstrate it.

What is Autoregression?​

Under the hood, an LLM is a statistical prediction model that is trained to generate a completion for a given text of any (practical) length. We can say that an LLM is a function that accepts text up to a pre-defined length (a context) and outputs a single token out of a pre-defined vocabulary. Once it has generated a new token, it feeds it back into its input context, and generates the next one, and so on and so forth, until something tells it to stop, thus generating (hopefully) coherent sentences, paragraphs, and pages of text.

For a deeper walkthrough of this process, see our recent post on autoregression.

Convergent or Divergent?​

What LeCun is saying, then, can be unpacked as follows.

1. Given the set C of all completions of length N (tokens),
2. Given the subset A ⊂ C of all "acceptable" completions within C (A = C - U, where U ⊂ C is the subset of unacceptable completions)
3. Let Ci be the completion we are now generating, token by token. Assume that Ci currently contains K<N completed tokens such that Ci is (still) an acceptable completion (Ci ∈ A)
4. Suppose some independent constant E (for error) as the probability of generating the next token such that it causes Ci to diverge and become unacceptable (Ci ⊂ U)
5. Then, generating the next token of Ci at K+1 is (1-E) likely to maintain the acceptability of Ci as a valid and correct completion
6. Likewise, generating all remaining tokens R = N - K such that Ci stays acceptable has the probability of (1-E)^R

In Simpler Terms​

If we always have, say, a 99% chance to generate a single next token such that the completion stays acceptable, then generating 100 next tokens brings our chance down to 0.99^100, or roughly 36%. If we generate 1,000 tokens, then by this logic there is a 0.0004% chance that our final completion is acceptable!

Do you see the problem here? Many of us have generated 1k completions that have been perfectly fine. Could we all have landed on the lucky side of 0.0004%, or is something else going on? Moreover, what about techniques like Chain-of-Thought (CoT) and reasoning models? Notice how they generate hundreds if not thousands of tokens before converging to a response that is often more correct.

The problem here is precisely with assuming that E is constant. It is not.

LLMs, due to their attention mechanism, have a way to bounce back even from initial completions that we would find unacceptable. This is exactly what techniques like CoT or CoV (Chain-of-Verification) do—they lead the model to generate new tokens that will actually increase the completion's likelihood to converge and ultimately be acceptable.

We know it first hand from developing the Attentive Reasoning Queries (ARQs) technique which we use in Parlant. We get the model to generate, on its own, a structured thinking process of our design, which keeps it convergent throughout the generation process.

Depending on your prompting technique and completion schema, not only do you not have to drop to 0.0004% acceptance rate; you can actually stay quite close to 100%.

Under the hood, an LLM is a statistical prediction model that is trained to generate a completion for a given text of any (practical) length. We can say that an LLM is a function that accepts text up to a pre-defined length (a context) and outputs a single token out of a pre-defined vocabulary. Once it has generated a new token, it feeds it back into its input context, and generates the next one, and so on and so forth, until something tells it to stop, thus generating (hopefully) coherent sentences, paragraphs, and pages of text.

Let's break that down carefully, step by step. An LLM:

1. Accepts an input context
2. Predicts a single token out of a vocabulary
3. Feeds that token back into the input context
4. Repeats the process again until something tells it to stop

On Accepting an Input Context​

It must be understood that LLMs, for various reasons, do not look at their input (nor output for that matter) as text per se, but as numbers.

Each of these numbers is called a token, and each token directly corresponds to some language fragment, depending on the architecture of the model. These tokens form the basic unit by which LLMs understand and predict text. The set of all supported values of these tokens is called the model's vocabulary.

In some LLMs, a token corresponds to a proper word. In theory, it could even correspond to full sentences or paragraphs. In practice, however, tokens are most commonly word-parts (i.e., not even full words). As such, while vocabulary would have been a perfect name for models in which tokens correspond to words, it's important to remember that in most LLMs today the vocabulary is a set of word-parts. Among other benefits, this allows us to keep the vocabulary relatively small (around 200k word-part tokens).

When you prompt an LLM, the prompt first undergoes tokenization so that the LLM can understand it using its own language—its vocabulary. Tokenization breaks down a text into a series of tokens, or, again, numbers, each representing a unique word-part in the vocabulary. To illustrate this concept:

tokens = tokenizer.encode("I like bananas")

for t in tokens:
  print(f"{t} = {tokenizer.decode(t)}")

# Output (for example):
#   4180 = "I"
#   5 = " "
#   918 = "lik"
#   5399 = "e"
#   5 = " "
#   882 = "ba"
#   76893 = "nana"
#   121 = "s"

On Predicting the Next Token​

The next step is to do what the model is supposed to do: predict the next token, also known as generating a completion for the input context (or at least the fundamental building block of this completion process).

We won't go into the internal prediction and attention mechanisms here. Instead, I want to focus on the very last stage of the prediction process.

When an LLM has done its best to "figure out" the meaning in its input context, it provides what's called a probability distribution over its vocabulary. This means that every single token is assigned the likelihood that it, among all others, should be chosen as the next predicted token in the completion—a great honor indeed.

The following snippet illustrates what that distribution might look like:

context_tokens = tokenizer.encode("The quick brown fox jumps over the lazy dog ")
next_token_probabilities = prompt(context_tokens)

for token, probability in next_token_probabilities:
  print(f"{token} ({tokenizer.decode(token)}) = {probability}")

# Output (for example):
#   ...
#   5 (" ") = 0.0001 (0.01%)
#   ...
#   121 ("s") = 0.002 (0.2%)
#   ...
#   4180 ("I") = 0.008 (0.8%)
#   ...
#   882 ("ba") = 0.013 (1.3%)
#   ...
#   918 ("lik") = 0.0004 (0.04%)
#   ...
#   1000 ("dog") = 0.975 (97.5%)
#   ...
#   76893 ("nana") = 0.0006 (0.06%)

Once we have this probability distribution, we need to actually decide which one to use. A naive approach would simply choose the one with the highest probability. It turns out, however, that this is a mistake. Not only can it cause models to repeat themselves robotically (a rather uninspiring application of such complex beasts); worse yet, due to biases and issues that often lurk within them, the most probable token—in their eyes—is not necessarily the ones that most of us would deem most reasonable. This is because any such statistical machine has inherent flaws and inaccuracies in its representation of our complex world.

Thus, we don't simply choose the token with the highest probability. Instead, we choose one using various sampling techniques that introduce randomization into the choice process, while respecting the assigned probabilities (kind of like rolling a loaded dice that's heavier on the sides that represent the more likely tokens).

Two things we must take note of here:

1. The token with the highest probability won't necessarily get selected at every prediction iteration—it will just be more likely to get selected
2. The token with the highest probability isn't even necessarily the most reasonable one from a Human/AI alignment perpsective

On Feeding the Predicted Token Back into the Context​

The final step of the process—before rinsing and repeating—is to the take the newly generated token and feed it back into the input context, appending it to the end. The completion process goes something like this (conceptually):

context_tokens = tokenizer.encode("The quick brown fox jumps over the lazy dog ")

while True:
  next_token_probabilities = prompt(context_tokens)
  next_token = weighted_random_choice(next_token_probabilities)
  context_tokens.append(next_token)

  if next_token == STOP_TOKEN:
    break

This iterative feed-back loop is directly related to how autoregressive models are trained. During the training process, they are essentially asked to predict or "fill out" the next token in the context. Once they improve on that, the training process moves on to the next token, and so forth. And this is the principle the inference process follows as well.

This is what autoregression is!