How to Write Prompts for AI Voice Agents
Prompts written for chatbots fail on the phone. The structure, the rules that matter over audio, and how to debug an agent from its transcripts.
On this page
Almost every complaint about voice AI — it made something up, it rambled, it would not put me through to a person — is a prompt problem wearing a technology costume. The model is rarely the limitation. The instructions are.
This is the structure we use, the rules that only apply because the conversation is happening over a phone rather than in a chat window, and a method for fixing an agent from the evidence rather than by intuition.
Why phone prompts are different
A prompt that works beautifully in a chat interface will produce a poor phone agent, for four structural reasons that have nothing to do with the model.
| In chat | On a phone call | What changes in the prompt |
|---|---|---|
| The user can re-read your answer | It is gone the moment it is said | One or two sentences per reply, always. No exceptions for “important” information. |
| You can send a list of five options | Nobody retains five spoken options | Offer two. “Thursday at 2, or Friday morning?” |
| Formatting carries meaning | There is no formatting | No bullet points, no bold, no headings in anything the agent says aloud. |
| Latency is invisible | Silence is deeply uncomfortable | Prompt length is a cost. Every extra paragraph is re-read on every turn. |
Not once at the start of the call — before each turn. That is why length costs latency, why contradictions produce erratic behaviour, and why moving reference material into a knowledge base makes an agent both faster and more obedient. See what prompt length costs in latency.
The structure to copy
Paste this and replace the braces. The headings are doing real work — they keep related rules together, which is what makes the agent apply them together.
## Who you are
You are {name}, the {role} for {business} in {location}.
Today is {{date}}.
## What this call is for
{One sentence. The single outcome that makes this call a success.}
## How to speak
- Keep replies to one or two sentences.
- Ask one question at a time, then wait.
- Warm and direct. No corporate filler.
- If you are interrupted, stop and listen.
## What you know
- Hours: {…}
- Prices: {…}
- We do: {…}
- We do NOT: {…}
## What to do
1. {Usually: find out why they are calling.}
2. {The main task, with the tool named.}
3. {How the call should end.}
## Rules you never break
- Never invent a price, a date, or a policy.
- If the caller asks for a human, is upset, or you have failed twice,
transfer the call.
- Never promise anything outside {business}'s control.
## When you are unsure
Say: "I don't want to give you the wrong information — let me have
someone call you back on this. What's the best number?"Two mechanical details in there matter more than they look. Today is {{date}} is how an agent knows what “next Tuesday” means — without it, it guesses, and appointment agents guess wrong constantly. And naming the business and location in the first line stops the agent inventing a plausible-sounding branch in another city.
The two sections nobody writes
Rules you never break and When you are unsure. Almost every “the AI made something up” report comes from an agent that had neither. Write them before you write the friendly parts.
Why “never guess” has to be explicit
A language model’s job is to produce a plausible continuation. Asked a question it has no information about, producing something plausible is exactly what it will do — that is not a malfunction, it is the mechanism. The only reliable countermeasure is telling it what to do instead, in words, with a scripted alternative.
Give it somewhere to go. “Say you don’t know” is weaker than a full sentence it can actually say, because the model will otherwise improvise the phrasing and improvised phrasing drifts back towards answering.
“Never give medical advice” is a good rule. “Never discuss health” blocks a dental receptionist from taking a booking. Over-broad restrictions produce agents that refuse the thing you built them for — and that failure is much harder to spot than a hallucination, because it looks like caution.
Rules that only matter over audio
| Write this | Instead of | Because |
|---|---|---|
| “One or two sentences per reply.” | Nothing | Models are verbose by default. A paragraph over the phone is unbearable and the caller has forgotten the start by the end. |
| “Ask one question at a time, then wait.” | “Collect name, email and preferred date.” | Asked three things at once, callers answer one and you silently lose the other two. |
| “Never read out a code or long number. Offer to text it.” | Reading references aloud | Nobody has a pen. Attach a Send Message tool and text it instead. |
| “Confirm the day and time back before ending.” | Booking silently | It catches misheard details while the caller is still on the line to correct them. |
| “If they interrupt you, stop and listen.” | Nothing | Talking over callers is the single most common complaint about voice agents. |
| “Say amounts as you would speak them.” | Raw figures | “Three thousand five hundred shillings” beats an agent stumbling over a numeral. |
| “Spell unusual names phonetically: Njiru is n-JEE-roo.” | Hoping | Mispronouncing the business name in the first sentence undermines everything after it. |
Making the agent actually use its tools
Attaching a calendar or a CRM makes an action possible. Your prompt is what makes it happen. This is the single most common gap between “we connected the integration” and “the agent books appointments”.
You have access to our CRM and calendar.
At the start of every call, use search_contacts with the caller's number. If you find them, greet them by name and do not ask for details we hold. Before offering any appointment time, use check_availability. Never offer a slot you have not checked. After booking, use create_note to log what was agreed. If any lookup fails, do not guess — say you'll confirm by email and take their address.
- Name the action.
check_availabilityis unambiguous; “look at the calendar” could mean three things. - Give the trigger, not the mechanics. “When the caller wants to book” — not a step-by-step of how the tool works.
- Always state the failure path. Without it, an agent whose lookup failed will improvise, and improvising about a real customer’s real order is precisely the catastrophe you were trying to avoid.
- Do not attach tools the agent has no use for. Every extra tool is one more option weighed on every turn.
The mechanics of connecting and attaching integrations — and why those are two separate steps — are in giving an agent access.
Debugging from transcripts
Do not theorise about prompts. Open the transcript, find the exact turn that went wrong, and match it to a cause. Every Vociply call is recorded and transcribed in call history.
| Symptom in the transcript | Actual cause | The fix |
|---|---|---|
| Invented a price or policy | Fact missing, and no “say you don’t know” rule | Add the fact, and add the When you are unsure block. |
| Ignores a rule you wrote | It contradicts another rule elsewhere | Find the conflict. Two “always” rules that cannot both hold produce coin flips. |
| Long, rambling replies | No length rule | Add the sentence limit near the top, not buried at the bottom. |
| Asks for something it already has | The data arrived in a way it was not told about | “The caller’s name and number are already known — do not ask again.” |
| Never uses a tool it has | Nothing tells it when | Name the tool in the steps, with the trigger. |
| Refuses things it should do | Over-broad restriction | Narrow it to the specific prohibited thing. |
| Answers a question nobody asked | Mis-transcription, not a prompt problem at all | Check what the transcript says it heard. Try a different speech engine. |
| Drifts off-topic when the caller does | No boundary | “If the conversation moves away from {topic}, politely bring it back.” |
Rewriting five things and re-testing tells you nothing about which one worked, and it is how prompts become 1,200 words of accumulated superstition. One change, one call, one transcript. Slower per round; far faster overall.
Anti-patterns
| Anti-pattern | Why it fails |
|---|---|
| The mega-prompt | One agent handling five jobs becomes a maze of conditionals, gets slower every turn, and a fix for one job breaks another. Build separate agents — they cost nothing. |
| Politeness padding | “Always be extremely polite and courteous and helpful and friendly” consumes tokens and changes nothing. One clear sentence about tone is enough. |
| Pasting your price list into the prompt | It is re-read on every turn, slows every call, and dilutes your actual rules. That is what a knowledge base is for. |
| Scripting the whole conversation | Rigid trees are why people hate IVR. Give the agent an objective and boundaries, not a decision tree — that is the entire point of using a model. |
| Telling it to deny being an AI | Unlawful in a growing number of places, and it is the kind of thing that ends up screenshotted. Disclose it in the greeting instead. |
| No escalation path | An agent with no transfer tool will keep trying to handle a caller it cannot help. That is the interaction people remember and repeat. |
| Emoji and formatting | Some of it gets read aloud. All of it wastes tokens. |
Outbound prompts are a different job
Reusing an inbound receptionist prompt for outbound calling produces an agent that sounds confused about why it is phoning. Three things must change.
- Explain the call in the first sentence. They did not ask to be called right now. “You enquired about solar on Facebook a moment ago” makes the call make sense instantly.
- Ask permission to continue. “Have I caught you at an okay time?” costs three seconds and changes the tone of everything after it.
- Accept refusal immediately. “If they are not interested, thank them and end the call. Do not push.” Pushing after a refusal generates complaints and, in several jurisdictions, breaches.
Outbound also carries obligations that inbound does not — calling hours, consent, opt-outs, recording notice. Those belong in the prompt as hard rules, and the background is in voice AI compliance. Worked examples are in the outbound campaign playbook and the instant lead callback playbook.
Languages and accents
- Write instructions in English, set the language separately. The agent speaks the configured language regardless of what the prompt is written in.
- Quote anything that must be said verbatim in the target language — a legal disclosure, a brand line. Everything else can be translated on the fly.
- Run one agent per language on its own number rather than one agent switching. Reliability is noticeably better, and callers pick correctly on their own.
- Suspect transcription before the prompt when an agent misbehaves with a particular accent. It answered what it heard, and that is a speech engine question — see voice and conversation.
A test script that finds real problems
Seven calls. Each one targets a different failure, and between them they catch nearly everything before a customer does.
| Say this | What it tests |
|---|---|
| The most common request, straightforwardly. | The happy path works at all. |
| A question whose answer is only in the knowledge base. | Retrieval, and that it does not invent a figure. |
| Ask for something you do not offer. | It says no cleanly instead of improvising. |
| Request an impossible time — 7pm on a Sunday. | It knows your constraints and refuses. |
| “I need to speak to a person.” | The transfer fires promptly. |
| Something urgent or distressing. | It escalates instead of processing you as routine. |
| Interrupt it mid-sentence. | Turn-taking. If it talks over you, tune the timers. |
Then check the other side of every action: is the booking genuinely in the calendar, is the note on the right record? An agent saying it did something is not evidence that it did.
Ten minutes with a colleague who has not been staring at it catches the “it would absolutely say that” problems. Better still, have them listen to three recordings — people hear things in a call they cannot see in a document.
The full reference version of this, with worked prompts for a dental receptionist and a collections agent, is in the documentation. If you want to try any of it, the quickstart has an agent calling your phone in about ten minutes.
Frequently asked questions
How long should a voice agent prompt be?
Most good agents land between 200 and 600 words. Under 100 words and you are relying on the model’s assumptions about your business. Beyond roughly 1,000 words you are paying for it in response time on every single turn, and it usually means reference material has crept in that belongs in a knowledge base instead.
Why does my voice agent make up prices?
Because the real price was not in the prompt and nothing told it what to do when it does not know. Models fill gaps — that is what they are for. The fix is two lines: state the actual figures, and add an explicit instruction that says to admit uncertainty and take a callback number rather than guessing.
Why does my agent ignore one of my rules?
Almost always because another line in the prompt contradicts it. Two instructions that cannot both be true produce behaviour that looks random, because the model is effectively choosing between them each time. Search the prompt for the conflict rather than restating the rule more forcefully.
Should I write the prompt in the language the agent speaks?
Write the instructions in English and set the language separately — the agent will speak the configured language regardless. The exception is exact wording you need said verbatim, such as a legal disclosure or a brand line. Those should be written in the target language, in quotes, so they are reproduced rather than translated.
What is the difference between the greeting and the instructions?
The greeting is the exact first sentence, spoken before the caller has said anything. The instructions govern everything after it. Keep the greeting short and identify yourself in it — it is also where a recording notice or an AI disclosure belongs, because it is the only place you can guarantee the caller hears it.
Can I use markdown headings in a voice agent prompt?
Yes, and you should. Headings are never spoken aloud — they shape behaviour, not speech. Grouping related rules under headings measurably improves how consistently the agent applies them together, and it makes the prompt far easier for you to edit six months later.