Build an AI Voice Agent with ElevenLabs: Phone, Web & WhatsApp (2026)

Why Voice Agents, Why Now

Two things changed. First, latency collapsed: ElevenLabs' conversational stack now responds fast enough (sub-second, with the company targeting sub-100ms on the voice side) that the awkward walkie-talkie pause that killed earlier voice bots is gone. Second, the agent layer matured — the same system prompt + knowledge base + tool-calling pattern you'd use for a chat bot now drives a voice, so there is no separate "voice app" to engineer.

The result is a real migration: support widgets, booking lines, and internal assistants that were chat-first in 2024–2025 are shipping voice-first in 2026. For solo builders, the practical upside is simpler — a voice agent feels like a product, not a form. The same assistant that reads flat in a chat window books the appointment when it speaks.

What You're Building

One agent, three deployment surfaces. The agent itself is a bundle of:

• A voice — from the ElevenLabs library, designed from a text prompt, or cloned (with consent) from a real one.
• A brain — the LLM plus your system prompt: personality, rules, escalation behavior.
• Knowledge — docs, FAQs, or site pages the agent can answer from, so it isn't bluffing.
• Tools — webhooks the agent can call mid-conversation: check availability, create a ticket, look up an order.

Deployment is a switch, not a rebuild: embed the web widget with a copy-paste snippet, attach a Twilio number for phone, or connect WhatsApp for messaging-native customers.

The Build, Step by Step

1. Create the agent. In the ElevenLabs dashboard, create a new agent and write the system prompt like a job description: who it is, what it can do, what it must never do, and when to hand off to a human. Short, concrete rules beat long prose.
2. Pick the voice deliberately. Test 3–4 candidates with your actual greeting text, not lorem ipsum. Warmth beats polish for support lines; crisp and quick wins for utility agents. (Voice design from a text description is the fastest path to something distinctive.)
3. Upload knowledge. Feed it your FAQ, pricing page, and policies. Then try to break it: ask questions the docs don't answer and confirm it says so instead of inventing an answer.
4. Wire one tool before launch. Even a single webhook — "create a lead in my CRM" — is the difference between a novelty and an employee. Start with one, add more once call logs show what users actually ask for.
5. Deploy the widget first. Web embed is zero-risk and gives you transcripts to tune the prompt against. Phone comes next: connect a Twilio number in the agent's phone settings, set the greeting, and test the escalation path end-to-end before publishing the number.
6. Read your transcripts weekly. The first two weeks of real conversations will rewrite your prompt for you — every confused exchange is a missing rule or a missing document.

E

ElevenLabs

Top Pick

Text-to-speech and voice AI platform

✓ Free tier available

Free plan: 10k chars/mo, no CC required

Take the free ElevenLabs course →

Paid from $5/mo

What It Costs (Honestly)

Prototyping is effectively free. Production has two meters: your plan tier and conversation usage. As of mid-2026, the $5/month Starter tier is enough to build and run a personal or low-traffic agent — that is the tier our free Jarvis course targets. Business lines with real call volume are priced per conversation minute, so estimate monthly minutes before you publish a phone number; a support line doing a few hundred short calls a month typically lands in the low hundreds of dollars, which is still a fraction of a human answering service.

The hidden cost to plan for is iteration time, not dollars: budget a week of transcript-reading and prompt tuning before you judge the agent's quality.

Start With the Jarvis Build

If this is your first voice agent, don't start with a business line — start with your own assistant. Our free course builds a Jarvis-style personal agent on exactly this stack: custom voice, real-time conversation, and tool integrations, one lesson at a time. Everything transfers directly to customer-facing agents afterward.