PyAI Omni: All-in-one speech-to-speech voice agent model with a fused LLM brain, tool calling, and knowledge grounding
Omni is a hybrid speech-to-speech engine with a fused LLM brain that hears, thinks, and speaks natively, so there is no STT-LLM-TTS pipeline to stitch. It runs at human conversational pace (~390 ms median voice-to-voice, in-region) with barge-in and native 8 kHz telephony tuning, over a single WebSocket.
Answered by a PyAI Omni agent · may be recorded.
- Price
- $0.05/min
- Endpoint
wss://api.pyai.com/v1/omni- Scope
omni:session- Model id
pyai-omni-realtime
Watch Omni run a call, end to end
Acme Dental’s front-desk agent - every step powered by PyAI.
Hear <-> Speak repeats every turn, with barge-in - then the call becomes data.
Each step is a PyAI model. Press play to watch them hand off - or tap any stage to jump.
- Hear
pyai-hear - Omni brain
pyai-omni-realtime - Speak
pyai-speak
Transcript, summary, intent, and disposition - emitted as JSON when the call ends.
What you get
The whole agent in one model
One hybrid speech-to-speech model with a fused LLM brain replaces the STT, VAD, turn-detection, LLM, RAG, and TTS chain other platforms stitch together.
Tool calling that does real work
Enable prebuilt hosted tools by name or declare your own functions in the configure frame; the brain calls them mid-call and speaks the result.
Grounded in your knowledge
Point Omni at your own knowledge endpoint and it pulls the right facts on every turn: fast, fail-open, and customer-hosted.
Voices with emotion, even yours
Speak back in expressive stock voices, or clone a voice for free and drop the voice_id into the configure frame.
Built for the phone
Purpose-built for 8 kHz telephony at ~390 ms median voice-to-voice, in-region. Early measurement, not an SLA.
Capabilities
- Hybrid speech-to-speech
- Fused LLM brain
- Tool / function calling
- Knowledge-base grounding
- Emotion-aware voices
- Built-ins: transfer, DTMF, hold, end-call
- Voice-to-voice ~390 ms p50 (in-region) + barge-in
- Telephony-native (8 kHz)
- OpenAI-compatible REST on-ramps
~390 ms median voice-to-voice (utterance-end to first audio, in-region) with barge-in; an early measurement, not an SLA. Per-second billing includes hearing, reasoning, retrieval, tool calls, and speech.
Everything in the box
Everything Omni comes loaded with
One JSON configure frame after connect sets the whole agent; PyAI stores nothing per agent.
voice_idAny stock voice, your free clone, or a prompt-designed voice.
personaThe system prompt: role, tone, and rules for the call.
kb_endpointYour knowledge endpoint; Omni grounds every turn against it.
tools[]Custom functions the brain can call mid-call, declared inline.
greetingThe first line spoken the instant the call connects.
consent_lineA spoken consent and disclosure line for compliance flows.
Migration path
Already on Vapi, Retell, or OpenAI Realtime?
Keep your agent logic, then adapt realtime audio and control events to Omni's documented native kind-byte protocol.
import Vapi from "@vapi-ai/web";
const vapi = new Vapi(process.env.VAPI_PUBLIC_KEY);
vapi.start({
transcriber: { provider: "deepgram", model: "nova-3" },
model: { provider: "openai", model: "gpt-4o" },
voice: { provider: "11labs", voiceId: "burt" },
});// One socket: STT + reasoning + retrieval + TTS.
const ws = new WebSocket(
"wss://api.pyai.com/v1/omni?format=pcm16&rate=24000",
["pyai-key." + apiKey],
);
ws.onmessage = (e) => handleOmniFrame(e.data);Keep your agent logic, then adapt audio and control events to Omni's documented native framing. Full Vapi migration guide
Start in minutes
# Omni uses its documented native kind-byte protocol.
url = "wss://api.pyai.com/v1/omni?format=pcm16&rate=24000"
# audio up, agent audio down - one socket, whole agent.FAQ
What can Omni actually do out of the box?
Everything a phone agent needs in one model: hear the caller, reason with a fused LLM brain, call your custom tools/functions, ground answers in your knowledge base, and speak back in emotion-aware voices with natural turn-taking and barge-in. Built-in call control (transfer_to_human, send_dtmf, play_hold, collect, end_call) ships in the box, and you tune voice, persona, greeting, language, and a consent line per call via one configure frame.
Does Omni support tool / function calling?
Yes, three ways. Enable prebuilt hosted tools by name with zero setup (knowledge search, web search, weather, currency, unit conversion, math, date/time, geocoding, news). Register your own as a server tool and Omni calls your signed webhook from a network-isolated executor, which works on phone calls, since your code never has to be on the socket. Or run tools client-side in a browser app. Omni's brain fills the arguments from the conversation, calls the tool mid-call, and speaks the result, with sensible timeouts and fail-open behavior so audio never stalls. Your endpoints and secrets never touch the model. Live call control (transfer to a human, send DTMF, play hold, hang up) runs in your telephony transport today and is handled for you on managed Telephony.
How does knowledge-base grounding work?
Point Omni at your own knowledge endpoint in the configure frame. On each turn the engine fetches the relevant facts (fast and fail-open) and grounds its answer in them. The knowledge stays customer-hosted (you don't ship your data to PyAI), and retrieval is folded into the call rather than added as a separate RAG hop.
Can Omni use my own voice, with emotion?
Yes. Use any expressive stock voice, or bring your own: voice cloning enrollment and prompt-to-voice design are free. Drop the resulting voice_id into the configure frame and Omni speaks with it: emotion-aware, in the voice that fits your brand and language.
How is this different from Vapi, Retell, ElevenLabs, or Bland?
Those platforms orchestrate a pipeline (STT, VAD, turn detection, an LLM, RAG, and TTS), and you pay the latency and fragility of every hop. Omni is a single hybrid speech-to-speech model with a fused LLM brain that does all of it in one call, purpose-built for telephony.
I'm already on another provider, how hard is the switch?
REST speech calls keep their OpenAI-compatible shape. Realtime agents need a wire adapter because Omni uses a binary kind-byte protocol rather than OpenAI Realtime JSON events. Start with the native Omni protocol guide and verify audio and control framing before production.
Why does telephony / 8 kHz matter?
Phone lines carry 8 kHz narrowband audio, far less detail than studio sound. Models trained on clean wideband audio lose accuracy on real calls. Omni is built for narrowband, so it stays accurate on the lines you actually run on.
Which realtime URL should I use?
Use wss://api.pyai.com/v1/omni?session_label=<label>&format=pcm16&rate=24000. Omni uses its documented native kind-byte framing; REST transcription and speech keep their OpenAI-compatible shapes.
Build with Omni today.
An instant sandbox key lets you test without billing. Create an account when you want saved projects, and fund a live key for production traffic.