Mandarin voice agents that hear the speaker live
A Mandarin voice agent may run on a smart speaker, inside a shopping app, in a game or on a support line, and the person talking wants an answer without delay. PyAI Hear is a Mandarin speech to text API that streams simplified Chinese while the user speaks. Hear works on its own in your assistant or as the listening layer of PyAI Omni, the speech to speech agent. Teams also use live Mandarin transcription for captions and voice notes. The same engine also works as a Mandarin speech to text generator for the transcripts your team keeps after each session.
Key takeaways
- First partial: simplified Chinese text reaches your assistant near 200 ms after the user starts talking.
- Accuracy, honestly scoped: Hear posts 1.59% WER on clean audio and 4.76% on 8 kHz, accented or noisy audio in PyAI's English benchmarks at industry parity, so treat both as English numbers.
- Input: PCM16 over a WebSocket, sent in frames near 20 ms each.
- Cost: $0.001/min, whole minute billing, no charge for an empty transcript and no platform fee.
- Omni timing: inside Omni, a Mandarin turn is closed near 196 ms after silence, and the spoken reply arrives near 390 ms median.
- Migration: a whisper-1 client switches by editing one base URL, and $50 of credit comes with no credit card.
How spoken Mandarin becomes live text
Mandarin voice assistant speech to text runs one live loop, whether the microphone is in a speaker, a phone app or a web page.
The user speaks into a device, an app, a browser mic or any line you connect, and your code grabs PCM16 audio.
Frames near 20 ms long stream to Hear on a WebSocket while the sentence continues.
Close to 200 ms after speech starts, the first simplified Chinese partial lands and tightens as more audio arrives.
Your assistant parses the text and carries out the request. Inside Omni, end of turn comes near 196 ms.
What Hear handles in live Mandarin speech
Spoken Mandarin brings unspaced text, sound-alike syllables and noisy rooms. Each point applies to speakers, apps and kiosks equally.
Partials while speaking
Hear streams characters before the user finishes, so a home or shopping assistant can begin matching a device or product early.
No spaces, many homophones
Syllables such as shi map to many characters. PyAI publishes no Mandarin score, so check how your own sentences are segmented and which characters come back.
Kitchens, streets and TV sound
A television, a range hood or traffic often runs behind a request. Record sessions in those settings and compare the transcripts before release.
Times and codes as digits
With numerals handling on, alarm times, prices and order codes arrive as digits. Omni's turn detector waits while a long code is still being read.
Five custom terms
Up to 5 custom vocabulary terms keep a device nickname, a brand or a district written in the characters your system uses.
Turns with timestamps
Verbose JSON tags speaker turns and timestamps, so a session log separates the user from the assistant line by line.
Accuracy and speed
Hear's figures were measured on English audio and match the industry. A Mandarin score has not been published, so stream a batch of your own Mandarin recordings through the sandbox before launch.
Where Mandarin voice agents put Hear to work
Mandarin speech to text for voice agents fits products where speaking is quicker than typing pinyin. Six jobs, each weighted the same.
Smart speaker and home control
Turn spoken commands for lights, music and alarms into actions across connected home devices.
Shopping app voice search
Let shoppers say a product, size and budget instead of typing characters on a small keyboard.
Game characters
Let players talk to non-player characters in Mandarin and have the game react to what was said.
Warehouse voice logging
Let pickers speak bin numbers and counts into a headset so the stock system updates hands free.
Language practice app
Transcribe a learner's spoken Mandarin so the app can compare the characters against the target sentence.
Customer support phone line
Answer a Mandarin caller on a help line and route the request before an agent picks up.
Formats, limits and output
| Input | A WebSocket carrying PCM16 audio from the device, app or browser, or a recorded clip up to 25 MB. Point a whisper-1 client at the PyAI base URL and it connects. |
|---|---|
| Output | Simplified Chinese partials while the user talks, then the final line. Choose text, JSON or verbose JSON, the last adding timestamps and speaker turns. |
| Modes | Sync file call for short clips, live streaming for apps, devices and voice sessions, and background jobs for long recordings that run 92 to 247 times real time. |
| Tuning | Set Mandarin as the language, enable numerals so times, prices and order codes land as digits, and add up to 5 custom vocabulary terms like a device name or a brand. |
| Billing | $0.001/min of audio, rounded to the whole minute. An empty transcript costs nothing, and a busy engine returns a retry, not a charge. |
Add Mandarin speech to text to your own agent
Hear understands OpenAI style transcription requests, so a Mandarin assistant on whisper-1 only needs the PyAI base URL. Reference: PyAI SDKs.
SDK
Within your service, the SDK opens one socket per user session and delivers each simplified Chinese partial to your handler.
MCP
Agent tooling can reach Hear through the MCP server, for example to turn a spoken Mandarin reminder into an entry in your scheduling tool.
CLI
Feed the CLI a recorded Mandarin clip and watch characters stream into the terminal while you refine the vocabulary list.
Where Hear fits
PyAI is voice infrastructure: Hear listens, Omni handles the turn and Speak answers in Mandarin, all in one stack. Hear is not a chatbot builder, and Hear does not choose what your assistant answers. Pair your own dialog code with Hear, or let Omni run the entire voice loop. The full English-measured results are listed on PyAI benchmarks.
Mandarin speech to text for voice agents FAQ
How soon does Hear return Mandarin text to an assistant?
Hear returns the first simplified Chinese partial close to 200 ms after the user starts talking and updates the characters as speech continues. Inside Omni, the end of a turn is detected near 196 ms after silence, and the complete voice to voice reply arrives around 390 ms median.
Has PyAI published a Mandarin accuracy figure for Hear?
No. Hear's published results are 1.59% WER on clean audio and 4.76% on 8 kHz, accented or noisy audio, both measured on English at industry parity. To judge Mandarin, stream recordings from your own speakers or apps through a sandbox key and review segmentation and character choice.
Can Hear slot into a Mandarin assistant that already exists?
Yes. Hear accepts whisper-1 style requests, so an existing assistant edits only its base URL. Stream PCM16 over the WebSocket, upload files up to 25 MB, or submit background jobs, then route the simplified Chinese text into the dialog logic your product already uses.
What is the price of live Mandarin recognition?
Mandarin audio costs $0.001 per minute, billed by the whole minute, and there is no platform fee. Empty transcripts are free. A new account includes $50 of credit with no credit card required, and sandbox keys never bill while your team prototypes.
Which fits a Mandarin product better, Hear or Omni?
Hear fits when your product controls the dialog and only needs Mandarin audio converted to text. Omni fits when you want listening, turn detection and spoken output in one service, closing turns near 196 ms and replying voice to voice near 390 ms median.
Prices checked November 2026. Hear figures from pyai.com/benchmarks.