# Speech and calling with PyAI

Use PyAI to add hearing, speech, a realtime voice agent, or managed calling to an
AI application. Start with the outcome below, then read the live contract.
This guide works for coding agents and MCP hosts, including Cursor and compatible
ChatGPT or Grok integrations. Host support and installation differ; a listing in
this guide does not mean PyAI is preinstalled or endorsed by a host.

| Job | Product | Starting point |
| --- | --- | --- |
| Transcribe a recording or audio stream | Hear | `pyai hear recording.wav --json`; MCP `transcribe_audio` |
| Speak text or use a stock/cloned voice | Speak | `pyai speak "Hello" -o hello.wav --json`; MCP `synthesize_speech` |
| Run a full duplex voice conversation | Omni | `pyai.omni.connect({ rate, configure })` in the official SDK |
| Detect answering machines, IVR or screening | AMD | MCP `get_amd_call`; use the AMD SDK for live detection |
| Review call compliance with evidence | Trace | MCP `list_trace_findings`; CLI `pyai trace findings --json` |
| Retrieve a call summary and action items | Recap | MCP `get_recap`; CLI `pyai recap get CALL_ID --json` |
| Bind a hosted number or place an outbound call | Telephony | `pyai recipes calling --json`; MCP `discover_tools` with `calling` |
| Save a reusable receptionist or campaign profile | Agents (Live Beta) | MCP `create_agent`; `pyai agents create --data @agent.json --json` |
| Render expressive, directed narration | Cast | MCP `get_cast_capabilities` then `render_cast` |
| Translate an existing recording | Dub | `pyai dub --help`; MCP `get_dub_capabilities` |

Use Hear and Speak independently when your application owns the conversation
framework. Use Omni when PyAI should run the hearing, reasoning, tools and speech
in one realtime session. Creating an Agent profile does not start a call.

## Discover, authenticate, execute

1. MCP: connect to `https://api.pyai.com/mcp`, then call `get_started`, `whoami`,
   and `discover_tools`. Use the connected server's `tools/list` as authoritative.
   Local MCP can reuse `pyai login` or an authorized `PYAI_API_KEY` environment variable.
2. CLI: install the official `@pyai/sdk`, inspect `pyai --version`, then run
   `pyai schema --json` and `pyai schema --openapi --json`. Pin the package version
   in CI. A new CLI feature requires an SDK release; website docs alone do not
   update an installed executable.
3. Verify the workspace and scopes using `whoami`. Read voices and capabilities
   before selecting IDs. A sandbox key has bounded scopes and daily limits; it
   does not authorize buying numbers or dialing real destinations.
4. Preserve returned resource IDs. For writes with an `Idempotency-Key`, reuse
   the same key and identical body for a retry of that operation. Never redial
   or resubmit a job after an uncertain timeout without inspecting its state.

MCP `openapi_get` returns an operation index and a `sha256` digest. Pass
`path` and `method` to retrieve one operation with its referenced schemas.
Pass the digest back as `sha256` to detect a changed deployed contract. Request
`include_document=true` only when the complete document is needed.

## Omni: use the official SDK

<!-- omni-starter:start -->
Requires SDK 0.7.1 for `--template omni`. The runnable starter uses ESM,
Node 20.19+ with an explicit `ws` transport, or Node 22+. MCP 0.5.0
requires Node 22+. The SDK's REST client continues to support Node 18+.

```sh
npm install @pyai/sdk@0.7.1
npx pyai init voice-demo --template omni
cd voice-demo
npm install
```

Initialization is offline; installing dependencies and running `main.mjs` are
separate steps. For an existing checkout of this example, run `npm install`
in its directory instead.

Inject `PYAI_API_KEY` through your environment, then supply a 24 kHz PCM16 mono
WAV (at most 20 seconds) saying “Please look up the office opening time.”

```sh
node main.mjs caller.wav
# Optional interruption clip, sent while reply audio is queued:
node main.mjs caller.wav interruption.wav
```

The generated project contains the complete runnable source, including its
Node 20 WebSocket import, WAV reader, one paced input stream, read-only tool,
and bounded capture. It waits for configuration and greeting playback to drain.
It sends caller PCM or silence in each slot, never overlapping silence timers.

Running it consumes Omni and Hear usage under the injected key. It saves a
private report and WAV files. The report separates received audio from an
answer recovered by Hear from captured bytes; synthesis text alone cannot pass.
The playback sink is simulated. Interruption clears that queue; physical speaker
playback remains a separate test. Capture uses two seconds of quiet after the
queue drains, because the protocol has no reply-end marker.

This section and the CLI project are generated from the same tested example.
Release readiness: https://pyai.com/agents/bot-release.json
<!-- omni-starter:end -->

For a client-executed lookup, declare `side_effect: "read"` in its tool definition:

```js
const officeHoursTool = {
  name: "lookup_office_hours",
  description: "Read the office opening time.",
  side_effect: "read",
  parameters: { type: "object", properties: {} },
};
// Include officeHoursTool in configure.tools. In onToolCall, return the
// actual lookup result using session.toolResult(frame.call_id, { result }).
```

An omitted `side_effect` is treated as an action. Action results need a positive
completion acknowledgement, for example `{ ok: true, receipt_id: actualReceiptId }`,
returned inside `toolResult`'s `result`. Send that only after the operation has
completed; a queued request or transport acknowledgement is insufficient.
On failure, return `{ error: "Operation failed" }` instead of claiming success.

For a browser, mint an origin-locked ephemeral token on your server with
`pyai.omni.createSession`, then pass that token to `omni.connect`. A hosted widget
uses its documented public widget token. Never put a live API key in browser
code, prompts, logs, or URL query strings.

Wire-reference JSON: https://api.pyai.com/omni-frames.json
Website mirror: https://pyai.com/omni-frames.json
Full protocol: https://docs.pyai.com/realtime/omni-protocol
Existing Twilio Media Streams integration: use `@pyai/twilio` and the official
[complete server example](https://github.com/atomsai/pyai-platform-backend/tree/main/examples/twilio-omni-voice-agent).

## Managed inbound and outbound calling

Use existing owned numbers before buying a new one. Requires `telephony:manage`;
post-call audio/transcripts/summaries additionally require `omni:read`.

```bash
pyai whoami --json
pyai agents list --json
pyai numbers list --json
pyai numbers search --country US --limit 5 --json
pyai schema agents create --json
pyai agents create --data @agent.json --json
pyai numbers bind NUMBER_ID --data @binding.json --dry-run --json
pyai numbers bind NUMBER_ID --data @binding.json --json
```

`agent.json` uses the live Agent schema, including `name` and
`persona_system_prompt`. `binding.json` is `{"agent_id":"RETURNED_AGENT_ID"}`.
Replace all placeholder IDs with returned values. Binding changes where inbound
calls go; it does not dial. Call the owned number through your authorized test
setup, then retrieve inbound sessions with `pyai omni calls --session-label AGENT_ID --json`.

If a number must be purchased, inspect the provisioning requirements and current
price, complete any requested account/regulatory steps, then run:

```bash
pyai numbers buy --data @number.json --idempotency-key PURCHASE_KEY --dry-run --json
pyai numbers buy --data @number.json --idempotency-key PURCHASE_KEY --confirm --json
```

`number.json` must match the live `POST /v1/telephony/numbers` request schema.
The confirmation flag represents user authorization for that purchase.
Setup: https://console.pyai.com/telephony

For an explicitly authorized real outbound call:

```bash
pyai calls create --data @call.json --idempotency-key CALL_REQUEST_KEY --dry-run --json
pyai calls create --data @call.json --idempotency-key CALL_REQUEST_KEY --confirm --json
pyai calls wait RETURNED_CALL_ID --wait-timeout 120 --poll-interval 3 --json
pyai calls get RETURNED_CALL_ID --json
```

`call.json` requires `from_number_id`, `to` (E.164), `agent_id`, and `use_case`.
Choose the use case from the live enum. Outbound availability, credit, identity
and policy checks still apply. This command dials a real destination and can
incur charges. There is no simulated `calls test` command or invented test flag.

Call statuses include `queued`, `dispatching`, `ringing`, `in_progress`,
`completed`, `failed`, `canceled`, and `dispatch_unknown`. An unknown dispatch
outcome stops automated waiting for inspection; it must never trigger redial.
A timeout ends the local wait, not the phone call.

When `artifacts.status` is `ready`, use `artifacts.omni_call_id` to fetch the
session. Recording availability is separate from call completion:

```bash
pyai omni get OMNI_CALL_ID --json
pyai omni summary OMNI_CALL_ID --json
pyai omni transcript OMNI_CALL_ID --json
pyai omni recording OMNI_CALL_ID --out call.wav --json
```

Recording is off by default and may be unavailable. Preserve the returned content
type when storing audio. Recap is a separate configured product; do not invent a
Recap or recording URL from a telephony request ID.

MCP equivalents: `telephony_list_numbers`, `telephony_search_numbers`,
`create_agent`, `telephony_bind_number`, `telephony_buy_number`,
`telephony_create_call`, `calls_get`, `omni_get_call`, `omni_get_summary`,
`omni_get_transcript`, and `omni_get_recording` (`response: "url"`). Generated
request fields are under `body`; `idempotency_key` and `confirm` are top-level.
Recording resource URLs require REST authentication; they are not public links.

## Predictable outcomes

MCP structured results add `ok`, `code`, and `docs_url` while retaining existing
product fields and audio blocks. `data` holds the existing catalog data or the
product payload. Inspect the operation's product fields for job/call state;
`ok` means the tool completed successfully, not that an asynchronous job finished.

- `needs_human`: purchase or dialing approval is missing; inspect `next_action`
  and `console_url`. No purchase or call was submitted.
- `authentication_required`, 401, or `insufficient_scope`: repair the connection.
- 402: resolve billing in the authorized workspace. Do not retry unchanged.
- 429: honor `error.retry_after` with a bounded wait.
- `contract_changed`: re-read the operation before constructing a request.
- Uncertain write timeout: retain IDs and inspect existing state before retrying.

CLI exit codes: 0 success, 1 API/job failure, 2 input/configuration or missing
confirmation, 3 authentication/permission, 4 network/timeout, 130 interrupted.
With `--json`, success goes to stdout and errors to stderr. Poll with a deadline;
never treat all products as having the same status enum.

Machine status: https://status.pyai.com/status.json
Live REST contract: https://api.pyai.com/openapi.json
MCP guide: https://pyai.com/mcp-agent-guide.md
CLI guide: https://pyai.com/cli-agent-guide.md