← Back to overview
Flagship · Voice agent · feature-complete

Vaani

A real-time, code-mixed voice + WhatsApp agent for BFSI servicing and soft-collections. Speaks Hindi / Hinglish across voice, WhatsApp and telephony; answers grounded questions over policy docs, calls validated backend tools, and hands off to a human on low confidence or sensitive requests.

Pipecat · Saaras v3 · Sarvam-105B · Bulbul v3 RAG · pgvector WhatsApp + telephony Evals-first VPC-deployable · PII-redacted
★ View on GitHub Watch a call →
Saaras STTSarvam-105B toolsBulbul TTSRAG · pgvectorWhatsApp + telephonylead-qual / KYC / bookingPII redactionevals · WER · funnelHITL Saaras STTSarvam-105B toolsBulbul TTSRAG · pgvectorWhatsApp + telephonylead-qual / KYC / bookingPII redactionevals · WER · funnelHITL
vaani · live call · +91 ••• collections line turn latency ~1.2s
Silero VAD idle · Saaras streaming ready

Scripted playback of a real turn flow: Saaras STT → RAG grounding → Sarvam-105B tool calls → Bulbul TTS, with barge-in and escalation.

Under the hood

The Pipecat pipeline

Caller ──(WebRTC / SIP)──► TRANSPORT (Daily / LiveKit / Exotel) │ audio ▼ VAD (Silero) → Saaras STT (codemix, streaming) → ROUTER │ ├─ Sarvam-30B fast smalltalk / clarify └─ Sarvam-105B tool-calling + grounded answers ├─ RAG retrieve (pgvector) → groundedness guard └─ tool calls → FastAPI mock CBS / LOS backend │ ▼ response text → Bulbul v3 TTS (streaming) → caller ▲ barge-in / interruption handling throughout low confidence ─► HUMAN-IN-THE-LOOP (warm transfer) OpenTelemetry traces + PII redaction span the whole pipeline

Latency budget < ~1.5s

  • Streaming STT partials; tokens streamed straight into the Bulbul WebSocket.
  • Fast path stays on Sarvam-30B; only escalates to 105B when tools / reasoning are needed.
  • First TTS byte ~sub-250ms.

Grounding & tools

  • Answers only from retrieved KB chunks — deflects to a human when unsure.
  • Tools: account status, EMI, outstanding balance, callback, disposition, escalate.
  • Every tool-call argument validated against a Pydantic schema before execution.

Named failure modes

  • Noisy code-mix WER, hallucinated tool args, turn-taking latency spikes, double-talk.
  • Rupee / name normalisation before TTS; PII redaction with Presidio.
  • Per-call budget + max-turn guard against runaway cost.

Evals-first harness

  • WER (jiwer), tool-call accuracy, groundedness (RAGAs / DeepEval).
  • p50 / p95 latency and cost per interaction from telemetry.
  • Lite suite wired into CI as a regression gate.
Multi-channel · Samvaad shape

One brain, every channel

Voice, WhatsApp (voice + chat in one thread), and telephony — sharing one memory, so a caller can switch mid-journey.

WhatsApp — voice + chat

  • Chat replies in text; voice notes get a spoken (TTS) reply.
  • Powered by Meta's Business Calling API shape.

Hybrid WhatsApp + telephony

  • A phone call and a WhatsApp chat on the same session_id share memory.
  • Switch channels without losing context.

Multimodal ingest

  • Audio → Setu sarvam_transcribe.
  • Image / document → Setu sarvam_parse_document.
  • Text → passthrough.

Custom memory

  • Workflow progress, user context, and execution variables across turns.
  • EMI amount, scheduled callback, etc. carried forward.
Guided flows

Structured, multi-turn tasks

A slot-filling engine drives each use-case with per-slot validation and completion tracking.

Lead qualification

  • Captures name, loan type, income, city.
  • Scores the lead (qualified ≥ 60/100).

KYC / application

  • Guided form-completion with PAN and DOB validation.
  • Re-asks on invalid input.

Booking

  • Branch, date, and time (24-hour) capture.
  • Confirmation on completion.

Post-call bus

  • On call end: summary + structured disposition + audit trail.
  • Kafka-shaped bus → Postgres-shaped records.
Evals-first

Every build is scored

The vaani-evals runner reports these on the curated suite; a lite version gates CI.

0
WER
0
tool-call acc
0
groundedness
0
intent acc
p50/p95
latency tracked
₹/turn
cost per interaction
funnel
drop-off + A/B lift
0
test coverage
Run it

Mock mode — no keys, no infra

bash
# clone + install
git clone https://github.com/manasasb2000/vaani-voice-agent.git
pip install -e '.[dev]'

# play a scripted BFSI collections call
vaani

# score the agent (WER, tool-call acc, groundedness, latency, cost)
vaani-evals

# full local stack: app + redis + postgres(pgvector) + kafka
docker compose up --build