SARA: Speech-to-Action AI Agent

Why We Built This

Most voice assistants handle simple commands well. "Set a timer." "What's the weather?" Single intent, single action. They break when conversations get real: multi-step instructions, context that carries across turns, actions that depend on the outcome of previous actions.

SARA (Speech-Action Reasoning Agent) is our internal exploration of what a production-grade speech-to-action agent looks like. Not a chatbot with a microphone. An agent that listens, reasons about what you want done, and executes structured actions in sequence.

What It Does

• Real-time speech recognition with streaming transcription. You don't wait for silence detection to start processing.
• Intent decomposition that breaks complex spoken instructions into discrete, ordered actions
• Context tracking across conversation turns. "Do the same thing for last month" works because the agent remembers what "the same thing" means.
• Action execution against connected systems (APIs, databases, tools) with confirmation and error handling
• Conversational clarification when instructions are ambiguous. The agent asks instead of guessing.

How It Works

The architecture has four layers:

• Speech layer: WebSocket-based streaming audio capture, real-time transcription via Deepgram, voice activity detection to handle natural pauses without premature cutoff
• Understanding layer: LLM-based intent parsing that converts transcribed speech into a structured action plan. Handles multi-step instructions, references to previous context, and implicit actions.
• Execution layer: Tool-calling framework that maps parsed intents to specific API calls or system actions. Each action returns a result that feeds into the next step.
• Response layer: Generates natural spoken confirmation of what was done, including any clarification questions if the instruction was ambiguous. Text-to-speech output via streaming for low-latency responses.

Technical Decisions

Streaming over batch: We process audio as it arrives rather than waiting for the user to finish speaking. This shaves 1-2 seconds off perceived latency, which matters when you're having a conversation. The tradeoff is handling partial transcripts and mid-sentence corrections, which adds complexity to the intent parsing layer.

Action planning over direct mapping: Instead of mapping each utterance to a single command, SARA generates an action plan (an ordered list of steps) and executes them sequentially. If step 3 fails, it can re-plan from that point rather than failing the entire instruction.

Sub-2 second response target: End-to-end latency from the user finishing a sentence to hearing the first word of the response. This required careful pipeline optimization: streaming transcription, fast LLM inference (GPT-4o with function calling), and streaming TTS output.

Tech Stack

Why This Matters for Clients

SARA isn't a client project. It's a capability demonstration. The underlying patterns (real-time speech processing, agentic task execution, multi-step planning) apply to any product where voice is the input:

• Voice-driven data queries ("Show me last quarter's sales by region, then compare to the year before")
• Hands-free workflow automation in field operations
• Accessible interfaces for users who can't interact with screens
• Customer service agents that handle multi-step requests without transfers

If you're building something that needs voice as an input channel, the architecture decisions we made here translate directly.