AI & Voice
Jun – Jul 2026
At Goodwish
Custom Robot Voice
A real-time voice stack — browser and robot clients, streaming PCM audio, and an LLM that can actually press buttons.
Role
Full-stack engineer across backend, admin portal and clients
Stack
- NestJS
- Next.js
- React
- TypeScript
- Flutter
- Prisma
- PostgreSQL
- Redis
- Docker
- Tailwind CSS
- Vitest
Evidence
- Commits
- 74
- Client platforms
- 3
- Evidence range
- Jun – Jul 2026
The problem
What this had to solve.
A physical robot needed to hold a spoken conversation and then act on it — answer from a knowledge base, run a command, change what is on its screen. That means audio has to move continuously in both directions with low enough latency to feel like speech rather than a walkie-talkie, across a browser client and a Flutter app on the robot, while an organisation-level admin portal manages bots, knowledge and analytics behind it.
Architecture
How it actually works.
Waiting for a complete utterance before responding leaves a pause long enough that people talk over the robot. Audio is chunked into short PCM frames and streamed both ways across one persistent WebSocket gateway, with playback beginning before generation finishes. Session lifecycle has a single owner, so a reconnect mid-utterance cannot orphan a partial buffer.
What I built
My contribution.
- 01
Built a cross-platform voice ecosystem comprising a NestJS backend, Next.js administration portal, browser voice client and Flutter robot application.
- 02
Implemented real-time voice gateways, PCM audio streaming and playback, AI tool calling, command execution, content caching, slideshows, device synchronisation and deployment events.
- 03
Added organisation-aware SaaS support, bot and knowledge management, analytics, rate limiting, sanitisation, fetch retries, session logging and public API guards.
- 04
Added backend and frontend unit tests, CI/CD and Docker configuration, and refactored duplicated voice and streaming logic into reusable services.
The hard part
Latency is a design constraint, not a metric
Request/response breaks down completely for speech: if you wait for a full utterance, transcribe it, call the model, synthesise a reply and then send it, the pause is long enough that people start talking over the robot. Audio had to be chunked and streamed through a persistent WebSocket gateway in both directions, with playback beginning before generation finishes. The subtle bug was state — two clients, a browser and the robot, plus reconnects mid-utterance, meant partial audio buffers could outlive the session that owned them. I pulled the duplicated streaming logic out of both clients into shared services with one clear owner of session lifecycle, which is also what made the reconnect behaviour testable.
Interface
What it looks like in use.
A live WebSocket voice session in the browser, with spoken replies as subtitles. The reply in this capture is staged from the bot’s seeded FAQ.