The client: a UK AI company
Our client is a UK AI company that offers conversational avatars to businesses for customer service. It wanted an avatar that could hold a real-time spoken conversation with a customer under the client's own branding, while all speech and language processing ran on its own Azure services, so conversation data was processed in UK and EU regions.
Why hosted avatar platforms weren't enough
Hosted avatar platforms do everything in one box: listening, thinking, speaking and rendering. That makes them quick to try but hard to control. Conversation data leaves your environment, you can't choose the model behind the answers, and every minute of video is billed by a third party. The client needed the intelligence on its own Azure services, the avatar answering from its own knowledge, and replies fast enough to feel like a conversation. Then it wanted to go further and stop depending on a hosted avatar platform at all.
What we built: a real-time AI video avatar, then a self-hosted one
- Azure SpeechSpeech to text
- Azure OpenAIAnswer from the client's knowledge (RAG)
- Azure SpeechText to speech
- The customer asks a question out loud
- Azure turns it into an answer and a voice
- The avatar speaks it, live in the browser
Azure environment in UK and EU regions
A dedicated Azure setup for the avatar: App Service for the API, Azure OpenAI and Azure Speech, Key Vault for every key and secret, and Blob Storage for temporary audio, all deployed in UK and EU regions.
Speech-to-answer pipeline
A .NET 8 API that streams the customer's microphone audio to Azure Speech for transcription, sends the text to Azure OpenAI, and turns the reply back into natural speech with Azure Speech text-to-speech.
Answers from the client's own knowledge (RAG)
Retrieval over the client's own knowledge base, so the avatar answers from the client's content rather than from the model's general knowledge alone.
Real-time branded avatar with D-ID
D-ID used only as the rendering layer: synthesised speech streams to a custom, branded avatar, and the video returns to the browser over WebRTC. A web app with a press-to-speak button ties it together, with replies in around 2–3 seconds.
Self-hosted avatar renderer
The next stage replaced D-ID with the client's own renderer: an open-source lip-sync model that TechTose customised to the client's requirements in Python and PyTorch, driven by the same Azure speech and OpenAI pipeline and streamed over WebRTC.
GPU deployment on RunPod
We started on the client's own NVIDIA GPU servers, then moved the renderer, containerised with Docker, to high-end RunPod GPU servers, where the avatar runs smoothly in real time.
How we built the AI avatar pipeline in two stages
We built it in two stages. First, a working end-to-end pipeline with a hosted renderer, to prove the conversation felt natural and the replies came back fast enough: speech and language on Azure, with D-ID only drawing the face. Once that worked, we swapped the renderer for a self-hosted one without rebuilding the rest, because listening, answering, speaking and rendering were separate services from the start. Keys stayed in Key Vault, speech and language processing stayed in UK and EU regions, and client details stayed confidential under the NDA.
How AI powers every step of the conversation
AI runs at every step of the conversation. Azure Speech turns the customer's voice into text and the reply back into speech, Azure OpenAI writes the answer using retrieval over the client's own knowledge, and a lip-sync video model turns the speech into a talking avatar: D-ID in the first stage, and a customised open-source model on RunPod GPUs in the second.
Where the AI video avatar project stands
Our work on the avatar was completed in 2026.
What the real-time AI video avatar delivers
- A real-time, branded AI video avatar replying in around 2–3 seconds with D-ID rendering
- Speech and language processing on the client's own Azure services in UK and EU regions
- Answers grounded in the client's own knowledge base
- A self-hosted avatar renderer on GPU servers, with no hosted avatar platform in the loop
Why the client and product aren't named
Many companies hire us to build the product their business depends on. Before we see a line of code, they ask us to keep it private, and we agree. So this page describes the work and the problem it solved, but not who it was for.
How we handle confidential work
- We sign your NDA before you share code, data or plans.
- Your code and accounts are yours. We work inside them, and our access ends when the work does.
- We never name a client, product or person publicly without written permission.
- We describe the work, not the client, as we have on this page.
Want the details? On a call we can walk you through this project, within what the NDA allows.
Questions about this AI video avatar project
Why isn't the client named?
We signed an NDA, so the client's name, its product and its code stay confidential. We protect every client's work the same way. If you'd like to know more, book a call and we'll share what the NDA allows.
How does a real-time AI video avatar work?
The customer speaks, and speech-to-text turns the audio into text. A large language model writes the answer, here Azure OpenAI with retrieval over the client's own knowledge. Text-to-speech turns the answer into a voice, and a lip-sync video model renders the avatar speaking it, streamed to the browser over WebRTC.
Can you build an AI avatar without D-ID or other hosted avatar platforms?
Yes. In the second stage of this project, TechTose replaced D-ID with a customised open-source lip-sync model running on RunPod GPU servers, while keeping the same Azure speech and language pipeline.
How fast does the avatar reply?
With D-ID rendering, replies arrived in around 2–3 seconds. The self-hosted renderer depends on the GPU behind it: on high-end RunPod servers it runs smoothly in real time.
Does TechTose build AI avatar and voice AI products?
Yes. This project covered the full voice-to-video pipeline, from speech recognition and language models to real-time avatar rendering on GPUs. Most projects start with a free Scan of your app, code or idea, which comes back as a written report in 48 hours.