Typing on your phone is slow, while sending a voice note is the most natural thing in the world. If your personal agent only understands text, you’re using half of what it can do. With an agent that listens, you talk while you walk or drive, and by the time you finish, a task is already underway. This article explains how it works under the hood, what actually goes wrong, and the rules we use to keep it from making mistakes.
What happens between the audio and the task
A language model works with text, not sound. Turning a voice note into a task takes four steps, always in this order:
- The audio arrives. In our case, through Telegram: you send a voice note just like you would to any contact.
- The file is downloaded. Telegram delivers it as .oga, a compressed audio container.
- It gets transcribed. A speech recognition model (we use Whisper, through Groq) turns the audio into text.
- The agent treats that text as if you had typed it. From there it’s just another instruction: the agent carries it out, asks a question, or replies.
There’s nothing magical about the fourth step, and that’s the best part of the design: the agent doesn’t need to know anything special about voice. All the work is in making sure the text arrives complete and correct.
What really changes day to day
To give you an idea of how much this gets used: on November 9, 2026, we counted the audio files in our agent’s inbox and found 315. It’s not a demo feature; it’s how we give instructions most of the time.
What we gain from it is concrete:
- More context in less time. Explaining a request out loud usually includes the nuances that get trimmed out when you type because of laziness.
- You can give an instruction at any moment. In the car, in a queue, between two meetings.
- A single audio can carry several tasks. Research something, write a text, and notify someone, all in a thirty-second voice note.
When the audio isn’t understood: three real failures
What follows comes from things that actually happened to us, not from a manual. Automatic transcription is very good, but it isn’t infallible, and an agent that acts on a bad transcript does the wrong thing with complete confidence.
1. Forcing the language wrecks the others
It’s common to set transcription to Spanish “just to be safe.” If the speaker uses Spanish, Portuguese and English, that fixed hint is a trap: one audio in Portuguese came out transcribed as “y y y y.” The fix was to remove the hint and let the model detect the language on its own. Since then, audio in all three languages comes out right.
2. The transcript that comes out too short
A 28-second audio of about 585 KB returned a single word with the fast model: “Thanks.” With the large model, the full request came out: research, write, publish and send two emails. If the agent had replied to “Thanks.”, all of that would have been lost without anyone noticing.
That gives us a simple rule: if the length of the audio doesn’t match how short the transcript is, the speaker didn’t say little, the transcription failed. Retry with the large model before replying with anything.
3. Proper names and numbers
This is where every voice system makes the most mistakes: surnames, company names, figures, addresses. A “fifteen thousand” that was really “fifty thousand” completely changes what needs to be done. That’s why the agent confirms before acting whenever a doubtful name or figure changes the outcome of the task.
The rules we use
- Acknowledge receipt instantly. The first thing the agent does when an audio arrives is reply that it received it. That way you know the channel is alive before the transcription finishes.
- Never pretend it heard. If the transcription fails, it says so and asks for the message in text. An invented reply to an audio it didn’t understand is worse than a warning.
- Treat the transcript as written text, but with judgment. It’s carried out as is, unless a possible error changes what’s about to be done.
- Check that the audio is being handled. We have a watchdog that raises an alert if an audio, photo or document arrives and ten minutes go by without the agent processing it.
An example: from audio to closed task
Imagine you walk out of a meeting and send this voice note: “Note that the supplier said they’d send the quote on Friday, draft a follow-up email for me for Monday, and look for a cheaper option.” That’s three things in a single sentence.
The agent immediately replies that it received the audio. Then it transcribes, separates the three instructions, logs the first one in its notebook, leaves the email draft ready for you to review, and starts looking for alternatives. When it’s done, it sends you the result in a new message so that you get the notification. You didn’t open any app, didn’t type anything, and didn’t have to remember anything on Monday.
What matters is that the audio doesn’t stay behind as a lost file in a chat: it becomes work that gets recorded.
What you need to set it up
Less than it seems:
- A channel for sending audio (Telegram or WhatsApp both work).
- A transcription service. There are cloud options, and also models that run on your own server.
- A rule in the agent’s instructions: what to do when an audio arrives and what to do when the transcription goes wrong.
The technical part comes together in an afternoon. What takes longer is fine-tuning the rules above through real use, because the failures don’t show up until they happen.
What voice doesn’t solve
- It doesn’t replace text for everything. An exact figure, an address or a link is better sent in writing.
- A long, rambling audio produces a messy task. Talking fast doesn’t remove the need to think about what you’re asking for.
- There’s noise. A busy street or a crowded restaurant lowers transcription quality, and the agent can only work with what reaches it.
How to try it
Start with small tasks that are easy to check: “Note that I’m calling this supplier on Thursday,” “find three options for this and send them to me.” Once you see that the transcription gets your voice and accent right, move up to longer requests. And review the first few transcripts: it’s the way to catch the names it struggles with early on.
If you’d like an agent you can talk to instead of type to, get in touch and we’ll work out together how to set it up for the way you work.
The figure in this article (315 audio files) is a count of our own agent’s inbox as of November 9, 2026. We don’t give figures for time saved because we haven’t measured them rigorously.
Versión en español: Hablarle a tu agente por voz: audios que se convierten en tareas
