Audio and speech models

Plain English

Audio AI covers three big jobs: listen (speech recognition), speak (text-to-speech), and create (music or sound effects). Your phone's voice typing, navigation prompts, and some music tools all sit on neural models trained on recordings and transcripts.

Sound is a wave over time. Models must handle rhythm, accent, background noise, and emotion—not just words on a page. That makes audio both similar to language modeling (sequences) and different because pitch and timing carry meaning.

Generative audio can clone a voice from a short sample, which helps accessibility and dubbing but also enables fraud if safeguards are weak.

How it works

Speech recognition (ASR) typically converts audio spectrograms or raw waveforms into token sequences—letters, subwords, or words—using encoder-decoder or transformer architectures. Training pairs audio clips with transcripts; data quality and accent coverage strongly affect fairness.

Text-to-speech (TTS) runs the opposite direction: text → acoustic features → waveform. Modern systems use neural vocoders and duration predictors so speech sounds natural rather than robotic. Some approaches generate spectrograms then invert them; others model audio tokens directly.

Music and ambient sound generation may use diffusion, autoregressive token models, or hybrid setups—similar ideas to /learn/diffusion-models but in the time-frequency domain.

• ASR: audio in → text out (dictation, captions, agents).
• TTS: text in → spoken audio out (assistants, accessibility).
• Voice conversion: change speaker identity while keeping words.
• Audio tokens: compress waveforms so language-model-style prediction applies.

Going deeper

Whisper-style models showed that large-scale weakly supervised training on diverse audio improves robustness. Multimodal models (/learn/multimodality) fuse audio encoders with /learn/transformers so assistants can hear and talk in one stack.

Deployment concerns mirror image generation: consent for voice cloning, watermarking, and detecting synthetic speech in call centers. For agent use cases, audio often pairs with /learn/tool-use-and-function-calling and real-time streaming infrastructure.

Common misconceptions

Speech recognition is solved for every accent and environment.

Performance gaps persist across dialects, noisy rooms, and specialized vocabulary unless systems are trained and tested on those conditions.

Text-to-speech always sounds human.

Long-form emotional delivery, singing, and subtle pauses still expose synthetic speech, though quality improves yearly.

Key facts

• Audio models operate on time-series signals, often via spectrograms or learned tokens.
• ASR and TTS are inverse problems sharing similar neural building blocks.
• Transformers and convolutional front ends both appear in production stacks.
• Voice cloning raises strong consent and fraud concerns.
• Multimodal assistants combine speech with language models for dialogue.