Image, video and audio
The terms behind models that generate and edit images, video, speech and music.
- AI avatar
- An AI avatar is a generated video character, often based on a real person, that speaks a script or live responses with matching face, lip and head movements.
- AI image upscaling
- AI image upscaling, or super-resolution, uses a neural network to enlarge an image and fill in plausible detail, so the result looks sharper than simple resizing would make it.
- ControlNet
- ControlNet is a neural network add-on that lets a text-to-image diffusion model follow a structural guide, such as an edge map, depth map or human pose, in addition to the text prompt.
- Diffusion model
- A diffusion model is a generative model that learns to turn random noise into data, such as an image, by removing a little noise at a time over many steps.
- Guidance scale (CFG)
- The guidance scale, or CFG scale, is a setting in diffusion image generators that controls how strongly the output follows the prompt. Higher values follow the prompt more closely but reduce variety.
- Image-to-image
- Image-to-image is the task of generating a new image from an existing one plus a prompt, keeping some of the original's structure while changing style, detail or content.
- Image-to-video
- Image-to-video is the task of turning a still image into a short video clip, usually with the image as the first frame and a text prompt describing the motion.
- Inpainting and outpainting
- Inpainting is filling a masked region of an image with new generated content that fits the rest of the picture. Outpainting is the same idea applied outside the edges, extending the image beyond its frame.
- Lip sync (AI video)
- AI lip sync is the use of a model to change the mouth movements of a person in a video or image so they match a new audio track, such as a translated or rewritten line.
- Speaker diarization
- Speaker diarization is the task of splitting an audio recording by speaker, labelling each stretch of speech with who said it. It answers the question "who spoke when".
- Speech-to-speech model
- A speech-to-speech model takes spoken audio in and produces spoken audio out directly, without first converting the speech to text and back. It is used for live voice conversation and translation.
- Speech-to-text
- Speech-to-text, also called automatic speech recognition (ASR), is software that converts spoken audio into written text. It powers transcription, captions, voice commands and the listening half of voice agents.
- Text-to-3D
- Text-to-3D is the task of generating a 3D object or scene from a written prompt. Image-to-3D does the same from one or more photos. The output can be viewed from any angle.
- Text-to-image
- Text-to-image is the task of generating a new picture from a written description, called a prompt. Most current text-to-image models are diffusion models conditioned on the text.
- Text-to-music
- Text-to-music is the task of generating music audio from a written description, such as genre, mood, instruments and tempo. Some systems also write and sing lyrics.
- Text-to-speech
- Text-to-speech (TTS) is software that turns written text into spoken audio. Modern TTS uses neural networks that predict the sound of speech from the text, then render it as a waveform.
- Text-to-video
- Text-to-video is the task of generating a short video clip from a written prompt. Models must produce frames that each look right and also move consistently from one frame to the next.
- Voice cloning
- Voice cloning is the use of AI to make a text-to-speech system speak in a specific person's voice, learned from a recording of that person. Some systems need only a short sample.