Skip to content

AI music and sound design for video: score, effects and the mix

Write music briefs that follow the story instead of looping under it, check what the model actually made, and mix music, ambience and effects around the voice, with eight real scores and the briefs behind them.

Updated

Most AI music sounds like stock music because it is asked for like stock music: “upbeat corporate track, 60 seconds”. Music for a video works when it follows what happens on screen. It enters when something changes, makes room when someone speaks, and ends when the story does. Below are eight scores Bald Rhino’s agent composed with ElevenLabs Music for real productions, each over a scene from its own production. The timeline is the track’s measured loudness, and each tick marks where its brief asked for a change. Press play and the brief’s words appear as the music reaches them.

Historical drama · 3-minute score

Bronze Age epic

Asked for at 0:00

Opens with near silence and a very low sustained drone, a single deep frame drum heartbeat entering slowly.

0:00 / 3:00

The brief

Instrumental cinematic score for an ancient Chinese Bronze Age epic drama, 180 seconds, no vocals. Opens with near silence and a very low sustained drone, a single deep frame drum heartbeat entering slowly. A lone bowed string melody (erhu-like, sorrowful, sparse) enters around 20 seconds over the drone. Dry wooden bone clappers punctuate sparingly. Around 70 seconds the music thins to a hush for an intimate dialogue; keep it very quiet and sustained through 70 to 150 seconds with a slow tension rise and low bronze bell tones. At about 150 seconds the frame drum quickens and the strings climb to a sharp urgent build, then a sudden complete stop at 170 seconds followed by a long drone fade. Austere, restrained, dusty, never heroic-pop, no electronic synths.

1. Watch the scene before you write a note

Play the cut with its existing sound first. Find the moments that matter: an entrance, a discovery, a hesitation, an action, a reversal, a release. For each one, decide whether sound should begin, grow, pull back or stop there. Music can reveal something, hold a space, build expectation, contradict the picture or leave room for a reaction. It does not have to run all the time, and it does not have to mark every cut.

Mark each cue against the actual line, gesture or cut that motivates it, in the edit you have. The second a script estimated is often wrong after a retake or a trim. One continuous scene may need one cue that runs across several generated clips: the place where one video generation ended is not a musical boundary. And a cold open with no music, only the sound of the place or one unaccompanied line, is a choice too.

2. Write the brief as a change over time

A brief that names a genre gets a loop. A brief that describes what happens gets a score. Say what the music is for, its texture and instruments, its rhythmic feel, how it develops and how it ends. Give a tempo or key when another cue has to join it. Describe the arrangement in order, say what must stay out, and leave the model room to compose the details that don’t matter to the story.

A close-up of a man's eyes, firelight reflected in them, in a dark Bronze Age hall.
A score written as a story
The brief

Instrumental cinematic score for an ancient Chinese Bronze Age epic drama, 180 seconds, no vocals. Opens with near silence and a very low sustained drone, a single deep frame drum heartbeat entering slowly. A lone bowed string melody (erhu-like, sorrowful, sparse) enters around 20 seconds over the drone. Dry wooden bone clappers punctuate sparingly. Around 70 seconds the music thins to a hush for an intimate dialogue; keep it very quiet and sustained through 70 to 150 seconds with a slow tension rise and low bronze bell tones. At about 150 seconds the frame drum quickens and the strings climb to a sharp urgent build, then a sudden complete stop at 170 seconds followed by a long drone fade. Austere, restrained, dusty, never heroic-pop, no electronic synths.

  • Opens with near silence and a very low sustained drone

    The score starts from almost nothing, so the drama has somewhere to go.

  • Around 70 seconds the music thins to a hush for an intimate dialogue

    The music makes room exactly where the words matter.

  • a sudden complete stop at 170 seconds followed by a long drone fade

    The ending is a dramatic choice, so the brief writes it as one: a cut to almost nothing.

  • never heroic-pop, no electronic synths

    What must stay out is named too. A synth would break the period in one bar.

When a track needs clearly separate stages, each with its own length and arrangement, Bald Rhino writes a composition plan instead of one paragraph (ElevenLabs’ composition plan guide): ordered sections, each with its own duration, lyrics and styles to include and to avoid, the first section’s styles setting the tone for the whole track. For an instrumental track, keep words that could be sung out of the plan and put “vocals” among each section’s styles to avoid.

3. Build emotion with change and contrast

Emotion in a score comes from what changes, not from a mood word repeated. These are starting points to shape for the story, not presets:

What the audience should feel A sound choice that helps What to protect
Curiosity turning into unease Start sparse and light; let a low texture or an unsettled harmony creep in as the discovery changes meaning. Don’t announce the danger before the audience is meant to suspect it.
Rising tension Tighten the repetition, add rhythmic activity or texture, delay the resolution. A quiet, persistent sound builds pressure too. Keep the action and the words readable. Constant full intensity leaves nowhere to go.
Joy or celebration A buoyant rhythm, an open texture, a theme that returns; let the energy answer how the characters move. A small private joy may want restraint, not a fanfare.
Surprise Set up an expectation, then interrupt or withdraw it; decide whether the reveal needs a hit, a new texture or only its own sound. A riser can give the surprise away; an oversized hit turns a discovery into a scare.
Comedy Let a confident theme collapse, a failed attempt lose momentum, a familiar motif come back changed. Give the audience time to get the joke. Don’t sting every line the same way.
Intimacy, sadness, relief Thin it out, expose one small sound, simplify a motif the audience already knows. Music must not drown a restrained performance or force one reading of an open moment.

The moonlit meeting above does this in miniature: a magical shimmer for the moment something happens, a sparse melody that leaves room for the two of them to speak, a second instrument answering the first, and a swell only at the end.

4. Check what came back

The seconds you write in a brief are a request, not a result. The timeline in each track above is its measured loudness. In the Bronze Age score the music thins at 1:10 and the build cuts off at 2:50, exactly where the brief asked. The brand film asked for “a breath” at 0:50, with the drums dropping out, and its level does not dip there; neither does its high end, where drums live. Measurements cannot tell you whether a change is musical, so listen: here, does the breath happen or not?

When a change must land on a frame, check it in the actual audio, then fit the edit to the music or the music to the edit. If the change is missing, regenerate that section or plan the track in sections. Shorten or join cues along their phrasing: find matching beats and harmony and listen across the join, because a crossfade at an arbitrary point can double an attack or clash two chords. Leave enough music after the planned end for a natural decay, unless an abrupt stop is the point.

5. Keep one musical identity

Choose a motif, a palette, a rhythmic character and an acoustic space for the whole piece, and carry them into each cue. A shared tempo and key help a join, but check the actual beat and harmony: matching numbers do not guarantee that two generations connect. When an idea returns, change its orchestration, register or density as the story changes; one exposed instrument after a busy passage makes the aftermath readable.

For a real reprise, reuse the passage you already have. Describing the same instruments to a new generation will not reproduce its melody. And when your own score carries a scene, ask the video model for dialogue, ambience and effects without music of its own, so two scores don’t compete.

6. Ambience is the place; effects are events

Treat ambience as the continuing sound of the location. Keep it running under cuts that stay in the same place, and change it when distance, enclosure, weather or the situation changes: a riverbank with birds, the same bank gone quiet, and its aftermath are related but different. A bird call that stops can change expectation before anything enters the frame. For a looping bed, play several repeats and listen for clicks, a recurring event or a fade at the join.

Give each foreground effect a source, an action and a material, then its distance, intensity and tail:

A single soft paper page turn, close and dry, in a quiet room;
one turn only, no other sounds.

Generate separate events when they need their own timing or level. Place the audible moment, the impact or the scrape, on the action, not the start of the file, which may begin with silence. Match the perspective: a sound behind a door or across a room should not sound as present as one next to the camera. And not every word, pan or transition needs a whoosh.

7. Mix for what matters in each moment

Keep speech, ambience, effects and music on separate tracks while you edit. Then balance around whatever carries each passage: bring the voice forward when its meaning matters, and let music or an effect take over during an action with no words. There is no universal voice-to-music ratio. When something gets lost, check its level, its timing and what competes with it before you raise everything else; thinning the conflicting layer often works better.

Lower the music under whole phrases, with a natural dip and recovery. Fast, deep dips around every word make the music pump. Keep the room tone where removing a sound would leave a hole, and keep an intended silence silent. Set the final loudness for where the video will play only after the balance works: mastering cannot fix a late cue.

8. Listen to the whole thing

Watch the full passage with picture, then isolate the layer you suspect. Check entrances and exits, phrase endings, ambience across cuts, effects in sync, every word complete, nothing masked or distorted, tails not cut off. Compare versions at the same playback level so the louder one doesn’t win by default. If the audience may watch on a phone, listen on a small speaker and in mono: an important moment should not rest only on a low rumble.