ACE-Step 1.5 Music Generation

Open-source music generation via tools/music_gen.py.

Cloud providers:

• acemusic (default) — Official ACE-Step cloud API with XL Turbo (4B) model + 5Hz LM thinking mode. Free API key from acemusic.ai/api-key. No GPU required.
• modal — Self-hosted ACE-Step 2B Turbo on Modal. Requires MODAL_MUSIC_GEN_ENDPOINT_URL.
• runpod — Self-hosted ACE-Step 2B Turbo on RunPod. Requires RUNPOD_ACESTEP_ENDPOINT_ID.

Setup

# acemusic (recommended — free, best quality, no GPU)
echo "ACEMUSIC_API_KEY=your_key" >> .env
# Get key at https://acemusic.ai/api-key

# Self-hosted (optional fallback)
python tools/music_gen.py --setup             # RunPod
modal deploy docker/modal-music-gen/app.py    # Modal

Quick Reference

# Basic generation (uses acemusic XL Turbo by default)
python tools/music_gen.py --prompt "Upbeat tech corporate" --duration 60 --output bg.mp3

# Generate 4 variations, pick the best
python tools/music_gen.py --prompt "Calm ambient piano" --duration 30 --variations 4 --output ambient.mp3

# Fast mode (disable thinking)
python tools/music_gen.py --no-thinking --prompt "Quick draft" --duration 30 --output draft.mp3

# With musical control
python tools/music_gen.py --prompt "Calm ambient piano" --duration 30 --bpm 72 --key "D Major" --output ambient.mp3

# Scene presets (video production)
python tools/music_gen.py --preset corporate-bg --duration 60 --output bg.mp3
python tools/music_gen.py --preset tension --duration 20 --output problem.mp3
python tools/music_gen.py --preset cta --brand digital-samba --duration 15 --output cta.mp3

# Vocals with lyrics
python tools/music_gen.py --prompt "Indie pop jingle" --lyrics "[verse]\nBuild it better\nShip it faster" --duration 30 --output jingle.mp3

# Cover / style transfer
python tools/music_gen.py --cover --reference theme.mp3 --prompt "Jazz piano version" --duration 60 --output jazz_cover.mp3

# Repaint a weak section
python tools/music_gen.py --repaint --input track.mp3 --repaint-start 15 --repaint-end 25 --prompt "Guitar solo" --output fixed.mp3

# Continue from existing audio
python tools/music_gen.py --continuation --input track.mp3 --prompt "Continue with jazz piano" --output extended.mp3

# Stem extraction
python tools/music_gen.py --extract vocals --input mixed.mp3 --output vocals.mp3

# Fall back to self-hosted
python tools/music_gen.py --cloud modal --prompt "Background music" --duration 60 --output bg.mp3

Fixing "Samey" Output

If generated music sounds repetitive or lacks variety, try these in order:

1. Use acemusic cloud (default) — the XL Turbo 4B model is significantly more capable than the 2B model on Modal/RunPod
2. Keep thinking mode on (default for acemusic) — the 5Hz LM enriches sparse prompts into detailed musical descriptions
3. Generate variations — --variations 4 generates 4 takes, pick the best
4. Use stochastic inference — --infer-method sde adds randomness (same seed gives different results)
5. Vary BPM and key across scenes — don't use the same preset for every scene
6. Write sparser prompts — "Upbeat indie rock" gives the model more creative freedom than a hyper-detailed description
7. Vary seeds — omit --seed to let each generation be unique

Creating a Song (Step by Step)

1. Instrumental background track (simplest)

python tools/music_gen.py --prompt "Upbeat indie rock, driving drums, jangly guitar" --duration 60 --bpm 120 --key "G Major" --output track.mp3

2. Song with vocals and lyrics

Write lyrics in a temp file or pass inline. Use structure tags to control song sections.

# Write lyrics to a file first (recommended for longer songs)
cat > /tmp/lyrics.txt << 'LYRICS'
[Verse 1]
Walking through the morning light
Coffee in my hand feels right
Another day to build and dream
Nothing's ever what it seems

[Chorus - anthemic]
WE KEEP MOVING FORWARD
Through the noise and doubt
We keep moving forward
That's what it's about

[Verse 2]
Screens are glowing late at night
Shipping code until it's right
The deadline's close but so are we
Almost there, just wait and see

[Chorus - bigger]
WE KEEP MOVING FORWARD
Through the noise and doubt
We keep moving forward
That's what it's about

[Outro - fade]
(Moving forward...)
LYRICS

# Generate the song
python tools/music_gen.py \
  --prompt "Upbeat indie rock anthem, male vocal, driving drums, electric guitar, studio polish" \
  --lyrics "$(cat /tmp/lyrics.txt)" \
  --duration 60 \
  --bpm 128 \
  --key "G Major" \
  --output my_song.mp3

3. Repaint a weak section

If the chorus sounds weak, regenerate just that section:

python tools/music_gen.py --repaint --input my_song.mp3 --repaint-start 20 --repaint-end 35 --prompt "Powerful anthemic chorus, big drums" --output fixed.mp3

4. Continue/extend a track

python tools/music_gen.py --continuation --input my_song.mp3 --prompt "Continue with gentle acoustic outro" --output extended.mp3

Key tips for good results

• Caption = overall style (genre, instruments, mood, production quality)
• Lyrics = temporal structure (verse/chorus flow, vocal delivery)
• UPPERCASE in lyrics = high vocal intensity
• Parentheses = background vocals: "We rise (together)"
• Keep 6-10 syllables per line for natural rhythm
• Don't describe the melody in the caption — describe the sound and feeling
• Use `--seed` to lock randomness when iterating on prompt/lyrics

Controlling vocal gender

The model doesn't reliably follow "female vocal" or "male vocal" on its own. Use both of these together:

1. In the prompt: Be explicit — "solo female singer, alto voice" or "female vocalist only, breathy intimate voice". Adding an artist reference helps (e.g., "Brandi Carlile style").
2. In the lyrics: Add [female vocal] tags before each section:

[female vocal]
[Verse 1]
Walking through the morning light...

[female vocal]
[Chorus - anthemic]
WE KEEP MOVING FORWARD...

Just saying "female vocal" in the prompt alone is often ignored. The combination of prompt + lyrics tags is what works.

Duets and vocal trading

For duets with male/female vocals trading verses, use both the prompt and per-section lyrics tags:

• Prompt: "duet, male and female vocals trading verses, warm harmonies on chorus"
• Lyrics: Tag each section with who sings it:

[Verse 1 - male vocal, storytelling]
First verse lyrics here...

[Chorus - male and female duet, harmonies]
Chorus lyrics here...

[Verse 2 - female vocal, wry]
Second verse lyrics here...

[Bridge - male vocal, spoken]
Spoken bridge...

[Bridge - female vocal, sung]
Sung response...

This reliably produces vocal trading between sections and harmonies on shared parts.

Scene Presets

Task Types

text2music (default)

Generate music from text prompt + optional lyrics.

cover

Style transfer from reference audio. Control blend with --cover-strength (0.0-1.0):

• 0.2 — Loose style inspiration (more creative freedom)
• 0.5 — Balanced style transfer
• 0.7 — Close to original structure (default)
• 1.0 — Maximum fidelity to source

extract

Stem separation — isolate individual tracks from mixed audio. Tracks: vocals, drums, bass, guitar, piano, keyboard, strings, brass, woodwinds, other

repainting (acemusic only)

Regenerate a specific time segment within existing audio while preserving the rest.

python tools/music_gen.py --repaint --input track.mp3 --repaint-start 15 --repaint-end 25 --prompt "Guitar solo" --output fixed.mp3

continuation (acemusic only)

Extend existing audio by continuing from where it ends.

python tools/music_gen.py --continuation --input track.mp3 --prompt "Continue with jazz piano" --output extended.mp3

Prompt Engineering

Caption Writing — Layer Dimensions

Write captions by layering multiple descriptive dimensions rather than single-word descriptions.

Dimensions to include:

• Genre/Style: pop, rock, jazz, electronic, lo-fi, synthwave, orchestral
• Emotion/Mood: melancholic, euphoric, dreamy, nostalgic, intimate, tense
• Instruments: acoustic guitar, synth pads, 808 drums, strings, brass, piano
• Timbre: warm, crisp, airy, punchy, lush, polished, raw
• Era: "80s synth-pop", "modern indie", "classical romantic"
• Production: lo-fi, studio-polished, live recording, cinematic
• Vocal: breathy, powerful, falsetto, raspy, spoken word (or "instrumental")

Good: "Slow melancholic piano ballad with intimate female vocal, warm strings building to powerful chorus, studio-polished production" Bad: "Sad song"

Key Principles

1. Specificity over vagueness — describe instruments, mood, production style
2. Avoid contradictions — don't request "classical strings" and "hardcore metal" simultaneously
3. Repetition reinforces priority — repeat important elements for emphasis
4. Sparse captions = more creative freedom — detailed captions constrain the model
5. Use metadata params for BPM/key — don't write "120 BPM" in the caption, use --bpm 120

Lyrics Formatting

Structure tags (use in lyrics, not caption):

[Intro]
[Verse]
[Chorus]
[Bridge]
[Outro]
[Instrumental]
[Guitar Solo]
[Build]
[Drop]
[Breakdown]

Vocal control (prefix lines or sections):

[raspy vocal]
[whispered]
[falsetto]
[powerful belting]
[harmonies]
[ad-lib]

Energy indicators:

• UPPERCASE = high intensity ("WE RISE ABOVE")
• Parentheses = background vocals ("We rise (together)")
• Keep 6-10 syllables per line within sections for natural rhythm

Video Production Integration

Music for Scene Types

Timing Workflow

1. Plan scene durations first (from voiceover script)
2. Generate music to match: --duration <scene_seconds>
3. Music duration is precise (within 0.1s of requested)
4. For background music spanning multiple scenes: generate one long track

Combining with Voiceover

Background music should be mixed at 10-20% volume in Remotion:

<Audio src={staticFile('voiceover.mp3')} volume={1} />
<Audio src={staticFile('bg-music.mp3')} volume={0.15} />

For music under narration: use instrumental presets (corporate-bg, ambient, lofi). For music-forward scenes (title, CTA): can use higher volume or vocal tracks.

Brand Consistency

Use --brand <name> to load hints from brands/<name>/brand.json. Use --cover --reference brand_theme.mp3 to create variations of a brand's sonic identity. For consistent sound across a project: fix the seed (--seed 42) and vary only duration/prompt.

Advanced Parameters

Technical Details

• acemusic cloud: XL Turbo 4B DiT + 4B LM, best quality, ~5-15s per generation
• Modal/RunPod: Standard Turbo 2B DiT, no LM, ~2-3s per generation
• Output: 48kHz MP3/WAV/FLAC
• Duration range: 10-600 seconds
• BPM range: 30-300

When NOT to use ACE-Step

• Voice cloning — use Qwen3-TTS or ElevenLabs instead
• Sound effects — use ElevenLabs SFX (tools/sfx.py)
• Speech/narration — use voiceover tools, not music gen
• Stem extraction from video — extract audio first with FFmpeg, then use --extract