Live Song Tracker
Point it at a band's studio recordings and a recording of one of their concerts. Every 3 seconds it says which song is playing and where in the studio version the band is, so an app can show the lyric being sung right now.
$ live-song-tracker --library studio/ --live antwerp_2025.wav live 5:36 disposable_teens at 0:11 (11.4s) confidence 0.35 [setlist: disposable_teens at 0:16] live 5:39 disposable_teens at 0:14 (14.3s) confidence 0.35 [setlist: disposable_teens at 0:19] live 5:42 disposable_teens at 0:17 (17.2s) confidence 0.37 [setlist: disposable_teens at 0:22] live 5:45 disposable_teens at 0:20 (20.5s) confidence 0.41 [setlist: disposable_teens at 0:25]
The setlist column is a hand-written estimate of when each song started. Those estimates can be off by up to 20 s, which is why it runs about 5 s ahead here.
PupuM2D, and what this adds
PupuM2D (Gu et al., 2026) is a self-supervised music encoder. It is very good at turning audio into vectors. It has no idea what a song, a library or a concert is. Everything that turns those vectors into "this song, at this second" lives here.
PupuM2D gives you
- An encoder in six sizes, 5M to 1.1B parameters, trained on 30 s clips at 24 kHz.
- 25 vectors per second of audio, each covering 8 frequency bands.
- Training code, with checkpoints meant to sit inside its own repository.
- No song identity, no time alignment, no live input.
This project adds
- A standalone loader:
pip install, pinned Hugging Face revision, teacher weights only, no training repo. - Position encoding that keeps the training spacing on clips shorter than 30 s.
- Half-second "sticks", library-mean centering and diagonal alignment against every studio song.
- An evidence tracker that follows the band through the song instead of guessing each window from scratch.
- A jump margin and a confidence gate, so wrong guesses are rare and flagged.
- A command line tool, a Python API, setlist checks and synced lyrics.
How it works
Embed
Every studio song is cut into 10 s windows and run through PupuM2D-Large once, up front. The model output averages layers 12 to 24, as the paper recommends. Each window gives 250 frames, one every 40 ms. Live audio goes through the same model: the last 10 s, every 3 s.
The model was trained on 30 s inputs, and timm spreads whatever it receives over the same position range, so a 10 s input would look like a slowed-down 30 s one. The loader crops the 30 s position grid instead, so neighbouring frames sit as far apart as they did in training.
Cut into sticks
Twelve frames are averaged into one 0.48 s stick. That is short enough to place a lyric line and long enough to smooth out single drum hits. A 10 s window holds 20 sticks; the last 10 frames are dropped.
Center
Every stick of every rock song points roughly the same way, so the raw cosine similarity between any two sticks is about 0.7 and the right moment barely stands out. Subtracting the library's average stick, then normalizing again, keeps only what makes one moment different from the rest. On one show this raised how often the true position was the single best match from 44% to 57%.
Align
The 20 live sticks slide along each studio song, one stick at a time. Every start position gets a score: the average similarity along that diagonal, live stick 1 against studio stick s, live stick 2 against s + 1, and so on. That is the stripe you can see in the centered heatmap.
Track
Songs repeat themselves. A single window often fits a later chorus better than the one being played, so picking each window's best start on its own makes the position jump around. The tracker keeps an evidence score for every studio stick of every song. Each step it does three things:
- Shifts the evidence 6 sticks forward, because the concert moved on 3 s.
- Multiplies it by 0.9, so an old mistake fades within a few steps.
- Adds the new window's alignment scores.
A wrong repeat scores high once, but its shifted evidence does not line up with where the next window scores high. The true position scores high at the same shifted spot step after step.
Gate
Two rules decide what the tracker reports.
- Jump margin. Last step's pick moved on by 3 s is where we expect to be now. A pick somewhere else, in another song or more than 6 s away, has to beat the evidence at that spot by 10%.
- Confidence. The evidence divided by its total weight is the average window similarity it stands for. Correct picks sit around 0.33 to 0.45. Wrong ones usually land on magnets, studio moments that match almost anything, at 0.15 to 0.25. Below 0.3 a match comes back with
sure=Falseand an app should show ♪ instead of a lyric.
Results on five shows
Share of 3 s steps where the reported position is within 3 s of the truth. Truth is the setlist start plus an offset fitted per song. A fitted tempo came out at 1.00 for every song, so live tempo was never the problem.
| Show | Jumps over 6 s, one window | Jumps, tracker |
|---|
With the 0.3 gate, 81 to 95% of correct picks are shown and 80 to 100% of wrong ones are hidden, on all five shows.
Limits
- Lock-in takes 2 to 6 steps after a song starts, and up to 13 when the live intro differs from the studio one.
- Live outros that run past the end of the studio song park on a magnet until the next song starts.
- It recognizes the songs you give it. Covers, medleys and songs missing from the library are not found.
- PupuM2D-Large has 307M parameters. One 10 s window every 3 s runs comfortably on a laptop GPU or Apple silicon; CPU works but slowly.
- Tried and dropped: searching only near the last position (it parks on magnets for good), stretching the studio side 0.9 to 1.1 for tempo, penalizing magnet sticks, and resetting evidence when the per-song score disagrees.
Install and run
pip install git+https://github.com/vampthedge/live-song-tracker live-song-tracker --library studio/ --live show.wav --cache studio.pt
Or from Python, with audio from a file, a microphone or a stream:
from live_song_tracker import Tracker, build_library, load_model
model = load_model()
tracker = Tracker(model, build_library("studio/", model, cache="studio.pt"))
match = tracker.step(last_10_seconds) # mono, 24 kHz, once every 3 s
if match and match.sure:
print(match.song, match.second)
No audio, setlists or lyrics ship with the package. Bring your own studio recordings; synced lyrics come from lrclib.net.
Add it with an AI assistant
Paste this into Claude Code, Cursor, Codex or a similar assistant, inside your own project.
Add live song recognition to this project with the live-song-tracker library
(https://github.com/vampthedge/live-song-tracker, MIT license). Read its README first.
1. Install it as a dependency with this project's package manager, e.g.
`uv add git+https://github.com/vampthedge/live-song-tracker` or
`pip install git+https://github.com/vampthedge/live-song-tracker`. It needs Python 3.11+.
2. Import it; do not copy its source into this project. The API:
- load_model(device=None) -> model. Downloads a 3.4 GB checkpoint on first use.
- build_library(folder, model, cache="library.pt") -> library. The folder holds one studio
recording per song; the file name is the song name.
- Tracker(model, library).step(window) -> Match(song, second, confidence, sure) or None.
window is the last 10 s of mono audio at 24 kHz as a 1D torch tensor; call it every 3 s.
- load_audio(path) decodes any audio file to that format.
- load_lyrics(path) and lyric_at(lines, second) read lrclib.net synced-lyrics JSON.
3. Feed the tracker from wherever this project gets audio (a file, a microphone, a stream):
keep a 10 s buffer, call step every 3 s, and show a song position or lyric only when
match.sure is true.
4. Ask me where my studio recordings are. Do not download music from YouTube or other sites.
5. Add a test that cuts a 10 s clip from one of my studio recordings, feeds it to a Tracker
three times 3 s apart, and checks the song name. Run it.
6. Commit the change on a new git branch and show me the diff summary.