Repository avatar
Multimedia
v0.2.1
active

vidx

io.github.renjithrrkj/vidx

Local video understanding and editing for agents, in small text instead of frames.

Documentation

vidx

tests license: MIT

Let AI agents edit video without burning tokens on frames.

For an agent to "watch" a 10-minute video at 1 frame per second costs ~67,000 tokens; an hour is ~400,000. vidx watches it once, locally, and gives the agent a compact text map instead: exact shots, keyframes, motion and colour per shot, on-screen text and watermarks, a transcript, speech/music/sfx stretches, beats, and text-searchable visuals. The agent reads a ~600-token overview, searches what it needs, edits with a few lines of script or a timeline file, previews at 480p, renders only what changed, and runs automatic QA. Pixels reach the agent only when it explicitly asks to look (peek).

Runs 100% locally. No paid APIs. Works with any MCP agent (Claude Code, Claude Desktop, …).

Measured on 3 real videos (8, 10, 60 min)transcript + framesvidx
Get the gist of the video7k–17k tokens~650–930
Find a moment, read it, look at it7k–17k~1.4k–1.8k
Cut dead air, render, QA3k–8k~0.8k–1k
Overview of a 10-min vs 60-min videogrows with length507 vs 564

6–18× fewer tokens in these tests, and the gap widens with longer video. Details, method and the cases where it does not win: TOKEN_REPORT.md.

Quickstart (Claude Code)

You need ffmpeg and uv:

brew install ffmpeg uv          # Linux: apt install ffmpeg, then curl -LsSf https://astral.sh/uv/install.sh | sh

Then install the plugin inside Claude Code. It adds the server and a skill that tells Claude how to use it:

/plugin marketplace add renjithrrkj/vidx
/plugin install vidx@vidx

Or add just the server: claude mcp add vidx -- uvx vidx mcp. Other MCP clients can find vidx in the MCP Registry as io.github.renjithrrkj/vidx.

No clone needed. Then ask your agent something like "index ~/Movies/talk.mp4, cut the dead air and make a 9:16 reel of the part where I talk about pricing".

The first index downloads Whisper small (~460 MB) and CLIP ViT-B-32 (~600 MB); run vidx setup to fetch them ahead of time. After that vidx never touches the network. The shot detector, OCR and speech detector ship inside their packages. Set VIDX_MODEL_DIR to keep all weights in one folder you can copy. Indexing costs CPU time, not tokens, and stays around 1 GB of RAM.

Claude Desktop / other MCP clients: add to your MCP config (desktop apps don't inherit your shell PATH, so give full paths):

"mcpServers": {
  "vidx": {
    "command": "/full/path/to/uvx",
    "args": ["--from", "git+https://github.com/renjithrrkj/vidx", "vidx-mcp"],
    "env": { "PATH": "/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin" }
  }
}

From source (to hack on it):

git clone https://github.com/renjithrrkj/vidx && cd vidx && uv sync
claude mcp add vidx -- uv run --directory "$PWD" vidx-mcp

Tested on macOS (Apple silicon). Linux should work (CPU/CUDA); Windows is untested.

The tools

ToolWhat it returns
index_video(path, force, captions, glitch)video_id, duration, shots, language, short summary
overview(video_id, level, limit, offset)table of contents (scenes cover the whole video in ≤limit lines)
query(video_id, text, kind, limit)ranked time ranges: speech / visual (CLIP) / on-screen text / event (silence>1.5, music, sfx, beat, laugh, glitch, unclear)
get_segment(video_id, start, end, fields)words, events, shots for one range (wordtimes = per-word stamps; detail = motion curve and peak, colour, audio level, keyframes, text boxes)
peek(target, times, size, grid)image: one contact sheet, ≤4×4 labeled tiles. times=["cuts"] on a version = both sides of every cut + a colour strip of all clips
edit(video_id, dsl, parent_version)DSL or a JSON timeline; version id, segment count, output duration, warnings, timeline file path
render(version_id, preset, out_path, preview)output path, duration, size, pieces encoded vs reused. preview=true = fast 480p
check(version_id)QA report ≤15 lines, OK when clean
diff(version_a, version_b)added / removed / moved source ranges, duration delta

Edit DSL

base all              # start from the full video (default) | base none
drop 30.0-31.2        # remove a time range (also 1:02.5 style)
drop s7               # remove a shot (or s7-s9)
drop silence>1.2      # remove every silence longer than 1.2s (also: black, freeze, loud)
keep 120.0-150.0      # add a range (with base none; line order = output order)
keep s3
snap words            # move cut points out of words into the nearest gap
pad 0.15              # handles around kept segments (never into a neighbouring word)
trans xfade 0.3       # crossfade at every join (default: hard cut)
crop 9:16 center
speed 1.1
match s3 0.8          # colour-match every clip to shot s3 (or a time), strength 0-1
caption 3.0 2.5 Some text           # output time, duration, text
still photo.jpg 3.0 at 12.5 motion zoom_in fit cover   # a still as a clip; `at end` also works
overlay logo.png 3.0 2.5 pos top_right width 0.2 fade 0.3 motion slide_left   # image layer; `all` = to the end
sfx impact_01 at 3.2 gain -6        # bundled sound at output time 3.2
music drone_02 gain -18             # bundled bed under the whole edit

Typical dead-air cleanup: drop silence>1.0 / snap words / pad 0.1. edit(..., parent_version="v3") appends to v3's script, so fixes from check are one-liners.

Images

A still is a clip: transitions, captions and check treat it like footage. It has no sound, so add music under it. Motions: none zoom_in zoom_out pan_left pan_right pan_up pan_down. fit cover fills the frame, fit contain shows the whole picture with bars. at is a time in the edit without stills, so adding one doesn't move the others.

An overlay is an image on top of the picture and under captions. Transparent PNGs stay transparent, and a layer can span cuts and transitions. pos is top_left top top_right left center right bottom_left bottom bottom_right or x,y as 0-1 of the frame, width is a share of the frame, and motion is none slide_left slide_right slide_up slide_down.

Anything Pillow reads works (jpg, png, webp). Quote paths with spaces. vidx keeps an upright copy in ~/.vidx/images, so phone photos aren't sideways and an edit still renders if the original moves. In a timeline file, use {"image": "/path/photo.jpg", "for": 3, "motion": "zoom_in"} as a clip, and "overlays": [...] on any clip.

Cuts

So an edit doesn't feel like one trick repeated, vidx has 27 named cuts (vidx cuts lists them):

  • Editorial: hard, smash (with an impact hit), punch_in / punch_big (zoom that hides a jump cut), j_cut / l_cut (sound leads or trails the picture).
  • Transitions: dissolve, dissolve_slow, dip_black, dip_white, flash, whip_left/right/up, zoom, blur, wipe, smooth, cover, reveal, iris, radial, squeeze, slices, wind, pixel, gray_fade.
cuts hype        # a style pack picks each cut from what happens at that join
cuts hype 7      # same pack, different roll
cut 3 smash      # set one join
cut whip_left    # set all joins

Packs: clean, documentary, comedy, hype, trailer. They look at each join: same shot on both sides (a jump cut, so punch in), speech crossing it (bridge with a J or L cut), fast motion (whip), a big colour change (dip or flash). They never repeat an effect twice in a row, and hard cuts stay the most common.

Train your own detector

Teach vidx a moment it doesn't know, from time ranges you label:

uv run vidx teach knockdown --pos abcd1234:12-14,abcd1234:88-90,ef012345:30-33 --neg abcd1234:50-70,ef012345:0-20,ef012345:40-60

This trains a small new layer (about 500 weights) on frozen CLIP embeddings plus motion, in under a minute. It reports precision and recall on held-out ranges. Then query(video_id, "knockdown") finds it in any video. When it gets something wrong, add that range to --neg and teach again. Use several videos: a detector taught on one video may learn that video rather than the idea. It sees one frame per second plus motion, so it suits things visible in a frame or a burst of movement, not a split-second contact.

Timeline files

Every edit, DSL or not, compiles to one JSON timeline, saved as ~/.vidx/<id>/v<N>.timeline.json. Change it and pass it back with edit(id, "@/path/to/file.json"), or write one directly for per-clip control:

{"clips": [
   {"in": 12.0, "out": 18.5, "captions": [{"at": 0.5, "for": 2, "text": "Hello"}]},
   {"in": 40.0, "out": 44.0, "video": "other_video_id", "speed": 1.2,
    "grade": {"contrast": 1.05, "match": {"from": 12, "to": 18}},
    "transition": {"type": "xfade", "dur": 0.3}},
   {"in": 90.0, "out": 95.0}],
 "audio": [{"sfx": "impact_01", "at": 3.2, "gain": -6}, {"music": "drone_02", "gain": -18}]}

Rendering caches each clip and each transition in ~/.vidx/cache (5 GB, VIDX_CACHE_GB). Change one shot and only that shot re-encodes; the rest is a stream copy. Renders land in vidx_renders/ next to the source (VIDX_OUTPUT_DIR to override).

Sounds: drone_01-06, impact_01-06, riser_01-05, whoosh_01-03. All synthesized by scripts/make_sfx.py, so they ship under this repo's MIT license.

Presets: original (as is), reel_9x16 (1080×1920, −16 LUFS, ≤90 s), yt_16x9 (1920×1080, −14 LUFS).

CLI (same output as the tools)

uv run vidx index ~/Movies/talk.mp4
uv run vidx overview 5450c7a6
uv run vidx query 5450c7a6 "whiteboard diagram" --kind visual
uv run vidx segment 5450c7a6 120 150
uv run vidx peek 5450c7a6 "12.5,s3,40" --out sheet.jpg
uv run vidx edit 5450c7a6 "drop silence>1.0;snap words;pad 0.1"
uv run vidx render v1 --preset reel_9x16 --preview
uv run vidx render v1 --preset reel_9x16
uv run vidx peek v1 cuts --out cuts.jpg
uv run vidx cuts             # the cut library
uv run vidx check v1
uv run vidx diff v1 v2
uv run vidx usage            # characters returned per call (~tokens)

Optional extras

  • uv sync --extra captions → index_video(..., captions=true) adds Florence-2 one-line shot captions.
  • uv sync --extra otio → every render also writes a .otio timeline (opens in Resolve / Premiere).
  • laugh events (heuristic audience-reaction detector) run by default: query(id, "laugh").
  • index_video(..., glitch=true) scans AI-generated clips for morphing and reports the last clean frame. These are suspects to confirm with peek: expect about one false alarm per handheld phone clip.

Measured (Apple M1, 8 GB, CPU whisper + MPS CLIP)

VideoIndex timeOverview size
10:00 1080p vertical stand-up set (1 take, Malayalam)shots 71 s, visual 23 s, transcript 120 s, audio 64 s ≈ 4.6 min2.6k chars (~650 tok)
7:51 720p short film (124 shots, letterboxed, Malayalam)2.6 min (+5 min with captions)3.0k chars (~760 tok)

A whole session on the 10-min video (23 calls: overviews, queries, segments, edits, 2 peeks) returned ~9.4k chars, roughly 2.3k tokens of text. Rendering the 9:41 silence-cut at 1080×1920 takes ~3 min, a 54 s reel ~10 s.

Malayalam and other languages that Whisper-small transcribes badly are indexed as an English translation (flagged in the summary). Set VIDX_WHISPER_MODEL=large-v3 for native-script transcripts.

A false start followed by a retake can get merged by whisper into one segment, leaving several seconds of real speech with no words and one word stretched across the gap. vidx re-transcribes stretches like that on their own and folds in whatever it finds. Seconds recovered this way show in the summary, and seconds still wordless after the retry are indexed as an unclear event so query(video_id, 'unclear') finds them instead of an agent reading them as safe to cut.

Config (env vars)

VIDX_HOME (default ~/.vidx), VIDX_WHISPER_MODEL (small; try medium for Malayalam), VIDX_WHISPER_LANGUAGE (force e.g. ml), VIDX_WHISPER_TASK (auto/transcribe/translate), VIDX_DEVICE (cuda/mps/cpu), VIDX_ENCODER (libx264 or h264_videotoolbox), VIDX_SHOT_THRESHOLD, VIDX_SILENCE_DB, VIDX_IDLE_UNLOAD_S (600: the server frees its speech and vision models after this many idle seconds; 0 keeps them loaded).

Layout

src/vidx/
  analyzers/   probe, shots (TransNetV2 + curves), detail (keyframes, OCR, watermarks), visual, transcript,
               audio events (+letterbox), audiomap (speech/music/sfx, beats), laugh, glitch, captions
               — pluggable list
  store.py     SQLite (~/.vidx/vidx.db) + per-video .npy embeddings
  query.py     overview / query / get_segment / peek
  dsl.py       edit script -> timeline (pure segment math)
  timeline.py  the JSON edit format and its validation
  render.py    timeline -> cached pieces -> ffmpeg
  color.py     Reinhard colour match as a 3D LUT
  cuts.py      the cut library and style packs
  detect.py    trainable detectors on frozen CLIP
  assets/sfx/  20 bundled sounds
  checks.py    QA      diff.py  version compare
  api.py       the 9 operations (+ usage log)   cli.py / mcp_server.py thin wrappers
  otio_export.py  .otio next to renders

Adding an analyzer: write a class with name, optional, and run(video, ctx) -> AnalyzerResult that writes rows to shots/words/events (or files in ~/.vidx/<id>/), then append it in analyzers/__init__.py:registry().

Tests

uv run pytest -m "not slow"   # DSL, segment math, snap/pad, diff, checks (no video needed)
uv run pytest                 # + end-to-end on a generated testsrc clip

See DECISIONS.md for the choices made along the way.

License

MIT, see LICENSE.

PYPI
vidx
Install Command
pip install vidx
Runtime: uvx