LLM plugin to turn a video into individual frames

Install this plugin in the same environment as LLM.

llm install llm-video-frames

Requires ffmpeg installed and available on the system PATH. On macOS, you can install it using Homebrew:

brew install ffmpeg

The llm-video-frames plugin provides a fragment loader that extracts individual frames from a video file using ffmpeg.

You can use the video-frames: fragment prefix to turn a video into a series of image attachments.

video-frames:<path>?fps=N&timestamps=1

• <path>: Path to the video file accessible to the environment where LLM runs.
• fps=N: (optional) Number of frames per second to extract. Defaults to 1 if omitted.
• timestamps=1: (optional) If set to 1, overlays the filename and timestamp on each extracted frame in the bottom-right corner.

Extract 1 frame per second (default) from video.mp4:

llm -f video-frames:video.mp4 'describe the key scenes in this video'

Extract 5 frames per second:

llm -f 'video-frames:video.mp4?fps=5' 'summarize the video'

Extract 2 frames per second with filename and timestamps overlayed on frames:

llm -f 'video-frames:video.mp4?fps=2&timestamps=1' 'list notable events with timestamps'

To set up this plugin locally, first checkout the code. Then create a new virtual environment:

cd llm-video-frames
python -m venv venv
source venv/bin/activate

Now install the dependencies and test dependencies:

python -m pip install -e '.[test]'

To run the tests:

python -m pytest