Library fork used to provide an advanced Qt Media Player based on FFmpeg library.
Note
I'm not the original author of this library, this repository is a fork from QtAvPlayer
The library allows to:
- Demuxes and decodes video/audio/subtitle frames.
- Muxes, encodes and saves the streams from different sources to one output file.
- Support FFmpeg Bitstream Filters and FFmpeg Filters including
filter_complex. - Multiple parallel filters for one input (one input frame produces multiple outputs).
- Decoding of all available streams at the same time.
- Support hardware acceleration.
- It is up to an application to decide how to process the frames:
- But there is experimental support of converting the video frames to QtMultimedia's QVideoFrame for copy-free rendering if possible.
Note: Not all Qt's renders support copy-free rendering. Also QtMultimedia does not always provide public API to render the video frames. And, of course, for best performance both decoding and rendering should be accelerated. - Audio frames could be played by
QAVAudioOutputwhich is a wrapper of QtMultimedia's QAudioSink
- But there is experimental support of converting the video frames to QtMultimedia's QVideoFrame for copy-free rendering if possible.
- Accurate seek, it starts playing the closest frame.
- Might be used for media analytics software like qctools or dvrescue.
- Implements and replaces a combination of FFmpeg and FFplay:
# FFmpeg command-line
ffmpeg -i we-miss-gst-pipeline-in-qt6mm.mkv -filter_complex "qt,nev:er,wanted;[ffmpeg];what:happened" - | ffplay -
# Using QML or Qt Widgets
./qml_video :/valbok "if:you:like[cats];remove[this-sentence]"Table of contents:
The fork is mainly due to the library being hard to build on mainline version due to multiple CMake hacks: source directory had to be set manually, some directories where includes and exported when building target.
So this fork aim to mainly fix those issues (and try some "workaround" to later propose a pull request to the original repository).
Current patches from this fork are all inside main branch, each features is represented by its own branch to ease synchronization with upstream project updates:
- Branch
refactor-library-structure: All details are explained in the pull request. This is the main refactor to ease building the library - Branch
add-log-categories:- Stop using
qDebug()and use log levels instead (info, warning, critical, etc...) - Allow to filter log according to the QtAvPlayer entity (player, demuxer, etc...), allowing more fine-grain for logs
- Stop using
Those symbols will be used:
- 💫: Untested (either lack of time, either because unable to access the device to perform tests)
- ❌: Not working
- 🌗: Partial support
- ✅: Tested and working
| / | Qt 5.12 -> 5.15.2 |
Qt 6.X |
HW accelerated feature | Comments |
|---|---|---|---|---|
| Linux (Ubuntu) | ✅ | ✅ | ✅ | - Using libva-drm (also require packages: libva-dev, libegl1-mesa-dev and libgles2-mesa-dev)- Using libva-x11 (also require packages: libva-dev, libva-x11-2 and libx11-dev)- Using libvdpau (also require package: libvdpau-dev) |
| Windows | ✅ | ✅ | ✅ | Use API d3d11 |
| MacOS | ✅ | ✅ | ✅ | Use APIs CoreVideo, IOSurface and Metal |
| iOS | 💫 | 💫 | 💫 | Use APIs CoreVideo, IOSurface and Metal |
| Android | 💫 | 💫 | 💫 | Use APIs MediaCodec and SurfaceTexture |
- QAVPlayer supports playing from an url or QIODevice or from avdevice:
player.setSource("http://clips.vorwaerts-gmbh.de/big_buck_bunny.mp4");
player.setSource("/home/lana/The Matrix Resurrections.mov");
// Dash player
player.setSource("https://bitdash-a.akamaihd.net/content/MI201109210084_1/m3u8s/f08e80da-bf1d-4e3d-8899-f0f6155f6efa.m3u8")
// Playing from qrc
QSharedPointer<QIODevice> file(new QFile(":/alarm.wav"));
file->open(QIODevice::ReadOnly);
QSharedPointer<QAVIODevice> dev(new QAVIODevice(file));
player.setSource("alarm", dev);
// Getting frames from the camera in Linux
player.setSource("/dev/video0");
// Or Windows
player.setInputFormat("dshow");
player.setSource("video=Integrated Camera");
// Or MacOS
player.setInputFormat("avfoundation");
player.setSource("default");
// Or Android
player.setInputFormat("android_camera");
player.setSource("0:0");
player.setInputOptions({{"user_agent", "QAVPlayer"}});
// Save to file
player.setOutput("output.mkv");
// Using various protocols
player.setSource("subfile,,start,0,end,0,,:/root/Downloads/why-qtmm-must-die.mkv");- Easy getting video and audio frames:
QObject::connect(&player, &QAVPlayer::videoFrame, [&](const QAVVideoFrame &frame) {
// QAVVideoFrame is comppatible with QVideoFrame
QVideoFrame videoFrame = frame;
// QAVVideoFrame can be converted to various pixel formats
auto convertedFrame = frame.convert(AV_PIX_FMT_YUV420P);
// Easy getting data from video frame
auto mapped = videoFrame.map(); // downloads data if it is in GPU
qDebug() << mapped.format << mapped.size;
// The frame might contain OpenGL or MTL textures, for copy-free rendering
qDebug() << frame.handleType() << frame.handle();
}, Qt::DirectConnection);
// Audio frames could be played using QAVAudioOutput
QAVAudioOutput audioOutput;
QObject::connect(&player, &QAVPlayer::audioFrame, [&](const QAVAudioFrame &frame) {
// Access to the data
qDebug() << autioFrame.format() << autioFrame.data().size();
audioOutput.play(frame);
}, Qt::DirectConnection);
QObject::connect(&p, &QAVPlayer::subtitleFrame, &p, [](const QAVSubtitleFrame &frame) {
for (unsigned i = 0; i < frame.subtitle()->num_rects; ++i) {
if (frame.subtitle()->rects[i]->type == SUBTITLE_TEXT)
qDebug() << "text:" << frame.subtitle()->rects[i]->text;
else
qDebug() << "ass:" << frame.subtitle()->rects[i]->ass;
}
}, Qt::DirectConnection);- Each action is confirmed by a signal:
// All signals are added to a queue and guaranteed to be emitted in proper order.
QObject::connect(&player, &QAVPlayer::played, [&](qint64 pos) { qDebug() << "Playing started from pos" << pos; });
QObject::connect(&player, &QAVPlayer::paused, [&](qint64 pos) { qDebug() << "Paused at pos" << pos; });
QObject::connect(&player, &QAVPlayer::stopped, [&](qint64 pos) { qDebug() << "Stopped at pos" << pos; });
QObject::connect(&player, &QAVPlayer::seeked, [&](qint64 pos) { qDebug() << "Seeked to pos" << pos; });
QObject::connect(&player, &QAVPlayer::stepped, [&](qint64 pos) { qDebug() << "Made a step to pos" << pos; });
QObject::connect(&player, &QAVPlayer::mediaStatusChanged, [&](QAVPlayer::MediaStatus status) {
switch (status) {
case QAVplayer::EndOfMedia:
qDebug() << "Finished to play, no frames in queue";
break;
case QAVplayer::NoMedia:
qDebug() << "Demuxer threads are finished";
break;
default:
break;
}
});- Accurate seek:
QObject::connect(&p, &QAVPlayer::seeked, &p, [&](qint64 pos) { seekPosition = pos; });
QObject::connect(&player, &QAVPlayer::videoFrame, [&](const QAVVideoFrame &frame) { seekFrame = frame; });
player.seek(5000)
QTRY_COMPARE(seekPosition, 5000);
QTRY_COMPARE(seekFrame.pts(), 5.0);If there is a frame with needed pts, it will be returned as first frame.
- FFmpeg filters:
player.setFilter("crop=iw/2:ih:0:0,split[left][tmp];[tmp]hflip[right];[left][right] hstack");
// Render bundled subtitles
player.setFilter("subtitles=file.mkv");
// Render subtitles from srt file
player.setFilter("subtitles=file.srt");
// Multiple filters
player.setFilters({
"drawtext=text=%{pts\\\\:hms}:x=(w-text_w)/2:y=(h-text_h)*(4/5):box=1:boxcolor=gray@0.5:fontsize=36[drawtext]",
"negate[negate]",
"[0:v]split=3[in1][in2][in3];[in1]boxblur[out1];[in2]negate[out2];[in3]drawtext=text=%{pts\\\\:hms}:x=(w-text_w)/2:y=(h-text_h)*(4/5):box=1:boxcolor=gray@0.5:fontsize=36[out3]"
}); // Return frames from 3 filters with 5 outputs- Step by step:
// Pausing will always emit one frame
QObject::connect(&player, &QAVPlayer::videoFrame, [&](const QAVVideoFrame &frame) { receivedFrame = frame; });
if (player.state() != QAVPlayer::PausedState) { // No frames if it is already paused
player.pause();
QTRY_VERIFY(receivedFrame);
}
// Always makes a step forward and emits only one frame
player.stepForward();
// the same here but backward
player.stepBackward();- Multiple streams:
qDebug() << "Audio streams" << player.availableAudioStreams().size();
qDebug() << "Current audio stream" << player.currentAudioStreams().first().index() << player.currentAudioStreams().first().metadata();
player.setAudioStreams(player.availableAudioStreams()); // Return all frames for all available audio streams
// Reports progress of playing per stream, like current pts, fps, frame rate, num of frames etc
for (const auto &s : p.availableVideoStreams())
qDebug() << s << p.progress(s);- Muxing the streams:
// Muxes all streams to the file without reencoding the packets.
// `QAVMuxerPackets` is used internally.
player.setOutput("output.mkv");
// Multiple players could be used to mux to one files
QAVPlayer p1;
QAVPlayer p2;
QAVMuxerFrames m;
// Wait until QAVPlayer::LoadedMedia
QTRY_VERIFY(p1.mediaStatus() == QAVPlayer::LoadedMedia);
QTRY_VERIFY(p2.mediaStatus() == QAVPlayer::LoadedMedia);
// Use all available streams from both players
auto streams = p1.availableStreams() + p2.availableStreams();
// Mux the streams to one file
m.load(streams, "output.mkv");
QObject::connect(&p1, &QAVPlayer::videoFrame, &p1, [&](const QAVVideoFrame &f) { m.enqueue(f); }, Qt::DirectConnection);
QObject::connect(&p1, &QAVPlayer::audioFrame, &p1, [&](const QAVAudioFrame &f) { m.enqueue(f); }, Qt::DirectConnection);
QObject::connect(&p2, &QAVPlayer::videoFrame, &p2, [&](const QAVVideoFrame &f) { m.enqueue(f); }, Qt::DirectConnection);
QObject::connect(&p2, &QAVPlayer::audioFrame, &p2, [&](const QAVAudioFrame &f) { m.enqueue(f); }, Qt::DirectConnection);
p1.play();
p2.play();- HW accelerations:
VA-APIandVDPAUfor Linux: the frames are returned with OpenGL textures.Video Toolboxfor macOS and iOS: the frames are returned with Metal Textures.D3D11for Windows: the frames are returned with D3D11Texture2D textures.MediaCodecfor Android: the frames are returned with OpenGL textures.
Note
Not all ffmpeg decoders or filters support HW acceleration. In this case software decoders are used.
- QtMultimedia could be used to render video frames to QML or Widgets. See examples
- Widget
QAVWidget_OpenGLcould be used to render to OpenGL. See examples/widget_video_opengl - Qt 5.12 - 6.x is supported
This library requires at least C++ 17 standard
Below, list of required dependencies:
| Dependencies | Packages | Comments |
|---|---|---|
| Qt | / | Library built with Qt framework |
| FFmpeg | ffmpeg |
/ |
Note
Some platforms required more packages, please refer to platform section for more details.
QtAvPlayer can be used as an embedded library in a subdirectory of your project (like a git submodule for example):
- In the root CMakeLists, add instructions:
add_subdirectory(qtavplayer) # Or if library is put in a folder "dependencies": add_subdirectory(dependencies/qtavplayer)- In the application/library CMakeLists, add instructions:
target_link_libraries(${PROJECT_NAME} PRIVATE qtavplayer)This library provide some CMake build options:
- Features:
QT_AVPLAYER_MULTIMEDIA(default:OFF): enables support ofQtMultimediawhich will used Qt packagesMultimedia.QTAVPLAYER_WIDGET_OPENGL(default:OFF): builds the widget based on OpenGL which will used Qt packagesOpenGLWidgetson Qt6 andQWidgetson Qt5.
- Hardware acceleration (see platforms support section for required packages):
QTAVPLAYER_HW_SUPPORT_WINDOWS(default:ON): enable HW acceleration for Windows platforms.QTAVPLAYER_HW_SUPPORT_MACOS(default:ON): enable HW acceleration for MacOS platforms.QTAVPLAYER_HW_SUPPORT_LINUX_VA_DRM(default:OFF): enable HW acceleration for Linux platforms usinglibva-drm.QTAVPLAYER_HW_SUPPORT_LINUX_VA_X11(default:OFF): enable HW acceleration for Linux platforms usinglibva-x11.QTAVPLAYER_HW_SUPPORT_LINUX_VDPAU(default:OFF): enable HW acceleration for Linux platforms usinglibvdpau.QTAVPLAYER_HW_SUPPORT_ANDROID(default:ON): enable HW acceleration for Android platforms.QTAVPLAYER_HW_SUPPORT_IOS(default:ON): enable HW acceleration for iOS platforms.
Some classes have more fine-grained logging category:
- Player backend (which is FFmpeg):
qtavplayer.backend - QAVPlayer:
qtavplayer.player - QAVVideoCodec:
"qtavplayer.videocodec"
Then we can enable/disable logs for those by using QLoggingCategory appropriate method:
// Manage Qt logging category
QLoggingCategory::setFilterRules(QStringLiteral(
"qtavplayer.backend.*=true\n" // Enable all logging level for "backend" category
"qtavplayer.player.debug=false\n" // Disable debug logging, all others are enabled
"qtavplayer.videocodec.debug=false\n"
"qtavplayer.videocodec.info=true"
));FFmpeg documentation is available on their website (and API documentation).
FFmpeg allow to set multiple options through their dictionary:
fflags: match flags defined underAVFMT_FLAG_*values, available at avformat header API docflags: match flags defined underAV_CODEC_FLAG_*values, available at avcodec header API doc- Context options: multiple context structure fields can be set through the dictionary, like
max_delay,probesize, etc.... Available fields can be seen at AVFormatContext API doc
QtAvPlayer library is released under MIT License, so this library too.