Skip to content

legerch/fork-qtavplayer

 
 

Repository files navigation

Library fork used to provide an advanced Qt Media Player based on FFmpeg library.

Note

I'm not the original author of this library, this repository is a fork from QtAvPlayer

The library allows to:

  • Demuxes and decodes video/audio/subtitle frames.
  • Muxes, encodes and saves the streams from different sources to one output file.
  • Support FFmpeg Bitstream Filters and FFmpeg Filters including filter_complex.
  • Multiple parallel filters for one input (one input frame produces multiple outputs).
  • Decoding of all available streams at the same time.
  • Support hardware acceleration.
  • It is up to an application to decide how to process the frames:
    • But there is experimental support of converting the video frames to QtMultimedia's QVideoFrame for copy-free rendering if possible.
      Note: Not all Qt's renders support copy-free rendering. Also QtMultimedia does not always provide public API to render the video frames. And, of course, for best performance both decoding and rendering should be accelerated.
    • Audio frames could be played by QAVAudioOutput which is a wrapper of QtMultimedia's QAudioSink
  • Accurate seek, it starts playing the closest frame.
  • Might be used for media analytics software like qctools or dvrescue.
  • Implements and replaces a combination of FFmpeg and FFplay:
# FFmpeg command-line
ffmpeg -i we-miss-gst-pipeline-in-qt6mm.mkv -filter_complex "qt,nev:er,wanted;[ffmpeg];what:happened" - | ffplay -

# Using QML or Qt Widgets
./qml_video :/valbok "if:you:like[cats];remove[this-sentence]"

Table of contents:

1. Library details

1.1. Fork Purposes

The fork is mainly due to the library being hard to build on mainline version due to multiple CMake hacks: source directory had to be set manually, some directories where includes and exported when building target.
So this fork aim to mainly fix those issues (and try some "workaround" to later propose a pull request to the original repository).

Current patches from this fork are all inside main branch, each features is represented by its own branch to ease synchronization with upstream project updates:

  • Branch refactor-library-structure: All details are explained in the pull request. This is the main refactor to ease building the library
  • Branch add-log-categories:
    • Stop using qDebug() and use log levels instead (info, warning, critical, etc...)
    • Allow to filter log according to the QtAvPlayer entity (player, demuxer, etc...), allowing more fine-grain for logs

1.2. Supported platforms

Those symbols will be used:

  • 💫: Untested (either lack of time, either because unable to access the device to perform tests)
  • ❌: Not working
  • 🌗: Partial support
  • ✅: Tested and working
/ Qt 5.12 -> 5.15.2 Qt 6.X HW accelerated feature Comments
Linux (Ubuntu) - Using libva-drm (also require packages: libva-dev, libegl1-mesa-dev and libgles2-mesa-dev)
- Using libva-x11 (also require packages: libva-dev, libva-x11-2 and libx11-dev)
- Using libvdpau (also require package: libvdpau-dev)
Windows Use API d3d11
MacOS Use APIs CoreVideo, IOSurface and Metal
iOS 💫 💫 💫 Use APIs CoreVideo, IOSurface and Metal
Android 💫 💫 💫 Use APIs MediaCodec and SurfaceTexture

2. Features

  1. QAVPlayer supports playing from an url or QIODevice or from avdevice:
player.setSource("http://clips.vorwaerts-gmbh.de/big_buck_bunny.mp4");
player.setSource("/home/lana/The Matrix Resurrections.mov");
// Dash player
player.setSource("https://bitdash-a.akamaihd.net/content/MI201109210084_1/m3u8s/f08e80da-bf1d-4e3d-8899-f0f6155f6efa.m3u8")

// Playing from qrc
QSharedPointer<QIODevice> file(new QFile(":/alarm.wav"));
file->open(QIODevice::ReadOnly);
QSharedPointer<QAVIODevice> dev(new QAVIODevice(file));
player.setSource("alarm", dev);

// Getting frames from the camera in Linux
player.setSource("/dev/video0");
// Or Windows
player.setInputFormat("dshow");
player.setSource("video=Integrated Camera");
// Or MacOS
player.setInputFormat("avfoundation");
player.setSource("default");
// Or Android
player.setInputFormat("android_camera");
player.setSource("0:0");

player.setInputOptions({{"user_agent", "QAVPlayer"}});
// Save to file
player.setOutput("output.mkv");
// Using various protocols
player.setSource("subfile,,start,0,end,0,,:/root/Downloads/why-qtmm-must-die.mkv");
  1. Easy getting video and audio frames:
QObject::connect(&player, &QAVPlayer::videoFrame, [&](const QAVVideoFrame &frame) {
    // QAVVideoFrame is comppatible with QVideoFrame
    QVideoFrame videoFrame = frame;
    
    // QAVVideoFrame can be converted to various pixel formats
    auto convertedFrame = frame.convert(AV_PIX_FMT_YUV420P);
    
    // Easy getting data from video frame
    auto mapped = videoFrame.map(); // downloads data if it is in GPU
    qDebug() << mapped.format << mapped.size;
    
    // The frame might contain OpenGL or MTL textures, for copy-free rendering
    qDebug() << frame.handleType() << frame.handle();
}, Qt::DirectConnection);

// Audio frames could be played using QAVAudioOutput
QAVAudioOutput audioOutput;
QObject::connect(&player, &QAVPlayer::audioFrame, [&](const QAVAudioFrame &frame) { 
    // Access to the data
    qDebug() << autioFrame.format() << autioFrame.data().size();
    audioOutput.play(frame);
}, Qt::DirectConnection);

QObject::connect(&p, &QAVPlayer::subtitleFrame, &p, [](const QAVSubtitleFrame &frame) {
    for (unsigned i = 0; i < frame.subtitle()->num_rects; ++i) {
        if (frame.subtitle()->rects[i]->type == SUBTITLE_TEXT)
            qDebug() << "text:" << frame.subtitle()->rects[i]->text;
        else
            qDebug() << "ass:" << frame.subtitle()->rects[i]->ass;
    }
}, Qt::DirectConnection);
  1. Each action is confirmed by a signal:
// All signals are added to a queue and guaranteed to be emitted in proper order.
QObject::connect(&player, &QAVPlayer::played, [&](qint64 pos) { qDebug() << "Playing started from pos" << pos;  });
QObject::connect(&player, &QAVPlayer::paused, [&](qint64 pos) { qDebug() << "Paused at pos" << pos; });
QObject::connect(&player, &QAVPlayer::stopped, [&](qint64 pos) { qDebug() << "Stopped at pos" << pos; });
QObject::connect(&player, &QAVPlayer::seeked, [&](qint64 pos) { qDebug() << "Seeked to pos" << pos; });
QObject::connect(&player, &QAVPlayer::stepped, [&](qint64 pos) { qDebug() << "Made a step to pos" << pos; });
QObject::connect(&player, &QAVPlayer::mediaStatusChanged, [&](QAVPlayer::MediaStatus status) { 
    switch (status) {
        case QAVplayer::EndOfMedia:
            qDebug() << "Finished to play, no frames in queue"; 
            break;
        case QAVplayer::NoMedia:
            qDebug() << "Demuxer threads are finished";
            break;
        default:
            break;
        }
});
  1. Accurate seek:
QObject::connect(&p, &QAVPlayer::seeked, &p, [&](qint64 pos) { seekPosition = pos; });
QObject::connect(&player, &QAVPlayer::videoFrame, [&](const QAVVideoFrame &frame) { seekFrame = frame; });
player.seek(5000)
QTRY_COMPARE(seekPosition, 5000);
QTRY_COMPARE(seekFrame.pts(), 5.0);

If there is a frame with needed pts, it will be returned as first frame.

  1. FFmpeg filters:
player.setFilter("crop=iw/2:ih:0:0,split[left][tmp];[tmp]hflip[right];[left][right] hstack");
// Render bundled subtitles
player.setFilter("subtitles=file.mkv");
// Render subtitles from srt file
player.setFilter("subtitles=file.srt");
// Multiple filters
player.setFilters({
    "drawtext=text=%{pts\\\\:hms}:x=(w-text_w)/2:y=(h-text_h)*(4/5):box=1:boxcolor=gray@0.5:fontsize=36[drawtext]",
    "negate[negate]",
    "[0:v]split=3[in1][in2][in3];[in1]boxblur[out1];[in2]negate[out2];[in3]drawtext=text=%{pts\\\\:hms}:x=(w-text_w)/2:y=(h-text_h)*(4/5):box=1:boxcolor=gray@0.5:fontsize=36[out3]"
}); // Return frames from 3 filters with 5 outputs
  1. Step by step:
// Pausing will always emit one frame
QObject::connect(&player, &QAVPlayer::videoFrame, [&](const QAVVideoFrame &frame) { receivedFrame = frame; });
if (player.state() != QAVPlayer::PausedState) { // No frames if it is already paused
    player.pause();
    QTRY_VERIFY(receivedFrame);
}

// Always makes a step forward and emits only one frame
player.stepForward();
// the same here but backward
player.stepBackward();
  1. Multiple streams:
qDebug() << "Audio streams" << player.availableAudioStreams().size();
qDebug() << "Current audio stream" << player.currentAudioStreams().first().index() << player.currentAudioStreams().first().metadata();
player.setAudioStreams(player.availableAudioStreams()); // Return all frames for all available audio streams
// Reports progress of playing per stream, like current pts, fps, frame rate, num of frames etc
for (const auto &s : p.availableVideoStreams())
    qDebug() << s << p.progress(s);
  1. Muxing the streams:
// Muxes all streams to the file without reencoding the packets.
// `QAVMuxerPackets` is used internally.
player.setOutput("output.mkv");

// Multiple players could be used to mux to one files
QAVPlayer p1;
QAVPlayer p2;
QAVMuxerFrames m;

// Wait until QAVPlayer::LoadedMedia
QTRY_VERIFY(p1.mediaStatus() == QAVPlayer::LoadedMedia);
QTRY_VERIFY(p2.mediaStatus() == QAVPlayer::LoadedMedia);
// Use all available streams from both players
auto streams = p1.availableStreams() + p2.availableStreams();
// Mux the streams to one file
m.load(streams, "output.mkv");

QObject::connect(&p1, &QAVPlayer::videoFrame, &p1, [&](const QAVVideoFrame &f) { m.enqueue(f); }, Qt::DirectConnection);
QObject::connect(&p1, &QAVPlayer::audioFrame, &p1, [&](const QAVAudioFrame &f) { m.enqueue(f); }, Qt::DirectConnection);
QObject::connect(&p2, &QAVPlayer::videoFrame, &p2, [&](const QAVVideoFrame &f) { m.enqueue(f); }, Qt::DirectConnection);
QObject::connect(&p2, &QAVPlayer::audioFrame, &p2, [&](const QAVAudioFrame &f) { m.enqueue(f); }, Qt::DirectConnection);

p1.play();
p2.play();
  1. HW accelerations:
  • VA-API and VDPAU for Linux: the frames are returned with OpenGL textures.
  • Video Toolbox for macOS and iOS: the frames are returned with Metal Textures.
  • D3D11 for Windows: the frames are returned with D3D11Texture2D textures.
  • MediaCodec for Android: the frames are returned with OpenGL textures.

Note

Not all ffmpeg decoders or filters support HW acceleration. In this case software decoders are used.

  1. QtMultimedia could be used to render video frames to QML or Widgets. See examples
  2. Widget QAVWidget_OpenGL could be used to render to OpenGL. See examples/widget_video_opengl
  3. Qt 5.12 - 6.x is supported

2.1. Requirements

2.1.1. C++ Standards

This library requires at least C++ 17 standard

2.2. Dependencies

Below, list of required dependencies:

Dependencies Packages Comments
Qt / Library built with Qt framework
FFmpeg ffmpeg /

Note

Some platforms required more packages, please refer to platform section for more details.

3. How to build

3.1. CMake usage

QtAvPlayer can be used as an embedded library in a subdirectory of your project (like a git submodule for example):

  1. In the root CMakeLists, add instructions:
add_subdirectory(qtavplayer) # Or if library is put in a folder "dependencies": add_subdirectory(dependencies/qtavplayer)
  1. In the application/library CMakeLists, add instructions:
target_link_libraries(${PROJECT_NAME} PRIVATE qtavplayer)

3.2. CMake options

This library provide some CMake build options:

  • Features:
    • QT_AVPLAYER_MULTIMEDIA (default: OFF): enables support of QtMultimedia which will used Qt packages Multimedia.
    • QTAVPLAYER_WIDGET_OPENGL (default: OFF): builds the widget based on OpenGL which will used Qt packages OpenGLWidgets on Qt6 and QWidgets on Qt5.
  • Hardware acceleration (see platforms support section for required packages):
    • QTAVPLAYER_HW_SUPPORT_WINDOWS (default: ON): enable HW acceleration for Windows platforms.
    • QTAVPLAYER_HW_SUPPORT_MACOS (default: ON): enable HW acceleration for MacOS platforms.
    • QTAVPLAYER_HW_SUPPORT_LINUX_VA_DRM (default: OFF): enable HW acceleration for Linux platforms using libva-drm.
    • QTAVPLAYER_HW_SUPPORT_LINUX_VA_X11 (default: OFF): enable HW acceleration for Linux platforms using libva-x11.
    • QTAVPLAYER_HW_SUPPORT_LINUX_VDPAU (default: OFF): enable HW acceleration for Linux platforms using libvdpau.
    • QTAVPLAYER_HW_SUPPORT_ANDROID (default: ON): enable HW acceleration for Android platforms.
    • QTAVPLAYER_HW_SUPPORT_IOS (default: ON): enable HW acceleration for iOS platforms.

4. How to use

4.1. Logging category

Some classes have more fine-grained logging category:

  • Player backend (which is FFmpeg): qtavplayer.backend
  • QAVPlayer: qtavplayer.player
  • QAVVideoCodec: "qtavplayer.videocodec"

Then we can enable/disable logs for those by using QLoggingCategory appropriate method:

// Manage Qt logging category
QLoggingCategory::setFilterRules(QStringLiteral(
    "qtavplayer.backend.*=true\n"           // Enable all logging level for "backend" category
    "qtavplayer.player.debug=false\n"       // Disable debug logging, all others are enabled
    "qtavplayer.videocodec.debug=false\n"
    "qtavplayer.videocodec.info=true"
));

5. FFmpeg library usage

FFmpeg documentation is available on their website (and API documentation).
FFmpeg allow to set multiple options through their dictionary:

  • fflags: match flags defined under AVFMT_FLAG_* values, available at avformat header API doc
  • flags: match flags defined under AV_CODEC_FLAG_* values, available at avcodec header API doc
  • Context options: multiple context structure fields can be set through the dictionary, like max_delay, probesize, etc.... Available fields can be seen at AVFormatContext API doc

6. License

QtAvPlayer library is released under MIT License, so this library too.

About

Free and open-source Qt Media Player library based on FFmpeg, for Linux, Windows, macOS, iOS and Android.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

 
 
 

Contributors

Languages

  • C++ 94.0%
  • CMake 4.4%
  • Other 1.6%