Intel Quick Sync Video is the technology used in intel integrated gpu's to enable hardware accelerated encoding and decoding of videos. It supports multiple codecs like H.264(avc), H.265 (hevc), MPEG-2 etc.
In this quick tutorial we take a quick look at how to use this technology inside ffmpeg to encode and decode videos.
Install non-free VA Driver for Encoding
Hardware accelerated video processing on intel gpus is provided through "Video Acceleration API" - VA-API on linux. We need to install the correct VAAPI drivers for enabling hardware acceleration on intel gpus.
There are 2 different packages of vaapi drivers for intel in the ubuntu repositories.
$ aptitude search "intel va driver" p intel-media-va-driver - VAAPI driver for the Intel GEN8+ Graphics family p intel-media-va-driver:i386 - VAAPI driver for the Intel GEN8+ Graphics family i intel-media-va-driver-non-free - VAAPI driver for the Intel GEN8+ Graphics family i A intel-media-va-driver-non-free:i386 - VAAPI driver for the Intel GEN8+ Graphics family $
The "intel-media-va-driver" package is usually installed by default if an intel gpu is detected on the system. The driver from this package can decode videos using hardware acceleration but cannot encode.
In order to encode using intel igpu via va-api we need to install the non-free driver package.
sudo apt install intel-media-va-driver-non-free
Note: The non-free drivers will automatically un-install the free driver package.
Check supported encoders/decoders: After installing the drivers, we can check which intel quick sync codecs are supported for encoding and decoding in ffmpeg.
List the encoders with the following command:
$ ffmpeg -encoders -hide_banner | grep -i qsv V..... h264_qsv H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10 (Intel Quick Sync Video acceleration) (codec h264) V..... hevc_qsv HEVC (Intel Quick Sync Video acceleration) (codec hevc) V..... mjpeg_qsv MJPEG (Intel Quick Sync Video acceleration) (codec mjpeg) V..... mpeg2_qsv MPEG-2 video (Intel Quick Sync Video acceleration) (codec mpeg2video) V..... vp9_qsv VP9 video (Intel Quick Sync Video acceleration) (codec vp9) $
In the above output we can see, that ffmpeg can encode videos with h264, hevc, mjpeg, mjpeg2 and vp9 codecs using intel gpu hardware acceleration.
List the decoders with the following command:
$ ffmpeg -decoders -hide_banner | grep -i qsv V....D av1_qsv AV1 video (Intel Quick Sync Video acceleration) (codec av1) V....D h264_qsv H264 video (Intel Quick Sync Video acceleration) (codec h264) V....D hevc_qsv HEVC video (Intel Quick Sync Video acceleration) (codec hevc) V....D mjpeg_qsv MJPEG video (Intel Quick Sync Video acceleration) (codec mjpeg) V....D mpeg2_qsv MPEG2VIDEO video (Intel Quick Sync Video acceleration) (codec mpeg2video) V....D vc1_qsv VC1 video (Intel Quick Sync Video acceleration) (codec vc1) V....D vp8_qsv VP8 video (Intel Quick Sync Video acceleration) (codec vp8) V....D vp9_qsv VP9 video (Intel Quick Sync Video acceleration) (codec vp9) $
We can see that more decoder codecs are supported than encoder codecs.
Encode with h264_qsv
After installing the packages we can now encode videos using intel quick sync technology in ffmpeg. All we need to do is to encode using the h264_qsv codec.
ffmpeg -loglevel verbose -i 2023-02-04_18-18-48.mkv -vcodec h264_qsv h264_qsv_encoded.mp4
The audio codec is automatically copied from the source file, if supported by ffmpeg.
Here is a modified version of the command:
ffmpeg -y -loglevel verbose -hide_banner -i 2023-02-04_18-18-48.mkv -vcodec h264_qsv h264_qsv_encoded.mp4
The hide_banner option hides the huge banner displaying configuration and build options. The -y option automatically instructs ffmpeg to overwrite existing output files without prompting for a confirmation.
Measure Time: If you want to measure the total running time for benchmark purpose, run the command with "time" command prepended
time ffmpeg -y -loglevel verbose -hide_banner -i 2023-02-04_18-18-48.mkv -vcodec h264_qsv h264_qsv_encoded.mp4
We can use the intel_gpu_top command to verify that the gpu hardware is actually being used for decoding:
FFmpeg Video Encoding with Intel GPU
Maintain Quality of Output
By default the h264_qsv codec renders the output video in a very low quality and generates a very small sized file. In order to encode the video in better quality we have to use the global_quality option.
ffmpeg -y -loglevel verbose -i 2023-02-04_18-18-48.mkv -ss 0 -t 10 -vcodec h264_qsv -global_quality 10 h264_qsv_encoded.mp4
Note: The more commonly known option for output video quality control is crf, but qsv does not support that.
List of supported options
A list of all the options supported by h264_qsv encoder and decoder can be viewed with the following commands. The hide_banner flag is optional.
ffmpeg -h encoder=h264_qsv -hide_banner ffmpeg -h decoder=h264_qsv -hide_banner
Decode using h264_qsv
To decode using the h264_qsv codec, specify it before the input file like this:
ffmpeg -hide_banner -vcodec h264_qsv -i h264_qsv_encoded.mp4 -f null -
Now ffmpeg will use h264_qsv for decoding the video, which will be hardware (gpu) accelerated.
The output should look something like this:
$ ffmpeg -hide_banner -vcodec h264_qsv -i h264_qsv_encoded.mp4 -f null -
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'h264_qsv_encoded.mp4':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
encoder : Lavf59.27.100
Duration: 00:00:10.00, start: 0.000000, bitrate: 86707 kb/s
Stream #0:0[0x1](und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709, progressive), 1920x1080 [SAR 1:1 DAR 16:9], 86560 kb/s, 60 fps, 60 tbr, 15360 tbn (default)
Metadata:
handler_name : VideoHandler
vendor_id : [0][0][0][0]
encoder : Lavc59.37.100 h264_qsv
Stream #0:1[0x2](und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 129 kb/s (default)
Metadata:
handler_name : SoundHandler
vendor_id : [0][0][0][0]
Stream mapping:
Stream #0:0 -> #0:0 (h264 (h264_qsv) -> wrapped_avframe (native))
Stream #0:1 -> #0:1 (aac (native) -> pcm_s16le (native))
Press [q] to stop, [?] for help
Output #0, null, to 'pipe:':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
encoder : Lavf59.27.100
Stream #0:0(und): Video: wrapped_avframe, nv12(tv, bt709, progressive), 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 60 fps, 60 tbn (default)
Metadata:
handler_name : VideoHandler
vendor_id : [0][0][0][0]
encoder : Lavc59.37.100 wrapped_avframe
Stream #0:1(und): Audio: pcm_s16le, 48000 Hz, stereo, s16, 1536 kb/s (default)
Metadata:
handler_name : SoundHandler
vendor_id : [0][0][0][0]
encoder : Lavc59.37.100 pcm_s16le
[h264_qsv @ 0x561e3ccca200] A decode call did not consume any data: expect more data at input (-10)
Last message repeated 2 times
frame= 600 fps=386 q=-0.0 Lsize=N/A time=00:00:10.00 bitrate=N/A speed=6.43x
video:277kB audio:1876kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
enlightened@enlightened:/media/enlightened/One Touch/mydata/ffmpeg$
Note the section titled Stream mapping
Stream mapping: Stream #0:0 -> #0:0 (h264 (h264_qsv) -> wrapped_avframe (native)) Stream #0:1 -> #0:1 (aac (native) -> pcm_s16le (native))
The first stream (0:0) is the video which is being mapped from h264_qsv to native.
To verify that hardware acceleration is indeed being used run the intel_gpu_top command again and it will would show video engine under load with high Busy value. On the other hand the cpu load would be less.
Play h264 videos with hardware acceleration
Video files can be played with ffplay command, however it doesn't have an option to choose a decoder directly to play the video. Instead we have to use the ffmpeg command to decode the video with our specified hardware accelerated decoder and then pipe the output to ffplay.
This way hardware acceleration can be used to decode h264 videos.
Here is a quick example:
$ ffmpeg -hide_banner -vcodec h264_qsv -i h264_qsv_encoded.mp4 -vcodec rawvideo -f matroska - | ffplay -hide_banner -i -
Note the output video codec is rawvideo. As for the container we are using matroska, as something has to be used.
VA-API support
Applications that utilise the intel qsv hardware acceleration on linux, do so via the va-api (Video Acceleration API), which was initially developed by Intel.
To check va-api support on your linux system, use the vainfo command:
sudo apt install vainfo
The output of the vainfo command would look something like this:
$ vainfo
libva info: VA-API version 1.17.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_17
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.17 (libva 2.12.0)
vainfo: Driver version: Intel iHD driver for Intel(R) Gen Graphics - 23.1.2 ()
vainfo: Supported profile and entrypoints
VAProfileNone : VAEntrypointVideoProc
VAProfileNone : VAEntrypointStats
VAProfileMPEG2Simple : VAEntrypointVLD
VAProfileMPEG2Simple : VAEntrypointEncSlice
VAProfileMPEG2Main : VAEntrypointVLD
VAProfileMPEG2Main : VAEntrypointEncSlice
VAProfileH264Main : VAEntrypointVLD
VAProfileH264Main : VAEntrypointEncSlice
VAProfileH264Main : VAEntrypointFEI
VAProfileH264Main : VAEntrypointEncSliceLP
VAProfileH264High : VAEntrypointVLD
VAProfileH264High : VAEntrypointEncSlice
VAProfileH264High : VAEntrypointFEI
VAProfileH264High : VAEntrypointEncSliceLP
VAProfileVC1Simple : VAEntrypointVLD
VAProfileVC1Main : VAEntrypointVLD
VAProfileVC1Advanced : VAEntrypointVLD
VAProfileJPEGBaseline : VAEntrypointVLD
VAProfileJPEGBaseline : VAEntrypointEncPicture
VAProfileH264ConstrainedBaseline: VAEntrypointVLD
VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
VAProfileH264ConstrainedBaseline: VAEntrypointFEI
VAProfileH264ConstrainedBaseline: VAEntrypointEncSliceLP
VAProfileVP8Version0_3 : VAEntrypointVLD
VAProfileVP8Version0_3 : VAEntrypointEncSlice
VAProfileHEVCMain : VAEntrypointVLD
VAProfileHEVCMain : VAEntrypointEncSlice
VAProfileHEVCMain : VAEntrypointFEI
VAProfileHEVCMain10 : VAEntrypointVLD
VAProfileHEVCMain10 : VAEntrypointEncSlice
VAProfileVP9Profile0 : VAEntrypointVLD
VAProfileVP9Profile2 : VAEntrypointVLD
$
Media players supporting vaapi
There are many media players available on linux that can use vaapi for hardware accelerated video decoding. These include mpv, smplayer etc.
MPV: MPV supports vaapi right away without needing any configuration. You can just launch mpv from command line to play a h264 video and it would automatically use h264 if supported.
$ mpv 2023-02-04_18-18-48.mkv (+) Video --vid=1 (h264 1920x1080 60.000fps) (+) Audio --aid=1 'simple_aac_recording' (aac 2ch 48000Hz) Using hardware decoding (vaapi). AO: [pipewire] 48000Hz stereo 2ch floatp [ffmpeg/video] h264: Increasing reorder buffer to 1 VO: [gpu] 1920x1080 vaapi[nv12] AV: 00:00:27 / 00:18:29 (2%) A-V: 0.000 Exiting... (Quit) $
Smplayer: SMplayer also supported hardware decoding with some configurations.
Open Smaplyer and go to "Options > Preferences > Performance" and select Hardware decoding: vaapi. Now play any h264 encoded video and intel gpu should be used for decoding.
To ensure that hardware decoding is being used check the output of intel_gpu_top command, and the monitor cpu usage. CPU usage should be low whereas the intel_gpu_top command should report load on video engine.
VLC: The most recent version of vlc player (3.0.18) has some issue with vaapi, and it unable to decode using hardware acceleration with intel quick sync.
Possible Errors
If you try to encode with the free intel va drivers installed then you would likely run into error messages like these:
... [h264_qsv @ 0x5627b4916e00] Selected ratecontrol mode is unsupported [h264_qsv @ 0x5627b4916e00] Current frame rate is unsupported [h264_qsv @ 0x5627b4916e00] Current picture structure is unsupported [h264_qsv @ 0x5627b4916e00] Current resolution is unsupported [h264_qsv @ 0x5627b4916e00] Current pixel format is unsupported [h264_qsv @ 0x5627b4916e00] some encoding parameters are not supported by the QSV runtime. Please double check the input parameters. ...
The full output would look like this:
$ ffmpeg -y -loglevel verbose -i 2023-02-04_18-18-48.mkv -threads 4 -vcodec h264_qsv h264_qsv_encoded.mp4
ffmpeg version 5.1.2-3ubuntu1 Copyright (c) 2000-2022 the FFmpeg developers
built with gcc 12 (Ubuntu 12.2.0-14ubuntu2)
configuration: --prefix=/usr --extra-version=3ubuntu1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libglslang --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librist --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --disable-sndio --enable-libjxl --enable-pocketsphinx --enable-librsvg --enable-libmfx --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-libplacebo --enable-librav1e --enable-shared
WARNING: library configuration mismatch
avcodec configuration: --prefix=/usr --extra-version=3ubuntu1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libglslang --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librist --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --disable-sndio --enable-libjxl --enable-pocketsphinx --enable-librsvg --enable-libmfx --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-libplacebo --enable-librav1e --enable-shared --enable-version3 --disable-doc --disable-programs --enable-libaribb24 --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libtesseract --enable-libvo_amrwbenc --enable-libsmbclient
libavutil 57. 28.100 / 57. 28.100
libavcodec 59. 37.100 / 59. 37.100
libavformat 59. 27.100 / 59. 27.100
libavdevice 59. 7.100 / 59. 7.100
libavfilter 8. 44.100 / 8. 44.100
libswscale 6. 7.100 / 6. 7.100
libswresample 4. 7.100 / 4. 7.100
libpostproc 56. 6.100 / 56. 6.100
[h264 @ 0x5627b48ffb80] Reinit context to 1920x1088, pix_fmt: yuv420p
[h264 @ 0x5627b48ffb80] Increasing reorder buffer to 1
Input #0, matroska,webm, from '2023-02-04_18-18-48.mkv':
Metadata:
ENCODER : Lavf59.27.100
Duration: 00:18:29.25, start: 0.000000, bitrate: 45046 kb/s
Stream #0:0: Video: h264 (High), 1 reference frame, yuv420p(tv, bt709, progressive, left), 1920x1080 (1920x1088) [SAR 1:1 DAR 16:9], 60 fps, 60 tbr, 1k tbn
Metadata:
DURATION : 00:18:29.250000000
Stream #0:1: Audio: aac (LC), 48000 Hz, stereo, fltp
Metadata:
title : simple_aac_recording
DURATION : 00:18:29.184000000
Stream mapping:
Stream #0:0 -> #0:0 (h264 (native) -> h264 (h264_qsv))
Stream #0:1 -> #0:1 (aac (native) -> aac (native))
Press [q] to stop, [?] for help
[h264 @ 0x5627b4b73480] Reinit context to 1920x1088, pix_fmt: yuv420p
[graph_1_in_0_1 @ 0x5627b4a3db40] tb:1/48000 samplefmt:fltp samplerate:48000 chlayout:stereo
[graph 0 input from stream 0:0 @ 0x5627b5406700] w:1920 h:1080 pixfmt:yuv420p tb:1/1000 fr:60/1 sar:1/1
[auto_scale_0 @ 0x5627b5409500] w:iw h:ih flags:'' interl:0
[format @ 0x5627b54072c0] auto-inserting filter 'auto_scale_0' between the filter 'Parsed_null_0' and the filter 'format'
[auto_scale_0 @ 0x5627b5409500] w:1920 h:1080 fmt:yuv420p sar:1/1 -> w:1920 h:1080 fmt:nv12 sar:1/1 flags:0x0
Last message repeated 3 times
[h264_qsv @ 0x5627b4916e00] Encoder: input is system memory surface
[AVHWDeviceContext @ 0x5627b57c80c0] Trying to use DRM render node for device 0, with matching kernel driver (i915).
[AVHWDeviceContext @ 0x5627b57c80c0] libva: VA-API version 1.17.0
[AVHWDeviceContext @ 0x5627b57c80c0] libva: User requested driver 'iHD'
[AVHWDeviceContext @ 0x5627b57c80c0] libva: Trying to open /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
[AVHWDeviceContext @ 0x5627b57c80c0] libva: Found init function __vaDriverInit_1_17
[AVHWDeviceContext @ 0x5627b57c80c0] libva: va_openDriver() returns 0
[AVHWDeviceContext @ 0x5627b57c80c0] Initialised VAAPI connection: version 1.17
[AVHWDeviceContext @ 0x5627b57c80c0] VAAPI driver: Intel iHD driver for Intel(R) Gen Graphics - 23.1.2 ().
[AVHWDeviceContext @ 0x5627b57c80c0] Driver not found in known nonstandard list, using standard behaviour.
[h264_qsv @ 0x5627b4916e00] Initialized an internal MFX session using hardware accelerated implementation
[h264_qsv @ 0x5627b4916e00] Using the variable bitrate (VBR) ratecontrol method
[h264_qsv @ 0x5627b4916e00] MFMode:2
[h264_qsv @ 0x5627b4916e00] Selected ratecontrol mode is unsupported
[h264_qsv @ 0x5627b4916e00] Current frame rate is unsupported
[h264_qsv @ 0x5627b4916e00] Current picture structure is unsupported
[h264_qsv @ 0x5627b4916e00] Current resolution is unsupported
[h264_qsv @ 0x5627b4916e00] Current pixel format is unsupported
[h264_qsv @ 0x5627b4916e00] some encoding parameters are not supported by the QSV runtime. Please double check the input parameters.
Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
[AVIOContext @ 0x5627b491bec0] Statistics: 0 bytes written, 0 seeks, 0 writeouts
[aac @ 0x5627b491d1c0] Qavg: 35450.430
[aac @ 0x5627b491d1c0] 2 frames left in the queue on closing
[AVIOContext @ 0x5627b4905480] Statistics: 1184366 bytes read, 0 seeks
Conversion failed!
enlightened@enlightened:/media/enlightened/One Touch/mydata/ffmpeg$
The fix for the above error is to install the non-free driver package.
Links and Resources
https://wiki.debian.org/HardwareVideoAcceleration
https://wiki.archlinux.org/title/Hardware_video_acceleration
https://en.wikipedia.org/wiki/Intel_Quick_Sync_Video
