Av1an

Av1an

Av1an is a video encoding framework. It can increase your encoding speed and improve CPU utilization by running multiple encoder processes in parallel. Target quality, VMAF plotting, and more, available to take advantage for video encoding.

The only downside to this, is that RAM usage increases significantly with each “worker” you add, so if you have a low amount of RAM (< 8GB), this might not be that useful to you.

Caution

Av1an does not ship with any dependencies except for it’s Docker image, so you will have to provide them on your own.

Prerequisites

You will need Python, FFmpeg and Vapoursynth to be installed regardless of your current operating system.

Installation

Windows

Script

Windows users can use this script which installs everything needed in a single portable folder. Do note that you will need to update the dependencies manually once installed.

Pre-built Binary

Av1an is available as a pre-built binary under the “latest” tag on GitHub releases, you can simply download that and place it wherever you want.

Compile from Source

To compile from source, it is easier to use mingw-w64 which comes with MSYS2. Once installed, open MinGW64 and run the following:

pacman -Syuu && pacman -S cmake git nasm python3 mingw-w64-x86_64-gcc mingw-w64-x86_64-cmake base-devel mingw-w64-x86_64-ffmpeg mingw-w64-x86_64-rust mingw-w64-x86_64-lld mingw-w64-x86_64-clang mingw-w64-x86_64-make

Then, before you do anything further. Download Vapoursynth portable (VapourSynth64-Portable-RXX.7z) from its GitHub release page. Make sure the version you chose is compatible with the current MinGW64 Python version. For example, R65 supports 3.8 and 3.11, the version from Pacman (currently) is 3.11, so you should choose that.

After downloading, open the 7z file and head to \sdk\lib64, copy all the libraries from there to MinGW64’s lib directory, this usually resides in C:\msys64\mingw64\lib.

Now finally, resume your session and run the following:

git clone https://github.com/master-of-zen/Av1an
cd Av1an
RUSTFLAGS="-C target-cpu=native" cargo build --release

The binary will be available in C:\msys64\home\USER\Av1an\target\release. The USER part is your Windows username.

macOS

Using MacPorts

Av1an is available in the MacPorts repository, to install just simply run:

port install av1an

Compile from Source

You need to have Git, Nasm, and Rust to be installed first. You can achieve this by using Homebrew or similar.

git clone https://github.com/master-of-zen/Av1an
cd Av1an
RUSTFLAGS="-C target-cpu=native" cargo build --release

Binary is located in Av1an/target/release, you can optionally place it somewhere like /usr/local/bin.

Linux

Package Managers

Av1an is available in the package manager of some distributions such as Arch Linux. If not, you can skip to the next part every Linux user is familiar with.

Compile from Source

Just like macOS, you need to have Git, Nasm, and Rust to be installed. Can be achieved by either using your package manager or Rustup for Rust.

git clone https://github.com/master-of-zen/Av1an.git
cd Av1an
RUSTFLAGS="-C target-cpu=native" cargo build --release

Binary is located in Av1an/target/release, you can optionally place it somewhere like /usr/local/bin or /usr/bin.

Docker

The following examples assume the file you want to encode is in your current working directory.

Windows

docker run --privileged -v "$(pwd):/videos" --user $(id -u):$(id -g) -it --rm masterofzen/av1an:latest -i S01E01.mkv {options}

Linux

docker run --privileged -v "${PWD}:/videos" -it --rm masterofzen/av1an:latest -i S01E01.mkv {options}

Manual Build

docker build -t "av1an" .

Run in the root directory of the repository. The dependencies will automatically be installed into the image, no manual installations necessary.

Note

To specify a different directory to use you would replace $(pwd) with the directory

docker run --privileged -v "/c/Users/masterofzen/Videos":/videos --user $(id -u):$(id -g) -it --rm masterofzen/av1an:latest -i S01E01.mkv {options}

The --user flag is required on Linux to avoid permission issues with the docker container not being able to write to the location, if you get permission issues ensure your user has access to the folder that you are using to encode.

Caution

The Docker image ships with the default dependencies.

Installing Dependencies

You will need a chunk method installed as the built-in ones are very slow.

Troubleshooting

“Error: The file ‘XXXXX.ivf’ could not be opened for reading: open file error.” with mkvmerge on Linux

Since mkvmerge opens all the encoded chunks at once, this obviously hits the default open file descriptor limit of 1024 on longer videos. You can temporarily increase it with ulimit -n 20000 to allow opening 20,000 files at once in that session.

Gray screen flashing for a single frame in output

This is a unique problem with Av1an. Since Av1an does random seeking for chunks, traditional keyframe-based decoding methods such as L-SMASH and FFMS2 may fail and result in a gray frame. There are currently only two known methods to solve this:

1. Create a lossless intermediary with x264 -qp 0, this always fixes all related seeking issues but will result in a huge file due to it’s lossless nature.
2. Using a linear-decoding chunk method such as bestsource, the downside to this is it’s mega slow. Tests shown encode speeds drop by a whopping 10-15x slower.
3. Using dgdecnv chunk method. dgdecnv and DGIndexNV is a proprietary CUVID-accelerated Vapoursynth “Source” decoder created by Donald Alan Graft where DGIndexNV is the standalone program for seeking and dgdecnv is the Vapoursynth plugin itself. It is especially made with random seeking in mind which makes it perfect for Av1an. Only downside is you need an NVIDIA GPU and DGIndexNV only supports x86_64 platforms.