# Compare encoding quality with metrics and visual checks Canonical URL: https://encode.wiki/guides/compare-encoding-quality/ Content version: ca42e61ede0a24f789d7287a0001244ad22f4abf Status: current License: CC BY-SA 4.0 Checksum (SHA-256): fdbca95c9ddfb83bff544f189e7c94473a678b7ba8aeaad9b1b4b82097aa84f9 Metrics narrow down settings, find bad frames and let you compare many encodes; your eyes make the final call. A fair comparison needs one reference, aligned frames, several quality levels, metrics that track perception, a summary that shows the worst parts, and a look at those parts. ## Make one reference Encode every candidate from the same file with the same preprocessing, and measure against that file, not the original release. Use a few 10–30 second clips that stress encoders: grain, dark gradients, foliage, fast motion, line art. Cutting them to lossless FFV1 gives every encoder and metric identical frames (`-map 0:v:0` keeps just the video): ```sh ffmpeg -ss 00:12:30 -i source.mkv -t 20 -map 0:v:0 -c:v ffv1 reference.mkv ``` ## Encode several quality levels A setting can win at one bitrate and lose at another, so encode each candidate at four or more levels and compare **rate–quality curves** (size or bitrate against score): ```sh for crf in 20 25 30 35; do ffmpeg -i reference.mkv -c:v libx264 -preset slow -crf $crf x264-crf$crf.mkv done ``` CRF values aren't comparable between encoders or even presets, so compare at equal size, not equal CRF. **BD-rate** condenses two curves into the average bitrate difference at equal quality; the [`bjontegaard`](https://github.com/FAU-LMS/bjontegaard) Python package computes it: ```python import bjontegaard as bd print(bd.bd_rate(rate_a, score_a, rate_b, score_b, method="akima")) ``` −10 means candidate B needs 10% fewer bits for the same score. It expects higher-is-better scores, so negate Butteraugli distances, and it only covers the quality range where the curves overlap. **Score encodes without film-grain synthesis.** Synthesised grain never matches the source pixel for pixel, so full-reference metrics punish it even when it looks better. For AV1 in FFmpeg, put `-export_side_data film_grain` before the encode's `-i` to decode it without the grain, and judge the grain by eye. If the encoder also denoised the source (SVT-AV1 `--film-grain-denoise 1`, aomenc's `--denoise-noise-level`), the decode is still a clean picture measured against a grainy reference, and the scores stay low however good it looks. ## Check that the frames line up ```sh ffprobe -v error -select_streams v:0 -count_frames -show_entries stream=width,height,nb_read_frames -of csv=p=0 x264-crf30.mkv ``` - Frame counts must match. FFmpeg's metric filters silently repeat the last frame of a shorter input, and a one-frame offset ruins every score. If everything scores terribly, check alignment first; [PSNR](/reference/metrics/psnr/) is a quick sanity check. - Scale a downscaled encode back to the reference resolution (bicubic) before measuring. - SSIMULACRA2, Butteraugli and CVVDP convert to RGB using the files' colour tags, so missing or wrong matrix, range or transfer tags give wrong scores. FFVship guesses missing tags; `FFVship --verbose` prints what it used for each file. ## Pick the metrics Use more than one. They fail in different ways, and where they disagree is worth a look. | Metric | Role | How to run it | | ----------------------------------------------------- | ------------------------------------------ | ----------------------------------- | | [SSIMULACRA2](/reference/metrics/ssimulacra2/) | Main perceptual score | FFVship or Vship (GPU), vszip (CPU) | | [Butteraugli](/reference/metrics/butteraugli/) 3-norm | High-fidelity and near-transparent encodes | FFVship or Vship | | [XPSNR](/reference/metrics/xpsnr/) | Fast, motion-aware first pass | FFmpeg `xpsnr` (reference first) | | [CVVDP](/reference/metrics/cvvdp/) | Temporal, colour and display-aware; HDR | `cvvdp`, FFVship | | [VMAF](/reference/metrics/vmaf/) NEG or v1 | Optional streaming-style score | FFmpeg `libvmaf` | | PSNR, [SSIM](/reference/metrics/ssim/) | Sanity checks only; they reward blur | FFmpeg `psnr`, `ssim` | ```sh FFVship --source reference.mkv --encoded x264-crf30.mkv --json x264-crf30-ssimu2.json FFVship --source reference.mkv --encoded x264-crf30.mkv -m Butteraugli --json x264-crf30-butter.json ffmpeg -i reference.mkv -i x264-crf30.mkv -lavfi xpsnr=stats_file=x264-crf30-xpsnr.log -f null - ``` ## Summarise with the worst frames A short ugly section is what viewers notice, and a mean hides it. For each encode keep the **mean**, a **low percentile** (the 5th for SSIMULACRA2, the 95th for Butteraugli, where lower is better) and the **worst frames** by number. FFVship prints the mean and percentiles; this lists the ten worst frames from its SSIMULACRA2 JSON: ```sh jq -c 'to_entries | sort_by(.value[0]) | .[:10] | map(.key)' x264-crf30-ssimu2.json ``` Plot curves for the low percentile as well as the mean. A setting that lifts the mean but lowers the percentile is trading consistency for average quality. ## Look at the worst sections Watch the scenes around the worst frames at normal speed, at the size people will watch them. Then step frame by frame and flip between the reference and each candidate at the same frame; flipping shows differences that side-by-side views hide. A [video player](/tools/video-players/) with frame stepping or [YUView](/tools/utilities/yuview/) works, or extract a frame: ```sh ffmpeg -i x264-crf30.mkv -vf "select=eq(n\,1234)" -frames:v 1 x264-crf30-1234.png ``` Look for banding, blocking, ringing, smeared texture, lost grain, colour shifts and flicker, and compare blind if you can. ## Keep it reproducible Record encoder versions and commands, the source and clip timestamps, and each metric tool with its settings (VMAF model, Butteraugli norm and intensity target, CVVDP display model). Otherwise nobody, including you later, can tell whether a difference came from the encoder or from the measurement.