Psychovisual encoding

Psychovisual quality (also called perceptual, subjective or visual quality) is quality as the human visual system perceives it. Psychovisual options in an encoder exist because the encoder’s own idea of quality, a simple error measure, often disagrees with what people see.

How much care an encoder puts into psychovisual quality is a good sign of its maturity. It is a large part of why x264 and x265 are so well regarded.

Why encoders need psychovisual tuning

An encoder chooses between coding options with rate–distortion optimization: it picks whichever option gives the least distortion for the bits spent. Distortion is usually measured as squared error, the same thing PSNR measures. The x265 documentation explains the problem:

Left to its own devices, the encoder will make mode decisions based on a simple rate distortion formula, trading distortion for bitrate. This is generally effective except for the manner in which this distortion is measured. It tends to favor blurred reconstructed blocks over blocks which have wrong motion. The human eye generally prefers the wrong motion over the blur and thus x265 offers psycho-visual adjustments to the rate distortion algorithm.

The same applies to detail. Viewers prefer a block with texture of roughly the right energy (amount of detail), even if the texture is not exactly the source’s, over a smooth block that is numerically closer. A blurred picture looks lifeless and smeared; a picture with slightly wrong grain or texture usually looks like the original. Psychovisual options push the encoder towards preserving visual energy.

Fidelity versus appeal

Jon Sneyers draws a useful distinction in Cloudinary’s blog:

Fidelity in images is about visually preserving the original; appeal is about hiding the compression artifacts.

At high bitrates the goal is fidelity: keep the source’s detail and grain, which is where psychovisual options shine. At very low bitrates the encoder cannot keep the detail anyway, and a little blur often looks better than blocky or noisy artifacts. That is why psychovisual strength should drop as bitrate drops: x265’s documentation notes that “the lower the bitrate, the lower the optimal psycho-visual settings”, and too much at low bitrates produces ringing and motion judder.

Common psychovisual options

Psy-RD adds a cost to blocks whose visual energy does not match the source, so the encoder stops choosing blurry prediction modes just because they are cheap. In x264 it is --psy-rd <rd>:<trellis> (default 1.0:0.0); the second value is psy-trellis. In x265 it is --psy-rd (default 2.0).

Psy-RDOQ (x265 --psy-rdoq, default 0, or 1.0 at the slow presets) biases quantization towards keeping more non-zero coefficients, which preserves energy in general rather than the source’s energy specifically. x265’s documentation describes the pair this way:

--psy-rd will add an extra cost to reconstructed blocks which do not match the visual energy of the source block. […] psy-rdoq is less accurate than psy-rd, it is biasing towards energy in general while psy-rd biases towards the energy of the source image.

Adaptive quantization (AQ) varies the quantizer within a frame. Flat areas and smooth gradients show errors far more readily than busy texture, so AQ moves bits towards them, which prevents banding and blocking in skies, walls and dark scenes. x264 enables variance AQ by default (--aq-mode 1); --aq-mode 3 adds a bias towards dark scenes.

In SVT-AV1, --tune 0 selects visual-quality tuning (the default, --tune 1, targets PSNR), --enable-variance-boost is a variance-based AQ for low-contrast areas, and --ac-bias is an energy-preserving bias similar to x264 and x265’s psy-rd. Several of these came from the community SVT-AV1-PSY fork, which is now archived; its work continues in mainline and in forks such as SVT-AV1-HDR and SVT-AV1-Essential.

Psychovisual options and metrics

Most metrics penalize psychovisual optimizations. Retained grain or texture that is not pixel-identical to the source counts as error, while a smooth, slightly blurred block scores well. That is why x264’s --tune psnr and --tune ssim simply turn psy-rd off (--no-psy), and why an encode tuned for PSNR, SSIM or VMAF can score higher while looking worse.

Use metrics to narrow down settings and find problem scenes, then decide by watching. Some newer metrics such as SSIMULACRA2 and Butteraugli track perceived quality better than PSNR, SSIM or VMAF, but none of them fully captures the value of psychovisual tuning. The quality comparison guide covers how to use metrics well.