Antialiasing

Aliasing is the stair-step look on diagonal and curved edges, and the crawling or shimmering of thin lines in motion. It is most common in animation and CG that was rendered or drawn without antialiasing, and in anything that was resized badly, sharpened, or upscaled from a lower resolution with a poor kernel. Live-action camera footage rarely needs it.

Look at the edge before you act. Horizontal teeth that appear only around moving objects are combing, which needs deinterlacing or IVTC, not antialiasing. Halos along an edge are ringing. Stair-steps that are present in a still frame are aliasing.

When to filter

Antialiasing works by interpolating edges from the pixels around them, so it softens and can bend fine detail. Use it on lines that visibly alias, ideally only on the scenes that need it, and keep it off everything else. If the aliasing came from an earlier resize, fixing the resize (or descaling, if the source was upscaled from a known resolution) is better than smoothing the damage afterwards.

FFmpeg has no dedicated antialiasing filter. A proper scaler (scale=...:flags=bicubic or lanczos) avoids creating aliasing when you downscale, but repairing existing aliasing is a VapourSynth job.

Antialiasing with vs-jetpack

vsaa wraps the edge-directed interpolators behind one interface. Each has an antialias method:

from vsaa import NNEDI3, EEDI3, SangNom

aa = NNEDI3().antialias(clip)   # the usual first try
aa = EEDI3().antialias(clip)    # slower, more parameters to tune
aa = SangNom().antialias(clip)  # the other classic option

All of them process both directions by default. To restrict one to vertical or horizontal edges, pass direction=NNEDI3.AADirection.VERTICAL (or HORIZONTAL).

Running the interpolator over the whole frame is blunt, because it touches everything that isn’t an edge too. based_aa supersamples the clip, antialiases it, scales it back, and merges the result into the original through an edge mask, so only edges change:

from vsaa import based_aa

aa = based_aa(clip)

The default supersampler is ArtCNN, a neural-network upscaler that needs its ONNX models (vsscale onnx download ArtCNN) and an inference backend that works on your machine. Without those, use a CPU supersampler:

from vsaa import NNEDI3, based_aa

aa = based_aa(clip, supersampler=NNEDI3())

rfactor (default 2.0) sets the supersampling factor and mask_thr the edge-mask threshold. Lower mask_thr to catch weaker edges, raise it to protect more of the picture. show_mask=True displays the mask so you can see what is being touched.

Checking the result

View an affected edge at 1:1 and then enlarged, comparing against the original at the same frame. Good antialiasing removes the stair-steps without thinning or blurring the line. Watch for lines that go soft, small text and thin details that vanish, and edges that flicker between frames because the mask switches on and off. If it only needs to apply to a few shots, trim those shots out, filter them, and splice them back.