Keying Difficult Footage

Primatte Keyer has a host of tools to deal with compression, spill and color correction but sometimes you are given source footage that is tremendously hard to key. Noise and lack of dynamic range can cause holes to appear in the matte or maybe an edge looks like it fluttering, which ruins the composite effect.

This page discusses how to deal with problem footage and get great looking output, even for low light shots. All the steps below use After Effects as the host application. While some of the functions covered can be done in other applications, we feel that After Effects is the best environment to work with tough footage.

We will talk about four main approaches to help with difficult keys:

- Smoothing the color channels
- Pre-keying with color correction
- Garbage matte prior to using Primatte
- Proper sampling in Primatte to avoid edge problems

Smoothing the color channels

Primatte features a Deartifacting pass that will create nice sharp edges even with sub-sampled footage from DV, HDV or MPEG-2 source formats. Sometimes this deartifacting will preserve edges but not generate a great key because the color information in the image has lots of noise.

Below left, an example of a poorly lit green screen example. This footage needs smoothing to help eliminate noise in the color channels. There is a large seam in the background and a lot of blocky noise artifacts in the green screen. Let's tackle the green screen smoothing first.

Below right, the Red Channel from this same image, so you can more easily see the blocking and the noise. Notice the aliasing along the shoulder and neck and the large blocks in the background.

In After Effects, you can easily smooth the color channels by using a Channel Combiner effect and a simple Channel Blur. This will allow you to smooth only the chroma channels in the image.

First, apply Channel Combiner set to RGB to YUV. Next, apply a Channel Blur with Blue and Green channels set 2.0 pixel values. (These are the correct settings for HDV or footage captures with a 4:2:2 codec. If you are using DV footage, then set the Channel Blur to 3.0 instead of 2.0.) Then apply a second Channel Combiner set to YUV to RGB. This will essentially smooth just the chroma channels and reduce the noise, which makes a nicer looking matte.

Below left, the original mage. Below right, the smoothed result after using these plug-ins. Full credit to Mark Christiansen, author of After Effects CS3 Studio Techniques, for this tip.

Pre-keying with color correction

The color values in the footage have a big impact on the final keyed output. Footage that is too dark or lack enough separation between the foreground colors and the background can give really bad results.

Below, our uncorrected shot again. Clearly this is very dark.

Let's apply some Exposure correction and Hue/Saturation changes with Magic Bullet Colorista and the After Effects Hue/Saturation tool. We will push Gain to blue and bring up the Exposure by two stops. The Hue and Saturation controls are set to boost the green and make it more pure. It useful to do these changes with the project set to 32-bit mode so the color correction has the greatest precision.

Below, the footage is greatly brightened, which will produce a better key.

Garbage matte prior to using Primatte

Our original footage has noticeably dark corners, and applying a simple garbage mask will remove those edges. A 'garbage mask' is any shape that you animate to cut out parts of the footage. It is called 'garbage' because you are cutting out the visual garbage but not trying to create a perfect fitting mask.

Removing the edges has the added benefit of not having to include those colors in the key that Primatte Keyer builds. By leaving those colors out, Primatte Keyer generates a more accurate matte. You don't have to create tight outlines, just move a couple of shapes around the subject. Animate by creating keyframes every 10-12 frames. Don't worry about animating points, just move shapes. You should be able to matte the footage in just a few minutes in After Effects.

Below, the original footage. Notice the darker edges and folds in the outside edges of the screen area. Cutting these out manually will allow a better key.

Below, the footage with two garbage mattes. Using two mask shapes makes quicker work because you only have to move the shapes, rather than the individual points on the mask shapes. We created these outlines for 30 seconds of green screen footage in about 10 minutes. If you use this footage as input source, the alpha channel created by the After Effects mask will get added to the matte that Primatte Keyer creates.

Proper sampling in Primatte to avoid edge problems

Our source footage has a bright greenish-yellow spot on the card fixed to the hat. If you try to include this in the key area, you will create a key with too much edge spill that looks rough and unconvincing. If you leave this sample out, you can fix it with Primatte's Alpha Cleaner and get a better key.

Below left, our key shows a dark area in the hat, which will appear as a big hole in matte. Below right, we turned on Alpha Cleaner with Plug Holes mode and did some control tweaking.

Here are the settings we used to remove the big specks but keep the edge untouched. Plug Holes even retained the fine transparency along the left side of the image, which keeps the hair in the shot looking realistic.