raw - What are the pros and cons of different Bayer demosaicing algorithms?

Bayer sensors use a pattern of red, green, and blue pixels, and merge those together into a final color image, with one pixel for every single color sensor. This could be done through a "naive" blending of neighbor sensors, but I've heard of more complicated approaches with names like AHD, HPHD, and AMaZE.

What are these other approaches, and what advantages do they bring? Do they have weaknesses beyond compute complexity?

I imagine that the approach used for in-camera JPEG is more tightly guarded, but clearly a lot of research and development goes into this area. Does the limited processing power available in-camera force any compromises in this area?

Answer

I was surprised a few months ago to find that my machine vision camera SDK used nearest neighbor "interpolation" in its built-in bayer function. It is the fastest, but worst type, giving hard edges, especially when you start doing math on the image channels for colour constancy or brightness invariance. I found this review of algorithms:

http://www.arl.army.mil/arlreports/2010/ARL-TR-5061.pdf

The next step up is bilinear and bicubic interpolations which can be computed quite fast because they amount just to convolution kernels. These give coloured saw-teeth on slanted edges - bilinear more than bicubic.

Can be seen in this paper, and with quantified quality data on 5 different algorithms:

http://research.microsoft.com/en-us/um/people/lhe/papers/icassp04.demosaicing.pdf

This is why they made edge direction based interpolations. However, these treat green as a more "important channel" (since it has the best resolution and account for most of our visual sensitivity and our eyes' resolution). And then they create blue and red as a function of the green channel, in a hue preserving fashion. This in turn makes green channel high frequency content more prone to errors. Complexity is higher as they have to detect what's going on and require multiple passes. Moire and maizes are common artifacts from these types of interpolation.

Here they show examples of Adaptive Homogeneity Demosaicing and bilinear versions with and without hue preserving and edge preserving addons:

http://math.auburn.edu/~kilgota/ahd_bayer.pdf

That paper favours AHD and doesn't show the negative part. On this page you can see the different pattern artifacts from Adaptive Homogeneity Demosaicing, Patterned Pixel Grouping, and Variable Number of Gradients (hover mouse over the names):

http://www.ruevski.com/rawhistogram/40D_Demosaicing/40D_DemosaicingArtifacts.html

In summary there are a number of assumptions employed in these algorithms, and artifacts occur when the assumption doesn't hold:

• Per channel smoothness. If the nearest neighbors are not the same, make the transition smooth. Artifact: saws/zippers, softness


• Brightness constancy with directed edges (bilinear direction). Artifacts: high frequency texture moire, colour fringing


• Hue constancy. if in a neighborhood the hue is the same, so if one channel changes the others have to follow. Artifacts: colour zippers on colour edges



• Constancy can be predicted from the greens. Artifact: mazes