Problem Statement

Given three black & white images that are taken at slightly different angles, how do you align the images automatically to create a coherent color image. When stacked naively on top of each other, you will get the image on the left.

Results

The three different images are aligned together as channels of a color image. The borders are automatically cropped out to produce vivid looking photographs.

More Results

Below are the results of other images being reconstructed. The preview shows the original images, click on them to see the rectified ones. Offsets information are shown in (red x offset, red y offset, green x offset, green y offset) format. The blue images were used as anchors.

Harvesters

Offsets: (14, 124, 17, 61)

  • Details

Emir

Offsets: (44, 104, 24, 49)

  • Details

Icon

Offsets: (23, 90, 17, 41)

  • Details

Lady

Offsets: (-9, 120, -4, 58)

  • Details

Self Portrait

Offsets: (36, 176, 29, 79)

  • Details

Three Generations

Offsets: (10, 112, 13, 55)

  • Details

Train

Offsets: (31, 89, 5, 49)

  • Details

Turkmen

Offsets: (27, 115, 20, 57)

  • Details

Village

Offsets: (23, 138, 12, 65)

  • Details

Uniform

Offsets: (38, 84, 27, 34)

  • Details

Monastery

Offsets: (6, 87, 10, 29)

  • Details

Milan

Offsets: (24, 107, 13, 57)

  • Details

Methodology

Images are aligned by iterating through different pixel offset values and calculating their Normalized Cross-Correlation (NCC) to find the offsets that best align the three color images. The blue channel was chosen to be the anchor, and the red and green channels are processed individually. After the alignment, the 3 color channels are stacked to form a color image. Below is an example of an image being aligned with this simple method with offsets ((3, 12, 2, 5)).

The method above works fine for small images, but for high-resolution images, it is prohibitively slow. The solution is to create an image pyramid, whereby the large image is blurred and downsampled multiple times. The image pyramid is then used for a coarse-to-fine search, whereby the deepest image in the pyramid (smallest and most blurry) is used to get a rough estimate for the alignment offset before searching on the larger images. The figure below shows an example of an image pyramid.

Bells & Whistles

PyTorch Implementation

The algorithm was implemented in PyTorch instead of Numpy. This is purely to get me familiar with PyTorch, which will be used extensively in future assignments

Border Trimming

A border detection and trimming algorithm was implemented. The raw images contained either black or white borders, which can disrupt the alignment algorithm which relies on finding image similarity. Therefore, before processing the borders are detected and trimmed away. To find the borders, average pixels values along the rows and columns are calculated, and the borders are selected based on when a certain amount of consecutive pixel values are not within specified boundaries that represent complete black or white borders. See the figure below for the pre-processing trim.

Additionally, a post-processing trim is added to eliminate any remaining black or white borders and the pixels that have been offset during alignment. Otherwise, it will create color artifacts around the borders. See the figure below for the post-processing trim.