CS180 Project #4A: IMAGE WARPING and MOSAICING


Shoot the Pictures

Acquire the images to be used on the projects by fix the center of projection (COP) and rotate the camera while capturing photos.

p1_visual


Recover Homographies

First, establishing point correspondences by mouse clicking with the tool from project 3.

p1_visual

To recover the parameters of the transformation, note that the transformation is a homography: \(Hp=p'\), where H is a 3x3 matrix with 8 degrees:

\[ \begin{bmatrix} a & b & c \\ d & e & f \\ g & h & 1 \end{bmatrix} \begin{bmatrix} x \\ y \\ 1 \end{bmatrix} = \begin{bmatrix} wx' \\ wy' \\ w \end{bmatrix} \]

Expanding the matrix multiplication, the linear systems are:

\[ \begin{bmatrix} x_1 & y_1 & 1 & 0 & 0 & 0 & -x_1'x_1 & -y_1'x_1 \\ 0 & 0 & 0 & x_1 & y_1 & 1 & -x_1'y_1 & -y_1'y_1 \\ x_2 & y_2 & 1 & 0 & 0 & 0 & -x_2'x_2 & -y_2'x_2 \\ 0 & 0 & 0 & x_2 & y_2 & 1 & -x_2'y_2 & -y_2'y_2 \\ &&&&...&&& \end{bmatrix} \begin{bmatrix} a \\ b \\ c \\ d \\ e \\ f \\ g \\ h \end{bmatrix} = \begin{bmatrix} x_1' \\ y_1' \\ x_2' \\ y_2' \\ ... \end{bmatrix} \]


Warp the Images

To warp each image towards the reference image with homograph matrix, first, we compute the projected positions of the corners from Image 2 to determine the bounding box of the final warped image. Then, utilizing inverse warping and scipy.interpolate.griddata, we interpolate the pixel values from Image 2 to their new positions in the destination image.


Image Rectification

To create image rectification, first marked the position of the four corners on the image, and then define the position of a matched rectangle. Use warpImage to produce the result.

p1_visual
p1_visual


Blend the images into a mosaic

After image warping, the naive overwriting would lead to strong edge artifacts:

p1_visual

To use weight averaging, first compute the distance transform with cv2.distanceTransform, and then compute the weighted linear combination of the two images by using the distance transforms as weights:

p1_visual
p1_visual More results are shown below.
p1_visual
p1_visual


Interest Point Detection

Using sample code harris.py, detect and locate the Harris corner points in a grayscale image.

p1_visual


Adaptive Non-Maximal Suppression

From the MOPS paper, interest points are suppressed based on the corner strength \(f_{HM}\) , and only those that are a maximum in a neighbourhood of radius \(r\) pixels are retained. As the suppression radius decreases from infinity, interest points are added to the list. In practice, the minimum suppression radius \(r_i\) given by\(r_i = min_j |x_i - x_j|, s.t. f(x_i) < c_{robust} f(x_j), x_j \in \mathcal{I}\) is calculated for each corner, then the results of the defined number of interest points are returned from the ordered list.

The two upper images show interest points with the highest corner strength, while the lower two images show interest points selected with adaptive non-maximal suppression with \(c_{robust}=0.9, n_{ip}=500\):

p1_visual


Feature Descriptor Extraction

As the paper suggests, a \(8 × 8\) patch of pixels are sampled around the sub-pixel location of the interest point, using a spacing of \(s = 5\) pixels between samples. Then the descriptor vector is normalised so that the mean is 0 and the standard deviation is 1. Here is an example of feature descriptor:

p1_visual


Feature Matching

With the use of kd-tree, a ratio threshold of \(e_{1-NN}/e_{2-NN} \) is used to find correct matches by Lowe's technique, based on the assumption that the 1-NN in some image is a potential correct match, whilst the 2-NN in that same image is an incorrect match.

p1_visual


RANdom SAmple Consensus

Use 4-point RANSAC as described in class to compute a robust homography estimate and a set of inliers of matches. Within each iteraction, four feature pairs are selected at random, then the exact homography H is computed. The set of inliers is computed where \( \text{dist}(p'_i, H p_i) < \epsilon \). Over the iteractions, the largest set of inliers are kept, and least-squares H estimate on all of the inliers is re-computed.

p1_visual

The final mosaic is presented below, with the warping and blending process same as stated above.

p1_visual

More results are attached below.

p1_visual
p1_visual