Our Idea

What is the idea?

The central idea is to fuse LiDAR and camera data together to produce robust, 3D object detection (x, y, z, r, p, y) for use in self-driving applications. The purpose of using both LiDAR and camera data is because the camera data can provide strong textural understanding of where cars are located in the image, while the LiDAR data can provide highly-accurate depth information. However, the difficulty with working with LiDAR point cloud data is that they are large in size (~100MB per point cloud) and are unstructured (as opposed to image data that has constant pixel dimensions). We project features from the camera data onto the LiDAR data, which places all of the information into a common embedding shape, allowing us to easily fuse the data via concatenation. This fused data is then processed with a resnet backbone and a YOLO detection head. The reason we used YOLO was because our network is aiming at online applications and this makes YOLO’s speed more favorable over a small increase in accuracy.

Data Preprocessing

The RGB images come from the KITTI dataset at 1242x375 pixels. We resize each image to a 448x448 pixel square to fit the pre-trained ResNet image encoder and augment the images with color jitter and gaussian noise to prevent overfitting. The point cloud comes from the KITTI dataset as a binary file. We translate the binary into an Np x 4 tensor of (x,y,z,R) where Np is the number of points in the cloud. For data augmentation, we add random gaussian noise to the x, y, z, and R values of each point.

Point Cloud Embedding

The point cloud embedding process is as follows. We create an L x W x N grid, where L is the length, W is the width, and N is the max number of lidar points that a grid can store. If there are less points in the cell than N, we pad them with zero and note down the number of real points. If there are more than N points per cell, we randomly select N points from it to get an estimated distribution. The voxel grid is then unrolled into a P x N x D matrix, where P is L x W and D is 4. The D dimension stores the (x, y, z, r) tuple for each lidar point. Next we do point augmentation to go from D = 4 to D = 9. The 5 new pieces of information are x, y, z offset from the x, y, z mean of points in their corresponding cell, and the x and y offset from each cell center point. This is then passed through several linear layers to extract learned features, such that the result matrix is P x N x C, where C is 128 or 256. We max along the N dimension and reshape back to L x W x C, which is critical because our point cloud embedding thus preserves its spatial relationship.

Detection Head

The detection head at the end of our architecture was a reimplementation of YOLO. YOLO’s end-to-end training allows us to quickly generate 3D bounding boxes from our input data, along with confidence scores for each of them. Similar to 2D YOLO, we did not directly predict the raw values, because they would be numerically stable. For example, objects can be 2, 20, or 120 meters away from our vehicle, orders of magnitude different. This would cause a number of problems with YOLO’s output because it isn’t flexible or precise enough to handle such large variations in some variables, but only differences from 0 to 1 in other variables. Therefore, to create some similarity between the YOLO outputs, the YOLO head predicts the offset from each cell corner that is normalized to the cell height and width. Furthermore, instead of predicting the shape of the bbox, we predict the shape offset from an ‘average’ shape, these are known as anchor boxes. Together, these changes result in YOLO outputting numbers roughly between 0 and 1 for all of YOLO’s diverse outputs. Lastly, predicting the yaw angle in Euler angles is difficult due to wrap around, for example the loss function will consider 0.1 degree and 359.9 degree to be far apart when in reality they are almost identical. Therefore, we predict the real and imaginary components of the yaw angle with complex decomposition. For the loss function, we first create an obj mask to mark down which of the cells contains a ground truth, and a noobj mask which is an inverse of the obj mask. We use the obj mask to ensure that we are only penalizing the model for having incorrect object-related losses when there is a ground truth result in the given cell. The noobj loss is used to penalize false positives. We use Mean Square Error on each of the loss types (obj confidence, no obj confidence, coordinate, shape, and angle), and tune the weights of each of these losses.

Learning Rate

Our initial learning rate of 1e-2 seemed to work well for our training purposes, but when plotted on a logarithmic scale it became clear that our learning rate only functioned correctly for the first 10 or 20 epochs. Therefore, we made an adjustment to our method and added a step-based learning rate scheduler that cut the learning rate by a tenth every ten epochs, down to a minimum of 1e-6. After implementing this learning rate scheduler, our model trained excellently, continuing to reduce its loss even after running for 30 or more epochs. In total, our loss dropped from an initial value of 1 all the way down to 1e-4. This is particularly important for our application as localizing 3d bounding boxes with sufficient IoU is difficult to achieve or improve and our YOLO detection head has a complex loss that requires minimizing along multiple dimensions.

Why does your idea make sense intuitively?

This idea makes intuitive sense because it combines the strengths of different types of sensors. Lidars are useful for determining the precise 3D coordinates of objects in a scene, and cameras are good for classifying objects. Fusing these two sensors can provide robust classification and localization. However, fusing these two sensors requires a good representation of each. Cameras are straight forward, but lidar embeddings are still an area of active research. By creating a latent embedding space that is the same for each sensor we make the process of fusing the data rather straightforward.

How does it relate to prior work in the area?

We took the ideas of sensor fusion in 3D LiDAR-based datasets and applied new and improved object detection algorithms to see the performance. The prior work listed has provided various sensor fusion methods with various learning methods, we built off of various successes found in their work to combine into a novel system for 3D object detection.