Real-world Instance-specific Image Goal Navigation: Bridging the Domain Gaps with Contrastive Learning

Overview

Focused task in this study. (Top) The robot identifies the position of an object shown in a query image provided by a user's mobile phone. (Bottom left) Domain gap that the image quality significantly differs between the image taken by the user's mobile phone and the object image observed by the real robot. (Bottom right) Contrastive learning to align images of the same instance with different image quality in latent space.

Few-shot cross-quality instance-aware adaptation (CrossIA)

The overall diagram of the proposed system. The proposed system automatically collects object images from RGBD images and camera pose time series data collected by a robot that explored the environment in advance. The object image database stores images along with their corresponding ID and position coordinates (e.g., ”ID:1, [3.7, 0.8]”). The fine-tuning module fine-tunes the pre-trained image encoder using the collected object images. The navigation module identifies the position of the object identical to the given query image using a fine-tuned image encoder.

Data Collection Module

The Data Collection Module constructs a 3D semantic map from sequence data of RGBD images and camera poses collected by the robot moving in 3D space. First, RGB images collected by the robot exploring the environment are inputted into the Multi-Scale Stage Network for deblurring. Next, they are inputted into the FastSAM for image segmentation. Then 3D semantic map is constructed using the segmentation results, depth images, and camera poses. Next, this module collects object images using a constructed 3D semantic map. During this process, 2D segmentation masks images from the 3D semantic map by ray tracing. Ray tracing generates a mask image from an arbitrary camera pose by sending pseudo-rays in the depth direction from the camera and capturing the instance ID of the first collision on the 3D map. This process ensures the consistent generation of pseudo-labels for images of the same instance. Then, the generated mask images are transformed into bounding boxes (BBoxes), and the regions of the transformed BBoxes are extracted from the RGB images to collect object images.

Fine-tuning Module

This module fine-tunes a pre-trained image encoder by the contrastive task between low-quality images observed by the robot and few-shot high-quality images provided by the user. SimSiam, a negative-free contrastive learning method, is shown to be capable of learning with a small batch size. In contrast, contrastive learning methods require negative pairs learning with a huge batch size. Therefore, we utilize SimSiam for fine-tuning. Additionally, for training, the user needs to capture few-shot images of objects they want the robot to locate with a mobile device and provide them to the robot. This requires the user to instruct the robot, but since it involves only a few images, it could not be considered a significant burden for the user. In this study, we assume that the system has up to five high-quality images for each instance and evaluate the experiments accordingly. Furthermore, it is shown that by minimizing the loss of contrastive learning and the loss function of the linear classifier for a pre-trained image encoder through fine-tuning, the variance of feature vectors between images with the same label decreases. This characteristic leads to an improvement in the accuracy of image classification tasks.

Navigation Module

The Navigation Module locates the object identical to the given query image. This is conducted by leveraging images of objects collected by the Data Collection Module, the 3D semantic map, and feature vectors extracted from the fine-tuned image encoder. First, the fine-tuned image encoder is utilized to compute the feature vectors \( x_{obs} \) and \( x_{query} \) for the object images collected by the Data Collection Module and the query image, respectively. Using these feature vectors, the similarity \( s \) between the query image and the observed object images is calculated using the cosine similarity. Then, the instance with the highest similarity to the query image is identified. Next, the module identifies the centroid coordinates of the object on the 3D map that is most similar to the query image, and the robot moves to close it.