Taichi Sakaguchi, Akira Taniguchi, Yoshinobu Hagiwara, Lotfi El Hafi, Shoichi Hasegawa, Tadahiro Taniguchi

Abstract

Improving instance-specific image goal navigation (InstanceImageNav), which locates the identical object in a real-world environment from a query image, is essential for robotic systems to assist users in finding desired objects. The challenge lies in the domain gap between low-quality images observed by the moving robot, characterized by motion blur and low-resolution, and high-quality query images provided by the user. Such domain gaps could significantly reduce the task success rate but have not been the focus of previous work. To address this, we propose a novel method called Few-shot Cross-quality Instance-aware Adaptation (CrossIA), which employs contrastive learning with an instance classifier to align features between massive low- and few high-quality images. This approach effectively reduces the domain gap by bringing the latent representations of cross-quality images closer on an instance basis. Additionally, the system integrates an object image collection with a pre-trained deblurring model to enhance the observed image quality. Our method fine-tunes the SimSiam model, pre-trained on ImageNet, using CrossIA. We evaluated our method's effectiveness through an InstanceImageNav task with 20 different types of instances, where the robot identifies the same instance in a real-world environment as a high-quality query image. Our experiments showed that our method improves the task success rate by up to three times compared to the baseline, a conventional approach based on SuperGlue. These findings highlight the potential of leveraging contrastive learning and image enhancement techniques to bridge the domain gap and improve object localization in robotic applications.

Approach

As shown in Figure 1, the proposed system is divided into three main modules. Firstly, the Data Collection Module constructs the 3D semantic map of the environment and then collects object images. Secondly, the Fine-tuning Module fine-tunes the pre-trained models using the collected object images and few-shot high-quality images provided by the user through contrastive learning. Our contrastive learning is called Few-shot Cross-quality Instance-aware Adaptation (CrossIA) . Lastly, the Navigation Module leverages the fine-tuned model and the semantic map to locate objects identical to the query image.

Figure 1: Overview of our method

Data Collection Module

The Data Collection Module constructs a 3D semantic map from sequence data of RGBD images and camera poses collected by the robot moving in 3D space. First, RGB images collected by the robot exploring the environment are inputted into the Multi-Scale Stage Network for deblurring. Next, they are inputted into the FastSAM for image segmentation. Then 3D semantic map is constructed using the segmentation results, depth images, and camera poses. Next, this module collects object images using a constructed 3D semantic map. During this process, 2D segmentation masks images from the 3D semantic map by ray tracing. Ray tracing generates a mask image from an arbitrary camera pose by sending pseudo-rays in the depth direction from the camera and capturing the instance ID of the first collision on the 3D map. This process ensures the consistent generation of pseudo-labels for images of the same instance. Then, the generated mask images are transformed into bounding boxes (BBoxes), and the regions of the transformed BBoxes are extracted from the RGB images to collect object images.

Fine-tuning Module

This module fine-tunes a pre-trained image encoder by the contrastive task between low-quality images observed by the robot and few-shot high-quality images provided by the user. SimSiam, a negative free contrastive learning method, is shown to be capable of learning with small batch size. In contrast, contrastive learning methods require negative pairs learning with a huge batch size. Therefore, we utilize SimSiam for fine-tuning. Additionally, for training, the user needs to capture few-shot images of objects they want the robot to locate with a mobile device and provide them to the robot. This requires the user to instruct the robot, but since it involves only a few images, it could not be considered a significant burden for the user. In this study, we assume that the system has up to five high-quality images for each instance and evaluate the experiments accordingly. Furthermore, it is shown that by minimizing the loss of contrastive learning and the loss function of the linear classifier for a pre-trained image encoder through fine-tuning, the variance of feature vectors between images with the same label decreases. This characteristic leads to an improvement in the accuracy of image classification tasks.

Figure 2: Fine-tuning module

Therefore, we also conduct fine-tuning of the image encoder by adding a linear classifier to SimSiam as shown in Figure 2. The label of instances required for calculating the loss of the linear classifier is obtained using pseudo-labels automatically generated by the data collection module. \( p_i^l \) and \( z_i^{l} \) are the outputs of SimSiam's projector and predictor of the object image \( x_i^l \) observed by the robot, respectively. \( p_i^{\prime l} \) and \( z_i^{\prime l} \) are the outputs of SimSiam's projector and predictor of the object image \( x_i^{\prime l} \) where \( x_i^{l} \) and \( x_i^{\prime l} \) are different images of the same object generated by data augmentation. \( y_{i, pred}^l \) is the prediction result when \( x_i^l \) is classified. \( p_i^h \) and \( z_i^h \) are the outputs of projector and predictor, respectively, when the high-quality image \( x_i^h \) is input to SimSiam, and \( y_{i, pred} ^h \) is the prediction result when \( x_i^h \) is classified. Therefore, fine-tuning the image encoder involves contrastive learning between high-quality and low-quality images, and the contrastive learning between images the robot observed. In this paper, \( \{(x_i^l, x_i^h, y_{i, true})\}_{i =1}^{N_c} \) is the set of high-quality image, low-quality image and their label and \( \{(x_i^l, y_{i_{true}})\}_{i=1}^{N_r} \) is the set of images and their labels of low-quality image.

Navigation Module

The Navigation Module locates the object identical to the given query image. This is conducted by leveraging images of objects collected by the Data Collection Module, the 3D semantic map, and feature vectors extracted from the fine-tuned image encoder. First, the fine-tuned image encoder is utilized to compute the feature vectors \( x_{obs} \) and \( x_{query} \) for the object images collected by the Data Collection Module and the query image, respectively. Using these feature vectors, the similarity \( s \) between the query image and the observed object images is calculated using the cosine similarity. Then, the instance with the highest similarity to the query image is identified. Next, the module identifies the centroid coordinates of the object on the 3D map that is most similar to the query image, and the robot moves to close it.