Zero-Shot Learning for Semantic Segmentation

Title Zero-Shot Learning for Semantic Segmentation
Summary Zero-Shot Learning for Semantic Segmentation
Keywords Deep Learning, Computer Vision, Semantic Segmentation

Prerequisites Excellent Programming Skills; Excellent Grasp of Deep Learning and Pytorch
Author Tiago Cortinhal
Supervisor Tiago Cortinhal, Eren Erdal Aksoy
Level Master
Status Open

Currently we are working of a GAN approach to be able to generate segmentation maps from a sensor modality to another (from/to LiDAR and RGB).

The currently available datasets are not perfectly aligned (SemanticKitti and Cityscape), to try to overcome this problem techniques like Zero-Shot Learning could be applied.

Zero-Shot Learning can be seen as a type of Domain Adaptation, where a given set of classes are used for training and we have another set of unseen classes to which we wish to have segmentation as well.

To be able to perform this, a kind of embedding of the classes need to be done, as to be able to adapt the pre-trained model to unseen classes by combining the embeddings of the learnt classes.

The project consists of using the pre-trained GAN generator model and apply Zero-Shot Learning.

 Research Questions:
   Can Zero-Shot Learning besides of learning unseen classes also improve the overall IoU (Intersection over Union)?
   Should Zero-Shot or N-Shot be applied in this scenario (N-Shot: N examples are showed during training)
   Compared with Domain transfer/finetuning techniques which could provide faster/better results?