Knowledge-based semantic enrichment in the context of deep learning

Suggested by: Dirk Tiede, Felix Kröber

Short description: 

Deep learning on remote sensing imagery has gained importance in recent years. Especially under the common supervised setting, large quantities of images with the spectral values of the various bands/channels are used as input data. These strongly data-driven learning approaches are opposed by the heritage of knowledge-based approaches, in which models are designed manually on the basis of expert knowledge. In this domain, semantic enrichment of the input data enables categorisation of the spectral reflectances and thus makes it possible to analyse and interpret them. Such a categorisation via the Spectral Image Automated Mapper (SIAM) forms the basis for models operating on the Austrian data cube sen2cube, for example. So far, there is little research on the interface between classical knowledge-based and data-driven approaches, combining the advantages of both methods. 

The aim of this work is to leverage semantic enrichment for training deep learners on remote sensing imagery. The research question is to what extent automated semantic enrichment of images increases the generalisability of deep learners and reduces the amount of required training samples. Different approaches to the integration of semantically enriched data can be investigated - e.g. the use of semantically enriched data instead of the original reflectances as input for deep learners or the pre-training or conditioning of deep learners by means of semantic enrichment. Standard tasks such as image classification or semantic segmentation will be targeted. Possible applications of the models are e.g. land use classifications but other applications can be considered as well. An essential step of this work is the creation of a benchmark dataset consisting of Sentinel-2 satellite data, its semantic enrichment realised with SIAM and corresponding label information. 

Suggested reading:

Cheng,G., Xie,X., Han,J., Guo,L., & Xia,G.S. (2020). Remote Sensing Image Scene Classification Meets Deep Learning: Challenges, Methods, Benchmarks, and Opportunities. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 13, 3735–3756. https://doi.org/10.1109/JSTARS.2020.3005403  

Minaee, S., Boykov, Y., Porikli, F., Plaza, A., Kehtarnavaz, N., & Terzopoulos, D. (2020). Image Segmentation Using Deep Learning: A Survey. http://arxiv.org/pdf/2001.05566v5  

Augustin, H., Sudmanns, M., Tiede, D., Lang, S., & Baraldi, A. (2019). Semantic Earth observation data cubes. Data, 4(3), 102. https://www.mdpi.com/2306-5729/4/3/102 

Start/finish: ASAP

Prerequisites/qualifications: 

• Basic familiarity with deep learning techniques for imagery 
• Understanding of remote sensing information extraction 
• Python programming skills (esp. deep learning frameworks such as pytorch)