Huge data sets are generated at increasing pace across various disciplines. It is becoming more and more common to acquire the information about the same phenomenon with a multitude of different sensors. For example, in medical diagnostics various imaging modalities are often combined (e.g., X-ray CT and magnetic resonance images) as well as totally different types of data (like one-dimensional electrocardiograms and four-dimensional sequences of volumetric ultrasound data). In remote sensing it is common to combine information from hyperspectral images, comprising several hundreds of bands, with optical and radar images of much higher spatial resolution, and in digital painting analysis optical images are commonly accompanied by radiographs, infrared and multispectral data. In all these and similar cases, the available heterogeneous data contain correlated versions of the same physical world, although each adding some unique piece of information. Extracting the relevant information from this wealth of data becomes a major challenge, which is also addressed in this project. We aim at developing new data representations and inference techniques for the processing of heterogeneous image data. As case studies, we use the application domains of digital painting analysis and remote sensing. Both share common practices in terms of multimodal data acquisition and similar research challenges in terms of content classification and separation, albeit each with its own specificities