The human brain excels when it comes to the efficient processing of multiple correlated
information streams. It continuously receives massive streams of sequential data from our
hearing, vision, touch and other senses. Nevertheless, it is capable to quickly process and learn
from these huge amounts of data, because it can efficiently abstract it into a more compact
representation and because it optimally exploits the correlations that exist between information
from its different senses. It is also very adaptive: the brain dynamically rewires itself all the time
and can learn to cope with sudden and lasting changes.
Unfortunately, artificial neural networks still cannot match the brain its performance when dealing
with multi-sensory information. Current approaches typically focus on specific combinations of
sensors and do not offer a generally applicable solution. They process the different sensor streams
in separation and only combine them at the highest levels of abstraction, while the human brain
appears to search for correlations already in early processing stages. Also, other recent
observations in the field of neuroscience involving dynamic adaptation remain unexplored for
applications in neural networks.
In my research, I will therefore draw inspiration from these insights to create a generic deep
architecture that can efficiently integrate information streams from multiple and noisy sensors,
while being able to adapt to changes in the sensor or the environment.