Panoptes targets to achieve photorealistic rendering of large dynamic scenery and to allow multiple users to consume immersive content from arbitrary viewpoints (called 6 degrees of freedom, or 6DoF). This requires capturing and processing the plenoptic function describing real scenes as a light field (LF), i.e., the entire set of incident light rays from any angle and at every point in space. The Panoptes project has the ambition to challenge the scientific and technological locks preventing the deployment of immersive wide baseline 6DoF visual content by designing a canonical light field representation and associated signal processing techniques that can be utilized throughout an end-to-end light field processing chain. The addressed research topics include light field acquisition, light field representations and modelling, and view synthesis (LF rendering). The employed methodology spans novel multidimensional signal processing methods and machine learning techniques. Panoptes will fundamentally advance the theory of light field acquisition, processing, representation, modelling, and rendering and contribute to the theoretical foundations for a new era in immersive visual communications.