Residential and commercial buildings consume 40% of the total energy, 70% of the total electricity and are responsible for 41% of the greenhouse gas emissions. Therefore, it is important to eliminate any energy wastage in buildings. The dwellers of these buildings, i.e. the knowledge workers, spend 86% of their time indoors, so this building environment has a large impact on their well-being. The management of these buildings is today done by model-based solutions with deterministic control algorithms. First attempts using machine learning have been made, however, they optimized specific components of buildings individually. A holistic approach that optimizes different building components simultaneously is needed. To do so, 4 research objectives are defined: 1) Design of BIM embedding to use BIM in machine learning to enable hybrid models for optimized building management; 2) Design of occupancy detection model that combines environmental sensor information and BIM; 3) Design of context-aware anomaly detection system based on BIM and context information to improve energy losses as a result of these anomalies and faulty sensors. Depending on context information the notion of what is anomalous also changes; and 4) Based on the detected occupancy, anomalies and collected user feedback, rules will be automatically derived to optimize the building for energy and well-being of its dwellers.