After AI models are trained and deployed, it is often hard to understand their results. The blooming field of explainable AI (XAI) seeks to make AI models understandable by using several statistical, mathematical, and computational techniques.These techniques lead to explanations of the model’s outputs. Philosophical analyses of XAI until now have focussed on problems other than explanation, despite it being a central concept in philosophy of science and in XAI. This project investigates the notion of explanation in the practice of XAI from the point of view of philosophy of science. The first step in the investigation is descriptive (how does explanation appear?). The aim of this step is to develop rigorous accounts of how explanation operates in different XAI techniques. The second step is prescriptive (how should explanation appear?), aimed at providing quality benchmarks for XAI-produced explanations. A creative combination of tools from philosophy of science and tools from XAI will be used to accomplish this descriptive and prescriptive investigation. Both disciplines stand to benefit. Philosophers learn how explanation works in a hitherto understudied and exciting domain (XAI), and XAI engineers benefit from the conceptual clarity brought by philosophical analysis. These benefits range from more precise benchmarks to better handles on challenges raised by the opacity of AI models, such as discrimination issues or the epistemic status of AI-driven scientific discoveries.