In the rapidly evolving landscape of healthcare innovation, artificial intelligence (AI) emerges as a transformative force with the potential to revolutionise diagnostics, treatment plans, and medical decision-making. The Medical Device regulation considers AI under the umbrella of Software as a Medical Device, failing to encapsulate the nuanced disparities that distinguish AI from traditional medical devices. As a consequence, at the day of today, the clinical trials in place are insufficient and dangerously unfit to test medical AI. Consequently, AI developers are stuck in overdemanding trials that might not be sustainable because of the different features of medical AI, and doctors are simultaneously dealing with systems they do not trust. Building up from a challenging and exciting interdisciplinary collaborative effort, the goal of this project is drafting a recommendation for an ad hoc legal framework for Medical AI clinical trials that balances clinical accuracy and efficiency with innovation. Succeeding in this intellectual challenge would signify setting a new benchmark for trustworthy AI in healthcare, positioning Belgium and the EU as pioneers in shaping the future integration of AI in healthcare. By implementing regulatory measures to ensure the trustworthiness of medical AI systems, we spark contemplation on whether medical AI could evolve into a standard of care, thereby grounding a duty to integrate and use AI into healthcare practices.