Technology that translates brain activity into speech could be transformative for patients deprived of it. Studies have reported impressive results when decoding intracranial recordings, called electrocorticography (ECoG), in response to performed or perceived speech. However, to serve the needs of the speech impaired, one should aim for non-vocalized speech, also called imagined or inner speech. Despite intensive efforts, this remains an elusive challenge. Several observations can be made which we believe prevent progress. Current speech decoders are developed and tested offline, per subject and speech mode, as the acutely implanted ECoG electrodes serve a clinical need and time for experiments is limited. We propose a new multiway decoder that accounts for the nonlinear relation between ECoG and speech features, that is fast to respond, and that facilitates the transfer of model knowledge across speech modes and subjects. We will then use the decoder, prior developed based on model transfer, to provide real-time audible feedback to the subject when attempting imagined speech, which we hypothesize to be crucial for the subject to master imagined speech production. When successful, this project can contribute to future developments based on chronic brain implants aimed at improving the quality of life of speech-impaired patients.