Carefully engineered radiofrequency (RF) pulses play a key role in a number of systems such as mobile phone, radar and magnetic resonance imaging. The design of an RF waveform, however, is often posed as an inverse problem with no general solution. As a result, various design methods, each with a specific purpose, have been developed on the basis of the intuition of human experts. In this work we propose an artificial intelligence (AI)-powered RF pulse design framework, DeepRF, which utilizes the self-learning characteristics of deep reinforcement learning to generate a novel RF pulse. The effectiveness of DeepRF is demonstrated using four types of RF pulses that are commonly used. The DeepRF-designed pulses successfully satisfy the design criteria while reporting reduced energy. Analyses demonstrate the pulses utilize new mechanisms of magnetization manipulation, suggesting the potentials of DeepRF in discovering unseen design dimensions beyond human intuition. This work may lay the foundation for an emerging field of AI-driven RF waveform design.
This is a preview of subscription content
Subscribe to Journal
Get full journal access for 1 year
only 7,71 € per issue
All prices are NET prices.
VAT will be added later in the checkout.
Tax calculation will be finalised during checkout.
Get time limited or full article access on ReadCube.
All prices are NET prices.
All processed data shown in the figures and table are available at https://github.com/SNU-LIST/DeepRF/tree/master/data.
Silver, D. et al. Mastering the game of Go without human knowledge. Nature 550, 354–359 (2017).
Wu, Y. et al. Google’s neural machine translation system: bridging the gap between human and machine translation. Preprint at https://arxiv.org/abs/1609.08144 (2016).
Yu, J. et al. Generative image inpainting with contextual attention. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, 5505–5514 (IEEE, 2018).
Segler, M. H., Preuss, M. & Waller, M. P. Planning chemical syntheses with deep neural networks and symbolic AI. Nature 555, 604–610 (2018).
Runge, F., Stoll, D., Falkner, S. & Hutter, F. Learning to design RNA. In Proc. International Conference on Learning Representations (2019).
Wang, H., Yang, J., Lee, H.-S. & Han, S. Learning to design circuits. In Proc. Conference on Neural Information Processing Systems (2018).
Popova, M., Isayev, O. & Tropsha, A. Deep reinforcement learning for de novo drug design. Sci. Adv. 4, aap7885 (2018).
Sutton, R. S. & Barto, A. G. Reinforcement Learning: An Introduction (MIT Press, 2018).
Bell, M. R. Information theory and radar waveform design. IEEE Trans. Inf. Theory 39, 1578–1597 (1993).
Simpson, D. H., Chin, C. T. & Burns, P. N. Pulse inversion Doppler: a new method for detecting nonlinear echoes from microbubble contrast agents. IEEE Trans. Ultrason. Ferroelectr. Freq. Control 46, 372–382 (1999).
Tarantola, A. Inverse Problem Theory and Methods for Model Parameter Estimation (SIAM, 2005).
Bloch, F. Nuclear induction. Phys. Rev. 70, 460 (1946).
Pauly, J., Le Roux, P., Nishimura, D. & Macovski, A. Parameter relations for the Shinnar–Le Roux selective excitation pulse design algorithm. IEEE Trans. Med. Imaging 10, 53–65 (1991).
Abragam, A. The Principles of Nuclear Magnetism (Oxford Univ. Press, 1961).
Conolly, S., Nishimura, D. & Macovski, A. Optimal control solutions to the magnetic resonance selective excitation problem. IEEE Trans. Med. Imaging 5, 106–115 (1986).
Khaneja, N., Reiss, T., Kehlet, C., Schulte-Herbrüggen, T. & Glaser, S. J. Optimal control of coupled spin dynamics: design of NMR pulse sequences by gradient ascent algorithms. J. Magn. Reson. 172, 296–305 (2005).
Rosenfeld, D. & Zur, Y. Design of adiabatic selective pulses using optimal control theory. Magn. Reson. Med. 36, 401–409 (1996).
Rund, A., Aigner, C. S., Kunisch, K. & Stollberger, R. Magnetic resonance RF pulse design by optimal control with physical constraints. IEEE Trans. Med. Imaging 37, 461–472 (2017).
Rund, A., Aigner, C. S., Kunisch, K. & Stollberger, R. Simultaneous multislice refocusing via time optimal control. Magn. Reson. Med. 80, 1416–1428 (2018).
Xu, D., King, K. F., Zhu, Y., McKinnon, G. C. & Liang, Z. P. Designing multichannel, multidimensional, arbitrary flip angle RF pulses using an optimal control approach. Magn. Reson. Med. 59, 547–560 (2008).
Vinding, M. S., Maximov, I. I., Tošner, Z. & Nielsen, N. C. Fast numerical design of spatial-selective RF pulses in MRI using Krotov and quasi-Newton based optimal control methods. J. Chem. Phys. 137, 054203 (2012).
Vinding, M. S., Guérin, B., Vosegaard, T. & Nielsen, N. C. Local SAR, global SAR, and power‐constrained large‐flip‐angle pulses with optimal control and virtual observation points. Magn. Reson. Med. 77, 374–384 (2017).
Loecher, M., Magrath, P., Aliotta, E. & Ennis, D. B. Time‐optimized 4D phase contrast MRI with real‐time convex optimization of gradient waveforms and fast excitation methods. Magn. Reson. Med. 82, 213–224 (2019).
Shang, H. et al. Multiband RF pulses with improved performance via convex optimization. J. Magn. Reson. 262, 81–90 (2016).
Vinding, M. S., Aigner, C. S., Schmitter, S. & Lund, T. E. DeepControl: 2D RF pulses facilitating B1+ inhomogeneity and B0 off-resonance compensation in vivo at 7T. Magn. Reson. Med. 85, 3308–3317 (2021).
Vinding, M. S., Skyum, B., Sangill, R. & Lund, T. E. Ultrafast (milliseconds), multidimensional RF pulse design with deep learning. Magn. Reson. Med. 82, 586–599 (2019).
Mirfin, C., Glover, P. & Bowtell, R. Optimisation of parallel transmission radiofrequency pulses using neural networks. In Proc. 26th Annual Meeting of ISMRM (2018).
Zhang, Y. et al. Multi‐task convolutional neural network‐based design of radio frequency pulse and the accompanying gradients for magnetic resonance imaging. NMR Biomed. 34, e4443 (2021).
Goodfellow, I., Bengio, Y., Courville, A. & Bengio, Y. Deep Learning (MIT Press, 2016).
Silver, M. S., Joseph, R. & Hoult, D. Highly selective π2 and π pulse generation. J. Magn. Reson. 59, 347–351 (1984).
Garwood, M. & DelaBarre, L. The return of the frequency sweep: designing adiabatic pulses for contemporary NMR. J. Magn. Reson. 153, 155–177 (2001).
Tannús, A. & Garwood, M. Adiabatic pulses. NMR Biomed. 10, 423–434 (1997).
Henderson, P. et al. Deep reinforcement learning that matters. In Proc. AAAI Conference on Artificial Intelligence (2018).
Glorot, X. & Bengio, Y. Understanding the difficulty of training deep feedforward neural networks. In Proc. International Conference on Artificial Intelligence and Statistics 249–256 (2010).
Vinyals, O. et al. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature 575, 350–354 (2019).
Zhu, B., Liu, J. Z., Koonjoo, N., Rose, B. R. & Rosen, M. S. Automated pulse sequence generation (AUTOSEQ) using Bayesian reinforcement learning in an MRI physics simulation environment. In Proc. 26th Annual Meeting of ISMRM (2018).
Walker-Samuel, S. Using deep reinforcement learning to actively, adaptively and autonomously control of a simulated MRI scanner. In Proc. 27th Annual Meeting of ISMRM (2019).
David, Y. et al. Reinforcement learning for online undersampling pattern optimization. In Proc. 27th Annual Meeting of ISMRM (2019).
Pineda, L., Basu, S., Romero, A., Calandra, R. & Drozdzal, M. Active MR k-space sampling with reinforcement learning. Proc. International Conference on Medical Image Computing and Computer-Assisted Intervention (2020).
Bahadir, C. D., Wang, A. Q., Dalca, A. V. & Sabuncu, M. R. Deep-learning-based optimization of the under-sampling pattern in MRI. IEEE Trans. Comput. Imaging 6, 1139–1152 (2020).
Meyer, C. H., Pauly, J. M., Macovskiand, A. & Nishimura, D. G. Simultaneous spatial and spectral selective excitation. Magn. Reson. Med. 15, 287–304 (1990).
Yip, C. Y., Fessler, J. A. & Noll, D. C. Iterative RF pulse design for multidimensional, small‐tip‐angle selective excitation. Magn. Reson. Med. 54, 908–917 (2005).
Cho, K. et al. Learning phrase representations using RNN encoder–decoder for statistical machine translation. In Proc. Conference on Empirical Methods in Natural Language Processing (2014).
Schulman, J., Wolski, F., Dhariwal, P., Radford, A. & Klimov, O. Proximal policy optimization algorithms. Preprint at https://arxiv.org/abs/1707.06347 (2017).
Paszke, A. et al. Automatic differentiation in pytorch. In Proc. Conference on Neural Information Processing Systems (2017).
Kingma, D. P. & Ba, J. Adam: a method for stochastic optimization. In Proc. International Conference for Learning Representations (2015).
Matson, G. B. An integrated program for amplitude-modulated RF pulse generation and re-mapping with shaped gradients. Magn. Reson. Imaging 12, 1205–1225 (1994).
Martin, J. B. et al. SigPy.RF: comprehensive open-source RF pulse design tools for reproducible research. In Proc. 28th Annual Meeting of ISMRM (2020).
Stockmann, J. P. et al. A 32‐channel combined RF and B0 shim array for 3T brain imaging. Magn. Reson. Med. 75, 441–451 (2016).
Shin, D. et al. DeepRF: (v1.0) (Zenodo, 2021); https://doi.org/10.5281/zenodo.5529394
This work was supported by the National Research Foundation of Korea (NRF-2021R1A2B5B03002783), Samsung Research Funding and Incubation Center of Samsung Electronics (SRFC-IT1801-09), RadiSen, INMC and IOER at Seoul National University.
Two patents are disclosed (in the order of patent applicant, names of inventors, application number, status of application, specific aspect of manuscript covered in patent application). Seoul National University (D.S. and J.L.), US Patent No. 17/168,274, patent pending; see the 'RF refinement module' section in the Methods. Seoul National University (D.S. and J.L.), Korea Patent No. 10-2020-0106569, patent pending, see the 'RF refinement module' section in the Methods.
Peer review information Nature Machine Intelligence thanks Peder Larson and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Shin, D., Kim, Y., Oh, C. et al. Deep reinforcement learning-designed radiofrequency waveform in MRI. Nat Mach Intell 3, 985–994 (2021). https://doi.org/10.1038/s42256-021-00411-1