The emergence of robotic body augmentation provides exciting innovations that will revolutionize the fields of robotics, human–machine interaction and wearable electronics. Although augmentative devices such as extra robotic arms and fingers are informed by restorative technologies in many ways, they also introduce unique challenges for bidirectional human–machine collaboration. Can humans adapt and learn to operate a new robotic limb collaboratively with their biological limbs, without restricting other physical abilities? To successfully achieve robotic body augmentation, we need to ensure that, by giving a user an additional (artificial) limb, we are not trading off the functionalities of an existing (biological) one. Here, we introduce the ‘neural resource allocation problem’ and discuss how to allow the effective voluntary control of augmentative devices without compromising control of the biological body. In reviewing the relevant literature on extra robotic fingers and arms, we critically assess the range of potential solutions available for this neural resource allocation problem. For this purpose, we combine multiple perspectives from engineering and neuroscience with considerations including human–machine interaction, sensory–motor integration, ethics and law. In summary, we aim to define common foundations and operating principles for the successful implementation of robotic body augmentation.
This is a preview of subscription content
Subscribe to Journal
Get full journal access for 1 year
only 7,71 € per issue
All prices are NET prices.
VAT will be added later in the checkout.
Tax calculation will be finalised during checkout.
Get time limited or full article access on ReadCube.
All prices are NET prices.
Bergamasco, M. & Herr, H. in Springer Handbook of Robotics (eds Siciliano, B. & Khatib, O.) 1875–1906 (Springer International Publishing, 2016).
Windrich, M., Grimmer, M., Christ, O., Rinderknecht, S. & Beckerle, P. Active lower limb prosthetics: a systematic review of design issues and solutions. Biomed. Eng. Online 15, 140 (2016).
Mendez, V., Iberite, F., Shokur, S. & Micera, S. Current solutions and future trends of robotic prosthetic hands. Annu. Rev. Control Robot. Auton. Syst. 4, 595–627 (2021).
Dollar, A. M. & Herr, H. Lower extremity exoskeletons and active orthoses: challenges and state-of-the-art. IEEE Trans. Robot. 24, 144–158 (2008).
Guggenheim, J., Hoffman, R., Song, H. & Asada, H. H. Leveraging the human operator in the design and control of supernumerary robotic limbs. IEEE Robot. Autom. Lett. 5, 2177–2184 (2020).
Salvietti, G. et al. Compensating hand function in chronic stroke patients through the robotic sixth finger. IEEE Trans. Neural Syst. Rehabil. Eng. 25, 142–150 (2017).
Kieliba, P., Clode, D., Maimon-Mor, R. O. & Makin, T. R. Robotic hand augmentation drives changes in neural body representation. Sci. Robot. 6, eabd7935 (2021).
Xie, H., Mitsuhashi, K. & Torii, T. Augmenting human with a tail. In Proc. 10th Augmented Human International Conference 2019 Vol. 35, 1–7 (ACM, 2019); https://doi.org/10.1145/3311823.3311847
Parietti, F. & Asada, H. H. Independent, voluntary control of extra robotic limbs. In Proc. 2017 IEEE International Conference on Robotics and Automation ICRA 5954–5961 (IEEE, 2017).
Di Pino, G., Maravita, A., Zollo, L., Guglielmelli, E. & Di Lazzaro, V. Augmentation-related brain plasticity. Front. Syst. Neurosci. 8, 109 (2014).
Makin, T., de Vigneromont, F. & Micera, S. Soft embodiment for engineering artificial limbs. Trends Cogn. Sci. 24, 965–968 (2020).
Mehring, C. et al. Augmented manipulation ability in humans with six-fingered hands. Nat. Commun. 10, 2401 (2019).
Penaloza, C. I. & Nishio, S. BMI control of a third arm for multitasking. Sci. Robot. 3, eaat1228 (2018).
Bernshtein, N. A. The Co-ordination and Regulation of Movements (Pergamon, 1967); http://books.google.com/books?id=F9dqAAAAMAAJ
Lisini Baldi, T. et al. Exploiting implicit kinematic kernel for controlling a wearable robotic extra-finger. Preprint at https://arxiv.org/pdf/2012.03600.pdf (2020).
Baldi, T. L., Farina, F., Garulli, A., Giannitrapani, A. & Prattichizzo, D. Upper body pose estimation using wearable inertial sensors and multiplicative Kalman filter. IEEE Sens. J. 20, 492–500 (2020).
Hussain, I., Meli, L., Pacchierotti, C., Salvietti, G. & Prattichizzo, D. Vibrotactile haptic feedback for intuitive control of robotic extra fingers. Proc. IEEE World Haptics Conference 394–399 (IEEE, 2015); https://doi.org/10.1109/WHC.2015.7177744
d’Avella, A., Saltiel, P. & Bizzi, E. Combinations of muscle synergies in the construction of a natural motor behavior. Nat. Neurosci. 6, 300–308 (2003).
Bräcklein, M., Ibáñez, J., Barsakcioglu, D. & Farina, D. Towards human motor augmentation by voluntary decoupling beta activity in the neural drive to muscle and force production. J. Neural Eng. 18, 016001 (2021).
Aoyama, T., Shikida, H., Schatz, R. & Hasegawa, Y. Operational learning with sensory feedback for controlling a robotic thumb using the posterior auricular muscle. Adv. Robot. 33, 243–253 (2019).
Guggenheim, J., Parietti, F., Flash, T. & Asada, H. Laying the groundwork for intra-robotic-natural limb coordination: is fully manual control viable?. ACM Trans. Hum. Robot. Interact. 9, 18 (2020).
Borzelli, D., Cesqui, B., Berger, D. J., Burdet, E. & d’Avella, A. Muscle patterns underlying voluntary modulation of co-contraction. PLoS ONE 13, e0205911 (2018).
Berger, D. J., Gentner, R., Edmunds, T., Pai, D. K. & d’Avella, A. Differences in adaptation rates after virtual surgeries provide direct evidence for modularity. J. Neurosci. 33, 12384–12394 (2013).
Gurgone, S. et al. Simultaneous control of natural and extra degrees-of-freedom by isometric force and EMG null space activation. In Converging Clinical and Engineering Research on Neurorehabilitation IV. Proceedings of the Fifth International Conference on Neurorehabilitation IV (INCR2020) (eds. Torricelli, D., Akay, M. & Pons, J. L.) 863–868 (2020); https://www.springer.com/gp/book/9783030703158
Lebedev, M. A. & Nicolelis, M. A. L. Brain–machine interfaces: from basic science to neuroprostheses and neurorehabilitation. Physiol. Rev. 97, 767–837 (2017).
Schwartz, A. B. Cortical neural prosthetics. Annu. Rev. Neurosci. 27, 487–507 (2004).
Omrani, M., Kaufman, M. T., Hatsopoulos, N. G. & Cheney, P. D. Perspectives on classical controversies about the motor cortex. J. Neurophysiol. 118, 1828–1848 (2017).
Fetz, E. E. Operant conditioning of cortical unit activity. Science 163, 955–958 (1969).
Carmena, J. M. et al. Learning to control a brain–machine interface for reaching and grasping by primates. PLoS Biol. 1, e42 (2003).
Ifft, P. J., Shokur, S., Li, Z., Lebedev, M. A. & Nicolelis, M. A. L. A brain-machine interface enables bimanual arm movements in monkeys. Sci. Transl. Med. 5, 210ra154 (2013).
Bashford, L. et al. Concurrent control of a brain–computer interface and natural overt movements. J. Neural Eng. 15, 066021 (2018).
Artoni, F., Delorme, A. & Makeig, S. Applying dimension reduction to EEG data by principal component analysis reduces the quality of its subsequent independent component decomposition. NeuroImage 175, 176–187 (2018).
Zhuang, K. Z. et al. Shared human–robot proportional control of a dexterous myoelectric prosthesis. Nat. Mach. Intell. 1, 400–411 (2019).
Johansson, R. S. & Flanagan, J. R. Coding and use of tactile signals from the fingertips in object manipulation tasks. Nat. Rev. Neurosci. 10, 345–359 (2009).
Bensmaia, S. J., Tyler, D. J. & Micera, S. Restoration of sensory information via bionic hands. Nat. Biomed. Eng. https://doi.org/10.1038/s41551-020-00630-8 (2020).
Clemente, F., D’Alonzo, M., Controzzi, M., Edin, B. B. & Cipriani, C. Non-invasive, temporally discrete feedback of object contact and release improves grasp control of closed-loop myoelectric transradial prostheses. IEEE Trans. Neural Syst. Rehabil. Eng. 24, 1314–1322 (2016).
Valle, G. et al. Hand control with invasive feedback is not impaired by increased cognitive load. Front. Bioeng. Biotechnol. 8, 287 (2020).
Guggenheim, J. W. & Asada, H. H. Inherent haptic feedback from supernumerary robotic limbs. IEEE Trans. Haptics 14, 123–131 (2020).
Amoruso, E. et al. Somatosensory signals from the controllers of an extra robotic finger support motor learning. Preprint at bioRxiv https://doi.org/10.1101/2021.05.18.444661 (2021).
Kim, J. H. & Lee, B.-H. Mirror therapy combined with biofeedback functional electrical stimulation for motor recovery of upper extremities after stroke: a pilot randomized controlled trial. Occup. Ther. Int. 22, 51–60 (2015).
Risi, N., Shah, V., Mrotek, L. A., Casadio, M. & Scheidt, R. A. Supplemental vibrotactile feedback of real-time limb position enhances precision of goal-directed reaching. J. Neurophysiol. 122, 22–38 (2019).
Vargas, L. et al. Object stiffness recognition using haptic feedback delivered through transcutaneous proximal nerve stimulation. J. Neural Eng. 17, 016002 (2019).
Wang, W. et al. Building multi-modal sensory feedback pathways for SRL with the aim of sensory enhancement via BCI. In Proc. 2019 IEEE International Conference on Robotics and Biomimetics (ROBIO) 2439–2444 (IEEE, 2019); https://doi.org/10.1109/ROBIO49542.2019.8961383
Raspopovic, S. et al. Restoring natural sensory feedback in real-time bidirectional hand prostheses. Sci. Transl. Med. 6, 222ra19 (2014).
Ganzer, P. D. et al. Restoring the sense of touch using a sensorimotor demultiplexing neural interface. Cell 181, 763–773 (2020).
D’Anna, E. et al. A closed-loop hand prosthesis with simultaneous intraneural tactile and position feedback. Sci. Robot. 4, eaau8892 (2019).
Dadarlat, M. C., O’Doherty, J. E. & Sabes, P. N. A learning-based approach to artificial sensory feedback leads to optimal integration. Nat. Neurosci. 18, 138–144 (2015).
Ortiz-Catalan, M., Mastinu, E., Greenspon, C. M. & Bensmaia, S. J. Chronic use of a sensitized bionic hand does not remap the sense of touch. Cell Rep. 33, 108539 (2020).
Merzenich, M. M. et al. Topographic reorganization of somatosensory cortical areas 3b and 1 in adult monkeys following restricted deafferentation. Neuroscience 8, 33–55 (1983).
Merzenich, M. M. et al. Somatosensory cortical map changes following digit amputation in adult monkeys. J. Comp. Neurol. 224, 591–605 (1984).
Jenkins, W. M., Merzenich, M. M., Ochs, M. T., Allard, T. & Guic-Robles, E. Functional reorganization of primary somatosensory cortex in adult owl monkeys after behaviorally controlled tactile stimulation. J. Neurophysiol. 63, 82–104 (1990).
Allard, T., Clark, S. A., Jenkins, W. M. & Merzenich, M. M. Reorganization of somatosensory area 3b representations in adult owl monkeys after digital syndactyly. J. Neurophysiol. 66, 1048–1058 (1991).
Wang, X., Merzenich, M. M., Sameshima, K. & Jenkins, W. M. Remodelling of hand representation in adult cortex determined by timing of tactile stimulation. Nature 378, 71–75 (1995).
Gindrat, A.-D., Chytiris, M., Balerna, M., Rouiller, E. & Ghosh, A. Use-dependent cortical processing from fingertips in touchscreen phone users. Curr. Biol. 25, 109–116 (2015).
Muret, D. & Makin, T. R. The homeostatic homunculus: rethinking deprivation-triggered reorganisation. Neurobiol. Learn. Plast. 67, 115–122 (2021).
Peng, G., Wang, Y. & Han, G. Information technology and employment: the impact of job tasks and worker skills. J. Ind. Relat. 60, 201–223 (2018).
Oertelt, N. et al. Human by design: an ethical framework for human augmentation. IEEE Technol. Soc. Mag. 36, 32–36 (2017).
The Transhumanist Reader (eds More, M. & Vita-More, N.) Ch. 4 (John Wiley & Sons, 2013); https://doi.org/10.1002/9781118555927
Raisamo, R. et al. Human augmentation: past, present and future. Int. J. Hum. Comput. Stud. 131, 131–143 (2019).
Buckingham, G. et al. The impact of using an upper-limb prosthesis on the perception of real and illusory weight differences. Psychon. Bull. Rev. 25, 1507–1516 (2018).
Blanke, O. Multisensory brain mechanisms of bodily self-consciousness. Nat. Rev. Neurosci. 13, 556–571 (2012).
Simon-Martinez, C. et al. Age-related changes in upper limb motion during typical development. PLoS ONE 13, e0198524 (2018).
Ciullo, A. S. et al. A novel soft robotic supernumerary hand for severely affected stroke patients. IEEE Trans. Neural Syst. Rehabil. Eng. 28, 1168–1177 (2020).
Wesselink, D. B. et al. Obtaining and maintaining cortical hand representation as evidenced from acquired and congenital handlessness. eLife 8, e37227 (2019).
Makin, T. R. & Bensmaia, S. J. Stability of sensory topographies in adult cortex. Trends Cogn. Sci. 21, 195–204 (2017).
Wu, F. & Asada, H. Bio-artificial synergies for grasp posture control of supernumerary robotic fingers. In Robotics: Science and Systems X (2014); https://doi.org/10.15607/RSS.2014.X.027
Hussain, I., Salvietti, G., Spagnoletti, G. & Prattichizzo, D. The Soft-SixthFinger: a wearable EMG controlled robotic extra-finger for grasp compensation in chronic stroke patients. IEEE Robot. Autom. Lett. 1, 1000–1006 (2016).
Abdi, E., Burdet, E., Bouri, M. & Bleuler, H. Control of a supernumerary robotic hand by foot: an experimental study in virtual reality. PLoS ONE 10, e0134501 (2015).
Saraiji, M. Y., Sasaki, T., Kunze, K., Minamizawa, K. & Inami, M. MetaArms: body remapping using feet-controlled artificial arms. In Proc. 31st Annual ACM Symposium on User Interface Software and Technology 65–74 (ACM, 2018); https://doi.org/10.1145/3242587.3242665
G.D. was funded by the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement no. 813713). S.S. was funded by the CHRONOS project, the Wyss Center for Bio and Neuroengineering and the Bertarelli Foundation. F.D.V. was funded by ANR grants nos. ANR-17-EURE-0017 FrontCog and ANR-16-CE28-0015 Developmental tool. T.R.M. was funded by an ERC Starting Grant (715022 EmbodiedTech) and a Wellcome Trust Senior Research Fellowship (grant no. 215575/Z/19/Z). G.S. and D.P. were supported by Progetto Prin 2017 ‘TIGHT: Tactile InteGration for Humans and arTificial systems’, protocol 2017SB48FP. S.M. was funded by the Swiss National Science Foundation through the National Centre of Competence in Research (NCCR) Robotics, the ERC under the European Union’s Horizon 2020 research and innovation programme (grant agreement no. 813713) and the Bertarelli Foundation.
The authors declare no competing interests.
Peer review information Nature Machine Intelligence thanks the anonymous reviewers for their contribution to the peer review of this work.
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Dominijanni, G., Shokur, S., Salvietti, G. et al. The neural resource allocation problem when enhancing human bodies with extra robotic limbs. Nat Mach Intell 3, 850–860 (2021). https://doi.org/10.1038/s42256-021-00398-9
Nature Communications (2022)