Restoring finger-specific tactile sensations with a sensory soft neuroprosthetic hand through electrotactile stimulation

this article


INTRODUCTION
Tactile feedback plays a vital role in neuroprosthetic hands, improving user acceptance and helping the user perceive stimuli from the environment [1][2][3][4][5][6] . Furthermore, tactile feedback can effectively decrease the burden on visual feedback, which causes fatigue for an amputee with a prosthesis [7,8] . Unfortunately, most current commercial prostheses are not capable of providing tactile feedback for users to deliver touching information [6,9,10] . The main challenges are designing an effective tactile sensor for neuroprosthetic hands to measure the touch with external objects and environments [11,12] and encoding and transferring the sensory information to the user with a feedback pathway that can be decoded easily by the human brain [13,14] . Therefore, it is essential to develop a sensory neuroprosthetic hand system for society.
With the development of soft robotics, soft tactile sensors have provided a promising method for endowing prosthetic hands with sensory capability [11,12,15] . Many soft tactile sensors with various sensing principles, including capacitive [16,17] , resistive [18,19] , optoelectronic [20,21] , magnetic [22,23] and triboelectric [24,25] , have been reported to show significant potential for implementation into prosthetic hands. Osborn et al. invented a multilayered electronic dermis that allows the prosthesis user to differentiate between non-painful and painful tactile stimuli [11] . Zhao et al. introduced a type of stretchable optical waveguide, which could be integrated into a prosthetic hand to perceive strain and force and showed the capability of feeling the shape and softness of tomatoes [15] . Li et al. developed a quadruple tactile sensor that integrated pressure sensing, material thermal conductivity sensing and bimodal temperature sensing capability, thereby allowing the robot hand to precisely recognize objects [26] . However, most studies focus on validating the performance of sensors. Extracting tactile information from sensor responses and delivering them to the human body have rarely been explored. Hence, it is essential to integrate tactile sensors into prosthetic hand systems to help amputee users better interact with their environment.
To resolve the challenge of tactile feedback for neuroprosthetic hands, some invasive and noninvasive methods have been proposed [13,14,27,28] . Through invasive technology, a direct interface to the peripheral nervous system can be built for intuitive control of the prosthesis. D'Anna et al. implemented a hybrid approach to restore both finger position and tactile information for transradial amputees by implanting transverse intra-fascicular multichannel electrodes in the ulnar and median nerves [13] . To reduce the complexity of invasive surgical procedures, sensory substitution through noninvasive methods has been a desirable alternative. Franceschi et al. transmitted mechanical information to subjects using noninvasive multichannel electrotactile stimulation [29] . The location of the stimulation region can also have an effect on the feedback. Due to the functional changes in the cerebral cortical maps of the body, amputation causes sensory reorganizations [30] . Furthermore, neighboring areas in the central body map expand over the former hand area [31] . Hence, we can find a specific region on the remnant limb mapping of the phantom hand that corresponds to each digit [31] . In our previous work, we demonstrated that stimulating the projected finger map (PFM) of the phantom hand on the stump for the amputee with electrical pulses can feedback fingerspecific tactile information from the prosthetic hand to forearm amputees [32] . Compared to sensory substitution, restoring close-to-natural tactile sensation and implementing a neuromorphic motor processing model [33][34][35][36] through a sensory prosthetic hand is promising.
The goal of this study is to present a method to restore finger-specific tactile sensations on the PFM of a unilateral forearm amputee with a sensory soft neuroprosthetic hand through electrotactile stimulation. This work builds on our preliminary research, where we first presented the prototype design of a soft neuroprosthetic hand integrated with five hydrogel-elastomer hybrid soft touch sensors [12] . Here, five soft touch sensors embedded in the fingertips are first adopted to measure the pressure changes of the soft neuroprosthetic hand with the touched objects. The measured pressure information is then accordingly encoded into electrotactile stimulation patterns to trigger an electrical stimulator, which outputs programmable electrical pulses via surface stimulation electrodes to the PFM of the phantom hand on the amputee's remnant limb. In this manner, the finger tactile sensation can be elicited, which can help the amputee restore a close-to-nature tactile sensation [37,38] and reduce the learning time for establishing the sensory substitution mapping relationship to distinguish the finger press state. Furthermore, we explore the sensory capability of our soft neuroprosthetic hand when interacting with different objects with various curvatures and hardness. The experimental results show that, based on the spatial encoding strategy, the amputee subject can instantaneously distinguish the tactile sensation of a single finger or multiple fingers with an accuracy of 98.57% and 91.71%, respectively. Otherwise, by programming the frequencies of the electrical pulses, the amputee subject can successfully discriminate the touching objects with different curvatures and hardness with an accuracy of 97.26% and 97.93%, respectively, in a blindfolded and acoustically-shielded environment. Finally, we demonstrate that the amputee subject can achieve closedloop control of the sensory soft neuroprosthetic hand by integrating myoelectric control interface and electrotactile feedback to achieve multilevel perception, which mimics the pain perception of humans to protect the soft finger from damage [11] .

Subject
The unilateral forearm amputee subject who participated in the experiment was recruited from Shanghai Liankang Prosthetics and Orthotics Manufacturing Co., Ltd., Shanghai, China. He used a cosmetic prosthetic hand and had five independent finger-projected areas corresponding to the phantom fingers on his forearm stump. All experiments were conducted in accordance with the declaration of Helsinki and approved by the Ethics Committee of Human and Animal Experiments of Shanghai Jiao Tong University. The amputee did not have any prior neuromuscular disorders and was informed about the experimental procedure and signed the informed consent forms prior to participation.

Experimental setup
The experimental setup is shown in Figure 1A. The subject sits near the soft neuroprosthetic hand and uses his own electromyogram (EMG) signals to control the soft neuroprosthetic hand. As the bending angles of the finger increase, the soft neuroprosthetic hand touches the object and the contact pressure (measured by the capacitive soft touch sensors) between the soft neuroprosthetic hand and the object increases, which leads to a consistent increase in the capacitance of the capacitive soft touch sensor. Simultaneously, we use an LCR meter (E4980AL, Keysight Technologies Inc., USA) with a customized multiplexer to record the measured capacitances of the five touch sensors, which are transmitted to the computer. A control algorithm written by MATLAB runs in the computer to continuously record the relative capacitance change with a sampling frequency of ~7 Hz, which is higher than the actuation frequency of the soft finger [12] . Additionally, we set a threshold of the relative capacitance change after prior experimental trials.
In the experiments, when three consecutively received values of the relative capacitance change are all higher than the threshold, the electrical stimulator with five isolators (Master-9, A.M.P.I. Company, Israel) is triggered to output the pre-set rectangular current pulses, which can be felt by the subject. As a result, we can approximately consider that the soft finger is located in a stable contact phase rather than a dynamic contact process. On this basis, we can partly avoid the misjudgment caused by the dynamic contact process. We then encode different stimulation parameters (current amplitude, pulse width and frequency) to map the different responses of the soft touch sensors. A custom surface electrode array with two uniformly distributed copper circles is connected to the isolator. To acquire a comfortable sensation during the stimulation, a layer of commercial hydrogel is sandwiched between the circular electrodes and the target skin on the forearm to reduce the uncomfortable prick sensation due to electric charge accumulation. This can also avoid chemical reactions during electrical stimulation. The electrode arrays are fixed on the target skin with tape to ensure optimal contact. This helps the amputee reestablish the closed-loop control path [ Figure 1B] like a non-disabled person [ Figure 1C].

Evaluation conducted prior to experiments
To better evoke a tactile sensation on the stump skin of the amputee through electrical stimulation, we identify the PFM of the phantom hand on the remnant limb of the subject with palpation [32] . It has been proved that PFM is always more sensitive than other sites [37] . To obtain the complete distribution of evoked tactile sensation sites, we first clean the stump skin with alcohol wipes. We then use a test stylus with a globular head to touch the skin on the entire stump to identify all evoked tactile sensation sites corresponding to each finger. The subject is required to respond to the touch feelings by identifying which finger is felt. A continuous outline of evoked tactile sensation sites is then depicted with a pen based on the subject's personal description. The continuous outline is referred to as PFM, which consists of the independent finger-projected areas (D1-D5) shown in Figure 2. The whole palpation process is conducted in a blindfolded environment to eliminate the effect of vision. The subject is then asked to reconfirm the PFM using the same stylus with his own contralateral hand. To prevent possible fatigue, several breaks are randomly provided during the whole process.
When the PFM regions are identified, we modulate stimulation parameters to evoke graded tactile sensation. In this experiment, we only adjust the frequency while the other two parameters (current amplitude and pulse width) are maintained at their nominal values. In the pilot test, the stimulation current is set as rectangular current pulses. We first clean the skin on the stump with alcohol wipes. The electrical stimulation electrode is then arranged on the PFM regions. As commonly used in the literature [27,37,39] , we set the stimulation patterns and parameters based on our prior tests. In the tests, the current amplitudes for all five channels are unified at 4 mA. However, the different stimulation sites on the upper limb of the subject have different sensitivities. Thus, under the same set of stimulation parameters, the current amplitude of each channel is then fine-tuned independently to ensure that the subject can perceive the same sensation level according to the verbal feedback from the amputee subject. It is known that the intensity of feeling is proportional to the velocity of vibration. Therefore, we maintained the pulse width at 250 μs based on our previous experimental results [37] and then changed the frequency from 1 to 75 Hz. The maximum frequency (f max ) is recorded when the subject first reports that he cannot discriminate the interval of the stimulation pulse. The frequencies of 5, 15 and 35 Hz below f max are finally chosen as constant values to indicate three levels of intensity after finishing the test of the amputee subject. After that, current pulses with these three groups of frequencies are output to ensure that the subject can clearly distinguish the difference between these three groups of electrical pulses. Each group lasts for 30 s to let them remember these feelings. Between two groups of electrical pulses, a 30 s rest is provided to prevent possible fatigue. Finally, we choose the electrical pulses with frequencies of 5, 15 and 35 Hz [ Table 1] o evoke graded tactile sensation for discriminating three groups of objects.

Design and fabrication of testing props
We design and manufacture two groups of test props to evaluate the characteristics of the capacitive soft touch sensors for tactile sensation. All test props are manufactured using a 3D printer (J750, Stratasys Co. Ltd., USA). One group of props is manufactured with the same materials (Vero Magenta, Stratasys Co., Ltd., USA) and the same height (H = 22.5 mm) but different curvature radii (i.e., 15, 1.5 and 0.015 mm), as shown in Figure 3A. Another group of props is manufactured with the same curvature but with three kinds of materials (Stratasys Co., Ltd., USA). The different hardnesses of these three different materials are 8A, 27A and 85D, respectively, as shown in Figure 3B.

Finger discrimination
Given that a prosthetic hand serves as an extension of the arm for an amputee, it is desirable to let the amputee feel the existence of fingers. To demonstrate the tactile sensation of the feeling of fingers, we first arrange stimulation electrodes on the PFM regions (D1-D5) corresponding with the soft touch sensor integrated from the thumb to the pinky finger [ Figure 4A]. When the touch sensor on the fingertip senses the mechanical interactions, the measurements activate the corresponding electrode through transcutaneous electrical stimulation. To reduce the coupling effect among each soft touch sensor, we set an effective activation threshold based on the prior result [12] . In the blindfolded and acoustically-shielded interaction experiment, we randomly touch a single finger or a combination of multiple fingers of the sensory soft neuroprosthetic hand. The subject can clearly discriminate the touched finger or finger combination and regain the evoked finger sensation close to the feeling of a native finger [ Figure 4B and Supplementary Movie 1]. The confusion matrixes show that it has a high accuracy of correction with 98.57% [ Figure 4C] and 91.71% [ Figure 4D] in distinguishing an individual finger and two fingers in combination, respectively.

Object discrimination
In addition to dexterous manipulation, human hands can simultaneously feel the touch when grasping diverse objects to determine their mechanical properties, such as shape and hardness, which remains a grand challenge for existing prosthetic hands. Here, we demonstrate that the unilateral forearm amputee with a sensory soft neuroprosthetic hand can discriminate different objects with various curvatures and hardness [ Figure 5A]. We encode the frequencies (5, 15 or 35 Hz) of the electrical pulses with fixed current amplitude (4.0 mA) and pulse width (250 μs) to map different ranges of ΔC/C 0 of the touch sensor (i.e., no stimulation when ΔC/C 0 ≤ 0.1, 5 Hz when 0.1 < ΔC/C 0 ≤ 0.25, 15 Hz when 0.25 < ΔC/C 0 ≤ 0.33 and 35 Hz when ΔC/C 0 > 0.33). As an example, the finger of the soft neuroprosthetic hand in a fixed position is applied with 70 kPa pneumatic pressure to touch three objects with the same material (Vero Magenta, Stratasys) but different radii of curvature (i.e., 15, 1.5 and 0.015 mm, Figure 3A), resulting in different ΔC/C 0 (i.e., 0.2, 0.3 and 0.9, Figure 5B). The blindfolded and acoustically-shielded interaction experiments demonstrate that the subject can successfully identify touching objects in a random sequence [ Figure 5B and Supplementary Movie 2] with an accuracy of 97.26% [ Figure 5C]. In another experiment on touching three objects with the same radius of curvature but different materials (with the hardness of 8A, 27A and 85D, Figure 3B), the ΔC/C 0 values of the touch sensor on the soft finger at 90 kPa pneumatic pressure are 0.12, 0.27 and 0.35, respectively. The blindfolded and acoustically-shielded interaction  experiments demonstrate that the subject can also successfully distinguish the material hardness based on the electrotactile feedback [ Figure 5B and Supplementary Movie 3]. The confusion matrix shows that the accuracy of correction can reach 97.93% [ Figure 5D].

Multilevel perception with closed-loop control
To develop a human-in-the-loop control system for closed-loop myoelectric control of the soft neuroprosthetic hand, we use electromyography sensors arranged on the remnant limb of the amputee to measure the surface EMG signals for intuitively controlling the soft neuroprosthetic hand to deliver different grasp types [12] . The decoded EMG signals are used to actuate the soft finger. Simultaneously, we encode the relative change of measured capacitances of the soft touch sensor, related to the touch pressure, by outputting electrical pulses. The amputee subject uses his own EMG signal to control the soft finger to achieve step bend [ Figure 6A]. As the bending angle increases, the contact pressure between the touch sensor and the object increases, leading to a consistent increase in the relative capacitance change of the soft touch sensor. When the sensor output reaches the threshold of 0.75, the electrical stimulator is triggered and outputs a pre-set stimulus with a frequency of 15 Hz to let the subject feel that the contact pressure is already sufficiently large [ Figure 6B and Supplementary Movie 4]. If he still controls the soft finger to bend, it results in damage to the soft finger. After receiving the stimulus, he uses his EMG signal to control the sensory soft neuroprosthetic hand to deflate the soft finger to the initial state. Thus, the contact pressure decreases and the relative capacitance change of the touch sensor decreases correspondingly. When the collected value drops below the threshold, the electrical stimulator stops outputting stimulus to let the amputee subject know that he is in a safe area of use. The whole process mimics the closed-loop control of human manipulation with multilevel perception.

CONCLUSIONS
In this study, we present a method to restore finger-specific tactile sensations on the PFM of a unilateral forearm amputee with a sensory soft neuroprosthetic hand through electrotactile stimulation. Five hydrogel-elastomer hybrid soft touch sensors integrated onto fingertips of the sensory soft neuroprosthetic hand are first adopted to measure the pressure changes when touching objects. The measured pressure information is then accordingly encoded into electrotactile stimulation patterns to trigger an electrical stimulator that outputs programmable electrical pulses via surface stimulation electrodes on the PFM of the amputee. To evoke the close-to-natural sensation, we arrange the electrodes on the PFM regions of the phantom hand. Based on the spatial encoded electrotactile feedback strategy, the amputee subject can instantaneously distinguish the tactile sensation of a single finger or multiple fingers with an accuracy of 98.57% and 91.71%, respectively. Furthermore, by programming the frequencies of the electrical pulses, the amputee subject can successfully discriminate the touching objects with different curvatures and hardness with an accuracy of 97.26% and 97.93%, respectively, in a blindfolded and acoustically-shielded environment. Finally, we demonstrate that the amputee subject can realize closed-loop control of the sensory soft neuroprosthetic hand by integrating myoelectric control interface and electrotactile feedback to achieve multilevel perception. Through this touch-pressure-triggered electrotactile stimulation, the sensory soft neuroprosthetic hand can establish consistent tactile sensations for amputees when touching objects. The soft neuroprosthetic hand combined touch sensors with electrotactile feedback can facilitate interaction with the environment and safe manipulation. To obtain rich tactile information during the grasping process, future studies need to integrate new architectures of high-density touch sensor arrays into the prosthetic hand system to decouple multiple properties of the touched object. Additionally, the miniaturized and sophisticated stimulation feedback device with the neuromorphic transmission model and advanced EMG control algorithm is also desirable to give better tactile feedback for the amputee in an untethered prosthetic hand system. Endowing sensory feedback for the prosthesis will further broaden the application of the prosthetic hand for amputee users.