Coupled mechanics in skin-interfaced electronics via computer vision methods

INTRODUCTION

Advanced materials and associated mechanics have opened pathways for transforming conventional, rigid, planar integrated circuits into stretchable, flexible, soft electronics^[1]. Theoretical studies, supported by mechanical modeling, such as finite element analysis (FEA), have revealed key features and characteristics of these unusual material structures, including the optimization of three-dimensional (3D) buckled electronics^[2]. The unique properties of these materials, which allow for conformal contact with the body, have led to numerous pioneering works on new classes of soft devices in biomedical applications known as bioelectronics. Skin-interfaced electronics, a specific class of soft electronics, have established the foundations for continuous clinical-grade monitoring and are well investigated and summarized in terms of materials selection, design, fabrication, and system integration^[3-5]. Nevertheless, the interaction between biosystems and soft electronics often results in non-trivial coupled mechanics, which requires further effort to characterize the biomechanics, bioelectronics, and their interactions.

This article underscores the latest advances in computer vision techniques and their impact on advancing soft-electronic systems, intending to refine the next generation of skin-interfaced electronics through a thorough characterization of associated biomechanics and, conversely, how these biomechanics influence electronic design. The process is iterative, encompassing the development of soft electronics, the identification of coupled mechanics, and their quantification using computer vision methods, as depicted in Figure 1. We delineate (i) pioneering computer vision techniques employed in skin-interfaced electronics; (ii) the interaction of mechanics in mechano-acoustic (MA) sensors; and (iii) the interconnected mechanics in haptic systems. Final remarks outline expected advancements in computer vision techniques and their projected applications across diverse areas within the soft electronics field.

Figure 1. Schematics showing the research diagram contributing to the identification of coupled mechanics^[8]. Copyright©2022, Nature Publishing Group, quantification via computer vision methods^[10]. Copyright©2022, National Academy of Science, and development of soft electronics^[10]. Copyright©2022, National Academy of Science.

COMPUTER VISION IN SOFT ELECTRONICS

Computer vision methods, also referred to as optical measurement systems, are employed across various research areas within continuum mechanics, including studies on fluid flows^[6], solid deformations^[7], and wave phenomena^[8]. These approaches provide non-contact, non-intrusive measurements with high spatial and temporal resolutions. Recently, they have been instrumental in offering robust mechanical insights for soft electronic devices, particularly in biomedical and biomechanical applications, revealing essential coupled mechanics between biological systems and soft electronic devices. The most representative computer vision techniques applied in soft electronics are summarized in Table 1. When employing one or a combination of these techniques, several factors must be considered, namely, applications, key outputs, the need for fiducial points, the suitable frame of reference, dimensionality, processing time, and resolution.

Particle image velocimetry (PIV) is an advanced optical measurement technique that allows for the detailed analysis of the flow velocity field by tracking the collective motion of tracer particles within a fluid^[9]. The method operates by observing these particles from an Eulerian reference frame. Essential to the PIV setup is introducing seeding particles into the flow, chosen for their ability to follow the fluid's motion with minimal impact - a property quantified by the Stokes number. A typical PIV system includes one or more high-resolution cameras synchronized with a laser illumination source. This synchronization is critical for capturing the scattered light from particles at precise intervals, particularly in a dual-pulse arrangement where two images are taken in rapid succession. These images are then dissected into smaller interrogation regions. Within each subregion, the displacement of particle groups is determined by employing spatial cross-correlation, providing a vector map of flow velocity and patterns essential for the resolution of complex turbulent flows. The technique is particularly useful for correlating cardiac or respiratory flows to sensors^[10].

Digital image correlation (DIC) is related to PIV in terms of the processing scheme, but it is used to quantify deformations on solid surfaces^[11]. Instead of seeding particles, DIC utilizes a random speckle pattern and specializes in characterizing spatial derivatives such as deformation and strain. The technique does not require a high-power illumination source or high-speed imaging, making it suitable for standard applications with relatively fast processing times. DIC can be easily combined with 3D reconstruction techniques to achieve 3D stereoscopic deformations on a curved surface, such as the neck and wrist, where the associated deformations can be highly three-dimensional^[12]. Applications in soft electronics include validating haptic and strain-related sensors^[8,13].

Particle tracking velocimetry (PTV) is particularly useful for characterizing fluid flows and structure motions by tracking particles and fiducial points in the Lagrangian frame of reference^[14]. Unlike PIV and DIC, PTV tracks particles in the global coordinate, allowing for estimating particle trajectories. With a multicamera setup, PTV can measure in 3D domains. It offers advantages for quantifying flows in drug delivery systems^[15] or validating motion-tracking devices^[16]. A relatively recent technique, the Markerless Pose Estimation (MPE), tracks a limited number of objects without fiducial points based on transfer learning with deep neural networks. The technique is ideal for monitoring animal behaviors, such as tracking multiple body parts of a mouse during an odor-guided navigation task and a fruit fly behaving in a 3D chamber^[17].

Eulerian Video Magnification (EVM)^[18], Structure from Motion (SfM)^[19], and area tracking^[20] represent advanced computer vision techniques that have substantial implications for soft electronics. These methods are particularly advantageous because they circumvent the need for fiducial points and can support approaches for various soft electronic concepts.

The combination of the above-mentioned computer vision methods could be implemented in skin-interfaced electronics concerning their functionalities and applications. For instance, PIV could be used to correlate respiratory flow with signals from the MA sensor [Figure 2A]. PTV would be useful for quantifying biomarkers requiring high temporal resolution, including vibrations of the neck during speech and respiratory activities, to address design and placement strategies of a wearable sensor [Figure 2B]. DIC can quantify mechanics with high spatial resolution, such as strains on the skin induced by haptic actuators or deformations on the neck during swallowing [Figure 2C]. EVM can enhance subtle motions either generated by a weak haptic actuator [Supplementary Video 1] or pulse wave velocity for developing a wearable pulse oximeter. Area tracking and MPE could be implemented to measure the deformation of organs or implantable devices where adding fiducial markers is difficult. The following sections discuss two sets of examples of computer vision methods used in skin-interfaced electronics: one involves measuring various biomarkers to validate a series of MA sensors, and the other involves quantifying the mechanics induced by various vibrotactile actuators in haptic devices.

Figure 2. Schematic showing computer vision experimental setup for optimizing design and manufacturing strategies for skin-interfaced electronics: (A) flow measurements during respiratory activities using PIV^[10]. Copyright©2021, National Academy of Science; (B) biomechanics on the neck during cardiopulmonary activities using 3D PTV^[24]. Copyright©2021, American Association for the Advancement of Science; (C) biomechanics on the neck during swallowing activities using 3D DIC^[25]. Copyright©2022, Nature Publishing Group. PIV: Particle image velocimetry; PTV: particle tracking velocimetry; 3D: three-dimensional; DIC: digital image correlation.

COUPLED MECHANICS IN MECHANO-ACOUSTIC SENSORS

Skin-interfaced sensor technologies offer a vast range of multimodal, clinical- and consumer-grade, continuous monitoring of physiological biomarkers with high accuracy and immunity to external noises in hospital and in-home settings. The MA device, a thin, soft sensor with a high-bandwidth accelerometer conformally coupled to the skin, has demonstrated its effectiveness in providing precise measurements of MA signals from subtle vibrations of the skin (~10^-3 m/s²) to large motions of the entire body (~10 m/s²)^[21]. When interfaced at unique anatomical locations, such as the suprasternal notch (SN) at the base of the neck, this technology offers a rich blend of MA information related to various classes of underlying body and physiological processes^[22]. In conjunction with the coupled biomechanics through computer vision methods, advanced versions of the MA system have been developed to monitor bio-signals tailored for infectious diseases [Figure 3A1-3]^[23], cardiopulmonary activities during significant body motions [Figure 3B1-3]^[24], swallowing patterns for dysphasia patients [Figure 3C1-3]^[25], and speech sounds [Figure 3D1-3]^[10].

Figure 3. Examples of Coupled Mechanics in Mechano-acoustic Sensors. (A1-3) Covid sensor correlated with droplet generation during respiratory activities using PTV^[23]. Copyright©2021, National Academy of Science; (B1-3) dual sensor focusing on the temporal variations of the neck for cardiopulmonary applications using 3D-PTV^[24]. Copyright©2021, American Association for the Advancement of Science; (C1-3) swallowing sensor focusing on the spatial variations of the neck for dysphasia payments using 3D DIC^[25]. Copyright©2022, Nature Publishing Group; (D1-3) speech sensor investigating the flow physics of plosive speech using PIV^[10]. Copyright©2022, National Academy of Science. PTV: Particle tracking velocimetry; 3D: three-dimensional; DIC: digital image correlation; PIV: particle image velocimetry.

The automated wireless version of the MA device [Figure 3A1] allows for capturing key respiratory symptoms of the infectious disease COVID-19^[23]. PTV has been instrumental in correlating the timing and intensity of respiratory activities from MA sensors with total droplet production and droplet dynamics from PTV experiments. In Figure 3A2, PTV measurements during coughing illustrate detected particles as gray circular symbols and grid-interpolated horizontal velocity of droplets as a colored contour. Figure 3A3 provides a sequence from the MA sensor marked by the automated algorithm (top) and the total number of droplet production determined through PTV analysis in sync with the intensity of MA signals (bottom) during coughing.

A later version of an MA sensor^[24] was developed to address data corruption resulting from motion artifacts [Figure 3B1], particularly associated with the mechanical characteristics of cardiopulmonary processes. PTV, combined with the 3D reconstruction technique SfM, has enabled the reconstruction of biomechanics in the neck with high temporal accuracy in three dimensions. Furthermore, 3D-PTV has been used to identify and validate the design strategy for canceling signals through the differential operation of time-synchronized dual accelerometers between SN and the sternal manubrium (SM). Additionally, 3D displacement and vector contour fields during cardiac activity based on Delaunay triangulation are illustrated in Figure 3B2. Figure 3B3 shows a time series of z-axis displacement at SN and SM over several cardiac cycles during a breath hold. Peak displacements at the SN are significantly larger than those at the SM, while both capture displacements associated with body motion, enabling efficient subtraction.

A variant of the modified MA sensor [Figure 3C1]^[25] was introduced to assist therapeutic treatments for patients with dysphagia by tracking swallows. In this context, 3D-DIC has effectively quantified rapid and overlapping biomechanics in the neck induced by the swallowing mechanism in the oral cavity, pharynx, and esophagus, which can vary widely with age, gender, and other factors^[26]. Additionally, 3D-DIC provides displacement fields and their temporal derivatives during swallowing motions with high spatial accuracy in the spatiotemporal domain. Figure 3C2 shows a representative 3D displacement map at the beginning of the esophageal phase, indicating that the most dominant signature of swallowing mechanisms occurs in laryngeal prominence (LP), followed by the SN. Figure 3C3 demonstrates the displacement and velocity profiles at LP and SN after a differential operation with the signals at SM, further supporting their fundamental swallowing mechanics. The results justify the development of an elongated version of the device that allows positioning it on the LP and SN or SN and SM for monitoring swallows across a broad range of individuals.

Lastly, an example introduces a version of a dual-in-plane MA device [Figure 3D1] designed to detect plosive speech patterns that generate a significant number of aerosols and droplets with intense turbulent background flows^[10]. PIV was used to correlate the background flow physics with MA signals. Figure 3D2 illustrates the formation of a vortex ring from plosive sounds with the velocity vector field and vorticity color contour. Figure 3D3 demonstrates the overall correlation between MA signals, droplet travel distance induced by background flows, and associated biomechanics in the neck with respect to sound power. The studies suggest that sternocleidomastoid (SCM), besides SN, is an ideal location for capturing plosive speech sounds and decoupling them from other bio-signals. MA signals from these locations, using a Dual-in-Plane (DiP) sensor, enhance machine learning-based classification of plosive sounds in various languages.

COUPLED MECHANICS IN HAPTIC SYSTEMS

On the other side of the spectrum, skin-integrated haptic technologies offer the capability to replicate sensations in virtual reality (VR) and augmented reality (AR) settings. Similar to MA sensor technologies, haptic interfaces need to softly laminate onto the curved skin surfaces to deliver sensory information through programmable patterns of localized mechanical vibrations in the spatiotemporal domain^[27]. Vibrations of the skin and the operating mechanisms of the actuator are closely linked with complexity due to (i) viscoelastic properties of the skin altering the characteristics of vibrotactile actuators by adding mechanical impedances and (ii) the vibrating direction and boundary conditions of vibrotactile sensors affecting the dynamics of wave propagation in the skin and vice versa. Therefore, a quantitative description of the mechanical effects is essential for designing such systems and understanding the fundamental aspects of resulting tactile sensations.

In recent studies on skin-integrated haptic systems, the investigation of the coupled mechanics between the skin and haptic actuators using computer vision methods has opened up the possibility of further optimizing device designs and gaining a fundamental understanding of our sensory perceptions^[8,28]. Complex patterns of the skin deformation induced by an array of eccentric rotating mass (ERM) actuators in a wireless haptic interface were quantified to investigate crosstalk in the system and the performance of the actuators in terms of the activation of mechanoreceptors such as Meissner and Pacinian corpuscles [Figure 4A1-3]^[8]. Strain distributions on skin phantoms induced by three representative vibrotactile actuators, including ERMs, linear resonant actuators (LRAs), and vibrotactile linear actuators (e.g., tactors), were explored and correlated with relevant perception studies to provide a fundamental understanding of sensory perceptions [Figure 4B1-3]^[28].

Figure 4. Examples of Coupled Mechanics in Haptic Systems. (A1-3) Haptic Interfaces across Large Areas of the skin optimized by the investigation of deformation and wave propagation of the actuator using the combination of 3D-DIC and PTV^[8]. Copyright©2022, Nature Publishing Group; (B1-3) Strain analyses of representative types of vibrotactile actuators on the phantom skin using 3D-DIC and Triangular Cosserat Point Element method^[28]. Copyright©2023, Elsevier. 3D: Three-dimensional; DIC: digital image correlation; PTV: particle tracking velocimetry.

A wireless, lightweight, flexible haptic interface [Figure 4A1] was developed to deliver spatiotemporal patterns of touch across large areas of the skin, controlled through smart devices in real-time^[8]. PTV and 3D-DIC were used to measure the wave propagation and 3D deformations resulting from structural interactions between the actuators and the surrounding skin. The 3D colored contour map in Figure 4A2 illustrates the deformation in the out-of-plane direction at a representative instant during the actuator’s vibration using 3D-DIC. Figure 4A3 displays the lateral component of wave propagation along the skin induced by the operation of an actuator using PTV. The results reveal that the mechanism of an ERM actuator shares some characteristics of Euler’s disk, and its induced wave propagation along the skin exhibits Rayleigh waves. These mechanical findings further validate the design strategy of devices, showing negligible crosstalk and delivering a strong, power-efficient perception.

Recently, the coupled mechanics of three representative types of vibro-tactile sensors, ERM, LRA, and tactor actuators, were investigated and compared to evaluate their performance in delivering sensory perceptions using 3D-DIC and the Triangular Cosserat Point Elements (TCPE) method for strain estimations^[28]. ERM actuators use a direct current (DC) motor, typically rotating laterally, with an off-center mass connected to the output shaft within a closed metal housing, whereas LRA and tactor actuators operate on alternating current (AC) based on voice coils and magnets suspended on springs, typically vibrating vertically [Figure 4B1]. Colored contours in Figure 4B2 and graphs in Figure 4B3 display strain distributions and center line profiles induced by an ERM actuator and tactor at different depths of the skin phantom, h₁ and h₂, with various contact areas L/L₀, indicating their distinctive characteristics. The strain results induced by the ERM actuator exhibit high magnitudes around the actuator’s border near the surface of the phantom skin, with substantial in-plane components. In contrast, the peak strain induced by the tactor occurs at the actuator’s center in-depth, oriented out of the plane of the skin. A combination of various types of actuators, along with the studies of associated coupled mechanics, may be necessary to achieve robust and sophisticated haptic sensations.

REMARKS AND FUTURE PERSPECTIVE

In conclusion, we have summarized the recent developments in computer vision methods and their applications in skin-interfaced electronics. These methods are not limited to skin-interfaced electronics but extend to other forms of soft or 3D electronics, including 3D passive microflying systems^[29,30], programmable surfaces^[23,31], and implantable devices^[13,15]. To fully harness the potential of these technologies, a comprehensive grasp of both the manufacturing intricacies of flexible electronics and the data processing schemes using computer vision methods is critical. Despite recent progress, several challenges persist in fully integrating computer vision technologies into soft electronics. For instance, volumetric measurements, such as 3D PTV or Digital Volumetric Correlation (DVC), still lack sufficient data resolution or require a large number of cameras, creating a bottleneck in quantification. Another challenge arises in the experimental setup, requiring a controlled optical environment, especially for simultaneous measurements using multiple techniques. This controlled environment may not be adequate for validating and quantifying the associated mechanics of soft electronic devices during situations where optical access is limited, such as in-vivo testing. However, these limitations address future opportunities. For example, image processing algorithms, such as “Shake-The-Box”^[32] and deep learning for PIV/PTV^[33] and DIC^[34], could be integrated to improve data resolution. Instead of relying on existing computer vision techniques, new methods tailored specifically for optimizing soft electronics could be developed. For instance, by combining infrared imaging techniques, thermal-mechanical behaviors associated with wearable devices could be investigated to address all possible safety hazards^[35]. Future research could involve developing (i) a novel haptic system targeted for multisensory engagement based on the measured coupled mechanics using PTV and DIC; (ii) a multimodal skin-integrated sensor for cardiac monitoring validated by EVM; (iii) a wearable motion sensor for orthopedic applications, calibrated with MPE; and (iv) soft tunable electronic camera systems embedded with computer vision processing capabilities. This transdisciplinary research, combining soft electronic technologies and computer vision measurements with a fundamental understanding of continuum mechanics, is key to realizing the next generation of soft electronic systems.