The integration of biometrics into autonomous vehicles

A. Biometrics technological overview

Although the topic of “biometrics” can be addressed in a number of ways, there are two fundamental points – the types or “what” information can be used, and the method or “how” that information is captured. Far beyond merely capturing an ink impression of an index finger or a partial of a thumb, today’s biomarkers cover a wide range of the human condition and are captured and stored in a myriad of ways that provide both great benefits and significant challenges. These are discussed below.

Biometric characteristics, the “whats,” are separated into two modalities, physiological and behavioral.¹ Physiological characteristics include those identified from the fingers and hands, veins, face, eyes, ears, odor, and DNA. In contrast to physiological traits, behavioral characteristics (or a combination of both physiological and behavioral traits) also are increasingly utilized by biometric systems. Behavioral characteristics are generally dynamic and can be affected by various factors, including age, illness, or emotional state.

(i) Hands

Fingerprints – Fingerprint recognition is one of the most well-known applications of biometrics. It is a commonly used physiological biometric for US Customs and Border Protection to control international border entry points.² It is also an integrated authentication feature in most cellphones on the market today. Fingerprint recognition involves recognizing the unique differences in patterns of certain characteristics of fingerprints, such as whorls, ridge patterns, and minutiae points (the points plotted to ridge endings and ridge discontinuities) which differentiate the fingerprints of different individuals.

Fingerprints are first acquired and imaged by either off-line or online techniques. Off-line techniques first require that the fingerprint be captured on a substrate, such as inked fingerprint on paper, and then digitized. Online acquisition, such as a live scan of the fingerprint with optical or capacitive digital imaging technologies, would directly create a digital image.

Most scanners, however, do not scan the entire finger at once and also do not create a full image from all the partial images.³ Using software algorithms, the features of the fingerprint (e.g. ridge orientation and frequency, ridges, and minutiae) are extracted and a biometric template is created. This template is a sequence of binary data that can be used to compare another sequence acquired from a subject for identity or authentication purposes. These templates may be either proprietary templates that are coded to distinct fingerprint recognition systems or standard templates that are interoperable between vendors. In order to achieve interoperability between competing fingerprint recognition systems, an initiative led by the US Department of Commerce’s National Institute of Standards and Technology (NIST) created standards for fingerprint recognition.⁴ Depending on the purpose for which the fingerprints are acquired, the biometric data may be stored locally on the device or on a secure portable smart card (e.g. for mobile banking) or on a server (e.g. for government identification purposes).

Palm & hand – Palm recognition also utilizes physiological measurements similar to those used in fingerprint recognition (e.g. matching minutiae points and ridge patterns).⁵Some law enforcement agencies, including those in Connecticut, Rhode Island, and California, have established palm print databases to identify potential criminal offenders.⁶Interestingly, at a recent security conference, researchers from NYU were able to create artificial fingerprints that contained some features of

fingerprints that were more common than others.⁷ The artificial fingerprints were able to fool the fingerprint sensor more than one in five times. These manufactured fingerprints were designed to target fingerprint scanners like those in cellphones.

Another related physiological biometric is hand geometry. Hand geometry recognition is the longest implemented biometric type. Commercially available systems for measuring hand geometry have been available since the early 1970s.⁸ Hand scanners were used in the 1996 Olympic Games to control access to the Olympic Village.⁹ It can involve the measurement of the length, width, thickness, and surface area of the hand, as well as the distance between knuckles, and the height or thickness of fingers. However, unlike fingerprints, hand geometry is not as unique and an individual’s hand geometry may change over time.

Both palm prints and hand geometry can be captured as high or low resolution images from charge-coupled cameras, digital scanners, webcams, contactless systems, and thermal, among others. As with fingerprint technology, the palm has certain identifying features, including ridges, valleys, and minutiae, that can be used to generate a biometric template. One advantage palm prints have over fingerprints is that the palm is larger, and thus has more information to use to create the biometric template. The captured images are preprocessed to smooth the image and enhance contrast. Depending on the system, a variety of algorithms can be used to extract features from an identified region of interest on the palm and create a biometric template. For hand geometry, the image of the hand is processed by an algorithm and converted to a numerical representation, which is then stored as the user’s biometric template. These biometric templates are stored in a database that can reside on the device’s memory, an identification smart card, or on a server.

Veins & face/hand temperature – Vein patterns in the hands or fingers are another biometric characteristic that can be used to authenticate identity. Financial institutions have utilized vein pattern biometrics in ATMs and for customers accessing safe deposit boxes.¹⁰ Vascular patterns are captured through the use of near-infrared light, which is readily absorbed by the deoxygenated blood carried by veins and renders the vein patterns visible. An additional benefit to vein pattern biometrics is that the vein pattern is stable over an individual’s lifetime and, unlike fingerprint or hand geometry recognition, can only be used to authenticate a living individual. Another related biometric technology uses infrared thermograms that recognize the pattern of heat radiation from the face or hand. Thermograms are unique to an individual since the thermal patterns are derived from the vascular structure of the individual. Thermograms can serve to not only identify or authenticate identity but to also verify that the biometric measurements are from that of a living individual.

Infrared and near infrared imaging is used to capture the unique patterns of heat that the individual radiates from their blood vessels in their face, hands, and the veins of his or her hands. This is a non-intrusive and non-invasive technology. This imaging of heat is then converted into a temperature, and the patterns are encrypted and stored in a similar manner as with templates previously discussed above.

(ii) Head

Face & ear – Biometric facial recognition is another technology widely used for authentication and identification purposes. Facial recognition technology is used worldwide by law enforcement agencies, including at least two separate FBI programs¹¹ and with social media platforms like Facebook. Facial attributes are captured by photometric or geometric sensors. The geometric method analyzes the shape and position of facial features (e.g. the distance between the eyes, cheekbones, chin, and nose) and relies upon distinguishing facial features. The photometric method converts the facial features into numerical values, creating a template based on the values, and then compares that template to the values for facial features from another image for identification or authentication purposes. Similarly, ear recognition is used in biometric technology because the shape of the ear is stable over time and its growth is almost linear with aging. As with facial recognition, ear recognition can be assessed based on matching distances between structural points in the ear or by matching based on the appearance of the ear.

Sensors are used to capture facial or ear features. These distinguishing facial features are analyzed with respect to their size or relative position to other facial features. The shape of the face is also an important feature. The spectral band of the sensor can be visible, infrared or thermal, and the image may be rendered as a 2D photograph, 3D image, or video. Different algorithms create a biometric template based on these distinctive features for later authentication use. In order to facilitate interoperability, NIST has also propagated voluntary consensus standards for the interchange of facial biometric data.¹² Similarly, ear recognition utilizes algorithms which extract distinctive features based on the shape of the ear and converts the images to a numerical format, which is incorporated into the individual’s biometric template.

Eye – The eye is also the source of multiple traits used in biometric systems. Iris recognition systems have been used in universities for access to on-campus dining halls and, recently, plans have been announced for facial and iris recognition for check-in and boarding purposes at the Dubai International Airport.¹³ The iris of the eye, the colored ring around the pupil, is considered the most accurate of biometric traits. The iris is also unique, even between the left and right eyes of the same individual, and there are many distinguishing features present in the iris (e.g. striations, rings, furrows, freckles; but generally not color) that can be utilized in an iris recognition system. The retina is another reliable and accurate trait of the eye that is used in biometric systems. The US military utilizes laptop computers and handheld identity detection equipment with retinal scanners in Iraq and Afghanistan to identify local suspects.¹⁴ The retinal vasculature is also considered a distinctive feature between individuals that is difficult to replicate. Similar to vein recognition, the eye is scanned with infrared light and the unique retinal vasculature is compiled into a template for the biometric system. However, due to the difficulties in image acquisition, the use of eye traits in biometric systems is not as well-adopted as fingerprint or facial recognition has been.

Although both the iris and the retina are essential parts of the eye, biometric data is gathered differently for the two. The iris is scanned with near infrared cameras to identify the distinctive textural details present in the iris. An algorithm converts the complex pattern in the iris into digital data that is stored in a database as a biometric template or used to compare against a stored template. In contrast to the iris, it is the vasculature of the retina that is of interest. In order to image the retina’s vasculature, visible light is beamed into the eye, where the retinal blood vessels absorb it. The amount of light reflected back changes as a result of the vasculature in the retina. These changes in the light pattern during the scan are then converted into code and stored in a database.

(iii) Intrinsic factors

Body odor – As with bloodhounds, body odor recognition relies upon identifying the unique chemical patterns of an individual’s scent.¹⁵ Every individual exudes an odor that is characteristic of its own chemical composition. These patterns of chemical composition are thought to be unaffected by the use of deodorant, diet or disease, and the detection methods are less intrusive than with biometric recognition systems involving the eye or fingerprints.

The characterization of an individual’s body odor is done by analyzing the air in the environment around the individual. A sensor reacts with the organic substances in the air and a chromatogram identifies the odor’s composition. The composition is then converted to digital format and stored in a database.

DNA – DNA patterns are distinct between individuals and, as such, DNA is a useful biometric modality for identification and authentication systems. However, the utility of DNA recognition in biometric systems outside of forensics is limited due to the lack of real-time recognition capabilities and the ease of sample contamination.

DNA identification involves measuring the lengths of short tandem repeat (STR) sequences present in the nuclear or mitochondrial DNA. The number of repeated DNA sequences in these STRs differs greatly between individuals. DNA is also inherently digital, and thus does not require an additional step to convert its data into another a template. However, the time required to complete a DNA analysis is prohibitively long for use in mass identification or authentication (e.g. border crossings). Care must be taken to not cross-contaminate the samples with another individual’s DNA.

(iv) Movement

Gait – The pattern of human locomotion, or gait, can be used for biometric systems based on behavioral characteristics. Algorithms are used to extract an individual’s gait features, both dynamic and static (such as body shape). Although gait recognition can be obtained non-intrusively, an individual’s gait can be altered by many outside factors (e.g. walking surface, footwear, clothing) or can change with age or weight variations.

Gait analysis can be performed with low-quality video footage of a person walking. In some instances, many cameras are placed all around the individual to capture all angles of a person’s gait. Sensors can also be placed on the floor to measure unique footstep patterns. The video footage can then be used to generate a blurred silhouette, which can also be used as the biometric template. Gait analysis involves not only dynamic gait motion but also static body appearance. Given that a person’s gait can be affected by a myriad of external issues (e.g. footwear, walking surface, etc.), there is a question as to how unique an individual’s gait actually is.

Signature – Another behavioral biometric is the way an individual signs their name. Signature recognition involves the measurement of the dynamic movements that an individual demonstrates as they sign their name. Such dynamic characteristics are difficult to mimic and include the direction of movement, the pressure exerted, stroke order and direction, speed and shape of the signature. However, an individual may have large variabilities in these dynamic movements between signatures, and signature recognition may be difficult.

The dynamic act of signing a signature can be measured and analyzed to isolate the unique dynamic movements used during the signing. Alternatively, the individual could provide a static sample signature, which is then turned into a digital image and analyzed by a software algorithm. Dynamic signature recognition is extremely difficult to replicate because the forger would have to physically copy the signer’s dynamic characteristics (e.g. acceleration, timing, pressure, etc.).

Keystroke – The behavioral biometric of keystroke rhythms is considered sufficiently distinct between individuals to use for identity verification purposes. Keystroke dynamics involve the manner and rhythm of an individual’s typing on a keyboard. An individual’s keyboard dynamic measurements may not be unique, but keystroke software can capture data based on the typing pattern, rhythm, strength and speed. Additional biometric parameters include the duration that a key is pressed, the dwell time, and the duration between releasing a key and pressing on the next key, flight time. These parameters are all utilized to generate a biometric template. Keystroke dynamics can, however, be affected by physical issues affecting the hands or muscles, emotional state, and the keyboard used. Monitoring of keystroke dynamics over the course of a session also allows for continuous verification of the individual’s identity. The Bank of Utah also incorporated keystroke dynamics software to enhance the security of its online banking platform.¹⁶ While keystroke dynamics are non-invasive and require no additional hardware, typing patterns are not as consistent as some believe. MIT found that keystroke patterns were affected by a change in the keyboard used, the keyboard layout, and physical discomfort in the hands.¹⁷

Gesture – More recently, gesture-based recognition systems have also been considered for biometric identification or authentication. Gesture recognition has been called “the mathematical interpretation of a human motion by a computing device.”¹⁸ Generally, gesture recognition tracks the movements of the hand or the face, but can also include tracking the head and/or body movements. Gesture parameters measured include acceleration, pressure, size, and time. Gesture recognition has been embraced in home game consoles, such as the Wii, Xbox, and Playstation, which have controllers with accelerometers and gyroscopes and readily respond to gestures. These same gaming companies also make their own gesture recognition software. Yamaha also introduced a gesture recognition feature on a motorcycle; it turned the engine on and off with a gesture.¹⁹ Algorithms used in gesture recognition software are 3D-based or appearance-based models. 3D-based models rely upon information gathered from the rest of the body.

(v) Combinations

Voice – Voice recognition is a combination of both physiological and behavioral characteristics. An individual’s voice results from the static physical aspects of the body that are responsible for generating sound, such as the mouth, jaw, larynx, throat, nasal cavity, or weight. The behavioral aspects may reflect factors including language, age, or physical or emotional state. Voice recognition technology is used by many financial services companies to authenticate their clients.²⁰
Unlike speech recognition, which only attempts to recognize sound waves based on samples of words spoken from a large variety of people of different characteristics and backgrounds (e.g. sex, age, race, geographic, etc.), voice recognition is used to authenticate an individual and requires a match between the voice of the individual and a unique digital template of that individual’s voice. For example, the US previously used a 24-hour voice-activated US-Canada border crossing for registered local residents of Scobey, Montana.²¹ These residents would pick up a telephone at the border gate, enter a preselected four-digit personal identification number (PIN), and utter a secret pass phrase, which had previously been recorded at the border post. Once authenticated by the voice recognition system, the driver could proceed across the border.

This digital template is a master voice print generated by voice recognition software that often requires the individual to repeat a phrase or series of numbers or words several times before the software will have enough data points to accept the voice print as a template. These spoken words are reduced to segments of tones (dominant frequencies) that are captured by the software, converted into a digital equivalent, and stored as a template. These tones digitally represent the individual’s unique voice template. Other voice recognition software utilizes the voice patterns of the individual, instead of repetitive phrases, to create the master voice print. This generation of the master voice print can be affected by outside influences, such as unnatural speech, background noise, and poor microphones. Voice recognition can also be affected by an individual’s condition (e.g. have a cold, be affected by medications, or mood). Another challenge with voice recognition is that software recognition may be fooled by a voice recording, but most systems have either incorporated some form of liveness detection or use a secondary input, such as a unique PIN.

B. Biometrics – Data storage and use

With the embrace of biometrics by many different industries for identification and authentication purposes and the increasing presence of technology that can passively collect biometric data (e.g. facial recognition and gait), the storage and security of the captured biometric data is of paramount concern since the physical and behavioral characteristics that underlie the biometric data is generally unchangeable. Biometric data is typically encrypted and can be stored (1) locally in an individual’s device (e.g. fingerprint in cell phone; biometric information stored on smart card), (2) on a centralized server that may reside inside or outside of the country, or (3) through a distributed data model, which can break the biometric data into separate files that are then stored in two or more locations (e.g. locally on a secure card or cell phone and on a server).²²

In general, the underlying biometric modality is captured and then the biometric system applies an algorithm, which may be proprietary to the system, and converts the original biometric data (e.g. fingerprint, facial recognition, vein pattern, etc.) into a numeric representation that is then used as the biometric template for comparison purposes. In order to authenticate or identify their identity, the individual’s biometric comparator is converted into a numeric template that is then compared to the original biometric template. These biometric templates are encrypted, and it is extremely unlikely that the biometric template could be used to reverse engineer the original biometric measured.

Local storage of encrypted biometric data can be compromised if the device or smart card is lost or stolen. Also, as even the US Office of Personnel Management learned, biometric data stored on a server can be vulnerable to a cybersecurity breach; over 5.6 million people’s fingerprints were taken during the breach.²³ Of the three options noted above, the distributed data model is the most secure at protecting biometric data from data breaches since all the parts of the data are stored in multiple locations. Nevertheless, if the servers that store the data are located in a foreign jurisdiction, then foreign law may govern how the biometric data is protected. For example, in the US there is no federal law that governs the gathering and management of intellectual property data. In most US states, biometric data can be collected and shared by businesses. Illinois, Washington and Texas have all enacted specific biometric privacy laws, and several states have included biometric data into their data breach notification laws.²⁴ In contrast, in the European Union, member states are required under E.U. data privacy law, the General Data Protection Regulation, to prohibit biometric data from being shared with a third party without consent, subject to a few exceptions.²⁵

C. Biometrics in the automotive industry

Many of these biometric technologies are being considered for incorporation into the next generation of automobiles to enhance safety, convenience, and customization of the driving experience. At the 2018 New York International Auto Show, Genesis, a Hyundai division, introduced the Essentia Concept car that incorporated fingerprint and facial recognition technology for vehicle entry.²⁶ Iris and voice recognition are also other biometric modalities that can be used for verification of identity for vehicle entry. The incorporation of biometric entry and biometric ignition systems would eliminate the need for keys and hackable key fobs and should further deter car theft.

Unfortunately, the incorporation of a fingerprint recognition system does not guarantee that the vehicle cannot be stolen. For example, car thieves in Malaysia in 2005 bypassed the fingerprint security measure by cutting off the end of the car owner’s index finger and using it to start the car.²⁷ Third-party companies that sell aftermarket parts have also created biometric car starter kits that require a registered fingerprint before the vehicle will start.²⁸ Jaguar has proposed the use of a facial and gait recognition system to unlock the car doors upon detecting the approach of an authorized user. The use of gait recognition would also prevent unauthorized access of the vehicle with a static picture of an authorized user.²⁹

Auto manufacturers are also integrating face and iris recognition technology with a vehicle camera system directed at the driver to detect fatigue or drowsiness. The car would sound an alert if fatigue or drowsiness is detected.³⁰ Moreover, if this technology is incorporated in an AV, the vehicle could take over operating the car once the driver shows signs of drowsiness.³¹ Biometrics to monitor the health of the driver have also been proposed, including infrared technology and Doppler sensors that monitor the driver’s facial temperature and heart rate.³² In an AV, the car could pull over to the side of the road or be programmed to call emergency services for assistance if the driver shows signs of being ill.

Biometrics can also be utilized for vehicle in-cabin preferences and personalization for vehicles with more than one driver. For example, in-cabin iris scanning technology can authorize a driver to start the car and automatically adjust the seats and mirrors and load music and GPS locations to the driver’s preset preferences.³³ The Jaguar facial and gait recognition system described above would also allow the automatic personalization of the vehicle functions and features upon recognition of an authorized user.³⁴

Voice recognition technology is now a relatively common feature in automobiles. Voice recognition in vehicles enable to driver to perform tasks (e.g. controlling navigation and music and answering connected cellphone) without taking their eyes off the road. In 2012, only 37 percent of new cars included a voice recognition system.³⁵ For cars manufactured in 2019, 55 percent of the new cars are anticipated to have a voice recognition system installed. By 2022, nearly 90 percent of all new vehicles are predicted to have voice recognition systems.³⁶

Gesture recognition is another biometric that the automotive industry is embracing for in-car controls.³⁷ A camera would be mounted on the steering wheel or dashboard to look for registered gestures that would then activate a processor to analyze the gesture commands and execute functions based on those gestures. The recognizable gestures would need to be performed within a defined space and would also be without contact to a touchscreen. Sensors have been developed to recognize such in-car gestures, including those that register 3D movements and positional data of the hand and proximity sensing. Hyundai’s HCD-14 Genesis concept sedan demonstrated 3D gesture recognition for controlling the dashboard’s navigation and volume and changing radio stations.³⁸