Interview with Prof. Yasue Mitsukura

What will a future in which human emotions can be analyzed and controlled look like?

Yasue Mitsukura (Professor, Faculty of Science and Technology)
Haruna Kawashima (KGRI Project Associate Professor)

What emotions are elicited in you as you read this text? You may be nervous, calm, or focused....Only you have the means to know what you yourself are feeling; emotions which are in themselves fleeting. The news that there is a technology which allows the observation of "human emotions," may occur to you as like something from a science fiction movie. However, this is already on the way to becoming "science fact."

Brain waves, the weak electrical currents that flow through our cerebral matter, alter their behavior from moment to moment in response to emotional changes. The KANSEI Analyzer is a device capable of analyzing and visualizing emotions in real time, by continuously capturing these changes. Yasue Mitsukura, a professor at the Department of System Design Engineering at the Keio University Faculty of Science and Technology has directly engaged with human emotions from engineering perspectives to create this device.

How will technology that perceives and records the vagaries of the human mind transform our lives? We spoke to Professor Mitsukura about the challenges involved with the KANSEI Analyzer and the potential it may hold for future society.

A device which "reads" a person's emotions from brain waves

When you are amused, when you are angry, when you are relaxed-- even at times when we are purportedly not thinking--our brains continue to do their work. This is because a variety of information is constantly being exchanged between neurons in the brain, in the form of electrical signals. These changes in electrical activity occurring in the brain when measured as a waveform on an electroencephalogram (EEG) are generally referred to as brain waves. In order to scrutinize brain waves from outside the body, these currents flowing through the brain must be measured using electrodes attached to the human cranium. However, this will involve eliminating interference (noise) to the waveform caused by body movements, slight displacements of the electrodes, perspiration, muscle movements, and similar factors to isolate and visualize the brain wave as an EEG.

The mention of brain waves may well bring to mind the phrases "alpha waves" and "beta waves." These are referred to as the "basic rhythms," and indicate brain waves with specific frequency components that stand in for the state of activity of the brain as a whole. EEG waveforms are initially comprised of a mixture of waves of differing frequencies ranging from broad (slow) to fast waves. For this reason, the particular characteristics of brain waves are ultimately determined by isolating these waves using frequency analysis to identify the degree to which wave components of particular frequencies are present in the original waveform. "To give one example, the hormone cortisol is activated in the body when a person is angry. Cortisol eventually causes changes in the body as it is circulated through the blood. This is naturally accompanied by changes to brain waves. If we can accurately measure the 'degree of change' and render this in a mathematical equation, we can decipher the emotion of anger from the EEG waveform," Professor Mitsukura explains.

The KANSEI Analyzer, which took 17 years to prefect, is a system comprised of electrodes embedded in a belt resembling a hairband, coupled with an application which operates after the manner of a tablet device. Real-time brain wave analysis will begin immediately when the device is put on by the user after initial set-up of the application. A graph showing the passage of time on the horizontal axis and numerical values from 0 to 100 on the vertical axis is displayed on the screen of the tablet device. This will show the momentary fluctuations of emotions such as "degree of concentration," "stress levels," "degree of interest," or "sleepiness." The KANSEI Analyzer is compact and easy-to-deploy when compared to regular measurement of EEGs, which require extremely bulky equipment. Indeed, the capacity to measure brain waves when standing or moving was previously unthinkable. Interference too can be eliminated in real time, meaning that we can even, for example, measure the brain waves of athletes during workouts" Professor Mitsukura proudly asserts.

This means modeling the human mind

Professor Mitsukura, who initially specialized in control system engineering, first encountered brain waves through a television soap opera she found herself watching. "The plotline dealt with a person whose thoughts are unwittingly transmitted to another protagonist, a concept I found sufficiently intriguing that it prompted me to do various research. This was how I became obsessed. The work of mathematically modeling changes in the mind and body that occur when we communicate with others is precisely the concern of my specialization, namely control systems".

Even if the single word "modeling" is used to describe this, human behavior is in fact profoundly complicated. Therefore, it must also encompass a wide range of research methods. Some studies are conducted in the lab with animals, while others turn their attention to the actual sites of daily life, and compare the lifestyles of the people living there with changes in brain wave measurements. This is a research method referred to as translational research, which links basic research to clinical development. "Such phenomenon which cannot be understood by looking at the data alone can sometimes be understood by making one's way on-site. We hope to model human behavior using the full gamut of our research methodologies, from basic to applied. The attitude that 'all phenomena which occur in the world can be modeled using differential equations' has long remained unchanged."

The KANSEI Analyzer, which came to fruition following the overcoming of numerous travails, is currently in the spotlight after being picked up by a large number of media outlets. Professor Mitsukura also mentions that they are in the process of collaborating with a number of companies and organizations. This is due to the fact that real-time measurements of emotions using the device can facilitate evaluations on the usability of a variety of products and services. "Completing this device was a major turning point in our research. It now has many users among people involved in fields which had previously had no relationship with EEGs. I have a real sense that we are transitioning from the research and development phase into the deployment and collaboration phase."

Using the KANSEI Analyzer, we have gained the means to discern a person's emotions by selecting frequency components corresponding to specific ones (i.e., stress) from brainwaves containing heterogeneous, concurrent signals. Professor Mitsukura is now taking her research a further step beyond the "reading" of brain waves, delving into what she describes as "the control of emotions." "If we can achieve an understanding of the brain waves which decrease or increase as a result of specific emotional changes, then we should also be able to dictate emotional states by controlling these using external electrical stimulation" says Professor Mitsukura. She reports that some of the studies now underway have actually confirmed changes such as reductions in stress. "While the sense of nervousness has its own important role to play, if one is more nervous than the situation requires, one's performance can suffer as a result. We believe that we can reduce nervousness and increase concentration at the most critical moments, and also increase relaxation when we want to get a decent night's sleep."

Challenges and ethical issues of EEG research

It is apparent that the KANSEI Analyzer, which can fully visualize those "emotions" which were previously the exclusive domain of the person experiencing them, has had a highly positive reception outside Japan. One reason, Professor Mituskura believes, may be located in the differences in attitudes and cultural contexts on societal fronts towards new technologies. "Naturally there will be differences in temperament between people of other nationalities and Japanese, and perhaps even the legal frameworks among countries will differ. The real-time perception of emotions has virtually no precedent worldwide, and we certainly aspire to somehow applying this technology to addressing social challenges. We speculate that perhaps one of the keys to what is possible using emotional analysis is emotional control. The next important step will be in ensuring that we get a firm handle on the ethical challenges, which will inevitably arise as we proceed on our journey."

For example, emotional recognition has high potential for deployment as a technology to facilitate communication with people who have difficulty discerning emotions, such as those with dementia. Emotional control is also a technology with the potential to alleviate the suffering experienced by people with depression and similar issues. However the argument could also be made that, because we can read emotions from brain waves, these constitute one piece of personal information and, depending on how they are used, emotional control techniques have considerable potential for abuse. During the research phase, Professor Mitsukura and colleagues ensure that subjects understand that there will be limits on how their information is used as well as increased trust by obtaining written consent. Nevertheless, this may prove insufficient when it comes to social implementation. She asserts that: "Developers have an inevitable inclination to altruistic views of human nature as they proceed with their research. The potential for abuses to occur rarely crosses their minds. This is exactly why discussions from legal and ethical perspectives are so important. It is also why a platform for discussion among experts which transcends not just the parameters of medicine and engineering but also the humanities and sciences is so imperative."

A platform for the creation of "breakthrough power" through communication

Professor Mitsukura is involved in several projects in her capacity as Vice Director of KGRI, which promotes interdisciplinary and educational initiatives across academic disciplines, based on the school philosophy of jitsugaku, or solving issues by thinking critically from a scientific approach in order to reveal the empirical truth. She reveals that she has high expectations of the KGRI platform in terms of her own ongoing research. "When we take in the full view from the basic research through to social implementation, the research no longer fits within the frameworks of science and technology or medical-engineering collaborations. This is precisely why I hope to use our platform to actively realize communication and work together with people whose specializations are in other fields, including law and economics, which had previously had no connection to KGRI."

Based on her experiences to date, Professor Mitsukura strongly emphasizes the importance of "an environment which rewards perseverance." "Many people abandon ship due to trifling matters, and this is a genuine pity. I want to provide such people with the push which allows them to break through to a point where a clear view opens up to them if they invest just a little more effort." She believes that KGRI is required precisely because it represents a platform for "breakthrough power," in creating opportunities for active communication among experts in diverse fields.

Professor Mitsukura says that, even as a person who has consistently worked on modeling the human mind and body, there remain many things that she has yet to understand, and that many challenges are yet to be surmounted. These include how to measure emotions with devices which are plain enough that the user has no sense that they are wearing them; whether there are simpler methods by which to control emotions; and the particular ways that technology can be used to reduce the numbers of those suffering from dementia or depression. "Modeling deploys information from the past to the present. However we do not at this stage have enough information to predict the future. I would sincerely love to contribute through my work to the super-aged society of 2040 by promoting information sharing across domains through KGRI."

Photo: Susumu Ishito