VR, Serious Games, Pattern Recall, Pattern Replication, EEG, GSR, HR, Biosignals
The incorporation of biosensors and physiological signals into a Virtual Reality (VR) experience opens up the possibility of creating adaptive, dynamic and personalised experiences for the user by generating a real-time feedback loop which tailors the player's experience, ensuring it maintains the user's engagement. This abstract describes the first of two stages in achieving this. This project is a part of a larger study which looks at the use of biosensors in VR.
To explore the possibility of creating a feedback loop using biosignals, a VR game has been developed using Unreal Engine 4 for the Oculus Rift. This game is a simple pattern replication game where a player has to memorise a pattern displayed on a screen in one room, move to a second room via two conjoined corridors and replicate the pattern on the screen in that room. The player will perform this task a number of times, each time the player will have to remember a longer pattern.
Two biosensors were incorporated into the game; the Empatica E4 and Myndplay's Myndband. The E4 looks, and can be worn like a wrist watch, it connects to the application via Bluetooth. The Myndband takes the form of a headband and also connects to the application via Bluetooth. The E4 measures Blood Volume Pulse (BVP) and Galvanic Skin Response (GSR), and the Myndband measures Electroencephalography (EEG). BVP gives a reading for a user's heart rate. The changes in a person's heart rate can help to infer a user's emotional state. GSR measures the changes in sweat gland activity that are reflective of the intensity of a person's emotional arousal. EEG is the recording of the electrical activity in the brain using electrodes placed on the scalp. The Myndband is a single probe EEG device which is placed on the forehead above the eye. The raw EEG signal can be broken down into a number of frequency bands, each representing a particular emotional state, furthermore the Myndband calculates values for attention and meditation, known as eSense values. These values could be useful in a feedback loop.
The purpose of the biosensors is to provide objective measures of the player's performance in the game. The physiological recordings taken from the biosensors can be graphed against events in the Virtual Reality game in order to determine the players emotional reactions to a particular event, such as success or failure replicating the pattern. In order to do this the VR application outputs a number of csv files corresponding to different data types; events, EEG, GSR and Heart Rate. The data in these files is timestamped so that the physiological signals can be synchronized with events in the game. This stage of the project focuses on gathering data which will be analysed in order to help create the feedback loop in the second stage of this project.
It is hoped that a feedback loop can be implemented which can adjust the difficulty of the pattern replication task in real-time, in order to keep the user engaged. If the player is frustrated or stressed by a particularly hard pattern, then the next pattern will be simpler. Similarly, if the player is bored or inattentive then the next pattern will be more difficult. It is believed that this will constantly keep the player on their toes, while not overwhelming them with difficult tasks.
The games virtual environment is modelled on a section of UCC's Western Gateway Building. It was essential that the environment have a high graphical and audio fidelity in order to ensure good ecological validity and immersion. To this end the games exact measurements of the rooms and corridors were taken. These measurements were used during the production of the 3D models, ensuring the virtual building faithfully represented the real building.
Testing has been carried out informally throughout the development process by colleagues with technical experience. There were two main aspects tested; testing of the VR experience and control scheme, and testing of the biosensors. Testing of the VR experience involved ensuring the effects of motion sickness were minimal, to ensure the control scheme was simple and easy to use and to establish that the instructions in the tutorial and experiment were easy to follow. The biosensors were tested to ensure the quality and consistency of the retrieved data.
It is planned to carry out experiments with between 20 and 40 people. Each participant will wear both biosensors for the duration of the experiment. They will first be presented with a tutorial which will help adjust the player to Virtual Reality and the control scheme. Following the tutorial is a free exploration section, this will help establish a baseline which can be compared to later on during the experiment. When the player is ready to proceed, they are teleported to the experiment section. The player will be presented with a set of instructions on a whiteboard upon arrival in the experiment section. The player is instructed to enter Room One, where they will be required to memorise a flashing pattern on a screen. They are then to move down two adjoining corridors into the second room. In this second room the player must replicate the pattern on an identical screen. If the player successfully replicates the pattern, they must then return to the first room where a new pattern will be presented to them. If they fail to replicate the pattern, then there may have to return to the previous room where the pattern will still be displayed. Several events are recorded in the events csv file; these include success and failure of entering patterns as well as events related to the tutorial section. Following the experiment, the subject will be asked to fill out a NASA Test Load Index survey which will access the pattern replication task.
Early anecdotal feedback from testers of the game indicate that the experience is immersive and enjoyable, with some reporting that the memorisation of patterns in VR is varyingly challenging. Following an analysis of the data produced in the first round of experiments it is planned to attempt to implement an algorithm which will determine a player's emotional state in order to provide a feedback loop which can adjust the difficulty of the pattern task.