Brain-Computer Interfaces based on multisensory Event-Related Potentials

This dissertation is organized along the four research questions in the form of scientific papers. We here give an overview of each chapter, their motivations, and the relation between them. Chapter 2: Controlling a tactile ERP-BCI in a dual-task. In this chapter we present the evaluation of what the costs of mental resources are to control a tactile ERP-BCI while at the same time performing a concurrent task using visual information. This is the first step towards applying a tactile ERP-BCI for navigation. For tasks like (serious) gaming cognitive resources are required, but when operating an ERP-BCI attending to stimuli also demands (cognitive) resources. We investigate whether or not these two tasks can be performed simultaneously, and what the effects on brain signals (and subsequently BCI performance) and task performance are. Chapter 3: Does bimodal stimulus presentation increase ERP components usable in BCIs? In this chapter we report the idea to increase ERP activity by means of bimodal (visual-tactile) stimulus presentation, with the goal to enhance BCI performance. Bimodal stimuli could evoke additional brain activity due to multisensory integration which may be of use in a BCI. We investigate effects of attending to bimodal visual-tactile (compared to unimodal) stimuli on the ERP. To this end we use stimulus pairs of tactile stimuli around the waist and visual stimuli embedded in a navigation environment presented on a display, corresponding in navigation direction. Chapter 4: Bimodal location-congruent ERP-BCIs: Increasing gaze-independent performance. In this chapter we further investigate bimodal (visual-tactile) ERP-BCIs and the role of location-congruency of the bimodal stimulus. Research has shown that bimodal stimuli do not necessarily have to be location-congruent to observe positive bimodal effects on task performance and brain activity, yet location-congruent bimodal stimuli may (further) positively affect task performance and ERP components. Whereas in chapter 3 we use a gaze-dependent setup as a first step and to compare results to traditional BCIs, in chapter 4 we take the next step by using a gaze-independent setup. In the latter case, the potential benefits of bimodal stimuli are expected to be greater as gaze-independent BCI performance is typically relatively low. Additionally, we study the effect of selectively attending to a modality in bimodal BCIs. Chapter 5: Control-display mapping in brain–computer interfaces. In this chapter we present our research on the effect of congruency regarding the relation between command options and stimuli in a BCI-context. When using a tactile ERP-BCI for navigation, mapping is required between navigation directions on a visual display and unambiguously corresponding tactile stimuli from a tactile control device: control-display mapping (CDM). Chapter 6: Discussion and conclusions. We discuss the results of the separate studies and integrate the studies to answer the main research question. Furthermore, we discuss the implications of our results, reflect on the usefulness of ERP-BCI for direct control and for other purposes, and make recommendations for future research. We finalise with some concluding remarks.