Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

# Novel fNIRS study on homogeneous symmetric feature-based transfer learning for brain–computer interface

## Abstract

The brain–computer interface (BCI) provides an alternate means of communication between the brain and external devices by recognizing the brain activities and translating them into external commands. The functional Near-Infrared Spectroscopy (fNIRS) is becoming popular as a non-invasive modality for brain activity detection. The recent trends show that deep learning has significantly enhanced the performance of the BCI systems. But the inherent bottleneck for deep learning (in the domain of BCI) is the requirement of the vast amount of training data, lengthy recalibrating time, and expensive computational resources for training deep networks. Building a high-quality, large-scale annotated dataset for deep learning-based BCI systems is exceptionally tedious, complex, and expensive. This study investigates the novel application of transfer learning for fNIRS-based BCI to solve three objective functions (concerns), i.e., the problem of insufficient training data, reduced training time, and increased accuracy. We applied symmetric homogeneous feature-based transfer learning on convolutional neural network (CNN) designed explicitly for fNIRS data collected from twenty-six (26) participants performing the n-back task. The results suggested that the proposed method achieves the maximum saturated accuracy sooner and outperformed the traditional CNN model on averaged accuracy by 25.58% in the exact duration of training time, reducing the training time, recalibrating time, and computational resources.

## Introduction

Brain–computer interface (BCI) offers an interaction between the brain and external devices through signals generated from the brain without the peripheral nervous system’s involvement1. BCI is among such neurofeedback methods that may enhance patients’ quality of life suffering from acute motor debilities due to tetraplegia, stroke, and other spinal cord injuries2. More BCI applications are in areas of neuro-rehabilitation, communication and control, motor therapy and recovery, brain monitoring, and neuro-ergonomics3,4,5. Non-invasive neuroimaging modalities like functional magnetic resonance imaging (fMRI), electroencephalography (EEG), magnetoencephalography (MEG), and functional near-infrared spectroscopy (fNIRS) are greatly used in BCI systems for brain imaging and functional assessment of activities. Portable non-invasive neuroimaging techniques are generally preferred owing to their ease of use with fewer imaging protocols. The commonly used neuroimaging methodologies in this context are EEG and fNIRS. Both the modalities are portable and lightweight and require a small setup than the other techniques6. The electrodes capture EEG signals due to current neurons’ current variation due to postsynaptic activities7. While fNIRS constructs the brain’s functional neuroimages using near-infrared (NIR) light and gauge hemodynamic response function (HRF) in form of change in concentration of oxy and deoxygenated hemoglobin (HbO and HbR) to estimate the brain activities. Just like fMRI, the fNIRS also measures the blood oxygen level dependence (BOLD).

Using the BCI systems out of the laboratory needs to address several challenges such as robust signal acquisition, extracting required information from raw brain signals, and accurate control or command generation through data classification8,9. Another challenge hindering the BCI systems is the need for lengthy recalibration due to the high dimensionality and low signal-to-noise ratio (SNR) of EEG and fNIRS signals10. Typically, each new session’s calibration time for these modalities-based BCI systems takes up to 20–30 min approximately11,12. That extended time exhausts the subjects and puts extra fatigue even before the actual experimentation starts or even before the BCI system became fully functional. Another important factor is the non-stationary nature of brain signals. The exact brain state depends on mental and psychosomatic conditions, concentration level, factors like drowsiness, fatigue, anatomical differences, and statistical variations in the data13,14. The artifacts like instrumental noise, motion artifacts and poor sensitivity in naturalistic and non-structured environments, and the experimental errors due to variations in the electrodes’ resistivity may also alter the acquired brain signals15,16,17,18. All these factors result in a complex classification problem. To successfully classify the correct brain states, brain signals classification and neurofeedback are implemented in four stages: first is pre-processing, then feature extraction, classification, and lastly command generation19,20. The extracted features from brain signals are used to train the classifier. Machine or deep learning (DL) classifiers are used to discriminate various states of brain data. The different studies tried to address these challenges by exploiting various methods and algorithms while maintaining accuracy and information transfer rate (ITR) in a significant range7,12,21,22,23. Deep learning (DL) algorithms have been vigorously applied in different BCI studies such as an artificial neural network (ANN)24,25, convolutional neural networks (CNN)26,27, deep belief network (DBN)28, long short-term memory (LSTM)29,30, and cascade CNN-LSTM31. Although DL algorithms have superior learning capabilities and can address complex classification problems, at the same time, these algorithms have posed a unique challenge of Big Data in the BCI domain19. DL’s inherent bottleneck is the requirement of the huge amount of training data and computational resources for training deep networks29. The collection of a very large amount of neuroimaging data is very complicated and expensive in terms of time and resources, making it very hard to develop a substantial-scale, high-quality marked dataset for DL models’ training. Moreover, it is difficult to approximate probability distributions of the feature vectors from low SNR signals, mostly in the case of machine learning (ML) algorithms, where only a few trials are performed for multi-dimensional brain signals. All these factors lead to the poor performance of trained classifiers on new session data. In this scenario “Transfer learning” proved to be an encouraging approach candidate to deal with these problems.

Transfer learning algorithms used in EEG-based BCI are primarily based on two approaches, one is importance sampling cross-validation methods20,32, and the second is instance selection methods33,34. Covariate shift adaptation (CSA) proposed in the study20 uses the importance sampling cross-validation to weigh the data from the target domain (other subjects). The final prediction function is estimated based on parts with high weights, and others are rejected. In various studies33,34, trials are selected on an active learning base and based on an instance selection approach close to the new subject’s few informative trials. Then, to train the BCI model, selected trials based on an instance selection approach are added to the new subject’s existing labeled trials. In35, Zhang et al. proposed a diagnosis to susceptibility to alcoholism was done via extracting features using deep learning algorithms combined with transfer learning. Most of the proposed transfer learning algorithms in the feature domain focus on improving common spatial patterns (CSP). CSPs are improved with modifications of either the covariance matrix using the estimation method36,37, or the CSP optimization function7,38. An extension of CSP, proposed by Samek et al. in39, transferred stationary information instead of discriminative information across multiple subjects by learning a stationary subspace. Similarly, to solve the Motor imagery (MI) for BCI, the authors in40 proposed a combination of Continuous Wavelet Transform (CWT) along with deep learning-based transfer learning. Many existing MI-based BCI transfer learning algorithms on the classification domain have used domain adaptation techniques41,42,43 and ensemble learning of classifiers7,23. Domain adaptation techniques use the source domain classifier for the target domain while adjusting its parameters according to target data. Moreover, multi-task learning is also used in BCI21,44, where the classification parameters are learned together from multiple subjects, resulting in minimization of the average total errors and differences among the parameters of the separate classifiers. This approach was a success to some extent. Still, it had its constraints as many parameters needed to be optimized simultaneously, making it computationally expensive. Similarly, that approach does not consider the similarities and dissimilarities between the data within subjects during the learning process. Several studies on BCI for the classification of different controlled and uncontrolled cognitive tasks29,45,46,47,48,49,50 have used fNIRS. Despite being getting popular, to the best of the authors’ knowledge, there is no study on applying transfer learning in fNIRS-based BCI. The application of symmetric homogeneous feature-based transfer learning in the fNIRS domain is novel. This study’s major takeaway is that optimization obtained through transfer learning is superior to traditional DL network training.

Symmetric feature-based transfer learning approach discovers underlying meaningful structures between the domains to find a common latent feature space that has predictive qualities while reducing the marginal distribution between the domains51. The exchange learning approach proposed by Prettenhofer addresses the complicated situation of a source space containing marked and unlabeled information and a physical space containing unlabeled information. The auxiliary correspondence learning procedure from Blitzer is applied to this issue. Supplemental correspondence learning depends on the manual meaning of turn works that catch correspondence between the source and target spaces. Viable rotate capacities should utilize highlights that happen as often as possible in the two areas and have significant prescient characteristics. Each turn work is transformed into a linear classifier using information from the source and target spaces. From these turn classifiers, correspondences between highlights are found, and an inactive component space is found out. The unused component space is utilized to prepare the last objective classifier. The paper by Prettenhofer uses this answer to take care of the issue of text order where the source is written in one language, and the objective is written in an alternate style. In this particular execution alluded to as cross-language essential correspondence learning (CLSCL), the rotate capacities are characterized by sets of words, one from the objective and one from the source, that speaks to coordinate word interpretations from one language to the next. The tests are performed on the utilization of report assumption characterization and archive point arrangement. English archives are utilized in the source, and other language reports are being used in the objective. The benchmark technique used in this test prepares a student on the marked source records; at that point deciphers the objective reports to the source language and tests the translated form. An upper bound technique is set up via preparing a student with the named target archives furthermore, experimenting with the objective reports. Standard order precision is estimated as the exhibition metric52.

## Methods

### Dataset and data acquisition

This study used an open-source meta-dataset acquired at the Technische Universität Berlin by Jaeyoung Shin et al. in 201739,53. The dataset includes fNIRS data of the scalp for different levels of mental workload acquired from 26 subjects. NIRScout (NIRx Medizintechnik GmbH, Berlin, Germany) was used for NIRS data acquisition using the configuration of 36 channels, according to the internationally recognized 10-5 system as shown in Fig. 1, at a sampling rate of 10.4 Hz. Sixteen optodes, a combination of sources with detectors were positioned at the frontal lobe across the region of AFz to AF8, and four channels were paced at C3, C4 for the motor cortex region. Four channels were places in the parietal region across P3 and P4. Likewise, four channels around the POz region for the occipital region. The distance between the source and the detectors was ensured to be 30 mm40. All participants were informed about the experimental procedure and gave written informed consent prior to the experiment. All the experiments were conducted in accordance with the Declaration of Helsinki and was approved by the Ethics Committee of the Institute of Psychology and Ergonomics, Berlin Institute of Technology.

Before the experimentation starts, all subjects were seated in a comfortable chair in front of a 24-in. LCD monitor, placed at a distance of 120 cm. It is pertinent to mention that the distance between the subject and the monitor is 120 cm. Subjects were asked to press numeric keys 7 and 8 to record their response and to ensure the subject’s engagement during data acquisition, with their index and middle finger, using a keypad attached to their right side. Furthermore, subjects were instructed to stay focused during the experiment by restricting their eye movement only to the monitor in order to avoid motion artifacts. The experiment protocol was designed to perform three cognitive tasks i.e., n-back, Discrimination Selection Response (DSR), and Word Generation (W.G) by each subject. The tasks were performed in descending order depending on task difficulty level as due to the long duration of tasks and data recording, the subject’s focus decreases with apparent stress and fatigue. First task A was completed, then C, and lastly B. In this study, only dataset A (n-back) is used and is explained in detail in the next section. So, first, the n-back task was performed followed by W.G and lastly, data acquisition was done for DSR. For further information on other datasets (DSR and W.G) and analysis39. The time sequence of the designed n-back experiment is shown in Fig. 2.

### Data pre-processing

The acquired data were first translated to the oxy and deoxy-hemoglobin (HbO and HbR) intensity variations to pre-process the fNIRS data. The conversions were made through the modified Beer-Lambert law (MBLL)41. The fNIRS raw data were acquired and sampled at 10 Hz. This dataset’s fundamental frequency was very low, so the down sampled was not fed into the Butterworth bandpass filter. Instead, low pass filtered is employed to avoid losing the essential frequency component42. The cutoff frequency of the filter was set to 0.2 Hz to remove the artifacts due to blood pressure, heartbeat and breathing and high frequency instruments.

### Proposed convolutional neural network model

In this study, a convolutional neural network (CNN) was used to classify three mental workloads (MWL) classes owing to its reputation and increase in use for different MWL classification studies26,27,29. CNN is a deep neural network that may integrate one or more convolutional layers with a pooling layer, batch norm layer, activation layer, dense layer, and at very last an output layer. The most important CNN layer, i.e., the convolutional layer, allows its inputs to pass through cascaded filters bank and performs simple convolution operations. Essentially convolution layers output feature maps extracted from the input due to convolution, i.e., shifting and multiplication of input signal and filter43. These feature maps are then used as an input to the next layer in the CNN architecture or as a set of definitive key features on which classification is performed in the last fully connected layers.

$$Output \; size\, \left(W,H\right)= \frac{(N-F)}{Stride }+1$$
(1)

where W and H are the width and height of the output activation map or feature map, N is the dimension of the input activation or feature map, F is the dimension of filter sliding over the input image or activation map, the stride is the number of steps taken while sliding filter. While the parameters of a layer are calculated using:

$$Parameters=\left(W*H*K\right)+K \,biases$$
(2)

W and H are the width and height of the output activation map or feature map, K is the number of filters, and K biases are the number of biases.

The mathematical formulation of CNN layers is well explained in44. During the CNN model training, both filter bank parameters and dense layer weights are adjusted throughout the period. The model precisely fits the training dataset with the least possible error. Successful implementation of CNN for a given dataset mainly relies on the fact that different data domains usually have some standard key features shared across all of its elements (such as images). But this is not the case when it comes to generalization in areas with high inter-subject unpredictability like brain signals acquired with EEG, fNIRS, fMRI methodologies, where data differ from subject to subject and depend on a lot of external and internal factors45. The research’s CNN models are based on a feed-forward CNN architecture comprising pairs of convolution and pooling layers46. After initial tests on different feed-forward CNN architectures, the chosen CNN architecture with complete parameters and structure is shown in Fig. 3. A fully connected feed-forward CNN network is selected with two convolution layers, a max-pooling layer, followed by a flattening and dense layer. Finally, a fully connected layer ends into the final output layer.

### Transfer learning

Since the study is based on homogeneous transfer learning, it is assumed that multiple fNIRS sessions previously acquired from different subjects or the same subject on the same or different tasks are already acquired. Throughout the literature, various research studies used different terminologies for similar concepts of transfer learning, types of transfer learning, and their mathematical formulation like domain adaptation, knowledge transfer, and transfer learning; following definition of transfer learning is used in this study:

A domain D consists of two essential parts, a feature space also known as latent space X and a marginal probability distribution (MPD) P(X), where feature vectors X = {× 1, . . . , xn}  X. In the case of BCI, the generation of command is the classification goal, and the channel readings are considered as features, then xi is the ith feature vector (instance) corresponding to the ith generated command, n is the numbers of feature vectors in X, and the X is the space of all possible feature vectors. For a given domain D, a task T can be defined as a label space Y and a predictive function F < . > . The predictive function F < . > is learned from the feature instance and corresponding label pairs [xi, yi] where xi X and yi Y. In the BCI problem, Y is the set of labels that might be rest, open, close commands, yi takes on one of the command values, and f(x) is the function approximator that predicts the label value for the command classification x. From the above definitions, a data domain is given by D = [X, P(X)], and a task is provided by T = [Y, F < . >]. Also, for consistency, we will represent source domain data as D.S., and by definition, it will be given by D.S. = [(xS1, yS1). . . , (xSn, ySn)], where xSi X.S. and it is the ith data point of D.S. and ySi Y.S. is the corresponding feature label for xSi. Likewise, the target domain data can be given as D.T. where D.T. = [(xT1, yT1). . . , (xTn, yTn)] where xTi X.T. and it is the ith data point of D.T. and yTi,  Y.T. is the corresponding class label for xTi. Now, the source task, the target task, the source predictive function, and the target predictive function can be represented by T.S., T.T., F.S. < . > , and F.T. < . > , respectively. Now we can define transfer learning as improving the F.T. < . > , target predictive function, by using the gathered information from source domain data D.S. and source task T.S., given source domain D.S and target domain D.T. with or without target tasks T.T. Transfer learning can be categorized into two types: (1) Homogenous transfer learning and (2) Heterogeneous transfer learning. Mathematically, the condition where the source and target domain features Xt and Xs are equal for transfer learning is called homogenous transfer learning47. Whereas the state where the source and target domain features Xt and Xs are not similar is called heterogeneous transfer learning. Homogenous and heterogeneous transfer learning is also called intra-domain and inter-domain transfer learning, respectively. This study performed homogenous transfer learning on fNIRS data and evaluated its performance and viability for deep learning networks.

## Methodology

The available dataset of 26 participants is divided into three subsets with an approximately 60:20:20 ratio. The first 16 participants’ data is used to train the CNN network to learn the task’s domain knowledge. This trained network is then used as parameters trained on D.S. transferred to D.T. The validity and viability of transfer learning are evaluated under the following assumption: the transfer learning efficiently transferred the source domain knowledge to the target domain, it required the reduced training iterations for deep learning models, and while transferring the learned domain knowledge, the transfer learning increases the achieved classification accuracies. We evaluated these assumptions by placing the remaining ten subject data into two groups and named them as the baseline and control groups. The baseline group is used for training conventional deep neural network models in a standard and widely adapted setting. The aim of this study is to learn the intra-subject varainces while performing the same task, as evident in the name homogenous transfer learning. We intended to learn features that maximally differentiate the n-back classes (0,2,3-back and rest) for the new subject in least amount of training data and time. For the training of the trained CNN network, we performed experiments with 70:30 split ratio, leave one out (LOO) and tenfold cross-validation methods. The result obtainerd with tenfold cross validation results were the best performing one. In contrast, the control group is retrained and fine-tuned on the pre-trained CNN model with domain knowledge Ds and Ts from the first 16 participants. The pre-trained model is fed with the control group data D.T. and trained with different epochs from 10 up to 60. The different experiments were performed for the retraining and fine-tuning process. First of all, the complete learnable parameters of the trained CNN were freezed except the last dense layer, and retraining was performed. Next, we repeated the same experiment by unfreezing the last two dense layers. The process is repeated up till the first convolutional layer. The retraining by unfreezing the last two dense layers yielded the best results and is used in further analysis. These obtained accuracies are compared with the baseline group accuracies. The statistical analysis is performed on the obtained accuracies, and the conclusion is discussed in the next section.

## Results

### Statistical analysis

The statistical analysis was performed between baseline and control groups (reference). Shapiro–Wilk test is used to gauge the normal distribution of baseline and control groups’ accuracies. It is a right-tailed normal distribution criterion with a null and alternate hypothesis as the H0 (null hypothesis): if the population is normally distributed and H1: if the population is not normally distributed48. For all epochs, the resulting p-value is (probability with data normal distribution with the confidence of 95%) > α (confidence level); therefore, the H0 is accepted. The quantile–quantile or QQ-plot is another method used for a graphical illustration of the Shapiro–Wilk test and shows the significance test run on the baseline group as shown in Fig. 4. Based on the Shapiro Wilk test results, it became established that the statistical significance measures with other scales like t-test and ANOVA are possible on the current accuracies. The paired t-test is calculated with the null hypothesis (H0): There is no significant difference between the baseline and control group accuracies, and the p-value > 0.05 and the alternative hypothesis (H1): the two populations are not equal, a significant difference between these accuracies and the p-value < 0.05. The two-tailed P-value (2.443e−8) is less than 0.0001 with the degree of freedom (DOF) = 9. By conventional criteria, this difference is considered to be extremely statistically significant. After analysis, the t-value comes out to be t = 17.8723 . The null hypothesis (H0) is rejected with p < 0.05, and the alternate hypothesis is accepted.

## Findings

This section presents the results of transfer learning for fNIRS-based BCI after experimentation and statistical analysis. The first 16 participants’ data is used to train the CNN network to learn the task’s domain knowledge. This pre-trained CNN model is used to re-train and fine-tune the learned parameters on the control group. The CNN network is trained and tested on both baseline and control groups with tenfold cross-validation. Figure 5 shows the accuracy of control group subjects with different training epochs (n = 10, 20, 30… 60), while Fig. 6 shows the accuracy of baseline group subjects trained on the randomly initialized CNN network. Tables 1 and 2 represent the accuracies of the control group and the baseline group, respectively. Both networks are trained with a range of epochs from 10 to 60 with an increment of 10 epochs per step. The optimality of network models is measured by the ‘accuracy’ metric that tells the percentage of true positives from all predictions. The average accuracies for the control group after each 10 step epochs were 51.42, 63.72, 73.78, 82.76, 90.43, and 94. %, while for the baseline group, the average accuracies were 52.14, 63.96, 64.89, 66.13, 67.83, and 68.95%. After 60 epochs, the training is stopped because the pre-trained CNN starts over-fitting. The results show that the proposed technique successfully transferred the learned knowledge and achieved the maximum accuracy of 97.83%. The control group’s saturated accuracy results are obtained earlier than the conventional CNN on the baseline group, which significantly reduces the number of training epochs and effectively reduces the time required to train the network. The proposed transfer learning method also outperformed the averaged accuracy achieved using the learned CNN model over the traditional CNN model by 25.58% in the exact duration of training time as shown in Fig. 7. The Keras is used for prototyping with the TensorFlow backend. The networks are trained on Nvidia GEFORCE GTX 1060 GPU, having 3 GB VRAM on spyder IDE. The number of neurons, the number of filters, the number of layers, their combinations, dropout, and max-pooling percentage, etc., all remain to be at best ‘hyper parameters’. For this study, the network architecture design process was as follows: create a network with a minimum number of parameters, a single convolutional layer, a single pooling layer, and one dense layer, then tune other hyperparameters. Add more layers and then tune network hyperparameters with grid search using the sklearn wrapper and choose the best performing network.

Machine or deep learning (DL) classifiers are used in various other studies to discriminate various states of brain data. The different studies tried to address these challenges by exploiting various methods and algorithms while maintaining accuracy and information transfer rate (ITR) in a significant range7,12,21,22,23. Deep learning (DL) algorithms have been vigorously applied in different BCI studies such as an artificial neural network (ANN)24,25, convolutional neural networks (CNN)26,27, deep belief network (DBN)28, long short-term memory (LSTM)29,30, and cascade CNN-LSTM31. Although DL algorithms have superior learning capabilities and can address complex classification problems, at the same time, these algorithms have posed a unique challenge of Big Data in the BCI domain19. DL’s inherent bottleneck is the requirement of a huge amount of training data and computational resources for training deep networks29. The collection of a very large amount of neuroimaging data is very complicated and expensive in terms of time and resources, making it very hard to develop a substantial-scale, high-quality marked dataset for DL models’ training. Moreover, it is difficult to approximate probability distributions of the feature vectors from low SNR signals, mostly in the case of machine learning (ML) algorithms, where only a few trials are performed for multi-dimensional brain signals. All these factors lead to the poor performance of trained classifiers on new session data. In this scenario “Transfer learning” proved to be an encouraging approach candidate to deal with these problems.

## Discussion

Transfer learning aims to produce an efficient model to map the learned knowledge from a source domain task to a different but related target domain task49,50. Training deep learning models only on target tasks may result in degraded performance due to insufficient data or labeled instances. Transfer learning improves the model’s ability to classify target instances by utilizing the source domain knowledge54. With the inherent constraints of collecting neuroimaging data and the high training iterations for deep learning models in BCI, transfer learning provides promising results. The collection of neuroimaging data is very complicated and expensive both in terms of time and resources, making it very hard to develop a substantial-scale, high-quality marked dataset for the training of deep learning models.

Usually while using machine learning algorithms on multi-dimensional brain signals, it is often difficult to approximate probability distributions of the features from low SNR signals with only a few trials. However, in this case the model’s ability to classify target instances can be improved by enhancing the training with supplementary labeled data from a related source domain. In last few years, various researchers have tested different transfer learning approaches on EEG-based BCI20,32,33,34. But the real challenge arises while distinguishing inherent cross-domain noise due to the varied distributions from the beneficial knowledge in a source domain and then applying that knowledge to a target domain. According to literature, transfer learning can be split into two main categories according to the feature space: homogeneous and heterogeneous transfer learning55. In homogeneous transfer learning, the feature spaces of the source and target domains is of the same dimension (Ds = Dt) while the data of both domains is represented by the same attributes (Xs = Xt) and labels (Ys = Yt). Thus, homogeneous transfer learning aims to bridge the gap in the data distributions experienced during cross-domain transfer55. While, in Heterogeneous Transfer Learning, the feature spaces between the source and target are non-equivalent and are non-overlapping i.e., Xs ≠ Xt and/or Ys ≠ Yt. The source and target domains may share no features or labels, and the feature spaces' dimensions also may differ. Thus, for cross-domain transfer, Heterogeneous Transfer Learning requires feature and label space transformations to bridge the gap for knowledge transfer and to handle the cross-domain data distribution differences.

For EEG-based BCI, both homogenous and heterogeneous transfer learning approaches are used in literature i.e., instance-based, feature-based, and parameter-based transfer learning7,19,49. Every transfer learning approach focuses on improving target prediction function using source and target domain data differently. Like, the instance-based transfer learning approach assumes that although the entire source domain cannot be used directly but some source domain data can be re-used for learning the target domain i.e., by combining the few target labeled data with some instances from the source domain, by some weight adjustments, if needed. While, the feature-representation transfer learning approach focuses on improving the construction of feature space for the target domain using the source domains’ data instead of combining target labeled data with source domain data to improve target prediction function19. The performance of the target task is thus enhanced by minimizing classification errors. Lastly, the parameter-based transfer learning relates target domain with ethe source domain by assuming that parameters and prior distributions are shared between the source’s functions and target tasks thus can be transferred to the target prediction function resulting in reduction of the classification errors.

In most of the transfer learning BCI approaches, some sort of knowledge is transferred between a source and target domain either by (i) finding some structure in the data that is invariant across datasets and known as stationary information transferred, (ii) finding some structure in a way the decision rules differ between different subjects and known as discriminative information transfer37. Here, the focus is on constructing discriminative systems by exploiting the features, filters, and classifiers to transfer stationary information. While in the case of discriminative information transfer, the aim is to construct more invariant systems that rely on common information across the source and target domains56. Due to the popularity of machine learning algorithms in BCI, various researchers opted for experimentation with transfer learning for the machine learning classifiers7,20,33,36. In20, authors proposed an instance-based transfer learning method, namely Bagged importance-weighted LDA (Bagged IWLDA), based on the covariant shift adaptation method. The purpose was to reduce the non-stationarities present in the recording of the different sessions. Another study33 proposed an instance-based transfer learning method based on active transfer learning (ATL) to transfer particular instances i.e., to find the most informative samples for labeling. This approach results a higher performance learning process with less labeling effort. In literature7,36 researchers have proposed different feature-based transfer learning methods for EEG-based BCI studies. Among all the presented methods in literature, spatial filters are most commonly used to learn the new feature representation for BCI transfer learning. Over the years different algorithms are designed to compute spatial features. While, Common Spatial Patterns (CSP), is the most commonly used algorithm of all for extracting discriminative features from EEG signals. Despite of its popularity among the researchers, the main bottleneck is its overfitting when there are only a few trials of data is available for training. Therefore, different improved approaches for CSP were proposed to overcome this limitation. In7 and36, linear discriminant analysis and stationary subspace-based CSPs were proposed, respectively. More precisely, it was proposed that using the data from a subset of source subjects could improve the CSP covariance matrix estimation. These studies showed that by using smart methods alongside CSP, this problem could be overcome.

This study proposed a novel symmetric homogeneous feature-based transfer learning methodology in the classification realm to increase the fNIRS-based BCI performance by reducing the training time, addressing the problem of insufficient data, and increasing the accuracy. The symmetric homogeneous feature-based transfer learning is applied in the following steps: A deep learning convolutional neural network (CNN) model is trained on multi-subject data acquired with the fNIRS system from subjects during Mental workload “n-back” tasks. Second, the trained CNN model parameters are transferred to train and fine-tune the unseen subjects’ data. Last, the transferred model’s learned feature space is utilized to regularize the re-training and fine-tuning process. The results confirmed that the proposed technique successfully transferred the learned knowledge and achieved the maximum accuracy of 96.5% with 20 epochs earlier than the conventional DNN method. The proposed transfer learning method also outperformed the averaged accuracy achieved using the learned CNN model over the traditional CNN model by 24.5% in the same duration of training time.

## Conclusion

In this research study, a feature-based homogenous transfer learning approach was explored for the classification domain to reduce the training and calibration time for the fNIRS-based BCI systems. We evaluated the validity and viability of transfer learning for the fNIRS-based BCI systems under the following different assumptions. First, the transfer learning efficiently transferred the source domain knowledge to the target domain and required reduced training iterations for deep learning models. Second, transfer learning minimizes the need for a large amount of data needed for training deep learning models for the target domain. We used 16 subjects to train the CNN network and named it a ‘learned CNN’ network that learns the source domain knowledge of the n-back dataset. Further, we split the remaining ten subjects into two groups, i.e., the control and baseline group. The control group is trained with the learned CNN network and baseline with a randomly initialized CNN network, and their accuracies are compared using statistical analysis. The results suggested that applying the proposed feature-based transfer learning algorithms could achieve the maximum saturated accuracy sooner than the baseline group, which reduces the training time. The proposed transfer learning method also outperformed the averaged accuracy achieved using the novel learned CNN model (94.52%) over the traditional CNN model (68.94%) by 25.58%. Thus, the proposed transfer learning methodology for fNIRS is a promising solution for both the problems of increased training iterations for deep learning models and limited training datasets for BCI.

The classification of different brain activities and training time of BCI models would remain an area of concern, leaving room for more research in using transfer learning methodology for fNIRS-based BCI. This study utilized the data acquired on the same task from different subjects. Future research work may explore the domains of the intrasession BCI dataset with heterogeneous transfer learning approaches. Additional experimentation of transfer learning for deep neural networks (DNN) designed explicitly for time-series data such as Long Short-Term Memory (LSTM) may be used to explore more optimal results with DNN compared to other machine learning classifiers. This study serves as a baseline study for future transfer learning research in fNIRS-based BCI.

## References

1. Khan, M. & Hong, K. Hybrid EEG–fNIRS-based eight-command decoding for BCI: Application to quadcopter control. Front. Neurorobot. 11, 6. https://doi.org/10.3389/fnbot.2017.00006 (2017).

2. Weyand, S., Takehara-Nishiuchi, K. & Chau, T. Weaning off mental tasks to achieve voluntary self-regulatory control of a near-infrared spectroscopy brain–computer interface. IEEE Trans. Neural Syst. Rehabil. Eng. 23, 548–561 (2015).

3. Hong, K. & Khan, M. Hybrid brain–computer interface techniques for improved classification accuracy and increased number of commands: A review. Front. Neurorobot. 11, 35. https://doi.org/10.3389/fnbot.2017.00035 (2017).

4. Barbosa, S., Pires, G. & Nunes, U. Toward a reliable gaze-independent hybrid BCI combining visual and natural auditory stimuli. J. Neurosci. Methods 261, 47–61 (2016).

5. Li, Y., Zhou, G., Graham, D. & Holtzhauer, A. Towards an EEG-based brain–computer interface for online robot control. Multimed. Tools Appl. 75, 7999–8017 (2015).

6. Hong, K. S., Khan, M. J. & Hong, M. J. Feature extraction and classification methods for hybrid fNIRS-EEG brain–computer interfaces. Front. Hum. Neurosci. 12, 246. https://doi.org/10.3389/fnhum.2018.00246 (2018).

7. Lotte, F. & Guan, C. Learning from other subjects helps reducing brain–computer interface calibration time. In 2010 IEEE International Conference on Acoustics, Speech and Signal Processing 2010 Mar 14, 614–617 https://doi.org/10.1109/ICASSP.2010.5495183.

8. Holtzer, R. et al. Online fronto-cortical control of simple and attention-demanding locomotion in humans. Neuroimage 112, 152–159. https://doi.org/10.1016/j.neuroimage.2015.03.002 (2015).

9. Naseer, N., Hong, M. & Hong, K. Online binary decision decoding using functional near-infrared spectroscopy for the development of brain–computer interface. Exp. Brain Res. 232, 555–564. https://doi.org/10.1007/s00221-013-3764-1 (2013).

10. Zhang, X. et al. A survey on deep learning-based non-invasive brain signals: Recent advances and new frontiers. J. Neural Eng. 18, 031002. https://doi.org/10.1088/1741-2552/abc902 (2021).

11. Millán, J. et al. Combining brain–computer interfaces and assistive technologies: State-of-the-art and challenges. Front. Neurosci. 4, 161. https://doi.org/10.3389/fnins.2010.00161 (2010).

12. Schölkopf, B, Platt, J. & Hofmann T. Reducing calibration time for brain–computer interfaces: a clustering approach. In Advances in Neural Information Processing Systems 19: Proceedings of the 2006 Conference, 753–760 (MIT Press, 2007).

13. Wronkiewicz, M., Larson, E. & Lee, A. Leveraging anatomical information to improve transfer learning in brain–computer interfaces. J. Neural Eng. 12, 046027. https://doi.org/10.1088/1741-2560/12/4/046027 (2015).

14. Jayaram, V., Alamgir, M., Altun, Y., Scholkopf, B. & Grosse-Wentrup, M. Transfer learning in brain–computer interfaces. IEEE Comput. Intell. Mag. 11(1), 20–31. https://doi.org/10.1109/MCI.2015.2501545 (2016).

15. Huppert, T., Diamond, S., Franceschini, M. & Boas, D. HomER: A review of time-series analysis methods for near-infrared spectroscopy of the brain. Appl. Opt. 48, D280. https://doi.org/10.1364/ao.48.00d280 (2009).

16. Ward, J. A. & Pinti, P. Wearables and the brain. IEEE Pervasive Comput. 18(1), 94–100. https://doi.org/10.1109/MPRV.2019.2898536 (2019).

17. Chiarelli, A. et al. Fiberless, multi-channel fNIRS-EEG system based on silicon photomultipliers: Towards sensitive and ecological mapping of brain activity and neurovascular coupling. Sensors 20, 2831. https://doi.org/10.3390/s20102831 (2020).

18. von Lühmann, A., Zimmermann, B. B., Ortega-Martinez, A., Perkins, N., Yücel, M. A. & Boas, D. A. Towards neuroscience in the everyday world: progress in wearable fNIRS instrumentation and applications. In Optics and the Brain 2020 Apr 20, BM3C-2. (Optical Society of America) https://doi.org/10.1364/BRAIN.2020.BM3C.2.

19. Azab, A. M., Toth, J., Mihaylova, L. S. & Arvaneh, M. A review on transfer learning approaches in brain–computer interface. Signal Process. Mach. Learn. Brain-Machine Interfaces. 1, 81–98. https://doi.org/10.1049/pbce114e_ch5 (2018).

20. Yan, L., Kambara, H., Koike, Y. & Sugiyama, M. Application of covariate shift adaptation techniques in brain–computer interfaces. IEEE Trans. Biomed. Eng. 57, 1318–1324. https://doi.org/10.1109/TBME.2009.2039997 (2010).

21. Tu, W. & Sun, S. A subject transfer framework for EEG classification. Neurocomputing 82, 109–116. https://doi.org/10.1016/j.neucom.2011.10.024 (2012).

22. Fazli, S. et al. Subject-independent mental state classification in single trials. Neural Netw. 22, 1305–1312. https://doi.org/10.1016/j.neunet.2009.06.003 (2009).

23. Mughal, N. E., Khalil, K. & Khan, M. J. fNIRS based multi-class mental workload classification using recurrence plots and CNN-LSTM. In 2021 International Conference on Artificial Intelligence and Mechatronics Systems (AIMS) 2021 Apr 28, 1–6 (IEEE) https://doi.org/10.1109/AIMS52415.2021.9466084.

24. Naseer, N., Qureshi, N., Noori, F. & Hong, K. Analysis of different classification techniques for two-class functional near-infrared spectroscopy-based brain–computer interface. Comput. Intell. Neurosci. 2016, 1–11. https://doi.org/10.1155/2016/5480760 (2016).

25. Trakoolwilaiwan, T., Behboodi, B., Lee, J., Kim, K. & Choi, J. Convolutional neural network for high-accuracy functional near-infrared spectroscopy in a brain–computer interface: Three-class classification of rest, right-, and left-hand motor execution. Neurophotonics 5, 1. https://doi.org/10.1117/1.NPh.5.1.011008 (2017).

26. Chiarelli, A., Croce, P., Merla, A. & Zappasodi, F. Deep learning for hybrid EEG-fNIRS brain–computer interface: Application to motor imagery classification. J. Neural Eng. 15, 036028. https://doi.org/10.1088/1741-2552/aaaf82 (2018).

27. Asgher, U., Khalil, K., Ayaz, Y., Ahmad, R. & Khan, M. J. Classification of mental workload (MWL) using support vector machines (SVM) and convolutional neural networks (CNN). In 2020 3rd International Conference on Computing, Mathematics and Engineering Technologies (iCoMET) 2020 Jan 29, 1–6 https://doi.org/10.1109/iCoMET48670.2020.9073799.

28. Ho, T., Gwak, J., Park, C. & Song, J. Discrimination of mental workload levels from multi-channel fNIRS using deep leaning-based approaches. IEEE Access 7, 24392–24403. https://doi.org/10.1109/ACCESS.2019.2900127 (2019).

29. Asgher, U. et al. Enhanced accuracy for multiclass mental workload detection using long short-term memory for brain–computer interface. Front. Neurosci. https://doi.org/10.3389/fnins.2020.00584 (2020).

30. Yoo, S., Woo, S. & Amad, Z. Classification of three categories from prefrontal cortex using LSTM networks: fNIRS study. In 2018 18th International Conference on Control, Automation and Systems (ICCAS), 1141–1146 (2018).

31. Ditthapron, A., Banluesombatkul, N., Ketrat, S., Chuangsuwanich, E. & Wilaiprasitporn, T. Universal joint feature extraction for P300 EEG classification using multi-task autoencoder. IEEE Access 7, 68415–68428. https://doi.org/10.1109/ACCESS.2019.2919143 (2019).

32. Zanini, P., Congedo, M., Jutten, C., Said, S. & Berthoumieu, Y. Transfer learning: A Riemannian geometry framework with applications to brain–computer interfaces. IEEE Trans. Biomed. Eng. 65, 1107–1116. https://doi.org/10.1109/TBME.2017.2742541 (2018).

33. Hossain, I., Khosravi, A. & Nahavandhi, S. Active transfer learning and selective instance transfer with active learning for motor imagery based BCI. In 2016 International Joint Conference on Neural Networks (IJCNN) 2016 Jul 24, 4048–4055 https://doi.org/10.1109/IJCNN.2016.7727726.

34. Hossain, I., Khosravi, A., Hettiarachchi, I. & Nahavandi, S. Multiclass informative instance transfer learning framework for motor imagery-based brain–computer interface. Comput. Intell. Neurosci. 2018, 1–12. https://doi.org/10.1155/2018/6323414 (2018).

35. Zhang, H., Silva, F. H. S., Ohata, E. F., Medeiros, A. G. & Rebouças Filho, P. P. Bi-dimensional approach based on transfer learning for alcoholism pre-disposition classification via EEG signals. Front. Hum. Neurosci. https://doi.org/10.3389/fnhum.2020.00365 (2020).

36. Kang, H., Nam, Y. & Choi, S. Composite common spatial pattern for subject-to-subject transfer. IEEE Signal Process. Lett. 16, 683–686. https://doi.org/10.1109/LSP.2009.2022557 (2009).

37. Samek, W., Meinecke, F. & Muller, K. Transferring subspaces between subjects in brain–computer interfacing. IEEE Trans. Biomed. Eng. 60, 2289–2298. https://doi.org/10.1109/TBME.2013.2253608 (2013).

38. Samek, W., Kawanabe, M. & Muller, K. Divergence-based framework for common spatial patterns algorithms. IEEE Rev. Biomed. Eng. 7, 50–72. https://doi.org/10.1109/RBME.2013.2290621 (2014).

39. Shin, J. et al. Simultaneous acquisition of EEG and NIRS during cognitive tasks for an open access dataset. Sci Data 5, 180003. https://doi.org/10.1038/sdata.2018.3 (2018).

40. Hwang, H., Lim, J., Kim, D. & Im, C. Evaluation of various mental task combinations for near-infrared spectroscopy-based brain–computer interfaces. J. Biomed. Opt. 19, 077005. https://doi.org/10.1117/1.JBO.19.7.077005 (2014).

41. Asgher, U. et al. Assessment and classification of mental workload in the prefrontal cortex (PFC) using fixed-value modified Beer–Lambert law. IEEE Access 7, 143250–143262. https://doi.org/10.1109/ACCESS.2019.2944965 (2019).

42. Zhang, Q., Strangman, G. & Ganis, G. Adaptive filtering to reduce global interference in non-invasive NIRS measures of brain activation: How well and when does it work?. Neuroimage 45, 788–794. https://doi.org/10.1016/j.neuroimage.2008.12.048 (2009).

43. LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444. https://doi.org/10.1038/nature14539 (2015).

44. Hussain, M., Bird, J. J. & Faria, D. R. A study on CNN transfer learning for image classification. In Advances in Computational Intelligence Systems. UKCI 2018. Advances in Intelligent Systems and Computing Vol. 840 (eds Lotfi, A. et al.) (Springer, 2019). https://doi.org/10.1007/978-3-319-97982-3_16.

45. Wittenberg, E., Thompson, J., Nam, C. S. & Franz, J. R. Neuroimaging of human balance control: A systematic review. Front. Hum. Neurosci. 11, 170. https://doi.org/10.3389/fnhum.2017.00170 (2017).

46. Lotte, F. et al. A review of classification algorithms for EEG-based brain–computer interfaces: A 10 year update. J. Neural Eng. 15, 031005. https://doi.org/10.1088/1741-2552/aab2f2 (2018).

47. Zhuang, F. et al. A comprehensive survey on transfer learning. Proc. IEEE 109(1), 43–76. https://doi.org/10.1109/JPROC.2020.3004555 (2020).

48. Azab, A., Mihaylova, L., Ang, K. & Arvaneh, M. Weighted transfer learning for improving motor imagery-based brain–computer interface. IEEE Trans. Neural Syst. Rehabil. Eng. 27, 1352–1359. https://doi.org/10.1109/TNSRE.2019.2923315 (2019).

49. Pan, S. & Yang, Q. A survey on transfer learning. IEEE Trans. Knowl. Data Eng. 22, 1345–1359. https://doi.org/10.1109/TKDE.2009.191 (2010).

50. Aggarwal, C. In Data Classification Algorithms and Applications (ed. Aggarwal, C.) 448–454 (Chapman and Hall/CRC, 2014).

51. Blitzer, J., McDonald, R. & Pereira, F. Domain adaptation with structural correspondence learning. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, 120–128 (2006).

52. Soni, A. N. Application and analysis of transfer learning-survey. Int. J. Sci. Res. Eng. Dev. 1(2), 272–278 (2018).

53. Shin, J. et al. Open access repository for hybrid EEG-NIRS data. In 2018 6th International Conference on Brain–Computer Interface (BCI), 1–4. https://doi.org/10.1109/IWW-BCI.2018.8311523 (2018).

54. Weiss, K., Khoshgoftaar, T. M. & Wang, D. A survey of transfer learning. J. Big Data 3, 9. https://doi.org/10.1186/s40537-016-0043-6 (2016).

55. Day, O. & Khoshgoftaar, T. M. A survey on heterogeneous transfer learning. J. Big Data 4, 29. https://doi.org/10.1186/s40537-017-0089-0 (2017).

56. Wang, P., Lu, J., Zhang, B. & Tang, Z. A review on transfer learning for brain–computer interface classification. In 2015 5th International Conference on Information Science and Technology (ICIST), 2015 Apr 24, 315–322 https://doi.org/10.1109/ICIST.2015.7288989.

## Acknowledgements

We would like to acknowledge the School of Mechanical and Manufacturing Engineering (SMME), National University of Sciences and Technology (NUST), Islamabad, Pakistan, for providing necessary support, laboratory equipment, and facilities to conduct this study. We would also like to acknowledge the EU H2020 MSCA-RISE 823904, Project : Enhancing Human Performance in Complex Socio-Technical Systems - ENHANCE (Grant Agreement Number 823904).

## Author information

Authors

### Contributions

Conceptualization, K.K. and U.A.; methodology, K.K. and U.A.; software, K.K., Y.A. and U.A.; validation, K.K., U.A.; formal analysis, K.K. and Y.A.; investigation, K.K., Y.A. and U.A.; resources, U.A. and Y.A.; writing—original draft preparation, K.K., Y.A. and U.A.; writing—review and editing, U.A. and Y.A.; visualization, U.A. and Y.A.; supervision, U.A. and Y.A.; project administration, U.A. and Y.A. All authors have read and agreed to the published version of the manuscript.

### Corresponding author

Correspondence to Yasar Ayaz.

## Ethics declarations

### Competing interests

The authors declare no competing interests.

### Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

## Rights and permissions

Reprints and Permissions

Khalil, K., Asgher, U. & Ayaz, Y. Novel fNIRS study on homogeneous symmetric feature-based transfer learning for brain–computer interface. Sci Rep 12, 3198 (2022). https://doi.org/10.1038/s41598-022-06805-4

• Accepted:

• Published:

• DOI: https://doi.org/10.1038/s41598-022-06805-4