EEG and fNIRS fail in different ways. EEG is fast and sensitive to muscle and movement; fNIRS is slow and less affected by them. A hybrid system can use one to cover the other. It can also add EMG to separate real from imagined movement, or eye tracking to gate an SSVEP selection.
BCILattice lets EEG, fNIRS, EMG and EOG live in the same study, with every step reading the sampling rate and channel structure from the recording it is processing, so a 10 Hz fNIRS file and a 500 Hz EEG file are each handled at their own rate.
Early, late and model-level fusion
On the BCILattice canvas, the Feature Concatenator and Data Merger blocks handle early fusion, the Ensemble Voter and Stacking Assembler handle late fusion, and Model Concat builds multi-branch networks. The Modality Fusion block combines EEG and fNIRS and reports whether combining actually helped, compared with each modality alone.
| Fusion | How | Trade-off |
|---|---|---|
| Early (feature) | Concatenate features from each modality into one design matrix | Simple; the higher-dimensional modality can drown the other. |
| Late (decision) | Train one model per modality and combine their outputs | Robust to one modality failing; ignores interactions. |
| Model-level | A network with a branch per modality joined inside | Learns interactions; needs the most data. |
Stimuli as data
Many experiments are multimodal in a second sense: the stimulus is text, an image, audio or video. BCILattice treats a stimulus set as data bound to the recording clock, so each item has an onset. Media blocks load, window and embed it, and the Encoding Model (TRF) block relates the stimulus to the brain response over time in either direction.
A paradigm runner presents stimuli with a measured onset timing band, rather than a claimed one, and paradigms export to PsychoPy or BIDS. When the same stimulus appears in both training and test data, the leakage is reported to you rather than silently allowed.
The fusion claim needs its own evidence
A fused model that scores 3 points higher than EEG alone on one split has not shown that fusion helps. Compare the fused model with each single-modality model on the same folds, and test whether the difference is real. The Statistical Comparison block tests whether model A is really better than model B or the gap is noise.
Questions
What is a hybrid BCI?
A hybrid BCI combines two or more signals or paradigms, such as EEG with fNIRS, EEG with EMG, or motor imagery with SSVEP, so that one compensates for the weaknesses of the other.
Why combine EEG and fNIRS?
EEG has millisecond timing but is sensitive to muscle and motion artifacts. fNIRS is slow but more robust and spatially more specific. Together they can classify more reliably than either alone, if the fusion is evaluated properly.
Can I analyse EEG and fNIRS in the same study in BCILattice?
Yes. A study can hold EEG, fNIRS, EMG and EOG recordings, and each step processes a recording at its own sampling rate and channel layout.
How do I know whether fusion improved my model?
Train each modality alone and the fused model on the same folds, then test the difference statistically instead of comparing single numbers.
Read the reference
Related
What Is fNIRS? Functional Near-Infrared Spectroscopy
Functional near-infrared spectroscopy (fNIRS) shines near-infrared light into the head and measures how much comes back, which tracks changes in oxygenated and deoxygenated haemoglobin in the cortex.
fNIRS Analysis Software
An fNIRS file holds a HbO and a HbR channel per source-detector pair. Every fNIRS-capable step here exposes a chromophore control, because decomposing or ranking both together mixes two different physiological signals.
EMG Analysis Software
RMS, mean absolute value, waveform length, slope sign changes and Willison amplitude are feature families in the catalogue, not something you script afterwards.
What Is a Brain-Computer Interface (BCI)?
A brain-computer interface (BCI) is a system that measures brain activity, decodes it, and turns it into a command for a computer or device, without using the muscles or nerves that normally carry that command.