EEG trials from the same person and session are not independent. They share electrode impedances, the participant's state, and slow drifts. A random k-fold split puts trials from minutes apart on both sides, and the model can score well by recognising the session instead of the task. This is the single most common reason a published BCI accuracy fails to replicate.
Choose the split that matches the claim. "Works for this person on this day" needs a within-session split. "Works the next day" needs a cross-session split. "Works for a new person" needs leave-one-subject-out.
Schemes and when to use them
These are the options on BCILattice's Validation Scheme block. Wired into a trainer, it trains once per fold and reports per-fold and aggregate metrics with a confidence interval, and the canvas suggests a scheme when you select the trainer.
| Scheme | Use when |
|---|---|
| Stratified k-fold | One subject, one session, balanced classes, and you only claim within-session performance. |
| Group k-fold | Multiple subjects or sessions; folds must not split a group. |
| Stratified group k-fold | Grouped and class-balanced. Classification only. |
| Leave-one-group-out (LOSO) | Each subject (or session) is the test set once. The standard cross-subject claim. |
| Time-series split | Ordered data where a fold must not train on the future, as in online-like evaluation. |
| Repeated stratified k-fold | Small datasets where a single 5-fold estimate is noisy. |
Cross-session: train on one session, test on the others
A cross-session claim is not a fold scheme at all. Some sessions train and the rest only test, and the number that matters is the drop from the same-session score to the other-session score. Indoor to outdoor, day one to day two, and the reverse direction are each their own result.
BCILattice's Evaluation Protocol block does this on the trainer: train on a named session (or each session in turn), test on every other session, and hold back part of the training session for the same-session score beside it. Every train-to-test pair is reported, with a train-by-test matrix. On the plain trainer it only evaluates; wired into the DANN or Fine-Tuning Trainer it adapts to each new session and reports the unadapted score beside it, and wired into a continual trainer it learns the sessions as a sequence.
Nested cross-validation for anything you tune
If you choose hyperparameters, features or a model by cross-validated score, that score is now optimistic: it was used for selection. Nested cross-validation runs an inner loop to select and an outer loop to score, so the reported number comes from folds no selection saw. BCILattice has a Nested CV block for exactly this, and Smart Training applies the same rule.
Leakage checklist
The Results page has a dedicated validation tab that reads the run design, infers the grouping, and flags a leaky split or an unnested search. The result keeps its real number, with the flag attached.
Fitted preprocessing
Scalers, PCA, CSP, feature selection and alignment must be fitted inside training folds.
Overlapping windows
Sliding windows or overlapping epochs from one trial must stay together.
Augmented copies
Augment only the training fold, after the split.
LOSO needs pooled subjects
A run that holds one subject has nothing to leave out. In BCILattice, set the data source to combined subjects so the trainer sees real subject groups.
Questions
What is leave-one-subject-out cross-validation?
LOSO trains on all subjects but one and tests on the held-out subject, repeating until every subject has been the test set. It estimates how well a model works for a new person.
Why is my EEG classification accuracy too high?
Most often because trials from the same session or subject appear in both training and test data, or because preprocessing such as CSP, scaling or feature selection was fitted on all the data before splitting.
What is nested cross-validation?
Cross-validation with two loops: the inner loop selects hyperparameters or features, and the outer loop evaluates the selected model on data the selection never saw, giving an unbiased performance estimate.
Should I use k-fold or LOSO for BCI?
Use the split that matches your claim. Within-subject k-fold supports within-session claims only. LOSO supports claims about new users. Report which one you used.
Read the reference
Related
Reproducible BCI Research
A methods section is a summary written from memory weeks after the run. A study that carries its own stages, parameters and fold scores is the run itself.
EEG Machine Learning
A node canvas of 119 blocks covers the pipeline from design matrix to evaluation — and the evaluation half is built to make an inflated score hard to produce by accident.
BCI Domain Adaptation
CORAL, MMD alignment, Transfer Component Analysis, a DANN discriminator, feature alignment and subject adaptation — assembled on the same canvas as the model they feed.
BCI for Beginners: How to Get Started with EEG and BCI
You do not need a headset to start. A public dataset, a laptop and a clear first question will teach you more than a new device and no plan.