Tobii Glasses Pro 3 has become fully integrated with our Photon Cap. It is a portable eye-tracker for academic, commercial, and industrial applications that can be used to capture deep insights into human behavior in real-world environments.
Its discreet and ultra-lightweight design perfectly complements the mobility and convenience of the Photon Cap, opening up a range of research scenarios. Integrating accurate and reliable measurement of eye movements and relative changes in hemoglobin concentration represents a new chapter in multimodal recording.
To illustrate the potential of combining these methods, we conducted a pilot study on motor skill acquisition in sports, using dance as a specific example. We explored the simultaneous application of functional Near-Infrared Spectroscopy (fNIRS) and eye-tracking technologies, detailing how the collected data is analyzed and processed.
Introduction
Expertise modifies how we experience, both at the level of observable behavior and cognitive processes such as memory, attention, and perception (Neumann et al., 2016). This can be seen in differences in eye movements as well as in levels of neuronal activity (Gegenfurtner et al., 2011; Burns et al., 2019). These effects have also been observed in professional dancers (Ponmanadiyil & Woolhouse, 2018). They tend to have faster saccades and more focused fixations than novices, which may suggest that they can anticipate movements and process choreographic information more quickly (Stevens et al., 2010). Furthermore, at the cerebral level, modulation of Event-Related Desynchronization (ERD) in the alpha and beta waveband has been observed in professionals depending on whether the observed movement was known or unknown (Orgs et al., 2008). It still remains unknown whether expertise in the area of a specialized style translates into the performance of other dance styles.
To date, research has been conducted on the observation of experimental stimuli, with the researcher sitting in front of a monitor and watching a video of the dancer performing a movement. The lack of replication of real-life conditions when observing dance in a dance hall means that it remains uncertain whether the effects observed so far will be repeated under natural conditions.
We designed a demo study to demonstrate how to integrate the two devices. The subject of this demo study is the relationship between neuronal brain activity and oculomotor processing during the observation or execution of dance movement in experts, depending on their dance specialization.
Methods
Participants
The study involved two professional dancers from two different dance styles (classical dance and contemporary dance). Both had more than 10 years of experience and formal training, and each specialized in one of the styles. In addition, the person presenting the moves was also a professional dancer who actively danced both styles, having formal training and over 10 years of experience.
Stimuli
Thirty visual stimuli were developed, half of them associated with a specific style of dance (contemporary or classical). The duration of these stimuli ranged from 3 to 7 seconds. Several movements were drawn from the research conducted by Casale et al. in 2023. The individual in the video performing the dance movements subsequently served as a movement presenter in the experiment. The objective behind creating these stimuli was to utilize them as cues for the instructor, guiding her on which movement to perform at any moment.
Fig 2. An example of a dance movement from classical dance that has been watched and then performed by the instructor.
Aparature
Eye-tracking
The Tobii Pro Glasses 3 mobile eye-tracker (Tobii AB, Sweden) recorded eye movements at a sampling frequency of 100 Hz. The Glasses 3 Controller software (Tobii AB, Sweden) was employed to observe the recorded signal in real time.
fNIRS device
Optical signals were recorded on two wavelengths (760 nm and 850 nm) using a continuous wave fNIRS Photon Cap C20 (Cortivision Sp. z o.o., Poland, Lublin). The device was equipped with 16 emitters and 10 detectors. The sampling frequency was 16 Hz.
fNIRS optodes montage
Relative measurements of oxygenated and deoxygenated hemoglobin concentrations were obtained from three areas of interest: inferior frontal gyrus (IFG), and middle temporal gyrus (MTG). These areas comprise the action-observation network (AON) associated with expertise towards movement (Liew et al., 2013). As a result, the montage included 26 channels.
Synchronizations
Initially, both devices were fitted, and hair management was performed to improve the quality of the fNIRS signal. The Tobii Glasses software was then launched and eye movement recording began. The Cortiview software was then switched on and, just before the recording started, the subject was asked to look at the RECORD button, which would act as a prompt to start recording on both devices. The experiment was then run in PsychoPy, which sent the markers in sync with the Cortiview software.
Experimental procedure
The experimental procedure consisted of two tasks: watching the movement presented and then performing it. At the beginning, the instructor watched one of the movements on the monitor and then performed it. At the same time, the subject observed the presented movement and then repeated the movement previously presented by the instructor. Eye movements and relative levels of oxygenated and deoxygenated hemoglobin concentrations were recorded during both movement observation and execution.
Data preparation
To connect the markers for both data analysis software (Cortiprism and Tobii Pro Lab), the following steps were performed:
- A .snirf file was uploaded to Cortiprism and data analysis continued for fNIRS
2. The function “Export preprocessed signal to .csv” to export markers from fNIRS to Tobii Pro Lab was used.
3. In order to convert event markers from CortiPrism to Tobii Pro Lab automatically we can use a short Python script. Change the path to your exported fNIRS CSV file. Then type the name for a destination folder.
4. The value of the add_time variable, which indicates the elapsed time between the start of eye-tracker recording and fNIRS, has also been changed.
5. Run the script.
6. The .json files, representing exported markers, need to be stored in the meta folder found within the individual recording from Tobii Pro Glasses 3.
7. Then, to analyze the eye-tracking data, import the entire folder from your recording into Tobii Pro Lab.
Proposed analysis method
The data presented can be analyzed separately in two dedicated software applications. After processing this data, averaging it, and then exporting it, a joint analysis can be performed in statistical packages.
Cortiprism
Below, we propose a pipeline that can be used to analyze the given files. First, load the .snirf file with the raw data into Cortiprism v1.3 software (Cortivision sp z o.o., Poland, Lublin). Then, convert the raw signal to optical density. Subsequently, discard channels that do not meet the desired quality criteria. We used Scalp Coupling Index (SCI) based channel rejection (Pollonini et al., 2016) with the recommended threshold of 0.8. In the next step, motion artifacts were corrected using the Temporal Derivative Distribution Repair (TDDR) function (Fishburn et al., 2019). In order to remove the physiological signal contribution: Apply a low pass filter with a value of 0.2. Then apply a short channel correction to minimize the influence of signal from superficial layers of brain tissue (Gagnon et al., 2012). Convert the data to relative changes in oxygenated and deoxygenated hemoglobin concentrations using a modified Beer-Lambert Law (MBLL) with a partial pathway factor (ppf) fixed at 1 for each subject (Whiteman et al., 2017). At the end, use the Correlation Based Signal Improvement (CBSI) algorithm (Cui et al., 2010).
Fig 6. Averaged HRF waveform from a -2 to 7s window for one of the channels in the IFG area, for the observation of classical and contemporary dance movement. The red solid line indicates a trial average for HbO in the contemporary dance observation condition. Similarly, the red dashed line indicates a trial average for HbR in the contemporary dance condition. The yellow solid line represents HbO in the classical dance condition. Similarly, the yellow dashed line represents HbR in the classical dance observation condition. In both cases, the shaded areas represent the standard deviation of results.
Tobii Pro Lab
After creating your project in Tobii Pro Lab with marker inputs:
- Click on the “add snapshot”
-
Go to the Analyze tab and then to the AOI tool. Prepare the areas of interest (AOIs) from which you want to analyze data. Name them using tags.
3. Go back to the settings and then change the gaze filter from raw to fixation, this will give you filtered data.
4. Next, return to Snapshot, and select the ‘Show snapshot’ option. An uploaded snapshot will appear next to your snapshot, which already has an AOI defined.
5. Define the window for the individual marker from which fixations will be analyzed.
6. Map sequentially appearing fixations onto a static snapshot. Repeat this for the two motion observation markers.
7. Export the data, from the ‘Metrics export’ tab, and then analyze it together with the data from fNIRS!
Applications – Social neuroscience
By simultaneously capturing eye movements and brain activity, we can comprehensively understand how individuals perceive, process, and respond to social information in real-world settings. Applying these technologies in hyperscanning paradigms during face-to-face interactions or to explore group dynamics in the learning process can provide deeper insights into understanding social interactions.
Applications – Sport science
Tracking both the eye movements and brain activity of athletes simultaneously coaches and researchers can achieve a more profound understanding of the cognitive and perceptual abilities that drive sports performance. Combining gaze patterns, saccadic movements along with neural activity related to attention and motor planning can be used to explore optimal visual strategies that can help create personalized training programs. The use of both technologies could allow for increased performance in sports such as archery, shooting, golf and darts.
Applications – Neurodevelopment
The combination of these two tools can also provide a better understanding of neurodevelopmental processes. Both clinicians and researchers can gain valuable insights into the neural mechanisms of child development. The use of eye-tracking technology to track mutual gaze in the mother-child relationship, together with the search for neural correlates of bonding and attachment, could play an important role in investigating the role of the caregiver in child development. This may allow the development of appropriate support programs to promote healthy attachment and emotional development.
Bibliography
Burns, E. J., Arnold, T., & Bukach, C. M. (2019). P-curving the fusiform face area: Meta-analyses support the expertise hypothesis. Neuroscience & Biobehavioral Reviews, 104, 209–221. https://doi.org/10.1016/j.neubiorev.2019.07.003
Casale, C., Moffat, R., & Cross, E. S. (2023). Aesthetic evaluation of body movements shaped by embodiment and arts experience: Insights from behaviour and fNIRS. OSF. https://doi.org/10.31234/osf.io/n24kc
Cui, X., Bray, S., & Reiss, A. L. (2010). Functional near-infrared spectroscopy (NIRS) signal improvement based on negative correlation between oxygenated and deoxygenated hemoglobin dynamics. NeuroImage, 49(4), 3039–3046. https://doi.org/10.1016/j.neuroimage.2009.11.050
Gagnon, L., Cooper, R. J., Yücel, M. A., Perdue, K. L., Greve, D. N., & Boas, D. A. (2012). Short separation channel location impacts the performance of short channel regression in NIRS. NeuroImage, 59(3), 2518–2528. https://doi.org/10.1016/j.neuroimage.2011.08.095
Gegenfurtner, A., Lehtinen, E., & Säljö, R. (2011). Expertise differences in the comprehension of visualizations: A meta-analysis of eye-tracking research in professional domains. Educational Psychology Review, 23(4), 523–552. https://doi.org/10.1007/s10648-011-9174-7
Gemignani, J., & Gervain, J. (2021). Comparing different pre-processing routines for infant fNIRS data. Developmental Cognitive Neuroscience, 48, 100943. https://doi.org/10.1016/j.dcn.2021.100943
Koh, P. H., Glaser, D. E., Flandin, G., Kiebel, S., Butterworth, B., Maki, A., Delpy, D. T., & Elwell, C. E. (2007). Functional optical signal analysis: A software tool for near-infrared spectroscopy data processing incorporating statistical parametric mapping. Journal of Biomedical Optics, 12(6), 064010. https://doi.org/10.1117/1.2804092
Liew, S.-L., Sheng, T., Margetis, J. L., & Aziz-Zadeh, L. (2013). Both novelty and expertise increase action observation network activity. Frontiers in Human Neuroscience, 7. https://doi.org/10.3389/fnhum.2013.00541
Neumann, N., Lotze, M., & Eickhoff, S. B. (2016). Cognitive expertise: An ALE meta-analysis. Human Brain Mapping, 37(1), 262–272. https://doi.org/10.1002/hbm.23028
Orgs, G., Dombrowski, J.-H., Heil, M., & Jansen-Osmann, P. (2008). Expertise in dance modulates alpha/beta event-related desynchronization during action observation. European Journal of Neuroscience, 27(12), 3380–3384. https://doi.org/10.1111/j.1460-9568.2008.06271.x
Ponmanadiyil, R., & Woolhouse, M. H. (n.d.). Eye movements, attention, and expert knowledge in the observation of Bharatanatyam dance. Journal of Eye Movement Research, 11(2), 10.16910/jemr.11.2.11. https://doi.org/10.16910/jemr.11.2.11
Whiteman, A. C., Santosa, H., Chen, D. F., Perlman, S., & Huppert, T. (2017). Investigation of the sensitivity of functional near-infrared spectroscopy brain imaging to anatomical variations in 5- to 11-year-old children. Neurophotonics, 5(01), 1. https://doi.org/10.1117/1.nph.5.1.011009
Zhou, X., Sobczak, G., McKay, C. M., & Litovsky, R. Y. (2020). Comparing fNIRS signal qualities between approaches with and without short channels. PLOS ONE, 15(12), e0244186. https://doi.org/10.1371/journal.pone.0244186