Multimodal Assessment is Emerging: What Should You Know?

The rapid advancement of digital technologies has brought about significant possibilities in educational practices, particularly in assessing students' learning. Whereas previously, assessment was limited to single modality methods, with new and emerging technologies, educators can leverage the principles of Universal Design for Learning (UDL) to understand better what students know. Multimodal Data (MMD) offers valuable insights into complex learning processes such as facial expressions, voice patterns, body language, and physiological responses. This enables a more nuanced understanding of student learning processes and emotional engagement.

This blog explores the current uses of Multimodal Learning Analytics (MMLA) and possible future applications. It will also address the ethical and practical considerations arising from this shift.

What is Multimodal Assessment & Multimodal Learning Analytics?

In the broadest sense, Multimodal Assessment refers to evaluating student learning in various media formats: videos, audio recordings, infographics, and interactive presentations instead of relying solely on text-based assessments. When integrated with emerging technologies such as Generative Artificial Intelligence, tracking technology (e.g., Eye tracker, Motion Tracker), and sensor technology (PPG Sensor, HRV Sensor, EEG Sensor), this concept extends beyond mere diverse media. Multimodal Learning Analytics (MMLA) uses these technologies to capture and analyze a wide range of data about student learning.

Current Applications of MMLA

According to Blikstein & Worsley (2016), current applications of MMLA are various by types of analysis.

Type of Analysis	Sensors/Tools	Purpose
Speech Analysis	Speaker, Wireless headset	Emotion, communication behavior
Action/Gesture Analysis	Camera, Skeletal tracking sensor (e.g., PoseNet, Kinect), Hand Tracking sensor (e.g., Leap Motion)	Body posture, movement, collaboration
Affective State Analysis	Physiological sensor ( e.g., PPG Sensor, HRV Sensor, EEG Sensor)	Emotions, cognitive load, stress-level
Neurophysiological Markers	Physiological sensor ( e.g., PPG Sensor, HRV Sensor, EEG Sensor)	Attention, cognitive load, stress-level
Eye Gaze Analysis	Eye trackers (e.g., WebGazer, Tobii)	Attention, cognitive load, engagement

Key Considerations for Implementation

While multimodal learning analytics (MMLA) offers significant potential, several key considerations must be addressed for successful implementation. Data privacy and ethical concerns are paramount, requiring solid personal and biometric data protections, transparent consent, and data usage updates. The complexity of integrating diverse data types, such as text, speech, and physiological signals, must be managed with systems that provide clarity and usability. Additional support and training for teachers, particularly in understanding and interpreting the analytics, will be needed to maximize their utility in debriefing sessions.

It is essential to consider how education can offer specialized learning solutions in conjunction with technological advancements that are ’adaptive, flexible, and future-ready.’ This involves examining the educational advantages that may justify the investment of resources needed to mitigate the risks associated with integrating technological innovations.

Future Directions for MMLA

Multimodal assessment has profound implications for the broader educational landscape. First, it enables a deeper understanding of the learning process by capturing real-time student engagement, emotion, and cognitive load data. For example, with MMLA by Smartbands, data may include environmental stressors such as students’ sleep, levels of happiness and frustration, and learning outcomes across multiple school subjects. With this abundant data, teachers could build Positive Behavioural Interventions and Support (PBIS) actions to prevent students from lacking well-being more precisely than the traditional approach. MMLA could also enhance the focus on School-Wide Positive Behavioural Support (SWPBS) when integrated with other recordable contextual details. For instance, light levels, noise, heat, humidity, and air pollution may prove pertinent for a comprehensive multimodal analysis. This allows educators to make informed decisions about the student's progress and necessary interventions in appropriate environments.

Conclusion

Multimodal Learning Analytics (MMLA) can enhance education by providing deeper insights into student engagement, cognitive processes, and emotional states. While promising, Multimodal Learning Analytics (MMLA) must be implemented carefully, addressing key ethical and practical concerns such as data privacy, clarity and usability, and appropriate teacher support and training.

References

Basham, J. D., Han, K., Zhang, L., & Yang, S. (2020). Considering the fourth industrial revolution in the preparation of learners with and without disabilities. In M. Yuen, W. Beamish, & V. S. H. Solberg (Eds.), Careers for students with special educational needs (pp. 31–46). Springer.

Blikstein, P. (2013, April). Multimodal learning analytics. In Proceedings of the third international conference on learning analytics and knowledge (pp. 102-106).

Blikstein, P., & Worsley, M. (2016). Multimodal learning analytics and education data mining: Using computational technologies to measure complex learning tasks. Journal of Learning Analytics, 3(2), 220–238.

Cosentino, G., & Giannakos, M. (2023). Multisensory interaction and analytics to enhance smart learning environments: a systematic literature review. IEEE Transactions on Learning Technologies, 16(3), 414-430.

Chan, R. Y. Y., Wong, C. M. V., & Yum, Y. N. (2023). Predicting behavior change in students with special education needs using multimodal learning analytics. IEEE Access, 11, 63238–63251.

DiMitri, D., Schneider, J., Specht, M., & Drachsler, H. (2018). From signals to knowledge: A conceptual model for multimodal learning analytics. Journal of Computer Assisted Learning, 34(4), 338–349. https://doi.org/10.1111/jcal.12288

Dhope, P., & Neelagar, M. B. (2022, February). Real-time emotion recognition from facial expressions using artificial intelligence. In 2022 2nd International Conference on Artificial Intelligence and Signal Processing (AISP) (pp. 1–6). IEEE.

Fabian Gunnars. (2024, May 16). Smartbands and behavioural interventions in the classroom: Multimodal learning analytics stress-level visualisations for primary education teachers. International Journal of Disability, Development and Education. https://doi.org/10.1080/1034912X.2024.2355625

Jin, Y., Echeverria, V., Yan, L., Zhao, L., Alfredo, R., Tsai, Y. S., ... & Martinez-Maldonado, R. (2024). FATE in MMLA: A student-centred exploration of fairness, accountability, transparency, and ethics in multimodal learning analytics. arXiv preprint arXiv:2402.19071.

Juergensen, R., Zangori, L., Friedrichsen, P., & Sadler, T. D. (2023). Teaching science to students with disabilities using socio-scientific issues. Journal of Science Education for Students with Disabilities. https://doi.org/10.14448/jsesd.15.0007

Lee-Cultura, S., Sharma, K., & Giannakos, M. (2022). Children’s play and problem-solving in motion-based learning technologies using a multi-modal mixed methods approach. International Journal of Child-Computer Interaction, 31, 100355.

Molenaar, I., de Mooij, S., Azevedo, R., Bannert, M., Järvelä, S., & Gašević, D. (2023). Measuring self-regulated learning and the role of AI: Five years of research using multimodal multichannel data. Computers in Human Behavior, 139, 107540.

Montanucci, G., & Peconi, A. (2023). Artificial intelligence and special educational needs: Exploring the potential for personalized and inclusive learning. In Conference Proceeding 16th International Conference Innovation in Language Learning (pp. 116–118). Filodiritto.

Ross, J., Curwood, J. S., & Bell, A. (2020). A multimodal assessment framework for higher education. E-learning and Digital Media, 17(4), 290–306.

Sharma, K., & Giannakos, M. (2020). Multimodal data capabilities for learning: What can multimodal data tell us about learning?. British Journal of Educational Technology, 51(5), 1450-1484.

Schneider, B., Hassan, J., & Sung, G. (2022). Augmenting social science research with multimodal data collection: The EZ-MMLA toolkit. Sensors, 22(2), 568. https://doi.org/10.3390/s22020568

Worsley, M., Martinez-Maldonado, R., & D'Angelo, C. (2021). A new era in multimodal learning analytics: Twelve core commitments to ground and grow MMLA. Journal of Learning Analytics, 8(3), 10–27.

Yan, L., Zhao, L., Gašević, D., & Martinez-Maldonado, R. (2022). Scalability, sustainability, and ethicality of multimodal learning analytics. In LAK22: 12th International Learning Analytics and Knowledge Conference (pp. 13–23).