Site Ecole des Mines, Centre de Robotique
AI-based interaction mechanism for adaptive sensorimotor augmented feedback
strategies
A doctoral position is open in the Center for Robotics, Ecole des Mines, PSL University, in the
framework of the PIA4 ReSource project « Conservatoire des Gestes et Savoir-faire des Métiers d’art
et de Fabrication », coordinated by the Manufacture des Gobelins. This project involving more than
20 partners, aims, among others, at developing innovative methods and tools for the preservation and
transmission of manual know-how in manufacturing industries and (eg. Luxury good industries).
Keywords: human sensing, human motion, timeseries, machine and deep learning, human-machine
interaction, augmented reality, feedback strategy, human learning and motor skills acquisition
Context: The acquisition of technical and dexterous motor skills requires the activation of complex
cognitive processes and learning mechanisms. Humans receive multisensory information, treat it,
make decisions and implement predictive or reactive mechanisms (feedforward control strategy) to
perform the task (1) . Making errors and learning from them by reducing the error gradient is one of
the most basic principles of sensorimotor learning. However, minimizing the average gradient doesn’t
guarantee the optimal performance because of the big variability of possible errors (2). The learner
needs to control spatial and temporal aspects of his/her motion, the degrees of freedom, the internal
and external forces applied etc. In parallel, during the learning process, the execution of the task
generates multisensory consequences (extrinsic feedback) that are processed again permitting to
continuously update the sensorimotor loop, to improve the performance and to acquire motor skills.
In other words, whatever we see, hear or touch may impact the way we perform and the way we learn
to perform.
Thanks to recent advances in interactive technologies this extrinsic feedback, that is also called
“augmented feedback” (AF), can be created artificially, providing thus complementary source
information or guidance that is helpful to the subject (3). This feedback is expected to improve user’s
capacity to integrate the most optimal sources of afferent information to perform the task. The AF use
different modalities and activates different senses, it can take various shapes, can be continuous or
terminal, can provide insights about the performance itself or about its result, it can have a positive
or a negative connotation, the frequency of its use can also vary. To design the most effective and
efficient mechanism for the activation of AF is a complex and challenging scientific task, that requires
not only the definition of the aforementioned parameters but also involves personal subjective
preferences.
Scientific objective: The goal of this thesis is to propose an AI-based mechanism for the automated
selection and generation of feedback following different feedback strategies that depend on their
predicted effectiveness and efficiency. By using motion capture sensors, the PHD candidate will collect
multisensory data of craft experts and their learners at project’s industrial (luxury goods
manufacturers) and regional partners (vocational training centers), contributing thus to the extension
of an existing Dataset (6,7). He/she will be expected then to propose a methodology in order to answer
to the following questions:
– What to provide feedback on? In order to answer this question the candidate needs to define
the performance evaluation strategy: to analyse the data captured, to define the most
important spatial and temporal errors, to measure the deviations of the learner from the
expert (9,10), to define the accepted tolerance aiming at their priorization and taking into
consideration that in motor skills acquisition and in the framework of embodied cognition, the
most significant variables are not always obvious and that a greater precision doesn’t
necessarily means a better gestural performance.
– When and what feedback to provide? This question refers to feedback design principles since
it can be of different types (knowledge of performance VS knowledge of result), use various
modalities (visual/auditory, unimodal/multimodal), be concurrent or terminal, regular or
irregular, implicit or explicit, positive or negative etc. (3).
– How to decide when and what feedback to provide on? This question focuses on the decision
making process that is expected to be automated by using machine and/or deep learning
models that have been broadly used to analyse, model, forecast human motion (DNN, LSTM,
RNNs, GRUs, GANs etc.) (8,9). This decision-making process concerns the automated feedback
selection (4) and is expected to be based on several criteria (mechanism adaptability) related
to the effectiveness and efficiency of the feedback strategy (5) that will be explored through
the previous 2 questions.
Requirements:
– Master degree in Computer Science, Human-Machine Interaction, Machine Learning or similar
relevant fields
– Knowledge of programming languages, e.g. Python, C/C++, Java, Max MSP would be appreciated
– Motivation for working in a multidisciplinary research project at the interface between
artificial intelligence, computer science and augmented reality
– A previous experience of working with human sensing will be appreciated
– Very good knowledge on signal processing
– Excellent level in French and English language (written and spoken)
Hiring Conditions
– 36-month full-time doctoral contract
– Start date: October 2024
– Teaching assignments will be carried out in the Centre for Robotics at Mines Paris, PSL
Université, 60 Boulevard Saint Michel, Paris, France
References:
(1) Wolpert, D. M., Diedrichsen, J., & Flanagan, J. R. (2011). Principles of sensorimotor
learning. Nature reviews neuroscience, 12(12), 739-751.
(2) “Skill Acquisition in Sport: Research, Theory and Practice” (2020) edited by Edited By Nicola
J. Hodges, A. Mark Williams
(3) Sigrist, R., Rauter, G., Riener, R., & Wolf, P. (2013). Augmented visual, auditory, haptic, and
multimodal feedback in motor learning: a review. Psychonomic bulletin & review, 20, 21-53.
(4) Rauter, G., Gerig, N., Sigrist, R., Riener, R., & Wolf, P. (2019). When a robot teaches humans:
Automated feedback selection accelerates motor learning. Science robotics, 4(27), eaav1560.
(5) Glushkova, A., Makrygiannis, D., & Manitsaris, S. (2023, July). Interactive sensorimotor
guidance for learning motor skills of a glass blower. In International Conference on Human-
Computer Interaction (pp. 29-43). Cham: Springer Nature Switzerland.
(6) https://www.kaggle.com/datasets/olivasbre/aimove
(7) Olivas-Padilla, B. E., Glushkova, A., & Manitsaris, S. (2023). Motion capture benchmark of real
industrial tasks and traditional crafts for Human Movement Analysis. IEEE Access.
(8) Vandevoorde, K., Vollenkemper, L., Schwan, C., Kohlhase, M., & Schenck, W. (2022). Using
Artificial Intelligence for Assistance Systems to Bring Motor Learning Principles into Real World
Motor Tasks. Sensors, 22(7), 2481.
(9) Olivas-Padilla, B. E., Manitsaris, S., & Glushkova, A. (2024). Explainable AI in human motion:
A comprehensive approach to analysis, modeling, and generation. Pattern Recognition, 151,
110418.
(10) Sedmidubsky, J., Elias, P., & Zezula, P. (2018). Effective and efficient similarity searching in
motion capture data. Multimedia Tools and Applications, 77, 12073-12094.