Internship M2 Center for Robotics, Mines Paris : Hybrid data-driven and human experience-based evaluation of motor performance in craft professions

Topic : Hybrid data-driven and human experience-based evaluation of motor

performance in craft professions

Lab : Center for Robotics

Place: Ecole des Mines, 60 Bvd Saint Michel, Paris

Dates & duration : 6 months, april to /september 2025

Funded by : ReSource PIA4 Project

Salary: 630€/month

Supervision : Alina Glushkova

 

Keywords : data science, machine learning, deep learning, human motion data,

timeseries, feature selection, hybrid models

Context : The progress in AI and data science contributes to the development of new

methods and tools for capturing/analyzing/modeling and assessing human movement in

diVerent contexts and applications such as sports, rehabilitation etc. Data analysis and

decision support tools for coaches/therapists are becoming increasingly popular. Craft

professions could benefit from this progress since they have common characteristics

with the precited application domains: over the years the craftsman “trains” him/herself,

activates cognitive processes and complex learning mechanisms, he/she makes

mistakes and learns from them by reducing the variability of these errors. Motion capture

technologies make it possible to capture digital signals and transform the gesture into

data, which encodes the tacit knowledge of experts. This data can be used to

analyze/model/evaluate the quality of performance and estimate the error gradients,

using machine/deep learning methods. However, machine-interpretable data are often

not the same as human-interpretable one, and decisions made by algorithms are not

easily accepted and understood by experts. Creating a hybrid approach that would

combine automatic preselection of parameters to be analyzed with feedback from the

domain expert (craftsman) would benefit from both data science and human expertise,

contributing to the acceptability of the system for human motion analysis.

In the framework of the ReSource project, several craft professions were captured with

multimodal technologies (egocentric and exocentric video, inertial sensors, contact and

stereo microphone) contributing thus to the creation of a publicly available dataset

(https://www.caor.minesparis.psl.eu/human-motion-capture-benchmark/).

Mission / Goals: The intern will develop a hybrid system, combining the automatic

evaluation of motor performance, using machine learning with an approach based on a

set of rules defined by the human expert.

• Dynamic selection of features to be evaluated for motor performance analysis:

To do this, he/she will first use the customized, collected aforementioned human

motion data (time series representing the angulations/rotations of subject’s joints)

and identify, using appropriate methods, the salient features to generate a specific

analysis by profession. Reinforcement learning, the « forward feature selection »

method or other discriminative deep learning methods such as eXtreme Gradient

Boosting make it possible to dynamically select the set of optimal features in

multidimensional data.

• Prediction of motor performance quality: Then it is necessary to predict the

quality of performance, by estimating the « accuracy score » of the preselected

features based on supervised machine learning methods (SVM, DT, etc.). The

evaluation with automatic prediction will then be completed by the prediction of

the quality of performance based on rules (« if-then ») predefined by the experts

through a series of interviews and elicitation process.

• Development of a hybrid model: Finally, a hybrid model will be developed to

merge the two evaluation perspectives. The trainee will choose the appropriate

method (depending on the size of the dataset, the nature of the data, etc.) for this

hybridization (e.g. Bayesian networks, MaxEnt, attention network, weighted

average « Ensemble » methods, etc.).

• Evaluation of the results: The trainee will also contribute to the analysis of the

results by comparing the evaluation carried out by the automatic approach, the

rule-based approach and the hybrid approach. If possible, he will also conduct a

qualitative evaluation with the experts/trainers to define the relevance of the

hybrid approach.

 

References :

Lee, M. H., Siewiorek, D. P., Smailagic, A., Bernardino, A., & Bermúdez i Badia, S. B. (2021, May). A

human-ai collaborative approach for clinical decision making on rehabilitation assessment.

In Proceedings of the 2021 CHI conference on human factors in computing systems (pp. 1-14).

X. Li, J. Yao, J. Ren and L. Wang, « A New Feature Selection Algorithm Based on Deep Q-Network, » 2021

40th Chinese Control Conference (CCC), Shanghai, China, 2021, pp. 7100-7105, doi:

10.23919/CCC52363.2021.9550745.

Frangoudes, F., Matsangidou, M., Schiza, E. C., Neokleous, K., & Pattichis, C. S. (2022). Assessing

human motion during exercise using machine learning: A literature review. IEEE Access, 10, 86874-

86903.

Bouchlaghem, Y., Akhiat, Y., Touchanti, K., & Amjad, S. (2024). A novel feature selection method with

transition similarity measure using reinforcement learning. Decision Analytics Journal, 11, 100477.