The last research axis of the MRIM team concerns the evaluation of information retrieval models. This work follows two main axes:
- the definition of evaluation frameworks,
- the building and distribution of test corpora.
These two axes make it possible to address two scientific obstacles: the longitudinal evaluation of an IRS, the provision of test collections for a reasonable human cost. On the first obstacle, we have defined an evaluation framework based on two key points: knowledge deltas that estimate the differences in the evaluation environment (in particular the corpus) over time, and result deltas, which quantify the differences in evaluation metrics for these evolving corpora and needs. The result deltas are based on bijective normalization functions using isotonic centered regressions, these elements make it possible to control that intra-corpus calculations can be projected into another corpus. Numerous experiments on state-of-the-art artificial data and on a corpus provided by Qwant have led to results published in major conferences in the field. In addition, we have designed evaluation campaigns and provided resources to the community. Thus, the MRIM team defined and proposed a protocol for building an evaluation corpus, in order to test the systems based on several corpora (documents and queries) acquired on different dates. The MRIM team established the relevance of the documents based on clicks obtained by Qwant. These evaluation corpora consist of more than 8 million documents, 5000 queries, and were published and shared during the LongEval evaluation campaign. In 2023 and 2024, more than 100 runs were submitted for evaluation (which represents 30 groups) and our corpus was downloaded more than 300 times. The MRIM team also published the protocol at one of the best information retrieval conferences ECIR in 2023 and 2024 and the results of the campaigns at the CLEF 2023 and CLEF 2024 conferences. This work was carried out as part of the ANR PRCI Kodicare project, with the University of Vienna in Austria and the company Qwant.
On a completely different point, the MRIM team experimented with a dialogue activity between an SRI and users in a mobile context dedicated to shows. The team thus created a dialogue AI in French to extend a play. The dialogue leads the user to solve puzzles from the play’s scenario. This project was in collaboration with the CEA and the startup Hoomano responsible for modeling the mood and animating the robot’s face in the mobile application.
The MRIM team successfully responded to calls for tenders between 2019 and 2024. We participated or have participated in: the national projects financed by the ANR Pantagruel, Flaubert, Guidance, the Partage project financed by the BPI, Guimuteic financed by the Single Interministerial Fund (FUI) and the European Regional Development Fund (ERDF), the PRCI Kodicare project financed partly by the ANR and partly by the Austrian FWF.