Can We Trust Recommender System Fairness Evaluation? The Role of Fairness and Relevance
Research output: Chapter in Book/Report/Conference proceeding › Article in proceedings › Research › peer-review
Documents
- Fulltext
Final published version, 1.23 MB, PDF document
Relevance and fairness are two major objectives of recommender systems (RSs). Recent work proposes measures of RS fairness that are either independent from relevance (fairness-only) or conditioned on relevance (joint measures). While fairness-only measures have been studied extensively, we look into whether joint measures can be trusted. We collect all joint evaluation measures of RS relevance and fairness, and ask: How much do they agree with each other? To what extent do they agree with relevance/fairness measures? How sensitive are they to changes in rank position, or to increasingly fair and relevant recommendations? We eempirically study for the first time the behaviour of these measures across 4 real-world datasets and 4 recommenders. We find that most of these measures: i) correlate weakly with one another and even contradict each other at times; ii) are less sensitive to rank position changes than relevance- and fairness-only measures, meaning that they are less granular than traditional RS measures; and iii) tend to compress scores at the low end of their range, meaning that they are not very expressive. We counter the above limitations with a set of guidelines on the appropriate usage of such measures, i.e., they should be used with caution due to their tendency to contradict each other and of having a very small empirical range.
Original language | English |
---|---|
Title of host publication | SIGIR 2024 - Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval |
Publisher | Association for Computing Machinery, Inc. |
Publication date | 2024 |
Pages | 271-281 |
ISBN (Electronic) | 9798400704314 |
DOIs | |
Publication status | Published - 2024 |
Event | 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2024 - Washington, United States Duration: 14 Jul 2024 → 18 Jul 2024 |
Conference
Conference | 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2024 |
---|---|
Land | United States |
By | Washington |
Periode | 14/07/2024 → 18/07/2024 |
Sponsor | ACM SIGIR |
Bibliographical note
Publisher Copyright:
© 2024 Owner/Author.
- fairness and relevance evaluation, recommender systems
Research areas
ID: 401936723