How to Robustly Combine Judgements from Crowd Assessors with AWARE
Research output: Contribution to journal › Conference article › Research › peer-review
Standard
How to Robustly Combine Judgements from Crowd Assessors with AWARE. / Ferrante, Marco; Ferro, Nicola; Maistro, Maria.
In: CEUR Workshop Proceedings, Vol. 2161, 01.01.2018, p. 1DUMMY.Research output: Contribution to journal › Conference article › Research › peer-review
Harvard
APA
Vancouver
Author
Bibtex
}
RIS
TY - GEN
T1 - How to Robustly Combine Judgements from Crowd Assessors with AWARE
AU - Ferrante, Marco
AU - Ferro, Nicola
AU - Maistro, Maria
PY - 2018/1/1
Y1 - 2018/1/1
N2 - We propose the Assessor-driven Weighted Averages for Retrieval Evaluation (AWARE) probabilistic framework, a novel methodology for dealing with multiple crowd assessors, who may be contradictory and/or noisy. By modeling relevance judgements and crowd assessors as sources of uncertainty, AWARE directly combines the performance measures computed on the ground-truth generated by the crowd assessors instead of adopting some classification technique to merge the labels produced by them. We propose several unsupervised estimators that instantiate the AWARE framework and we compare them with Majority Vote (MV) and Expectation Maximization (EM) showing that AWARE approaches improve both in correctly ranking systems and predicting their actual performance scores.
AB - We propose the Assessor-driven Weighted Averages for Retrieval Evaluation (AWARE) probabilistic framework, a novel methodology for dealing with multiple crowd assessors, who may be contradictory and/or noisy. By modeling relevance judgements and crowd assessors as sources of uncertainty, AWARE directly combines the performance measures computed on the ground-truth generated by the crowd assessors instead of adopting some classification technique to merge the labels produced by them. We propose several unsupervised estimators that instantiate the AWARE framework and we compare them with Majority Vote (MV) and Expectation Maximization (EM) showing that AWARE approaches improve both in correctly ranking systems and predicting their actual performance scores.
KW - AWARE
KW - Crowdsourcing
KW - Unsupervised estimators
UR - http://www.scopus.com/inward/record.url?scp=85051853548&partnerID=8YFLogxK
M3 - Conference article
AN - SCOPUS:85051853548
VL - 2161
SP - 1DUMMY
JO - CEUR Workshop Proceedings
JF - CEUR Workshop Proceedings
SN - 1613-0073
T2 - 26th Italian Symposium on Advanced Database Systems, SEBD 2018
Y2 - 24 June 2018 through 27 June 2018
ER -
ID: 216516836