Examining the Separation-Deviation Model's Effectiveness in Group Rating Problems
Author(s)
Goel, Zoya
Advisor(s)
Gupta, Swati
Editor(s)
Collections
Supplementary to:
Permanent Link
Abstract
Group rating problems are relevant when considering decision-making processes where individuals have to be evaluated by judges. We focus on a potential method for solving consensus rating problems - the Separation-Deviation (SD) model. This model considers a set of ratings given to individuals (which may be complete or incomplete) and returns a final score for each individual. We carry out experiments to determine how effective the SD model is at retrieving the "true" scores of each individual, and we compare its performance to a simple heuristic that finds the mean scores of each individual. We find that the SD model generally does not outperform this heuristic, regardless of whether judges give individuals biased or unbiased ratings. We also find that when we choose different convex functions for different individuals used in the SD model to correct for bias, we can improve the SD model's performance for minority groups that face bias. More work needs to be done applying the SD model to "true" score distributions that aren't just the normal distribution, and in situations where bias is applied more realistically.
Sponsor
Date
Extent
Resource Type
Text
Resource Subtype
Undergraduate Research Option Thesis