论文标题
评估有偏见的评估
Debiasing Evaluations That are Biased by Evaluations
论文作者
论文摘要
通常,通过征求人们对它们进行评分来评估一组项目。例如,大学要求学生对讲师的教学质量进行评分,并会议组织者要求作者提交评估评论的质量。但是,在这些应用程序中,如果学生在课程中获得较高的成绩,则通常给予更高的评级,并且如果将论文接受了会议,作者通常将评论评级更高。在这项工作中,我们将这些外部因素称为人们所经历的“结果”,并考虑到有关结果的一些信息,在给定等级中减轻这些结果引起的偏见的问题。我们将有关结果的信息提出为偏见的已知部分顺序。我们通过在此订购约束下解决一个正则优化问题来提出一种偏态方法,并提供了精心设计的交叉验证方法,该方法适应性地选择了适当的正则化量。我们提供有关算法的性能以及实验评估的理论保证。
It is common to evaluate a set of items by soliciting people to rate them. For example, universities ask students to rate the teaching quality of their instructors, and conference organizers ask authors of submissions to evaluate the quality of the reviews. However, in these applications, students often give a higher rating to a course if they receive higher grades in a course, and authors often give a higher rating to the reviews if their papers are accepted to the conference. In this work, we call these external factors the "outcome" experienced by people, and consider the problem of mitigating these outcome-induced biases in the given ratings when some information about the outcome is available. We formulate the information about the outcome as a known partial ordering on the bias. We propose a debiasing method by solving a regularized optimization problem under this ordering constraint, and also provide a carefully designed cross-validation method that adaptively chooses the appropriate amount of regularization. We provide theoretical guarantees on the performance of our algorithm, as well as experimental evaluations.