Understanding Mean Score Differences between the "e-rater"® Automated Scoring Engine and Humans for Demographically Based Groups in the "GRE"® General Test. ETS GRE® Board Research Report. ETS GRE®-18-01. ETS Research Report. RR-18-12

Item request has been placed! ×
Item request cannot be made. ×
  Processing Request
  • Source:
    ETS Research Report Series, Dec 2018. 33 pp.
  • Language:
    English
  • Publication Type:
    Journal Articles; Reports - Research
  • Additional Information
    • Author(s):
    • Availability:
      Full Text from ERIC Available online: https://eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=EJ1202928
      Educational Testing Service. Rosedale Road, MS19-R Princeton, NJ 08541. Tel: 609-921-9000; Fax: 609-734-5410; e-mail: RDweb@ets.org; Web site: https://www.ets.org/research/policy_research_reports/ets
    • Peer Reviewed:
      Y
    • ISSN:
      2330-8516
    • Subject Terms:
    • Subject Terms:
    • Subject Terms:
    • Abstract:
      Notable mean score differences for the "e-rater"® automated scoring engine and for humans for essays from certain demographic groups were observed for the "GRE"® General Test in use before the major revision of 2012, called rGRE. The use of e-rater as a check-score model with discrepancy thresholds prevented an adverse impact on the examinee score at the item or test level. Despite this control, there remains a need to understand the root causes of these demographically based score differences and to identify potential mechanisms for avoiding future instances of discrepancy. In this study, we used a combination of statistical methods and human review to propose hypotheses about the root cause of score differences and whether such discrepancies reflect inadequacies of e-rater, human scoring, or both. The human rating process was found to be influenced strongly by the scale structure and did not fully correspond to the e-rater scoring mechanism. The human raters appeared to be using conditional logic and a rule-based approach to their scoring, while e-rater uses linear weighting of all the features. These analyses have implications for future research and operational policies for the scoring of the rGRE.
    • Abstract:
      As Provided
    • Number of References:
      40
    • Physical Description:
      33
    • Education Level:
      Higher Education; Postsecondary Education
    • Journal Code:
      FEB2019
    • Publication Date:
      2019
    • Accession Number:
      EJ1202928
  • Citations
    • ABNT:
      RAMINENI, C.; WILLIAMSON, D. Understanding Mean Score Differences between the “e-rater”® Automated Scoring Engine and Humans for Demographically Based Groups in the “GRE”® General Test. ETS GRE® Board Research Report. ETS GRE®-18-01. ETS Research Report. RR-18-12. ETS Research Report Series, [s. l.], 2018. Disponível em: . Acesso em: 18 out. 2019.
    • AMA:
      Ramineni C, Williamson D. Understanding Mean Score Differences between the “e-rater”® Automated Scoring Engine and Humans for Demographically Based Groups in the “GRE”® General Test. ETS GRE® Board Research Report. ETS GRE®-18-01. ETS Research Report. RR-18-12. ETS Research Report Series. December 2018. http://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1202928&custid=s8280428. Accessed October 18, 2019.
    • APA:
      Ramineni, C., & Williamson, D. (2018). Understanding Mean Score Differences between the “e-rater”® Automated Scoring Engine and Humans for Demographically Based Groups in the “GRE”® General Test. ETS GRE® Board Research Report. ETS GRE®-18-01. ETS Research Report. RR-18-12. ETS Research Report Series. Retrieved from http://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1202928&custid=s8280428
    • Chicago/Turabian: Author-Date:
      Ramineni, Chaitanya, and David Williamson. 2018. “Understanding Mean Score Differences between the ‘e-Rater’® Automated Scoring Engine and Humans for Demographically Based Groups in the ‘GRE’® General Test. ETS GRE® Board Research Report. ETS GRE®-18-01. ETS Research Report. RR-18-12.” ETS Research Report Series, December. http://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1202928&custid=s8280428.
    • Harvard:
      Ramineni, C. and Williamson, D. (2018) ‘Understanding Mean Score Differences between the “e-rater”® Automated Scoring Engine and Humans for Demographically Based Groups in the “GRE”® General Test. ETS GRE® Board Research Report. ETS GRE®-18-01. ETS Research Report. RR-18-12’, ETS Research Report Series. Available at: http://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1202928&custid=s8280428 (Accessed: 18 October 2019).
    • Harvard: Australian:
      Ramineni, C & Williamson, D 2018, ‘Understanding Mean Score Differences between the “e-rater”® Automated Scoring Engine and Humans for Demographically Based Groups in the “GRE”® General Test. ETS GRE® Board Research Report. ETS GRE®-18-01. ETS Research Report. RR-18-12’, ETS Research Report Series, viewed 18 October 2019, .
    • MLA:
      Ramineni, Chaitanya, and David Williamson. “Understanding Mean Score Differences between the ‘e-Rater’® Automated Scoring Engine and Humans for Demographically Based Groups in the ‘GRE’® General Test. ETS GRE® Board Research Report. ETS GRE®-18-01. ETS Research Report. RR-18-12.” ETS Research Report Series, Dec. 2018. EBSCOhost, search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1202928&custid=s8280428.
    • Chicago/Turabian: Humanities:
      Ramineni, Chaitanya, and David Williamson. “Understanding Mean Score Differences between the ‘e-Rater’® Automated Scoring Engine and Humans for Demographically Based Groups in the ‘GRE’® General Test. ETS GRE® Board Research Report. ETS GRE®-18-01. ETS Research Report. RR-18-12.” ETS Research Report Series, December 1, 2018. http://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1202928&custid=s8280428.
    • Vancouver/ICMJE:
      Ramineni C, Williamson D. Understanding Mean Score Differences between the “e-rater”® Automated Scoring Engine and Humans for Demographically Based Groups in the “GRE”® General Test. ETS GRE® Board Research Report. ETS GRE®-18-01. ETS Research Report. RR-18-12. ETS Research Report Series [Internet]. 2018 Dec 1 [cited 2019 Oct 18]; Available from: http://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1202928&custid=s8280428