To the content
1 . 2022

HOW TO SET STANDARDS ON PERFORMANCE-BASED EXAMINATIONS: AMEE GUIDE NO. 85

Abstract

This AMEE Guide offers an overview of methods used in determining passing scores for performance-based assessments. A consideration of various assessment purposes will provide context for discussion of standard setting methods, followed by a description of different types of standards that are typically set in health professions education. A step-by-step guide to the standard setting process will be presented. The Guide includes detailed explanations and examples of standard setting methods, and each section presents examples of research done using the method with performance-based assessments in health professions education. It is intended for use by those who are responsible for determining passing scores on tests and need a resource explaining methods for setting passing scores. The Guide contains a discussion of reasons for assessment, defines standards, and presents standard setting methods that have been researched with performance-based tests. The first section of the Guide addresses types of standards that are set. The next section provides guidance on preparing for a standard setting study. The following sections include conducting the meeting, selecting a method, implementing the passing score, and maintaining the standard. The Guide will support efforts to determine passing scores that are based on research, matched to the assessment purpose, and reproducible.

McKinley D.W., Norcini J.J. How to set standards on performance-based examinations: AMEE Guide No. 85. Medical Teacher. 2014; 36 (2): 97–110. DOI: https://doi.org/10.3109/0142159X.2013.853119

References

1. American Educational Research Association, American Psychological Association, National Council on Measurement in Education, 1999. Standards for educational and psychological testing. American Educational Research Association, Washington DC.

2. Angoff W.H. Scales, norms, and equivalent scores. In: Thorndike R.L, ed. Educational measurement. Washington, DC: American Council on Education. 1971, p. 508–600.

3. Bandaranayake R.C. Setting and maintaining standards in multiple choice examinations: AMEE Guide No. 37. Med Teach. 2008; 30: 836–45.

4. Ben-David M.F. AMEE Guide No. 18: Standard setting in student assessment. Med Teach. 2000; 22: 120–30.

5. Beuk C.H. A method for reaching a compromise between absolute and relative standards in examinations. J Educ Measure. 1984; 21: 147–52.

6. Boulet J.R., De Champlain A.F., McKinley D.W. Setting defensible performance standards on OSCEs and standardized patient examinations. Med Teach. 2003; 25: 245–9.

7. Burrows P.J., Bingham L., Brailovsky C.A. A modified contrasting groups method used for setting the passmark in a small scale standardised patient examination. Adv Health Sci Educ Theory Pract. 1999; 4: 145–54.

8. Clauser B.E., Clyman S.G. A contrasting-groups approach to standard setting for performance assessments of clinical skills. Acad Med. 1994; 69: S42–S44.

9. Cohen-Schotanus J., van der Vleuten C.P.M. A standard setting method with the best performing students as point of reference: Practical and affordable. Med Teach. 2010; 32: 154–60.

10. De Gruijter D.N.M. Compromise models for establishing examination standards. J Educ Measure. 1985; 22: 263–9.

11. Dijkstra J., Van der Vleuten C.P.M., Schuwirth L.W.T. A new framework for designing programmes of assessment. Adv Health Sci Educ Theory Pract. 2010; 15: 379–93.

12. Downing S.M., Lieska N.G., Raible M.D. Establishing passing standards for classroom achievement tests in medical education: A comparative study of four methods. Acad Med. 2003; 78: S85–S87.

13. Downing S.M., Tekian A., Yudkowsky R. Procedures for establishing defensible absolute passing scores on performance examinations in health professions education. Teach Learn Med. 2006; 18: 50–7.

14. Ebel R. Essentials of educational measurement. 2nd ed. Englewood Cliffs, NJ: Prentice-Hall. 1972.

15. Frank J.R., Snell L.S., Cate O.T, Holmboe E.S., Carraccio C., Swing S.R., Harris P., Glasgow N.J., Campbell C., Dath D., et al. Competency-based medical education: Theory to practice. Med Teach. 2010; 32: 638–45.

16. Geisinger K.F. Using standard-setting data to establish cutoff scores. Educ Measure: Issu Pract. 1991; 10: 17–22.

17. Geisinger K.F., McCormick C.M. Adopting cut scores: Post-standardsetting panel considerations for decision makers. Educ Measure: Issu Pract. 2010; 29: 38–44.

18. Haladyna T., Hess R. An evaluation of conjunctive and compensatory standard-setting strategies for test decisions. Educ Assess. 1999; 6: 129–53.

19. Hambleton R.K., Jaeger R.M., Plake B.S., Mills C. Setting performance standards on complex educational assessments. Appl Psychol Measur. 2000; 24: 355–66.

20. Hambleton R.K., Slater S.C. Reliability of credentialing examinations and the impact of scoring models and standard-setting policies. Appl Measur Educ. 1997; 10: 19–28.

21. Hofstee W.K.B. The case for compromise in educational selection and grading. In: Anderson S.B., Helmick J.S., ed. On educational testing. San Francisco, CA: Jossey-Bass. 1983, p. 109–27.

22. Holmboe E.S., Sherbino J., Long D.M., Swing S.R., Frank J.R. The role of assessment in competency-based medical education. Med Teach. 2010; 32: 676–82.

23. Jaeger R.M. Selection of judges for standard-setting. Educ Measu: Iss Pract. 1991; 10: 3–14.

24. Kane M.T. The assessment of professional competence. Eval Health Prof. 1992; 15: 163–82.

25. Kane M.T. Validating interpretive arguments for licensure and certification examinations. Eval Health Prof. 1994; 17: 133–59; discussion 236–41.

26. Kane M.T. Validation. In: Brennan R.L., ed. Educational measurement. Westport, CT: American Council on Education and Praeger Publishers. 2006, p. 17–64.

27. Kaufman D.M., Mann K.V., Muijtjens A.M.M., van der Vleuten C.P.M. A comparison of standard-setting procedures for an OSCE in undergraduate medical education. Acad Med. 2000; 75: 267–71.

28. Linn R.L., Burton E. Performance-based assessment: Implications of task specificity. Educ Measure: Issu Pract. 1994; 13: 5–8.

29. Livingston S.A., Zieky M.J. Passing scores: A manual for setting standards of perfromance on education and occupational tests. Educ Testing Serv. Princeton, New Jersey. 1982.

30. Miller G.E. The assessment of clinical skills/competence/performance. Acad Med. 1990; 65: S63–S67.

31. Nedelsky L. Absolute grading standards for objective tests. Educ Psychol Measur. 1954; 14: 3–19.

32. Nestel D., Kneebone R., Black S. Simulated patients and the development of procedural and operative skills. Med Teach. 2006; 28: 390–1.

33. Norcini J., Burch V. Workplace-based assessment as an educational tool: AMEE Guide No. 31. Med Teach. 2007; 29: 855–71.

34. Norcini J.J. Principles for setting standards on certifying and licensing examinations. In: Rothman A.I., Cohen R., ed. The Sixth Ottawa Conference on Medical Education. Toronto: University of Toronto Bookstore. 1994, p. 346–7.

35. Norcini J.J. Work based assessment. Br Med J. 2003; 326: 753–5.

36. Norcini J., McKinley D. Assessment methods in medical education. Teach Teacher Educ. 2007; 23: 239–50.

37. Norcini J.J., Stillman P.L., Sutnick A.I., Regan M.B., Haley H.L., Williams R.G., Friedman M. Scoring and standard setting with standardized patients. Eval Health Prof. 1993; 16: 322–32.

38. Patil N.G., Saing H., Wong J. Role of OSCE in evaluation of practical skills. Med Teach. 2003; 25: 271–2.

39. Pell G., Fuller R., Homer M., Roberts T. How to measure the quality of the OSCE: A review of metrics – AMEE guide no. 49. Med Teach. 2010; 32: 802–11.

40. Raymond M.R., Reid J. Who made thee a judge? Selecting and training participants for standard setting. In: Cizek G.J., ed. Setting performance standards: Concepts, methods, and perspectives. Mahwah, NJ: Lawrence Erlbaum Associates. 2001, p. 119–58.

41. Reznick R.K., Blackmore D., Dauphine´e W.D., Rothman A.I., Smee S. Large-scale high-stakes testing with an OSCE: Report from the Medical Council of Canada. Acad Med. 1996; 71: S19–S21.

42. Rothman A.I., Cohen R. A comparison of empirically- and rationallydefined standards for clinical skills checklists. Acad Med. 1996; 71: S1–S3.

43. Schindler N., Corcoran J., DaRosa D. Description and impact of using a standard-setting method for determining pass/fail scores in a surgery clerkship. Am J Surg. 2007; 193: 252–7.

44. Smee S.M., Blackmore D.E. Setting standards for an objective structured clinical examination: The borderline group method gains ground on Angoff. Med Educ. 2001; 35: 1009–10.

45. Traub R.E. Facing the challenge of multidimensionality in performance assessment. In: Rothman A.I., Cohen R., ed. Proceedings of the Sixth Annual Ottawa Conference on Medical Education. Toronto: University of Toronto Bookstore. 1994, p. 9–11.

46. Wood T.J., Humphrey-Murto S.M., Norman G.R. Standard setting in a small scale OSCE: A comparison of the modified borderline-group method and the borderline regression method. Adv Health Sci Educ. 2006; 11: 115–22.

All articles in our journal are distributed under the Creative Commons Attribution 4.0 International License (CC BY 4.0 license)

CHIEF EDITOR
CHIEF EDITOR
Balkizov Zalim Zamirovich
Secretary General of the Russian Society of Medical Education Specialists, Director of the Institute of Training of Medical Education Specialists of the Russian Medical Academy of Continuing Professional Education, 125993, Moscow, Russian Federation, Professor of the Department of Vocational Education and Educational Technologies of the N.I. Pirogov RNIMU of the MOH of Russia, CEO of GEOTAR-Med, Advisor President of the National Medical Chamber, Moscow, Russian Federation

Journals of «GEOTAR-Media»