To the content
1 . 2022

HOW TO SET STANDARDS ON PERFORMANCE-BASED EXAMINATIONS: AMEE GUIDE NO. 85

Abstract

This AMEE Guide offers an overview of methods used in determining passing scores for performance-based assessments. A consideration of various assessment purposes will provide context for discussion of standard setting methods, followed by a description of different types of standards that are typically set in health professions education. A step-by-step guide to the standard setting process will be presented. The Guide includes detailed explanations and examples of standard setting methods, and each section presents examples of research done using the method with performance-based assessments in health professions education. It is intended for use by those who are responsible for determining passing scores on tests and need a resource explaining methods for setting passing scores. The Guide contains a discussion of reasons for assessment, defines standards, and presents standard setting methods that have been researched with performance-based tests. The first section of the Guide addresses types of standards that are set. The next section provides guidance on preparing for a standard setting study. The following sections include conducting the meeting, selecting a method, implementing the passing score, and maintaining the standard. The Guide will support efforts to determine passing scores that are based on research, matched to the assessment purpose, and reproducible.

McKinley D.W., Norcini J.J. How to set standards on performance-based examinations: AMEE Guide No. 85. Medical Teacher. 2014; 36 (2): 97–110. DOI: https://doi.org/10.3109/0142159X.2013.853119

References

1. American Educational Research Association, American Psychological Association, National Council on Measurement in Education, 1999. Standards for educational and psychological testing. American Educational Research Association, Washington DC.

2. Angoff W.H. Scales, norms, and equivalent scores. In: Thorndike R.L, ed. Educational measurement. Washington, DC: American Council on Education. 1971, p. 508–600.

3. Bandaranayake R.C. Setting and maintaining standards in multiple choice examinations: AMEE Guide No. 37. Med Teach. 2008; 30: 836–45.

4. Ben-David M.F. AMEE Guide No. 18: Standard setting in student assessment. Med Teach. 2000; 22: 120–30.

5. Beuk C.H. A method for reaching a compromise between absolute and relative standards in examinations. J Educ Measure. 1984; 21: 147–52.

6. Boulet J.R., De Champlain A.F., McKinley D.W. Setting defensible performance standards on OSCEs and standardized patient examinations. Med Teach. 2003; 25: 245–9.

7. Burrows P.J., Bingham L., Brailovsky C.A. A modified contrasting groups method used for setting the passmark in a small scale standardised patient examination. Adv Health Sci Educ Theory Pract. 1999; 4: 145–54.

8. Clauser B.E., Clyman S.G. A contrasting-groups approach to standard setting for performance assessments of clinical skills. Acad Med. 1994; 69: S42–S44.

9. Cohen-Schotanus J., van der Vleuten C.P.M. A standard setting method with the best performing students as point of reference: Practical and affordable. Med Teach. 2010; 32: 154–60.

10. De Gruijter D.N.M. Compromise models for establishing examination standards. J Educ Measure. 1985; 22: 263–9.

11. Dijkstra J., Van der Vleuten C.P.M., Schuwirth L.W.T. A new framework for designing programmes of assessment. Adv Health Sci Educ Theory Pract. 2010; 15: 379–93.

12. Downing S.M., Lieska N.G., Raible M.D. Establishing passing standards for classroom achievement tests in medical education: A comparative study of four methods. Acad Med. 2003; 78: S85–S87.

13. Downing S.M., Tekian A., Yudkowsky R. Procedures for establishing defensible absolute passing scores on performance examinations in health professions education. Teach Learn Med. 2006; 18: 50–7.

14. Ebel R. Essentials of educational measurement. 2nd ed. Englewood Cliffs, NJ: Prentice-Hall. 1972.

15. Frank J.R., Snell L.S., Cate O.T, Holmboe E.S., Carraccio C., Swing S.R., Harris P., Glasgow N.J., Campbell C., Dath D., et al. Competency-based medical education: Theory to practice. Med Teach. 2010; 32: 638–45.

16. Geisinger K.F. Using standard-setting data to establish cutoff scores. Educ Measure: Issu Pract. 1991; 10: 17–22.

17. Geisinger K.F., McCormick C.M. Adopting cut scores: Post-standardsetting panel considerations for decision makers. Educ Measure: Issu Pract. 2010; 29: 38–44.

18. Haladyna T., Hess R. An evaluation of conjunctive and compensatory standard-setting strategies for test decisions. Educ Assess. 1999; 6: 129–53.

19. Hambleton R.K., Jaeger R.M., Plake B.S., Mills C. Setting performance standards on complex educational assessments. Appl Psychol Measur. 2000; 24: 355–66.

20. Hambleton R.K., Slater S.C. Reliability of credentialing examinations and the impact of scoring models and standard-setting policies. Appl Measur Educ. 1997; 10: 19–28.

21. Hofstee W.K.B. The case for compromise in educational selection and grading. In: Anderson S.B., Helmick J.S., ed. On educational testing. San Francisco, CA: Jossey-Bass. 1983, p. 109–27.

22. Holmboe E.S., Sherbino J., Long D.M., Swing S.R., Frank J.R. The role of assessment in competency-based medical education. Med Teach. 2010; 32: 676–82.

23. Jaeger R.M. Selection of judges for standard-setting. Educ Measu: Iss Pract. 1991; 10: 3–14.

24. Kane M.T. The assessment of professional competence. Eval Health Prof. 1992; 15: 163–82.

25. Kane M.T. Validating interpretive arguments for licensure and certification examinations. Eval Health Prof. 1994; 17: 133–59; discussion 236–41.

26. Kane M.T. Validation. In: Brennan R.L., ed. Educational measurement. Westport, CT: American Council on Education and Praeger Publishers. 2006, p. 17–64.

27. Kaufman D.M., Mann K.V., Muijtjens A.M.M., van der Vleuten C.P.M. A comparison of standard-setting procedures for an OSCE in undergraduate medical education. Acad Med. 2000; 75: 267–71.

28. Linn R.L., Burton E. Performance-based assessment: Implications of task specificity. Educ Measure: Issu Pract. 1994; 13: 5–8.

29. Livingston S.A., Zieky M.J. Passing scores: A manual for setting standards of perfromance on education and occupational tests. Educ Testing Serv. Princeton, New Jersey. 1982.

30. Miller G.E. The assessment of clinical skills/competence/performance. Acad Med. 1990; 65: S63–S67.

31. Nedelsky L. Absolute grading standards for objective tests. Educ Psychol Measur. 1954; 14: 3–19.

32. Nestel D., Kneebone R., Black S. Simulated patients and the development of procedural and operative skills. Med Teach. 2006; 28: 390–1.

33. Norcini J., Burch V. Workplace-based assessment as an educational tool: AMEE Guide No. 31. Med Teach. 2007; 29: 855–71.

34. Norcini J.J. Principles for setting standards on certifying and licensing examinations. In: Rothman A.I., Cohen R., ed. The Sixth Ottawa Conference on Medical Education. Toronto: University of Toronto Bookstore. 1994, p. 346–7.

35. Norcini J.J. Work based assessment. Br Med J. 2003; 326: 753–5.

36. Norcini J., McKinley D. Assessment methods in medical education. Teach Teacher Educ. 2007; 23: 239–50.

37. Norcini J.J., Stillman P.L., Sutnick A.I., Regan M.B., Haley H.L., Williams R.G., Friedman M. Scoring and standard setting with standardized patients. Eval Health Prof. 1993; 16: 322–32.

38. Patil N.G., Saing H., Wong J. Role of OSCE in evaluation of practical skills. Med Teach. 2003; 25: 271–2.

39. Pell G., Fuller R., Homer M., Roberts T. How to measure the quality of the OSCE: A review of metrics – AMEE guide no. 49. Med Teach. 2010; 32: 802–11.

40. Raymond M.R., Reid J. Who made thee a judge? Selecting and training participants for standard setting. In: Cizek G.J., ed. Setting performance standards: Concepts, methods, and perspectives. Mahwah, NJ: Lawrence Erlbaum Associates. 2001, p. 119–58.

41. Reznick R.K., Blackmore D., Dauphine´e W.D., Rothman A.I., Smee S. Large-scale high-stakes testing with an OSCE: Report from the Medical Council of Canada. Acad Med. 1996; 71: S19–S21.

42. Rothman A.I., Cohen R. A comparison of empirically- and rationallydefined standards for clinical skills checklists. Acad Med. 1996; 71: S1–S3.

43. Schindler N., Corcoran J., DaRosa D. Description and impact of using a standard-setting method for determining pass/fail scores in a surgery clerkship. Am J Surg. 2007; 193: 252–7.

44. Smee S.M., Blackmore D.E. Setting standards for an objective structured clinical examination: The borderline group method gains ground on Angoff. Med Educ. 2001; 35: 1009–10.

45. Traub R.E. Facing the challenge of multidimensionality in performance assessment. In: Rothman A.I., Cohen R., ed. Proceedings of the Sixth Annual Ottawa Conference on Medical Education. Toronto: University of Toronto Bookstore. 1994, p. 9–11.

46. Wood T.J., Humphrey-Murto S.M., Norman G.R. Standard setting in a small scale OSCE: A comparison of the modified borderline-group method and the borderline regression method. Adv Health Sci Educ. 2006; 11: 115–22.

CHIEF EDITOR
CHIEF EDITOR
Balkizov Zalim Zamirovich
Professor of the Department of Professional Education and Educational Technologies, Associate Professor of the Department of Hospital Surgery of the N.I. Pirogov Russian National Research Medical University of the Ministry of Health of the Russian Federation, member of the Coordinating Council for Personnel Policy of the Ministry of Health of the Russian Federation, Adviser to the President of the National Medical Chamber, Executive Secretary of the Commission for Accreditation of Continuing Medical Education Events in the National Medical Chamber, Secretary General of the Russian Society of Medical Specialists General Representative of the Association for Medical Education in Europe, General Director of GEOTAR-Med
РОСМЕДОБР 2021
Вскрытие
Medicine today

Уважаемые коллеги, приглашаем принять участие в конференции "Бронхоэктазы: муковисцидоз и не только..." в Воронеже, которая состоится с 23 по 24 июня Организаторы: · Общероссийская общественная организация "Российское общество медицинских генетиков"; · НМИЦ...

7-8 июня 2022 года "Фёдоровские чтения" объединили молодых талантливых офтальмологов Летний сезон Общество офтальмологов России открыло Всероссийской научно-практической конференцией с международным участием "Фёдоровские чтения"! Это одно из немногих российских...

На 2022 год запланирован новый этап в развитии отечественной ринологии - впервые в России будет проведен RHINOLOGY WORLD CONGRESS 2022 IRS - ISIAN . Это грандиозное мероприятие объединит два ключевых события мировой ринологической науки: 22-й Конгресс Международного...


Journals of «GEOTAR-Media»