AMEE GUIDE # 119. THE FOUNDATIONS OF MEASUREMENT AND ASSESSMENT IN MEDICAL EDUCATION

• Tavakol Mohsen
• Reg Dennick

Abstract

As a medical educator, you may be directly or indirectly involved in the quality of assessments. Measurement has a substantial role in developing the quality of assessment questions and student learning. The information provided by psychometric data can improve pedagogical issues in medical education.

By measuring, we are able to assess the learning experiences of students. Standard setting plays an important role in assessing the performance quality of students as doctors in the future. Presentation of performance data for standard setters may contribute towards developing a credible and defensible pass mark. Validity and reliability of test scores are the most important factors for developing quality assessment questions. The analysis of assessment individual questions provide useful feedback for assessment leads in order to improve the quality of each question, and hence make students' marks fair in terms of the diversity and ethnicity. Item Characteristic Curves (ICC), Differential Item Function (DIF) analysis and option analysis will send signals to assessment leads to improve the quality of individual question.

Conflict of interests. The authors declare no conflict of interests.

References

- Allen M., Yen W. Introduction to measurement theory. Long Grove, IL: Waveland Press, 2002.

- American Educational Research Association, American Psychological Association & National Council on Measurement in Education. The standards for educational and psychological testing. Washington, DC: American Educational Research Association, 1999.

- Black P., Wiliam D. Assessment and classroom learning. Assess Educ. 1998; 5: 7–73.

- Bowers J., Shindoll R. A comparison of the Angoff, Beuk and Hofstee methods for setting a passing score. Iowa: The American College of Testing Program, 1989.

- Brookchart S., Nitko A. Assessment and grading in classrooms. Upper Saddle River, NJ: Pearson Education, 2008.

- Buckendahl C., Davis-Becker S. Setting passing standards for Credentialing programs. In: R.L. Brennan, G.J. Cizek (eds). London: Routledge, 2006.

- Camilli G. Test fairness. In: R. Brennan (ed.). Educational Measurement. USA: ACE, 2006.

- Cizek G. Reconsidering standards and criteria. J Educ Meas. 1993; 30: 93–106.

- Cizek, G. Setting passing scores. Educ Meas Issues Pract. 1996; 15: 20–31.

- Cizek, G. An introduction to contemporary standard setting. In: G. Cizek (ed.). Setting Performance Standards. New York: Routledge, 2012.

- Clauser B., Mee J., Baldwin S., Margolis M., Dillon G. Judges’ use of examinee performance data in an Angoff standard-setting exercise for a medical licensing examination: an experimental study. J Educ Meas. 2009; 46: 390–407.

- Crocker L., Algina J. Introduction to classical and modern test theory, Mason, Ohio: Cengage Learning, 2008.

- De Beer M. Use of differential item functioning (DIF) analysis for bias analysis in test construction. J Ind Psychol. 2004; 30: 52–8.

- Downing S. Threats to the validity of locally developed multiple-choice tests in medical education: construct-irrelevant variance and construct underrepresentation. Adv Health Sci Educ. 2002; 7: 235–41.

- Downing S., Tekian A., Yudkowsky R. Procedures for establishing defensible absolute passing scores on performance examinations in health Professions education. Teach Learn Med. 2006; 18: 50–7.

- Ebel R. Essentials of educational measurement. London: Prentice-Hall International, 1972.

- Epstein R. Assessment in medical education. N Engl J Med. 2007; 356: 387–96.

- Feher Waltz C., Stricland O., Lenz E. Measurement in nursing and health research. New York: Springer, 2010.

- Goodwin L. Changing conceptions of measurement validity: an update on the new standards. J Nurs Educ. 2002; 41: 100–6.

- Haladyna T., Hess R. An evaluation of conjunctive and compensatory standard-setting strategies for test decisions. Educ Assess. 1999; 6: 129–53.

- Haladyna T.M., Downing S. Functional distractors: implications for test-item writing and test design. 1988 [Electronic resource]. URL: http://files.Eric.Ed.Gov/fulltext/ed293851.pdf (date of access August 10, 2015)

- Hambleton R., Itoniak M., Copella J. Essential steps in setting Performance standards on educational tests and strategies for assessing the reliability of results. In: G. Cizek (ed.). Setting Performance Standards. London: Routledge, 2012.

- Hambleton R., Pitoniak M. Setting performance standards. In: R.L. Brennan (ed.). Educational Measurement. USA: American Council on Education, 2006.

- Henryssen S. Gathering, analyzing, and using data on test items. In: R. Thorndike (ed.). Educational measurement. Washington, DC: American Council on Education, 1971.

- Hoyt C.J. Test ability estimated by analysis of variance. Psychometrika. 1941; 6: 153–60.

- Hurtz G., Auerbach M. A meta-analysis of the effects of modification to the Angoff method on cut-off scores and judgment consensus. Educ Psychol Meas. 2003; 63: 584–601.

- Kane M. Validating high-stakes testing programs. Educ Meas Issues Pract. 2002; 21: 31–41.

- Kelley T., Ebel R., Linacre J. Item discrimination indices. Rasch Meas Trans. 2002; 16: 883–4.

- Kolen M. Scaling and norming. In: R. Brennan (ed.) Educational Measurement. Westport, CT: American Council on Education, 2006.

- Lane S., Stone C. Performance assessment. In: R. Brennan (ed.) Educational Measurement. USA: ACE, 2006.

- Margolis M., Clauser B. The impact of examinee performance information on judges’ cut scores in modified Angoff standard-setting exercises. Educ Meas Issues Pract. 2014; 33: 15–21.

- Mcdonald M. Guide to assessing learning outcomes. New York: Jones and Bartlett Learning, 2014.

- Mckinley D., Norcini J. How to set standards on performance-based examinations: AMEE Guide No. 85. Med Teach. 2014; 36: 97–110.

- Miller M., Linn R., Gronlund N. Measurement and assessment in teaching, Boston, Pearson, 2013.

- Norcini J., Dawson-Saunders B. Issues in recertification in North America. In: D. Newble, B. Jolly, R. Wakeford (eds). The Certification and Recertification of Doctors. Cambridge: Cambridge University Press, 1994.

- Schmeiser C., Welch C. Test development. In: R.L. Brennan (ed.). USA: American Council on Education, 2006.

- Shepard L. Classroom assessment. In: R. Brennan (ed.). Educational Measurement. Westport, CT: American Council on Education, 2006.

- Sireci S.S., Robin F. Using cluster analysis to facilitate standard setting. Appl Meas Educ. 1999; 12: 301–5.

- Taube K. The incorporation of empirical item difficulty data into the Angoff standard-setting procedure. Eval Health Prof. 1997; 20: 479–98.

- Tavakol M., Dennick R. Post-examination analysis of objective tests: AMEE Guide No. 54. [Electronic resource]. Dundee AMEE, 2001a. URL: www.amee.org

- Tavakol M., Dennick R. Post examination analysis of objective tests. Med Teach. 2011b; 33: 447–58.

- Tavakol M., Dennick R. Post-examination interpretation of objective test data: monitoring and improving the quality of high-stakes examinations: AMEE Guide 66. Med Teach. 2012; 34: 161–75.

- Tavakol M., Dennick R. Psychometric evaluation of a knowledge based examination using Rasch analysis: an illustrative guide: AMEE Guide No. 72. Med Teach. 2013; 35: 74–84.

- Tavakol M., Dennick R. Post-examination analysis: a means of improving the exam cycle. Acad Med. 2016a; 91: 1324.

- Tavakol M., Dennick R. Post-examination analysis: a means of improving the exam cycle. Acad Med. 2016b; 91: 1324.

- Volkan K., Simon S., Baker H., Todres I. Psychometric structure of a comprehensive objective structured clinical examination: a factor analytic approach. Adv Health Sci Educ. 2004; 9: 83–92.

- Zieky M., Perie M. A primer on setting cut scores on tests of educational achievement [Elecrtonic resource]. ETS, 2006. URL: https://www.ets.Org/Media/Research/pdf/Cut_Scores_Primer.pdf [date of access June 10, 2016]

All articles in our journal are distributed under the Creative Commons Attribution 4.0 International License (CC BY 4.0 license)