To the content
2 . 2021

AMEE GUIDE # 87. DEVELOPING QUESTIONNAIRES FOR EDUCATIONAL RESEARCH

Abstract

In this AMEE Guide, we consider the design and development of self-administered surveys, commonly called questionnaires. Questionnaires are widely employed in medical education research. Unfortunately, the processes used to develop such questionnaires vary in quality and lack consistent, rigorous standards. Consequently, the quality of the questionnaires used in medical education research is highly variable. To address this problem, this AMEE Guide presents a systematic, seven-step process for designing high-quality questionnaires, with particular emphasis on developing survey scales. These seven steps do not address all aspects of survey design, nor do they represent the only way to develop a high-quality questionnaire. Instead, these steps synthesize multiple survey design techniques and organize them into a cohesive process for questionnaire developers of all levels. Addressing each of these steps systematically will improve the probabilities that survey designers will accurately measure what they intend to measure.

Artino Jr. A.R., La Rochelle J.S., Dezee K.J., Gehlbach H. Developing questionnaires for educational research: AMEE Guide No. 87. 2014; 36: 6, 463–74. DOI: 10.3109/0142159X.2014.889814.

References

- American Educational Research Association (AERA), American Psychological Association (APA) & National Council on Measurement in Education (NCME). Standards for education and psychological testing. Washington, DC: American Educational Research Association, 1999.

- Artino A.R., Gehlbach H., Durning S.J. AM last page: Avoiding five common pitfalls of survey design. Acad Med. 2011; 86: 1327.

- Artino A.R., Gehlbach H. AM last page: avoiding four visual design pitfalls in survey development. Acad Med. 2012; 87: 1452.

- Beck C.T., Gable R.K. Ensuring content validity: An illustration of the process. J Nurs Meas. 2001; 9: 201–15.

- Christian L.M., Parsons N.L., Dillman D.A. Designing scalar questions for web surveys. Sociol Method Res. 2009; 37: 393–425.

- Colliver J.A., Conlee M.J., Verhulst S.J., Dorsey J.K. Reports of the decline of empathy during medical education are greatly exaggerated: a reexamination of the research. Acad Med. 2010; 85: 588–93.

- Cook D.A., Beckman T.J. Current concepts in validity and reliability for psychometric instruments: theory and application. Am J Med. 2006; 119: 166.e7–16.

- DeVellis R.F. Scale development: theory and applications. 2nd ed. Newbury Park, CA: Sage, 2003.

- Dillman D., Smyth J., Christian L. Internet, mail, and mixed-mode surveys: the tailored design method. 3rd ed. Hoboken, NJ: Wiley, 2009.

- Drennan J. Cognitive interviewing: verbal data in the design and pretesting of questionnaires. J Adv Nurs. 2003; 42 (1): 57–63.

- Fabrigar L.R., Wegener D.T. Exploratory factor analysis. New York: Oxford University Press, 2012.

- Fowler FJ. Survey research methods. 4th ed. Thousand Oaks, CA: Sage, 2009.

- Gehlbach H., Artino A.R., Durning S. AM last page: survey development guidance for medical education researchers. Acad Med. 2010; 85: 925.

- Gehlbach H., Brinkworth M.E. Measure twice, cut down error: a process for enhancing the validity of survey scales. Rev Gen Psychol. 2011; 15: 380–7.

- Kane M.T. Validation in educational measurement. 4th ed. Westport, CT: American Council on Education/Praeger, 2006.

- Karabenick S.A., Woolley M.E., Friedel J.M., Ammon B.V., Blazevski J., Bonney C.R., et al. Cognitive processing of self-report items in educational research: do they think what we mean? Educ Psychol. 2007; 42 (3): 139–51.

- Krosnick J.A. Survey research. Annu Rev Psychol. 1999; 50: 537–67.

- Magee C., Byars L., Rickards G., Artino A.R. Tracing the steps of survey design: a graduate medical education research example. J Grad Med Educ. 2013; 5 (1): 1–5.

- McCoach D.B., Gable R.K., Madura J.P. Instrument development in the affective domain: school and corporate applications. 3rd ed. New York: Springer, 2013.

- McIver J.P., Carmines E.G. Unidimensional scaling. Beverly Hills, CA: Sage, 1981.

- McKenzie J.F., Wood M.L., Kotecki J.E., Clark J.K., Brey R.A. Establishing content validity: Using qualitative and quantitative steps. Am J Health Behav. 1999; 23 (4): 311–8.

- Napoles-Springer A.M., Olsson-Santoyo J., O’Brien H., Stewart A.L. Using cognitive interviews to develop surveys in diverse populations. Med Care. 2006; 44 (11): s21–30.

- Pett M.A., Lackey N.R., Sullivan J.J. Making sense of factor analysis: the use of factor analysis for instrument development in health care research. Thousand Oaks, CA: Sage, 2003.

- Polit D.F., Beck C.T. Nursing research: principles and methods. 7th ed. Philadelphia: Lippincott Williams and Wilkins, 2004.

- Polit D.F., Beck C.T. The content validity index: are you sure you know what’s being reported? Critique and recommendations. Res Nurs Health. 2006; 29: 489–97.

- Rickards G., Magee C., Artino A.R. You can’t fix by analysis what you’ve spoiled by design: developing survey instruments and collecting validity evidence. J Grad Med Educ. 2012; 4 (4): 407–10.

- Rubio D.M., Berg-Weger M., Tebb S.S., Lee E.S., Rauch S. Objectifying content validity: Conducting a content validity study in social work research. Soc Work Res. 2003; 27 (2): 94–104.

- Schmitt N. Uses and abuses of coefficient alpha. Psychol Assess. 1996; 8: 350–3.

- Schwarz N. Self-reports: How the questions shape the answers. Am Psychol. 1999; 54: 93–105.

- Sullivan G. A primer on the validity of assessment instruments. J Grad Med Educ. 2011; 3 (2): 119–20.

- Sullivan G.M., Artino A.R. Analyzing and interpreting data from Likerttype scales. J Grad Med Educ. 2013; 5 (4): 541–2.

- Tourangeau R., Rips L.J., Rasinski K.A. The psychology of survey response. New York: Cambridge University Press, 2000.

- Waltz C.F., Strickland O.L., Lenz E.R. Measurement in nursing and health research. 3rd ed. New York: Springer, 2005.

- Watt T., Rasmussen A.K., Groenvold M., Bjorner J.B., Watt S.H., Bonnema S.J., et al. Improving a newly developed patient-reported outcome for thyroid patients, using cognitive interviewing. Qual Life Res. 2008; 17: 1009–17.

- Weng L.J. Impact of the number of response categories and anchor labels on coefficient alpha and test-retest reliability. Educ Psychol Meas. 2004; 64: 956–72.

- Willis G.B., Artino A.R. What do our respondents think we’re asking? Using cognitive interviewing to improve medical education surveys. J Grad Med Educ. 2013; 5 (3): 353–6.

- Willis G.B. Cognitive interviewing: a tool for improving questionnaire design. Thousand Oaks, CA: Sage, 2005.

All articles in our journal are distributed under the Creative Commons Attribution 4.0 International License (CC BY 4.0 license)

CHIEF EDITOR
CHIEF EDITOR
Balkizov Zalim Zamirovich
Secretary General of the Russian Society of Medical Education Specialists, Director of the Institute of Training of Medical Education Specialists of the Russian Medical Academy of Continuing Professional Education, 125993, Moscow, Russian Federation, Professor of the Department of Vocational Education and Educational Technologies of the N.I. Pirogov RNIMU of the MOH of Russia, CEO of GEOTAR-Med, Advisor President of the National Medical Chamber, Moscow, Russian Federation

Journals of «GEOTAR-Media»