The Art of the Question
New reforms create demand for skilled item writers.
Well-prepared residents fly through them in less than a minute. They read the question, consider the options, choose an answer, and move on.
They are multiple-choice questions, and there is nothing quick or easy about writing good ones. The question writer — called an item writer in the field of exam writing — spends hours formulating just one question to measure the test taker's clinical knowledge and diagnostic skill.
But the effort is considered time well spent. Under the guidance of the National Board of Medical Examiners (NBME), many professional medical societies, including the American College of Radiology and the American Roentgen Ray Society, have become increasingly involved in promoting sound medical testing practices. "Valid testing has a very important role to play," says Mark R. Raymond, Ph.D., NBME's principal assessment scientist. "If the tests are done right, the public is well served. If the questions are not well designed, they can allow people who really don't know the subject matter to get a passing grade."
Diplomates of the ABR also deserve to know that cognitive exams are a reliable way to determine who is qualified for clinical practice, notes ABR Executive Director Gary J. Becker, M.D., FACR. "Most [diplomates] do not know that the evidence base is solid and growing behind the correlation between their performance on certifying cognitive examinations and performance in practice," he says.
Good questions, and the item writers required to create them, are in demand — particularly in radiology because its certification programs are in a state of flux. New committees are developing test questions for the ABR's new core and certification exams for radiology residents. The question areas include medical physics, patient safety, 10 categories of organ systems, and six imaging modalities.
The new core exam will have 600-700 questions, and the certifying exam will have approximately 300. Each committee member who is helping develop the new computer-based exams has been asked to submit 20 cases for each of the next four years to create a sufficient pool of items for the test. (See October 2010 ACR Bulletin for related article, "Introducing the New Boards.") Each case has one or more items.
So, what is promoting this emerging market, and what does it take to become a good item writer?
Good questions, and the item writers required to create them, are in demand — particularly in radiology.
Effect of MOC
The ABR's Maintenance of Certification (MOC) program is one area fueling demand for talented item writers. Lifetime board certification was discontinued in 2002 in favor of a 10-year certification cycle that requires ABR diplomates, who were first certified after 2001, to pass a cognitive recertification exam every 10 years.
ABR diplomates are also required to earn 250 AMA PRA Category 1 Credits™ every 10 years and to complete two self-assessment modules (SAMs) annually. More than 200 SAMs, which each include a multiple-choice exam to test comprehension, are needed to accommodate the diplomates' varied professional needs.
Continuous professional development and MOC testing are necessary because of published evidence showing that a physician's professional performance is likely to deteriorate over time, according to Becker. "The facts have tremendous bearing on the mandate for continuous professional development and on the need for MOC as a framework to accomplish it," he says.
Developing Good Questions
Though academic radiologists will meet most of the demand for this field, anyone can learn to become a good item writer, says Janette Collins, M.D., M.Ed., FCCP, FACR, chair of radiology at the University of Cincinnati College of Medicine in Ohio. Mastery over a particular dimension of diagnostic radiology, radiation oncology, or medical physics, as well as willingness to learn the mechanics of good item writing are key requirements, she says.
The process begins with framing an objective, according to Jeffrey P. Kanne, M.D., associate professor of radiology at the University of Wisconsin, Madison, and member of ABR's Thoracic Certifying Committee. For questions designed for the ABR, Kanne selects subject matter that an individual seeking certification is expected to know after completing residency training. Question ideas are also drawn from personal clinical experience and review articles.
"Skilled item writers overcome the tendency [to only write] multiple-choice questions that require test takers to recall memorized material," Collins says. "Writers should draw from Bloom's taxonomy, a pyramid of increasingly sophisticated intellectual behaviors important in learning. Higher levels of learning can be tested with properly formed questions. For example, the item writer could provide a mammogram showing a cluster of calcifications and ask about the next logical step for the patient's management."
Raymond advises item writers to keep the question format as simple as possible. He prefers "single best answer" multiple-choice questions where a concise questions, often referred to as "the stem," precedes four or five possible options with one correct answer.
Another critical element for item writers to develop carefully is the lead-in. This element is the last sentence of the question's stem. According to Steven A. Haist, M.D., an internist and associate vice president of test development at NBME, item writers need to create a focused lead-in, structuring it as a question to produce a better item.
“A distractor must be plausible enough to become the choice for some test takers, but it must be implausible enough to lead well-informed test takers to still select the right answer.” — Jeffrey P. Kanne, M.D.
Developing alternate responses is also a vital skill for item writers. "Inventing wrong multiple-choice answers, also called distractors, [can be] the hardest part of item writing," Kanne says. "A distractor must be plausible enough to become the choice for some test takers, but it must be implausible enough to lead well-informed test takers to still select the right answer."
Other pitfalls for item writing include:
• Stating a question in the negative (e.g., "Which of the following is not ..."). These statements are often harder for test takers to understand than questions stated in the positive.
• Using "always," "never," or "sometimes." These terms rarely appear in the correct answer.
• Writing correct answers that are longer than the distractors.
• Repeating key words in the correct answer that are used in the stem.
• Creating distractors that are grammatically inconsistent with the stem.
Overall, item-writing is both a science and an art, Kanne notes. Newcomers may assume that item writing will be easy. However, the challenge of preparing a good question becomes apparent when an item writer tries to write one. "That is where the art comes in," he says.
Case S.M., Swanson D.B. Constructing Written Test Questions for the Basic and Clinical Sciences. National Board of Medical Examiners. 2001; third edition (revised).
Raymond M., Neiers R.B., Reid J.B. “Test-Item Development for Radiologic Technology.” American Registry of Radiologic Technologists 2003.
Collins J. “Writing multiple choice questions for continuing medical education activities and self-assessment modules.” RadioGraphics 2006; 26:543–551.
Frey G.D., Ibbott G., Morin R. et al. “The Life of ABR Physics Examination Questions.” The Beam. Available at: http://theabr.org/4beam/question.pdf. Spring 2009.
By James Brice