45 - Using Psychometrics to Identify Quality Assessments

Introduction: Psychometrics explores the science of measurement. To evaluate traditional continuing education (CE) programs and assessments, providers often study correct response frequencies on pretest and posttest questions. The discrimination index (DI) compares the top tier of learners’ percentage correct on a question(s) minus the bottom tier of learners’ percentage correct. Positive numbers indicate more discrimination and adding this tool could help providers with deeper evaluation of their assessments.

Objective: To explore the use of psychometrics to evaluate programs and assessments in continuing education

Methods: A live program was identified that had the same 4 pretest and posttest questions. The DI was calculated by defining learner tiers based on their percentage of practice dedicated to the topic area. The correct response frequencies were compared to DI values to determine quality of the program and assessment questions. The staff time needed to run the DI analysis was observed.

Results: There were 237 learners with usable assessment scores, with 125 in the top tier and 116 in the bottom tier. Overall correct response frequencies were 36% on the pretest and 47% on the posttest. The overall DIs were 0.08 on pretest and 0.21 on posttest. Question 1 had a decrease in correct response frequencies (pretest: 30%, posttest: 23%). The DIs for Q1 were (pretest: -0.09, posttest: 0.20). All other questions had a positive improvement in both correct response frequency and DI. It took an estimated 2 additional hours to prepare and analyze the data.

Conclusions: Question 1 had a drop in learners answering correctly, implying a discrepancy in the question or the program. However, the DI improved pretest to posttest, implying that the top tier (ie, pivotal) learners improved during the program. Using psychometrics, such as DI, could provide more information to CE providers when evaluating the quality of programming and assessments.