Filetype PDF | Posted on 04 Oct 2022 | 3 years ago
Authentication
digital resources
133x Filetype PDF File size 0.15 MB Source: journals.stfm.org
BRIEF
REPORTS
fmCASES National Examination as a
Pretest in a Family Medicine Clerkship
Dana R. Nguyen, MD; Jessica T. Servey, MD; LaTraia S. Scott, MD
BACKGROUND AND OBJECTIVES: Pretests have been shown to contribute foundation for future successful li-
to improved performance on standardized tests by serving to facilitate devel- censing examination performance.
opment of individualized study plans. fmCASES is an existing validated ex- Previous studies demonstrate the
amination used widely in family medicine clerkships throughout the country. ability of clerkship practice tests to
Our study aimed to determine if implementation of the fmCASES National predict performance on NBME sub-
Examination as a pretest decreased overall failure rates on the end-of-clerk- ject examinations, but show mixed
ship National Board of Medical Examiners (NBME) subject examination, and results of the impact on student
to assess if fmCASES pretest scores correlate with student NBME scores. performance improvement.4-5 Gen-
METHODS: One hundred seventy-one and 160 clerkship medical students in eral education concepts show that
different class years at a single institution served as the control and interven- practice tests can contribute to im-
tion groups, respectively. The intervention group took the fmCASES Nation- proved performance on standardized
al Examination as a pretest at the beginning of the clerkship and received tests via comprehension calibration,
educational prescriptions based on the results. Chi-square analysis, Pearson study plan development, and appli-
6-13
correlation, and receiver operating curve analysis were used to evaluate the cation of metacognitive strategy.
effectiveness and correlations for the intervention. fmCASES are online case-based
RESULTS: Students completing an fmCASES National Examination as a pre- modules originally created to meet
test failed the end-of-clerkship NBME exam at significantly lower rates than the Society of Teachers of Family
those students not taking the pretest. The overall failure rate for the interven- Medicine’s National Clerkship Cur-
riculum Objectives,14 and were found
tion group was 8.1% compared to 17.5% for the control group (P=0.01). High- to “foster self-directed and indepen-
er pretest scores correlated with higher NBME examination scores (r=0.55, dent study” and “emphasize and
P<0.001). 15
model clinical problem-solving.”
CONCLUSIONS: fmCASES National Examination is helpful as a formative During the time of this initiative, 146
assessment tool for students beginning their family medicine clerkship. This medical schools in the United States
tool introduces students to course learning objectives, assists them in identi- utilized fmCASES to teach or assess
fying content areas most in need of study, and can be used to help students student learners.16 Increased student
design individualized study plans. engagement in the online fmCASES
(Fam Med. 2018;50(2):142-5.) is associated with improved student
doi: 10.22454/FamMed.2018.853841 performance on end-of-clerkship ex-
aminations, and use of the virtual
cases is comparable to traditional
17-20
tandardized knowledge- between National Board of Medi- textbook learning. Evidence also
based examinations are the cal Examiners (NBME) subject ex- exists to suggest that the fmCAS-
Sstandard by which medical aminations and subsequent United ES National Examination provides
professionals must demonstrate a States Medical Licensing Examina- accurate and valid assessments of
1-3
minimum competence in order to tion (USMLE) scores. Thus, stu-
earn a medical license and achieve dent preparation and performance From the Uniformed Services University,
specialty-based credentials. Previous on gateway NBME subject exami- Department of Family Medicine (Drs Nguyen
and Servey), and Ft Belvoir Community
studies suggest a strong correlation nations can potentially provide a Hospital, Ft Belvoir, VA (Dr Scott).
142 FEBRUARY 2018 VOL. 50, NO. 2 FAMILY MEDICINE
BRIEF
REPORTS
student clinical knowledge in family results report was provided to all MCAT score were not significantly
medicine, and are comparable to the students 2 days after the exam. In- different between the groups.
17
use of NBME examinations. tervention group students attended Thirty of 171 students (17.5%)
Our study extended the appli- an hour-long study skills lecture di- failed the family medicine NBME
cation of fmCASES National Ex- recting them to prioritize study time on the first attempt; 160 students
amination outside of its use as an during clinical weeks on the fmCAS- from the class of 2017 took the fm-
end-of-clerkship evaluation. The spe- ES for which they received the low- CASES pretest and completed the
cific aims were: (1) to assess if imple- est scores. Other study management entirety of the family medicine
mentation of the fmCASES National strategies were also discussed. All clerkship. Among these students,
Examination as a tool for formative students completed the NBME ex- 13 (8.1%) failed the family medicine
assessment decreased group overall amination at the end of the clerk- NBME examination on the first at-
failure rates on the end-of-clerkship ship. tempt. Chi-square analysis between
NBME examination, and (2) to as- Intervention group students were these two groups indicates a signifi-
sess if fmCASES pretest scores cor- risk stratified based upon their pre- cant difference in pass rates, with
relate with student NBME scores. test score. Those who received a P=0.01 (Table 2). Multiple regres-
score below the group 20th percen- sion analysis was used to test if the
Methods tile were designated as at-risk for student pretest score was predictive
This was a retrospective cohort failure of the NBME examination, of the NBME score as compared to
study comparing outcomes of stu- and subsequently received academ- other demographic factors (Table
dents from two different medical ic counseling by one of two clinical 3). It was found that pretest signifi-
school classes. Participants includ- associate professor faculty members cantly predicted NBME score (β=.56,
ed students from the class of 2016 (current and previous FM clerkship P<0.001). Age, undergraduate GPA,
(Co16) and 2017 (Co17) who com- directors). Faculty assisted at-risk and MCAT scores were not predic-
pleted the 6-week family medicine students in development of individu- tive of NBME score.
clerkship at the Uniformed Services alized study plans. Study plans were Bivariate Pearson correlation
University (USU). Co16 USU stu- tailored toward the learning styles of analysis of the relationship be-
dents served as the control cohort, the learners, with focused study in tween pretest scores and NBME
and Co17 USU students served as weaker knowledge areas. In addition scores showed a correlation coeffi-
the intervention/comparison group. to completion of online fmCASES, cient of 0.55 with P<0.001 (Figure
The local Institutional Review Board study plan options included use of 1), indicating a statistically signifi-
approved this project as an exempt question banks, case-based learning cant moderate positive correlation
protocol. textbooks, directed topic reading, and between pretest scores and NBME
As part of a clerkship curriculum online video review. The clerkship di- examination scores.
change in January 2015, we imple- rector offered optional assistance to Receiver-operator curve (ROC)
mented the fmCASES National Ex- all clerkship students. analysis determined the optimal
amination as a 100-question pretest We determined an overall NBME pretest examination score with the
and formative assessment tool. Our failure rate for the control and inter- highest sensitivity and specificity
goals were: (1) to provide initial per- vention groups. In the intervention for failing the NBME examination.
sonal perspective and reflection on group we also examined individual Pretest examination scores ranging
topic areas likely to be covered on pretest and NBME scores for corre- from 40% to 78% were selected as
the NBME exam; (2) to use this per- lation. Chi-square analysis, multi- cutoff values based upon the range of
spective to serve as study focus areas variate linear regression, Pearson scores achieved by the pretest group.
during the following rotation weeks; correlation, and receiver-operator For each score cutoff value, sensitiv-
(3) to give students an objective mea- curve analysis were used to evaluate ity and specificity were calculated
surement of their knowledge prior to the effectiveness of the intervention. and the results were plotted as an
taking the NBME examination; (4) ROC curve. Figure 2 displays the
to identify students at risk for fail- Results ROC curve with optimal sensitivity
ure of the NBME examination; and Demographic data for the control and specificity cut-offs marked with
(5) to highlight individual knowledge and intervention groups is shown in an arrow. According to our data, a
areas in need of improvement. Table 1. The family medicine clerk- pretest score cutoff value of 60 opti-
Students in the intervention group ship was completed by 171 students mized sensitivity and specificity for
were given one of two versions of the hip in the class of 2016. This con- test failure.
fmCASES National Examination, a trol group did not take the pretest.
nationally-standardized and validat- The intervention group (Co17) had a Discussion
ed test on Med-U.org. The examina- significantly higher undergraduate The fmCASES National Examina-
tion was administered in proctored GPA, but characteristics of age and tion in conjunction with the case
classrooms in the online format. A breakdown results report served
FAMILY MEDICINE VOL. 50, NO. 2 FEBRUARY 2018 143
BRIEF
REPORTS
Table 1: Student Demographic Characteristics
Student Characteristics Class of 2016 Class of 2017 P Value
Total Number 171 *170
Mean Age 24.5 24.4 0.74
Male 111 114 0.68
Average Cumulative Undergraduate GPA 3.53 3.60 0.02
Average MCAT Score 31.1 31.3 0.45
Race/Ethnic Group
Black American 7 2
American Indian/Alaskan/Hawaiian 0 0
White 116 106 0.29
Asian/Pacific Islander 30 45
Puerto Rican 0 0
Mexican American/Chicano 3 2
Other Hispanic 2 3
Race not reported 13 12
* Only 160 students took both the pretest and family medicine NBME exam.
Table 2: Failure Rates Between Cohorts between the groups, the pretest score
Total Fail Failure % was a significant predictor of NBME
score after adjusting for age, MCAT
Control group 171 30 17.5% score, and undergraduate GPA. Oth-
Intervention group 160 13 8.1% er institutions could use this corre-
Total 331 43 lation and our ROC analysis to help
P=.01 identify students most likely to fail
Intervention Group Subanalysis the end-of-clerkship exam.
We theorize that the benefits
“Non at-risk” students 132 9 6.8% gained from our pretest intervention
“At-risk” students 28 4 14.3% are not purely secondary to the iden-
P=.19 tification of knowledge gaps. General
education theory suggests that test-
ing produces better organization of
Table 3: Multivariate Linear Regression Coefficients knowledge and improves the trans-
β P Value 13
fer of knowledge to new contexts.
Age -0.15 0.37 We also theorize that completion of
Cumulative undergraduate GPA 0.03 0.27 the pretest served as a cognizant or
mindful introduction to the clerk-
MCAT score 0.36 0.11 ship course objectives, and that de-
Pretest score 0.56 <0.001 tailed awareness of these objectives
helped students tailor their clinical
Dependent Variable: NBME Score encounters and academic study time
to subject areas most likely on the
as a helpful formative assessment clerkship students. Additionally, we final exam.
tool and guided individual NBME confirmed a moderate positive cor-
examination preparation in our relation between fmCASES pretest Conclusion
student population. Our findings scores and end-of-clerkship NBME Implementation of a formative as-
demonstrate that use of this exam examination scores. In general, low- sessment prior to the beginning of
as a pretest combined with study er fmCASES pretest scores were as- a clerkship can provide students an
plan counseling can decrease over- sociated with lower NBME scores. explicit roadmap for learning objec-
all failure rates on the end-of-clerk- Although there was a slight differ- tives and assist them in identifying
ship NBME examination for FM ence in demographic characteristics content areas in most need of study.
144 FEBRUARY 2018 VOL. 50, NO. 2 FAMILY MEDICINE
BRIEF
REPORTS
Figure 1. Pretest Score vs. Shelf Exam Score
Figure 1: Pretest Score vs Shelf Exam Score 5. Constance E, Dawson B, Steward D, Schrage
J, Schermerhorn G. Coaching students who
100 fail and identifying students at risk for fail-
) 95 r=0.55 ing the National Board of Medical Examiners
0 medicine subject test. Acad Med. 1994;69(10)
0
1 P<0.001 (suppl):S69-S71.
f 90
o 6. Balch WR. Practice versus review exams
t
u 85 and final exam performance. Teach Psychol.
(o 1998;25(3):181-185.
e 80
r
o
c 75 7. Zabrucky KM. Knowing what we know and do
S
not know: educational and real world implica-
w
a 70 tions. Procedia Soc Behav Sci. 2010;2(2):1266-
R
1269.
m 65
a
x 8. Glenberg AM, Wilkinson AC, Epstein W.
E
60
f The illusion of knowing: failure in the self-
el
h 55 assessment of comprehension. Mem Cognit.
S 1982;10(6):597-602.
50 9. Glenberg AM, Sanocki T, Epstein W, Morris C.
40 45 50 55 60 65 70 75 80 Enhancing calibration of comprehension. J Exp
Pretest Raw Score (out of 100) Psychol. 1987;116(2):119-136.
10. Bjork EL, Soderstrom NC, Little JL. Can
multiple-choice testing induce desirable dif-
ficulties? Evidence from the laboratory and the
Figure 2: ROC Curve classroom. Am J Psychol. 2015;128(2):229-239.
Figure 2. ROC Curve 11. Little JL, Bjork EL, Bjork RA, Angello G.
1.000 Multiple-choice tests exonerated, at least of
(Pretest Score=60%) some charges: fostering test-induced learning
0.800 and avoiding test-induced forgetting. Psychol
Sci. 2012;23(11):1337-1344.
y 12. Grühn D, Cheng Y. A self-correcting approach
t
i 0.600
v to multiple-choice exams improves students’
i
t
i learning. Teach Psychol. 2014;41(4):335-339.
s
n
e 0.400 13. Roediger HL III, Putnam AL, Smith MA. The
S benefits of testing and their applications to
0.200 educational practice. Psychol Learn Motiv.
2011;(55):1-36.
0.000 14. Society of Teachers of Family Medicine. About
0.000 0.200 0.400 0.600 0.800 1.000 the Family Medicine Clerkship Curriculum.
http://www.stfm.org/Resources/Resourcesfor-
1-Specificity MedicalSchools/STFMNationalClerkshipCur-
riculum/AbouttheFamilyMedicineClerkship-
Curriculum. Accessed 24 July 2017.
15. Leong SL. fmCASES: collaborative develop-
fmCASES National Examination is References ment of online cases to address educational
helpful in this context and can be 1. Zahn CM, Saguil A, Artino AR Jr, et al. Corre- needs. Ann Fam Med. 2009;7(4):374-375.
used to help students design indi- lation of National Board of Medical Examiners 16. MedU. Our Subscribers. http://www.med-u.org/
vidualized study plans. scores with United States Medical Licensing about/our-subscribers. Accessed July 24, 2017.
Examination Step 1 and Step 2 scores. Acad 17. Sussman, H. Does clerkship student perfor-
ACKNOWLEDGMENTS: The authors thank Med. 2012;87(10):1348-1354. mance on the NBME and fmCASES exams
Cara Olsen, PhD, USU Biostatistician, for as- 2. Dong T, Swygert KA, Durning SJ, et al. Is correlate? Poster presented at the Society
sistance with the statistical analysis on this poor performance on NBME clinical subject of Teachers of Family Medicine Conference
project, and Christian Ledford, PhD, USU De- examinations associated with a failing score on Medical Student Education; February
partment of Family Medicine, for assistance in on the USMLE step 3 examination? Acad Med. 2-5, 2012; Long Beach, CA.
preparing the manuscript. 2014;89(5):762-766. 18. Chessman A, Svetlana C, Mainous A, et
Presentations: Presented at the Society of 3. Hiller K, Franzen D, Heitz C, Emery M, al. fmCASES National Exam: Correlations
Teachers of Family Medicine Medical Student Poznanski S. Correlation of the National Board with Student Performance Across Eight Family
Education conference in February 9-12, 2017 of Medical Examiners Emergency Medicine Medicine Clerkships. Presented the Society of
in Anaheim, CA. Advanced Clinical Examination Given in Teachers of Family Medicine Conference on
Disclaimer: The views expressed herein July to Intern American Board of Emergency Medical Student Education; January 24-27,
are those of the authors and do not reflect the Medicine in-training Examination Scores: A 2013; San Antonio, TX.
official policy of the Department of the Army, Predictor of Performance? West J Emerg Med. 19. Shokar GS, Burdine RL, Callaway M, Bulik
Department of Defense, or the US Government. 2015;16(6):957-960. RJ. Relating student performance on a family
CORRESPONDING AUTHOR: Address corre- 4. Hemmer PA, Markert RJ, Wood V. Using in- medicine clerkship with completion of Web
spondence to Dr Nguyen, Department of Fam- clerkship tests to identify students with insuf- cases. Fam Med. 2005;37(9):620-622.
ily Medicine, Uniformed Services University, ficient knowledge and assessing the effect of 20. Demarco MP, Bream KD, Klusaritz HA, Margo
4301 Jones Bridge Road, Bethesda, MD 20814. counseling on final examination performance. K. Comparison of textbook to fmCases on fam-
301-295-3632. dana.nguyen@usuhs.edu. Acad Med. 1999;74(1):73-75. ily medicine clerkship exam performance. Fam
Med. 2014;46(3):174-179.
FAMILY MEDICINE VOL. 50, NO. 2 FEBRUARY 2018 145
no reviews yet
Please Login to review.
