Advancing Resident Skills in Evidence-Based Practice and Research: PICO and PubMed Search Educational Workshop

Background

: Evidence-based medicine (EBM) is a central component of modern clinical practice, requiring clinicians to define clinically relevant questions, appraise the best available evidence, apply it in practice, and evaluate its effectiveness. Despite emphasizing EBM in residency programs, residents often face barriers to engaging in EBM and research. Objective: To evaluate the impact of a targeted PICO (Population, Intervention, Comparison, Outcome) and PubMed article search workshop on residents’ confidence and skills in EBM and research. Methods: A one-hour workshop was conducted for residents at the internal medicine residency in Englewood Hospital and Medical Center in 2024. The workshop covered formulating PICO questions, PubMed searches, and using Medical Subject Headings (MeSH) terms.

Pre- and post-workshop surveys assessed participants’ confidence in EBM. Data were analyzed using the Wilcoxon signed-rank test, with statistical significance defined as a p-value < 0.05. Results: 57 potential participants were in the residency program. A total of 31 participants (54.4% of total potential participants) completed the pre-workshop survey, and 21 (38.9%) completed the post-workshop survey. Statistically significant improvements were observed across all assessed domains, with the global confidence score increasing from 2.65 pre-workshop to 3.32 post-workshop (p < 0.001). Conclusion: Our workshop significantly enhanced participants’ confidence and skills in EBM and research. It effectively addressed knowledge gaps and fostered EBM proficiency in the residency program. Further research is needed to replicate these findings in larger cohorts using structured interventions and validated assessment tools to explore the long-term impact of such workshops on residents’ scholarly activity and clinical practice.

Keywords : Evidence-Based Medicine, Evidence-Based Practice, Workshop Efficacy, Resident Scholarly Activity.

Introduction

Evidence-based practices (EBPs) were first introduced in 1992, laying the foundation for evidence-based medicine (EBM) across all medical disciplines 1,2 . EBM involves a five-step process: 1) defining a clinically relevant question, 2) searching for the best available evidence, 3) critically appraising the evidence, 4) applying the evidence in clinical practice, and 5) evaluating the performance of EBM 3 . This approach is essential in reducing variations in medical practice, which could further lower healthcare costs, as well as improve patient care quality and outcomes 4 . Research and scholarly activity are vital aspects of residency training, providing post-graduate residents with essential EBP skills for their medical careers. Studies indicate that residents who receive research training develop a deeper appreciation for EBM 5 .

The Accreditation Council for Graduate Medical Education (ACGME) mandates that residents and fellows engage in scholarly activities and evidence-based presentations 6 . To support this, residency programs are required to offer resources that facilitate residents’ participation and have adjusted their curricula to meet these needs. Accredited programs must advance residents’ knowledge and application of a scholarly approach to evidence-based patient care, ensuring they are proficient in locating, appraising, and integrating evidence from scientific studies relevant to their patients’ health issues 7 . Despite the emphasis on EBM and residents’ growing interest in research, scholarly activities participation remains limited. Residency programs have introduced initiatives like research rotations and resident research days.

Common barriers include a lack of time, insufficient mentoring, a shortage of research-trained residents, and gaps in knowledge and skills related to statistics and research methodology 5,8 . Possible interventions to overcome these barriers may include mentoring programs along with more protected research time 9 . Since its introduction, PICO (Population, Intervention, Comparison, Outcome) has been recognized as a cornerstone of EBM 10,11 Educators have employed various methods to enhance the use of PICO, improve literature search strategies, and ultimately advance EBP 12,13 . To support the development of residents’ EBP skills, we designed a dedicated workshop aimed at enhancing their ability to define clinically relevant questions, conduct effective PubMed searches and search for the best available evidence using PICO.

METHOD Theinterventioncomprisedaone-hourworkshopforincoming and current internal medicine residents and medical students during the designated curriculum period. The workshop was structured into three segments: the first segment offered an overview of evidence-based medicine (EBM), PICO questions, MeSH (Medical Subject Headings) terms, and PubMed; the second segment involved a real-time demonstration of PubMed article searching using a PICO question developed from a clinical scenario; and the third segment provided participants with hands-on practice in developing PICO questions from the clinical scenario, identifying relevant MeSH terms, and performing PubMed searches. The study was conducted within the Internal Medicine Residency Program at Hackensack University Medical Center/Englewood Hospital and Medical Center in 2024. The study protocol was reviewed and exempted by our Institutional Review Board (IRB, Title: Advancing Resident Skills in Evidence-Based Practice and Research: PICO and PubMed Search Educational Workshop; Number: XR-24-991).

Participation in the anonymous survey was voluntary, and participants were informed that the questionnaire was part of a medical education research project before completion. The pre-workshop survey was distributed to rotating medical students, incoming first-year residents, and residents from three different classes several weeks before the intervention workshop. It was structured into three sections: the first section gathered demographic information to identify respondents’ current training levels and two Likert-scale questionsaccessingbaselineresearchskills;thesecondsection comprised six Likert-scale questions evaluating residents’ confidence in using PICO questions for evidence-based medicine, conducting searches in PubMed, and formulating research questions; and the third section assessed residents’ perceptions of research challenges through one open-ended question, one Likert-scale question, and one multiple-choice question regarding perceived barriers to conducting research.

The Likert-scale questions employed a four-point scale, with responses ranging from 1 (not confident/difficult at all) to 4 (very confident/very difficult). The post-workshop survey, administered immediately after the workshop, replicated the second sections of the pre-workshop survey and included the current training level and an additional open feedback section. The intervention comprised a one-hour workshop for two groups: once during the orientation for incoming first-year residents and once for the remaining participants during a designated curriculum period. The workshop was structured into three segments: the first segment offered an overview of evidence-based medicine (EBM), PICO questions, MeSH (Medical Subject Headings) terms, and PubMed; the second segment involved a real-time demonstration of PubMed article searching using a PICO question developed from a clinical scenario; and the third segment provided participants with hands-on practice in developing PICO questions from the clinical scenario, identifying relevant MeSH terms, and performing PubMed searches.

Data were analyzed using the Statistical Analysis System (SAS, SAS Institute Inc., version 3.81). Given the non-parametric nature of the data, the Wilcoxon signed-rank test was applied to compare pre- and post-workshop survey results from non-matched participants. Six Likert-scale questions were evaluated from both the pre- and post-workshop surveys, addressing the following areas: confidence in performing evidence-based medicine (Q1), formulating a PICO question when faced with a clinical scenario (Q2), searching for a suitable article that fits the PICO question in PubMed (Q3), using MeSH terms in conducting an article search (Q4), knowing what study designs to applied the PICO (Q5), and developing a research question (Q6). The variable “Group” represented the survey condition (pre- or post-workshop), while “Q1” through “Q6” denoted the individual survey scores, ranging from 1 to 4.

“Group” was designated as the class variable, and “Q1” through “Q6” as the analysis variables. The null hypothesis posited no significant difference between preand post-workshop survey scores. Statistical significance was defined as a p-value of < 0.05.

Results

Table 1 presented the baseline characteristics of the respondents and the surveys. There were a total of 55 residents and 2 medical students rotating during the time of workshops. A total of 31 participants (54.4% of total potential participants) completed the pre-workshop survey, and 21 participants (38.9% of total potential participants)completed the post-workshop survey. Among the pre-workshop respondents, 2 (6.5%) were medical students, 14 (45.2%) were incoming first-year residents (PGY1), 6 (19.4%) were current PGY1 residents, 4 (12.9%) were second-year residents (PGY2), and 5 (16.1%) were third-year residents (PGY3). Regarding the question, “How confident are you in writing an abstract and presenting your scholarly work at a conference?” most participants (n=12, 38.7%) selected level 2.

Additionally, 4 participants (12.9%) chose level 1, 10 participants (32.3%) chose level 3, and 5 participants (16.1%) selected level 4. The average confidence level was 2.5. Of the 4 participants who selected level 1, 3 were incoming PGY1 residents, and 1 was a current PGY1 resident. For the question, “How confident are you in conducting original research (e.g., database, retrospective studies)?” the majority of participants (n=12, 38.7%) selected level 1, resulting in an average confidence level of 1.9, indicating low confidence. Additionally, 11 participants (35.5%) chose level 2, 7 participants (22.6%) selected level 3, and only 1 participant (3.2%) chose level 4, indicating the highest confidence. Among those who selected level 1, respondents included individuals from all classes, ranging from incoming PGY1 to PGY3.

Most residents attributed their lack of confidence to insufficient training, exposure, and experience in conducting research. When asked, “How difficult do you think it is to start research?” 13 residents selected level 3, 1 selected level 1, 12 selected level 2, and 5 selected level 4, with an average difficulty level of 2.7. Regarding the obstacles to starting research, the top two reasons were that 18 residents (58.1%) indicated difficulty in finding a research topic, and 17 (54.8%) stated they were unsure where to start or were unfamiliar with study design. Only 5 residents (16.1%) identified the literature review as a barrier, and just 1 resident (3.2%) stated they were not interested in research.

Table 1. Baseline Characteristics of the Survey and Respondents. Respondent n (%) Pre-workshop Survey Respondent 31 (100) Medical Students 2 (6.5) Incoming PGY1 14 (45.2) PGY1 6 (19.4) PGY2 4 (12.9) PGY3 5 (16.1) Post-workshop Survey Respondent 21 (100) Medical Students 2 (9.5) Incoming PGY1 14 (66.7) PGY1 3 (14.3) PGY2 1 (4.8) PGY3 1 (4.8) Pre-workshop Survey Baseline Characteristic Questions How confident are you in writing an abstract and presenting your scholarly work at a conference? Level 1 Least confident 4 (12.9) Level 2 12 (38.7) Level 3 10 (32.3) Level 4 Very confident 5 (16.1) How confident are you in conducting original research (e.g. database, retrospective studies) Level 1 Least confident 12 (38.7) Level 2 11 (35.5) Level 3 1 (3.2) Level 4 Very confident 1 (3.2) How difficult do you think it is to start a research?

Level 1 Least difficult 1 (3.2) Level 2 12 (38.7) Level 3 13 (41.9) Level 4 Very difficult 5 (16.1) Possible Barriers (Multiple Answeres) Difficulty finding a research question/topic 18 (58.1) Difficulty finding a mentor 13 (41.9) Difficulty finding colleagues to help 6 (19.4) Unfamiliar with research design 17 (54.8) Not knowing how to do a thorough literature review 5 (16.1) Not knowing how to start research 17 (54.8) Interested but no time to conduct research 12 (38.7) Not interested 1 (3.2) I would like to attend an after-hour workshop to enhance my research skills. Yes 26 (83.9) No 5 (16.1) The comparison of pre- and post-workshop survey responses revealed statistically significant improvements in participants’ confidence across all assessed domains (Table 2).

The global score increased from a mean of 2.65 (standard deviation (SD) 0.89) pre-workshop to a mean of 3.32 (SD 0.71) post-workshop, with a Z-score of 6.06 and a highly significant p-value of <0.001. Confidence in performing evidence-based medicine (Q1) improved from a mean of 2.87 to 3.28 (Z = 2.31, p = 0.01), and the ability to formulate a PICO question (Q2) increased from 2.80 to 3.33 (Z = 1.83, p = 0.03). Similarly, participants reported enhanced confidence in searching for suitable articles in PubMed (Q3), with mean scores rising from 2.66 to 3.42 (Z = 3.05, p < 0.01), and in using MeSH terms in article searches (Q4), which increased from 2.41 to 3.38 (Z = 3.47, p < 0.001).

Additionally, confidence in selecting appropriate study designs based on the PICO framework (Q5) improved from 2.45 to 3.14 (Z = 2.63, p < 0.01), and in developing a research question (Q6), scores increased from 2.64 to 3.33 (Z = 2.96, p < 0.01) (Figure 1). Table 2. The Comparison of Pre- and Post-workshop Survey. Pre-workshop Survey Post-workshop Survey Z score P value Mean SD Mean SD Global score 2.65 0.89 3.32 0.71 6.06 <0.001*** Q1. Perform EBM 2.87 0.66 3.28 0.45 2.31 0.01* Q2. Formulate a PICO question 2.80 0.99 3.33 0.64 1.83 0.03* Q3. Search for a suitable article that fits the PICO question in PubMed 2.66 0.90 3.42 0.72 3.05 <0.01** Q4.

Using MeSH terms in conducting an article search 2.41 0.90 3.38 0.84 3.47 <0.001*** Q5. Know what study designs to apply the PICO 2.45 0.91 3.14 0.71 2.63 <0.01** Q6. Develop a research question 2.64 0.82 3.33 0.77 2.96 <0.01** *P<0.05; **P<0.01; ***P<0.001 Figure 1. Box plots comparing pre-workshop and post-workshop levels of confidence.

Discussion

The results of our study demonstrated that PICO and PubMed article search workshop significantly enhanced participants’ confidence and skills across various domains of EBM and research. The statistically significant improvements in post-workshop survey responses indicate that the intervention successfully addressed knowledge gaps, particularly in formulating PICO questions, conducting PubMed searches, using MeSH terms, selecting appropriate study designs, performing EBM, and developing research questions. These findings demonstrated the importance of targeted educational interventions to increase exposure to the related tools and further bridge the gap of current research training in medical residency. The success of this workshop suggests that similar programs could be valuable additions to medical education curricula, helping better prepare future clinicians for scholarly activities and EBP.

Previous randomized controlled trials and meta-analyses have suggested that workshops and interventions may not be particularly effective, as residents might acquire EBM skills over the course of their residency training14,15 . This may explain why PGY3 residents appeared less motivated to participate in the workshop and complete the survey; they were more confident in their research and EBM skills compared to PGY1 residents. However, these findings should not discourage us from structured educational interventions. Other studies have demonstrated that a well-designed EBM curriculum can indeed improve residents’ attitudes, skills, and knowledge related to EBM 16,17 . Furthermore, the ACGME has recognized the importance of such training, mandating that residency programs provide resources to facilitate residents’ participation in scholarly activities and advance residents’ knowledge and application of a scholarly approach to EBP 7 .

Integrating structured EBM curricula or workshops into residency program curricula is crucial. In this way, all residents will be well-equipped to apply EBP in their clinical work regardless of their training level and continue to progress throughout their training. Our study revealed that most participants, particularly incoming and current PGY1 residents, reported low confidence in conducting original research and writing abstracts for scholarly presentations. Insufficient training and experience in research have led to a lack of confidence. Additionally, many residents faced significant obstacles to starting research, identifying a research topic, and unfamiliarity with the study design. Similar findings have been reported in other studies. The study revealed that residents were highly interested in research.

However, the lack of protected time and insufficient training in research methodology prevent them from participating 8 . Additionally, previous study indicated that the most frequently mentioned barriers to EBM for residents include limited available time, as well as challenges related to attitude, knowledge, and skills 18 . It’s crucial to provide dedicated time for residents to develop EBM and research skills. Our findings underscore the need for targeted educational interventions to address gaps in research skills and confidence. By tackling these barriers and enhancing research education, residency programs can foster a more research-oriented culture and promote EBP. Existing literature has suggested that targeted educational interventions can significantly enhance learners’ knowledge and skills, particularly when designed to be interactive and aligned with learners’ needs 17 .

Traditional journal clubs, for example, may not be as effective as interactive approaches19 . Our workshop was specifically designed to be interactive and address the needs identified in the pre-workshop survey. It was an essential first step in equipping residents with the tools to define clinically relevant PICO questions and search for the best available evidence. Further training is necessary to build on this foundation. The next step of EBM includes appraising evidence and applying it in clinical practice, as well as evaluating the performance of EBM. This will ensure that residents are well-prepared to engage in EBP and eventually contribute to meaningful medical research. Although our study offers important insights into the effectiveness of the workshop in improving EBM education for residents, there were still several limitations that should be considered.

Firstly, our small sample size was small and may not represent the broader resident population. Also, because of the study design using the survey, there could also be reporting bias with participants potentially providing socially desirable responses rather than accurately reflecting their experiences and confidence levels. In addition, due to the voluntary nature of participation, the discrepancy in the number of respondents between pre- and postworkshop surveys may introduce response bias. There might also be selection bias as more confident residents may not have chosen to attend the workshop, leading to an overrepresentation of those with lower research confidence. Furthermore, there’s a lack of validated tools to assess all aspects of EBM and educational outcomes comprehensively.

Future studies should aim to replicate these findings with a larger and more diverse cohort, employing standardized and validated assessment tools to enhance the robustness and applicability of the results.

Conclusion

The PICO and PubMed article search workshop significantly improved participants’ confidence and skills in EBM and research. Through targeted educational interventions, we successfully addressed key knowledge gaps, particularly in formulating PICO questions, conducting PubMed searches, using MeSH terms, and developing research questions. These improvements of the confidence level underscore the importance of interactive workshops in bridging research training gaps and enhancing residents’ ability to engage in scholarly activities and EBP. Acknowledgment The authors express their sincere gratitude to all individuals and institutions whose contributions facilitated the completion of this research. Ethics Approval The study is conducted according to the principles expressed in the Declaration of Helsinki. The study protocol was reviewed and exempted by our Institutional Review Board (IRB, Title: Advancing Resident Skills in Evidence-Based Practice and Research: PICO and PubMed Search Educational Workshop; Number: XR-24-991).

Conflict of Interest

All authors declare that there are no conflicts of interest to disclose. All authors declare that this research was conducted without any specific funding or financial support from any external organization. The article processing fee for this work, if any, is supported by the Department of Medicine at Englewood Hospital and Medical Center (350 Engle St, Englewood, NJ 07631. Contact number: 201-894-3510).

References

1. Connor L, Dean J, McNett M, et al. Evidence-based practice improves patient outcomes and healthcare system return on investment: Findings from a scoping review. Worldviews Evid Based Nurs. Feb 2023;20(1):6- 15. doi:10.1111/wvn.12621
2. Evidence-based medicine. A new approach to teaching the practice of medicine. Jama. Nov 4 1992;268(17):2420- 5. doi:10.1001/jama.1992.03490170092032
3. Tenny S, Varacallo M. Evidence Based Medicine. StatPearls. StatPearls Publishing Copyright © 2024, StatPearls Publishing LLC.; 2024.
4. Melnyk BM, Fineout-Overholt E. Evidence-based practice in nursing & healthcare: A guide to best practice. Lippincott Williams & Wilkins; 2022.
5. Chan JY, Narasimhalu K, Goh O, et al. Resident research: why some do and others don’t. Singapore Med J. Apr 2017;58(4):212-217. doi:10.11622/smedj.2016059
6. ACGME Specialty-Specific Program Requirements: Resident/Fellow Scholarly Activity (Last Access Aug 26 2024) https://wwwacgmeorg/globalassets/pdfs/ specialty-specific-requirement-topics/dio-scholarly_ activity_resident-fellowpdf.
7. ACGME Common Program Requirements (Residency) (Last Access Aug 26 2024) https://wwwacgmeorg/ globalassets/pfassets/programrequirements/ cprresidency_2023pdf.
8. Nair SC, Ibrahim H, Almarzoqi F, Alkhemeiri A, Sreedharan J. Addressing research barriers and facilitators in medical residency. J Family Med Prim Care. Mar 2019;8(3):1145-1150. doi:10.4103/jfmpc jfmpc_38_19.
9. Merino T, Rojas V, Fuentes-López E, et al. Barriers for research activities in residency programs: A mixmethods study. Medwave. Jan 3 2023;23(1):e2627. doi:10.5867/medwave.2023.01.2627
10. Richardson WS, Wilson MC, Nishikawa J, Hayward RS. The well-built clinical question: a key to evidence-based decisions. ACP J Club. Nov-Dec 1995;123(3):A12-3.
11. Stone PW. Popping the (PICO) question in research and evidence-based practice. Appl Nurs Res. Aug 2002;15(3):197-8. doi:10.1053/apnr.2002.34181
12. Milner KA, Cosme S. The PICO Game: An Innovative Strategy for Teaching Step 1 in Evidence-Based Practice. Worldviews Evid Based Nurs. Dec 2017;14(6):514-516. doi:10.1111/wvn.12255
13. Brown D. A Review of the PubMed PICO Tool: Using Evidence-Based Practice in Health Education. Health Promot Pract. Jul 2020;21(4):496-498. doi:10.1177/1524839919893361
14. Feldstein DA, Maenner MJ, Srisurichan R, Roach MA, Vogelman BS. Evidence-based medicine training during residency: a randomized controlled trial of efficacy. BMC Med Educ. Sep 1 2010;10:59. doi:10.1186/1472- 6920-10-59.
15. Ilic D, Maloney S. Methods of teaching medical trainees evidence-based medicine: a systematic review. Med Educ. Feb 2014;48(2):124-35. doi:10.1111/medu.12288
16. Halalau A, Holmes B, Rogers-Snyr A, et al. Evidencebased medicine curricula and barriers for physicians in training: a scoping review. Int J Med Educ. May 28 2021;12:101-124. doi:10.5116/ijme.6097.ccc0
17. Cusack L, Del Mar CB, Chalmers I, Gibson E, Hoffmann TC. Educational interventions to improve people’s understanding of key concepts in assessing the effects of health interventions: a systematic review. Syst Rev. May 2 2018;7(1):68. doi:10.1186/s13643-018-0719-4 18. van Dijk N, Hooft L, Wieringa-de Waard M. What Are the Barriers to Residents’ Practicing Evidence- Based Medicine? A Systematic Review. Academic Medicine. 2010;85(7):1163-1170. doi:10.1097/ ACM.0b013e3181d4152f.
18. Ilic D, de Voogt A, Oldroyd J. The use of journal clubs to teach evidence-based medicine to health professionals: A systematic review and meta-analysis. J Evid Based Med. Feb 2020;13(1):42-56. doi:10.1111/jebm.12370

This is a text version generated from the article. For the formatted version of record (with original tables & figures), download the PDF →