"It has no meaning to me." How do researchers understand the effectiveness of literature searches? A qualitative analysis and preliminary typology of understandings.

Item request has been placed! ×
Item request cannot be made. ×
loading   Processing Request
  • Additional Information
    • Source:
      Publisher: Wiley Blackwell Country of Publication: England NLM ID: 101543738 Publication Model: Print-Electronic Cited Medium: Internet ISSN: 1759-2887 (Electronic) Linking ISSN: 17592879 NLM ISO Abbreviation: Res Synth Methods Subsets: MEDLINE
    • Publication Information:
      Publication: : Chichester : Wiley Blackwell
      Original Publication: Malden, MA : John Wiley & Sons, 2010-
    • Subject Terms:
    • Abstract:
      This study aimed to address the question: what does "effectiveness" mean to researchers in the context of literature searching for systematic reviews? We conducted a thematic analysis of responses to an e-mail survey. Eighty-nine study authors, whose studies met inclusion in a recent review (2018), were contacted via e-mail and asked three questions; one directly asking the question: in literature searching, what does effective (or effectiveness in) literature searching mean to you? Thirty-eight (46%) responses were received from diverse professional groups, including: literature searchers, systematic reviewers, clinicians and researchers. A shared understanding of what effectiveness means was not identified. Instead, five themes were developed from data: (a) effectiveness is described as a metric; (b) effectiveness is a balance between metrics; (c) effectiveness can be categorized by search purpose; (d) effectiveness is an outcome; and, (e) effectiveness is an experimental concept. We propose that these themes constitute a preliminary typology of understandings. No single definition of effectiveness was identified. The proposed typology suggests that different researchers have differing understandings of effectiveness. This could lead to uncertainty as to the aim and the purpose of literature searches and confusion about the outcomes. The typology offers a potential route for further exploration.
      (© 2020 John Wiley & Sons, Ltd.)
    • References:
      Cooper C, Booth A, Varley-Campbell J, Britten N, Garside R. Defining the process to literature searching in systematic reviews: a literature review of guidance and supporting studies. BMC Med Res Methodol. 2018;18(1):85.
      Cooper C, Dawson S, Peters J, et al. Revisiting the need for a literature search narrative: a brief methodological note. Res Synth Meth. 2018;9(3):361-365.
      Cooper C, Varley-Campbell J, Booth A, Britten N, Garside R. Systematic review identifies six metrics and one method for assessing literature search effectiveness but no consensus on appropriate use. J Clin Epidemiol. 2018;99:53-63.
      Rader T, Mann M, Stansfield C, Cooper C, Sampson M. Methods for documenting systematic review searches: a discussion of common issues. Res Synth Meth. 2014;5(2):98-115.
      Lefebvre C, Manheimer E, Glanville J. Chapter 6: Searching for studies. In: The Cochrane Collaboration, ed. Cochrane Handbook for Systematic Reviews of Interventions [Internet]. Oxford: Cochrane Collaboration; 2011. http://handbook.cochrane.org/ Accessed December 7, 2017.
      Rethlefsen ML, Farrell AM, Osterhaus Trzasko LC, Brigham TJ. Librarian co-authors correlated with higher quality reported search strategies in general internal medicine systematic reviews. J Clin Epidemiol. 2015;68(6):617-626.
      McGowan J, Sampson M. Systematic reviews need systematic searchers. J Med Libr Assoc. 2005;93(1):74-80.
      Lefebvre C, Glanville J, Beale S, et al. Assessing the performance of methodological search filters to improve the efficiency of evidence information retrieval: five literature reviews and a qualitative study. Health Technol Assess. 2017;21(69):1-148.
      Haynes RB, Wilczynski N, McKibbon KA, Walker CJ, Sinclair JC. Developing optimal search strategies for detecting clinically sound studies in MEDLINE. J Am Med Inform Assoc. 1994;1(6):447-458.
      McDonald SJ, Lefebvre C, Clarke MJ. Identifying reports of controlled trials in the BMJ and the lancet. BMJ. 1996;313(7065):1116-1117.
      Robinson KA, Dickersin K. Development of a highly sensitive search strategy for the retrieval of reports of controlled trials using PubMed. Int J Epidemiol. 2002;31(1):150-153.
      Watson RJ, Richardson PH. Identifying randomized controlled trials of cognitive therapy for depression: comparing the efficiency of Embase, Medline and PsycINFO bibliographic databases. Br J Med Psychol. 1999;72(Pt 4):535-542.
      Jenkins M. Evaluation of methodological search filters-a review. Health Inf Libr J. 2004;21(3):148-163.
      Anderson LM, Oliver SR, Michie S, Rehfuess E, Noyes J, Shemilt I. Investigating complexity in systematic reviews of interventions by using a spectrum of methods. J Clin Epidemiol. 2013;66(11):1223-1229.
      Gough D, Thomas J, Oliver S. Clarifying differences between review designs and methods. Syst Rev. 2012;1(1):28.
      Munn Z, Stern C, Aromataris E, Lockwood C, Jordan Z. What kind of systematic review should I conduct? A proposed typology and guidance for systematic reviewers in the medical and health sciences. BMC Med Res Methodol. 2018;18(1):5.
      Noyes J, Gough D, Lewin S, et al. A research and development agenda for systematic reviews that ask complex questions about complex interventions. J Clin Epidemiol. 2013;66(11):1262-1270.
      Petticrew M. Time to rethink the systematic review catechism? Moving from ‘what works’ to ‘what happens’. Syst Rev. 2015;4(1):36.
      Kaltenthaler E, Cooper K, Pandor A, Martyn-St James M, Chatters R, Wong R. The use of rapid review methods in health technology assessments: 3 case studies. BMC Med Res Methodol. 2016;16(1):108.
      Khangura S, Konnyu K, Cushman R, Grimshaw J, Moher D. Evidence summaries: the evolution of a rapid review approach. Syst Rev. 2012;1(1):10.
      Nussbaumer-Streit B, Klerings I, Wagner G, Titscher V, Gartlehner G. Assessing the validity of abbreviated literature searches for rapid reviews: protocol of a non-inferiority and meta-epidemiologic study. Syst Rev. 2016;5:197.
      Tricco AC, Antony J, Zarin W, et al. A scoping review of rapid review methods. BMC Med. 2015;13:224.
      Watt A, Cameron A, Sturm L, et al. Rapid reviews versus full systematic reviews: an inventory of current methods and practice in health technology assessment. Int J Technol Assess Health Care. 2008;24(2):133-139.
      Booth A. How much searching is enough? Comprehensive versus optimal retrieval for technology assessments. Int J Technol Assess Health Care. 2010;26(4):431-435.
      Booth A. The number needed to retrieve: a practically useful measure of information retrieval? Health Inf Libr J. 2006;23(3):229-232.
      Booth A. Searching for qualitative research for inclusion in systematic reviews: a structured methodological review. Syst Rev. 2016;5:74.
      Egger M, Juni P, Bartlett C, Holenstein F, Sterne J. How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study. Health Technol Assess. 2003;7(1):1-76.
      Cooper C, Lovell R, Husk K, Booth A, Garside R. Supplementary search methods were more effective and offered better value than bibliographic database searching: a case study from public health and environmental enhancement. Res Synth Meth. 2018;9(2):195-223.
      Cooper C, Booth A, Britten N, Garside R. A comparison of results of empirical studies of supplementary search techniques and recommendations in review methodology handbooks: a methodological review. Syst Rev. 2017;6(1):234.
      Cooper C, Bou JT, Varley-Campbell J. Evaluating the effectiveness, efficiency, cost and value of contacting study authors in a systematic review: a case study and worked example. BMC Med Res Methodol. 2019;19(1):45.
      Cooper C, Varley-Campbell J, Carter P. Established search filters may miss studies when identifying randomised controlled trials. J Clin Epidemiol. 2019;112:12-19.
      Shemilt I, Simon A, Hollands GJ, et al. Pinpointing needles in giant haystacks: use of text mining to reduce impractical screening workload in extremely large scoping reviews. Res Synth Meth. 2014;5(1):31-49.
      Cooper C, Snowsill T, Worsley C, et al. Handsearching had best recall but poor efficiency when exporting to a bibliographic tool: case study. J Clin Epidemiol. 2020;123:39-48.
      O'Leary F. Is email a reliable means of contacting authors of previously published papers? A study of the emergency medicine journal for 2001. Emerg Med J. 2003;20(4):352-353.
      Gibson CA, Bailey BW, Carper MJ, et al. Author contacts for retrieval of data for a meta-analysis on exercise and diet restriction. Int J Technol Assess Health Care. 2006;22(2):267-270.
      Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006;3(2):77-101.
      Smith KB. Typologies, taxonomies, and the benefits of policy classification. Policy Stud J. 2002;30(3):379-395.
      Bailey K. Typologies and Taxonomies (Paper): An Introductionto Classification Techniques: an Introduction to Classification Techniques (Quantitative Applications in the Social Sciences). New York, NY: Sage; 1994.
      Brettle AJ, Long AF, Grant MJ, Greenhalgh J. Searching for information on outcomes: do you need to be comprehensive? Qual Health Care. 1998;7(3):163-167.
      Harbour J, Fraser C, Lefebvre C, et al. Reporting methodological search filter performance comparisons: a literature review. Health Inf Libr J. 2014;31(3):176-194.
      McGowan J, Sampson M, Salzwedel DM, Cogo E, Foerster V, Lefebvre C. PRESS peer review of electronic search strategies: 2015 guideline Statement. J Clin Epidemiol. 2016;75:40-46.
      Stoll CRT, Izadi S, Fowler S, Green P, Suls J, Colditz GA. The value of a second reviewer for study selection in systematic reviews. Res Synth Meth. 2019;10:539-545.
      Polanin JR, Pigott TD, Espelage DL, Grotpeter JK. Best practice guidelines for abstract screening large-evidence systematic reviews and meta-analyses. Res Synth Meth. 2019;10:330-342.
      Centre for Reviews and Dissemination. Systematic reviews - CRD's guidance for undertaking reviews in healthcare. York: Centre for Reviews and Dissemination, University of York; 2009. https://www.york.ac.uk/media/crd/Systematic_Reviews.pdf.
      Beverley CA, Booth A, Bath PA. The role of the information specialist in the systematic review process: a health information case study. Health Inf Libr J. 2003;20(2):65-74.
      Harris MR. The librarian's roles in the systematic review process: a case study. J Med Libr Assoc. 2005;93(1):81-87.
      Koffel JB. Use of recommended search strategies in systematic reviews and the impact of librarian involvement: a cross-sectional survey of recent authors. PLoS One. 2015;10(5):e0125931.
      Pham MT, Waddell L, Rajić A, Sargeant JM, Papadopoulos A, McEwen SA. Implications of applying methodological shortcuts to expedite systematic reviews: three case studies using systematic reviews from agri-food public health. Res Synth Meth. 2016;7(4):433-446.
    • Publication Date:
      Date Created: 20200605 Date Completed: 20210802 Latest Revision: 20210802
    • Publication Date:
      20220902
    • Accession Number:
      10.1002/jrsm.1426
    • Accession Number:
      32495989