Publisher: Wiley Blackwell Country of Publication: England NLM ID: 101543738 Publication Model: Print-Electronic Cited Medium: Internet ISSN: 1759-2887 (Electronic) Linking ISSN: 17592879 NLM ISO Abbreviation: Res Synth Methods Subsets: MEDLINE

Publication: : Chichester : Wiley Blackwell
Original Publication: Malden, MA : John Wiley & Sons, 2010-

Qualitative Research* ; Research Design* ; Review Literature as Topic*; Bayes Theorem ; Data Interpretation, Statistical ; Disabled Persons ; Educational Status ; Humans ; Regression Analysis ; Systematic Reviews as Topic

This article identifies and unpacks three key decision points in a systematic mixed research synthesis, a mixed methods approach to research synthesis. The research community has increasingly recognized the value of synthesis studies in expanding our understanding of phenomena. Mixed-methodology broadly has also gained ground as a pragmatic approach to research investigations. By leveraging mixed methods approaches in a synthesis review, systematic mixed studies review (SMSR) enables a broad and integrated summary of existing research on the topic. Yet SMSRs can be challenging to undertake given the complexity of working with research synthesis and mixed methods requirements. This methods guidance article presents information to assist novice researchers in navigating these critical decisions in SMSRs, and provides examples of how these were addressed by researchers in the two case studies provided. Implications of the SMSRs for the research community and future research directions are also discussed.
(© 2020 John Wiley & Sons Ltd.)

Chalmers I, Hedges LV, Cooper H. A brief history of research synthesis. Eval Health Prof. 2002;25(1):12-37. https://doi.org/10.1177/0163278702025001003.
Glass GV. Primary, secondary, and meta-analysis of research. Educ Res. 1976;5(10):3-8. https://www.jstor.org/stable/pdf/1174772.pdf?refreqid=excelsior%3A6397e9487648b40f34985fd6f99b94ae. Accessed December 13, 2017.
Grant MJ, Booth A. A typology of reviews: an analysis of 14 review types and associated methodologies. Heal Inf Libr J. 2009;26(2):91-108. https://doi.org/10.1111/j.1471-1842.2009.00848.x.
Henson RK. Effect-size measures and meta-analytic thinking in counseling psychology research. Couns Psychol. 2006;34(5):601-629. https://doi.org/10.1177/0011000005283558.
Makel MC, Plucker JA. Facts are more important than novelty. Educ Res. 2014;43(6):304-316. https://doi.org/10.3102/0013189X14545513.
Lorenc T, Pearson M, Jamal F, Cooper C, Garside R. The role of systematic reviews of qualitative evidence in evaluating interventions: a case study. Res Synth Methods. 2012;3(1):1-10. https://doi.org/10.1002/jrsm.1036.
Dixon-Woods M, Fitzpatrick R, Roberts K. Including qualitative research in systematic reviews: opportunities and problems. J Eval Clin Pract. 2001;7(2):125-133. https://doi.org/10.1046/j.1365-2753.2001.00257.x.
Oliver S, Harden A, Rees R, et al. An emerging framework for including different types of evidence in systematic reviews for public policy. Evaluation. 2005;11(4):428-446. https://doi.org/10.1177/1356389005059383.
Petticrew M, Rehfuess E, Noyes J, et al. Synthesizing evidence on complex interventions: how meta-analytical, qualitative, and mixed-method approaches can contribute. J Clin Epidemiol. 2013;66(11):1230-1243. https://doi.org/10.1016/j.jclinepi.2013.06.005.
Popay J, Rogers A, Williams G. Rationale and standards for the systematic review of qualitative literature in health services research. Qual Health Res. 1998;8(3):341-351. https://doi.org/10.1177/104973239800800305.
Heyvaert M, Hannes K, Onghena P. Using Mixed Methods Research Synthesis for Literature Reviews. 1st ed. Thousand Oaks, CA: Sage Publications, Ltd; 2017 doi:10.4135/9781506333243.
Heyvaert M, Maes B, Onghena P. Mixed methods research synthesis: definition, framework, and potential. Qual Quant. 2013;47(2):659-676. https://doi.org/10.1007/s11135-011-9538-6.
Harden A, Thomas J. Mixed methods and systematic reviews: examples and emerging issues. In: Tashakkori A, Teddlie C, eds. SAGE Handbook of Mixed Methods in Social & Behavioral Research. 2nd ed. Thousand Oaks, CA: SAGE Publications, Inc.; 2010:749-774. https://doi.org/10.4135/9781506335193.
Sandelowski M, Voils CI, Barroso J. Defining and designing mixed research synthesis studies. Res Sch. 2006;13(1):29 http://www.ncbi.nlm.nih.gov/pubmed/20098638. Accessed February 27, 2019.
Pluye P, Hong QN. Combining the power of stories and the power of numbers: mixed methods research and mixed studies reviews. Annu Rev Public Health. 2014;35(1):29-45. https://doi.org/10.1146/annurev-publhealth-032013-182440.
van Grootel L, van Wesel F, O'Mara-Eves A, Thomas J, Hox J, Boeije H. Using the realist perspective to link theory from qualitative evidence synthesis to quantitative studies: broadening the matrix approach. Res Synth Methods. 2017;8(3):303-311. https://doi.org/10.1002/jrsm.1241.
Frantzen KK, Fetters MD. Meta-integration for synthesizing data in a systematic mixed studies review: insights from research on autism spectrum disorder. Qual Quant. 2016;50(5):2251-2277. https://doi.org/10.1007/s11135-015-0261-6.
Brown SD, Lent RW. A social cognitive view of career development and counseling. In: Brown SD, Lent RW, eds. Career Development and Counseling: Putting Theory and Research to Work. Hoboken, NJ: John Wiley & Sons, Inc.; 2005:101-127.
Lent RW, Brown SD, Hackett G. Social cognitive career theory. In: Brown D, ed. Career Choice and Development. 4th ed. San Francisco, CA: Jossey-Bass; 2002:255-311.
Hong QN, Pluye P. A conceptual framework for critical appraisal in systematic mixed studies reviews. J Mix Methods Res. 2018:155868981877005;13:446-460. https://doi.org/10.1177/1558689818770058.
Johnson RB, Onwuegbuzie AJ. Mixed methods research: a research paradigm whose time has come. Educ Res. 2004;33(7):14-26. https://doi.org/10.3102/0013189X033007014.
Morse JM. Principles of mixed methods and multi-method research design. In: Teddlie C, Tashakkori A, eds. Handbook of Mixed Methods in Social and Behavioral Research. Thousand Oaks, CA: SAGE Publications; 2003:189-208.
Sandelowski M, Voils CI, Leeman J, Crandell JL. Mapping the mixed methods-mixed research synthesis terrain. J Mix Methods Res. 2012;6(4):317-331. https://doi.org/10.1177/1558689811427913.
Dixon-Woods M, Agarwal S, Jones D, Young B, Sutton A. Synthesising qualitative and quantitative evidence: a review of possible methods. J Heal Serv Res Policy. 2005;10(1):45-53. https://doi.org/10.1258/1355819052801804.
Khan KS, Kunz R, Kleijnen J, Antes G. Five steps to conducting a systematic review. J R Soc Med. 2003;96(3):118-121. https://doi.org/10.1177/014107680309600304.
Tricco AC, Tetzlaff J, Moher D. The art and science of knowledge synthesis. J Clin Epidemiol. 2011;64(1):11-20. https://doi.org/10.1016/J.JCLINEPI.2009.11.007.
Pai M, McCulloch M, Gorman JD, et al. Systematic reviews and meta-analyses: an illustrated, step-by-step guide. Natl Med J India. 2004;17(2):86-95. http://europepmc.org/abstract/MED/15141602.
Harden A, Thomas J. Methodological issues in combining diverse study types in systematic reviews. Int J Soc Res Methodol. 2005;8(3):257-271. https://doi.org/10.1080/13645570500155078.
Higgins JPT, Thomas J, Cumpston M, Li T, Page MJ, Welch VA. Cochrane Handbook for Systematic Reviews of Interventions. 6.0.; 2019. www.training.cochrane.org/handbook.
Cooper HM. Research Synthesis and Meta-Analysis: A Step-by-Step Approach. 4th ed. Thousand Oaks, CA: Sage Publications; 2010.
Harden A, Thomas J, Cargo M, et al. Cochrane qualitative and implementation methods group guidance series-paper 5: methods for integrating qualitative and implementation evidence within intervention effectiveness reviews. J Clin Epidemiol. 2018;97:70-78. https://doi.org/10.1016/J.JCLINEPI.2017.11.029.
Boaz A, Ashby D, Denyer D, et al. A multitude of syntheses: a comparison of five approaches from diverse policy fields. Evid Policy A J Res Debate Pract. 2006;2(4):479-502. https://doi.org/10.1332/174426406778881755.
Levitt HM, Bamberg M, Creswell JW, Frost DM, Josselson R, Suárez-Orozco C. Journal article reporting standards for qualitative primary, qualitative meta-analytic, and mixed methods research in psychology: The APA Publications and Communications Board task force report. Am Psychol. 2018;73(1):26-46. https://doi.org/10.1037/amp0000151.
Appelbaum M, Cooper H, Kline RB, Mayo-Wilson E, Nezu AM, Rao SM. Journal article reporting standards for quantitative research in psychology: the APA publications and communications board task force report. Am Psychol. 2018;73(1):3-25. https://doi.org/10.1037/amp0000191.
Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;6(7):e1000097. https://doi.org/10.1371/journal.pmed.1000097.
Sanford C, Newman L, Wagner M, Cameto R, Knokey A-M, Shaver D. The post-high school outcomes of young adults with disabilities up to 6 years after high school. Key Findings from the National Longitudinal Transition Study-2 (NLTS2) (NCSER 2011-3004). Menlo Park, CA; 2011.
Noblit GW, Hare RD. Meta-ethnography: synthesizing qualitative studies. Counterpoints. 1999;44:93-123. https://www.jstor.org/stable/pdf/42975557.pdf?refreqid=excelsior%3Ac8b057e688b82754ef71cc18e6599f8e. Accessed February 15, 2018.
Dixon-Woods M, Bonas S, Booth A, et al. How can systematic reviews incorporate qualitative research? A critical perspective. Qual Res. 2006;6(1):27-44. https://doi.org/10.1177/1468794106058867.
Gough D, Thomas J, Oliver S. Clarifying differences between review designs and methods. Syst Rev. 2012;1(1):28. https://doi.org/10.1186/2046-4053-1-28.
Gough D, Thomas J, Oliver S. Clarifying differences between reviews within evidence ecosystems. Syst Rev. 2019;8(1):170. https://doi.org/10.1186/s13643-019-1089-2.
Hong QN, Pluye P, Bujold M, Wassef M. Convergent and sequential synthesis designs: implications for conducting and reporting systematic reviews of qualitative and quantitative evidence. Syst Rev. 2017;6(1):61. https://doi.org/10.1186/s13643-017-0454-2.
Munn Z, Peters MDJ, Stern C, Tufanaru C, McArthur A, Aromataris E. Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC Med Res Methodol. 2018;18(1):143. https://doi.org/10.1186/s12874-018-0611-x.
Levac D, Colquhoun H, O'Brien KK. Scoping studies: advancing the methodology. Implement Sci. 2010;5(1):69. https://doi.org/10.1186/1748-5908-5-69.
Davis K, Drey N, Gould D. What are scoping studies? A review of the nursing literature. Int J Nurs Stud. 2009;46(10):1386-1400. https://doi.org/10.1016/j.ijnurstu.2009.02.010.
Peterson J, Patricia Pearce CF. Understanding scoping reviews: definition, purpose, and process. doi:https://doi.org/10.1002/2327-6924.12380.
Pham MT, Rajić A, Greig JD, Sargeant JM, Papadopoulos A, McEwen SA. A scoping review of scoping reviews: advancing the approach and enhancing the consistency. Res Synth Methods. 2014;5(4):371-385. https://doi.org/10.1002/jrsm.1123.
Schultz A, Goertzen L, Rothney J, et al. A scoping approach to systematically review published reviews: adaptations and recommendations. Res Synth Methods. 2018;9(1):116-123. https://doi.org/10.1002/jrsm.1272.
Pluye P, Grad RM, Dunikowski LG, Stephenson R. Impact of clinical information-retrieval technology on physicians: a literature review of quantitative, qualitative and mixed methods studies. Int J Med Inform. 2005;74(9):745-768. https://doi.org/10.1016/j.ijmedinf.2005.05.004.
Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O, Peacock R. Storylines of research in diffusion of innovation: a meta-narrative approach to systematic review. Soc Sci Med. 2005;61(2):417-430. https://doi.org/10.1016/j.socscimed.2004.12.001.
Teddlie C, Tashakkori A. Foundations of Mixed Methods Research: Integrating Quantitative and Qualitative Approaches in the Social and Behavioral Sciences. Thousand Oaks, CA: Sage Publications; 2009.
Harden A, Brunton G, Fletcher A, Oakley A. Teenage pregnancy and social disadvantage: systematic review integrating controlled trials and qualitative studies. BMJ. 2009;339:b4254. https://doi.org/10.1136/BMJ.B4254.
Thomas J, Harden A, Oakley A, et al. Integrating qualitative research with trials in systematic reviews. BMJ. 2004;328(7446):1010-1012. https://doi.org/10.1136/bmj.328.7446.1010.
Eppi. EPPI-Centre methods form conducting systematic review. Evid Policy Pract Inf Co-ord Cent. 2007;2007:27 https://eppi.ioe.ac.uk/cms/LinkClick.aspx?fileticket=hQBu8y4uVwI=&tabid=88.
EPPI-Centre. https://eppi.ioe.ac.uk/cms/Default.aspx?tabid=53.
Aromataris EMZ. Joanna Briggs Institute Reviewer's Manual. 2017. https://reviewersmanual.joannabriggs.org/.
Sandelowski M, Voils CI, Knafl G. On quantitizing. J Mix Methods Res. 2009;3(3):208-222.
Pace R, Pluye P, Bartlett G, et al. Testing the reliability and efficiency of the pilot mixed methods appraisal tool (MMAT) for systematic mixed studies review. Int J Nurs Stud. 2012;49(1):47-53. https://doi.org/10.1016/J.IJNURSTU.2011.07.002.
Onwuegbuzie AJ, Frels R. 7 Steps to a Comprehensive Literature Review: A Multimodal And Cultural Approach. Thousand Oaks, CA: Sage; 2016.
Johnson B, Christensen LB. Educational Research: Quantitative, Qualitative, and Mixed Approaches. Thousand Oaks, CA: Sage Publications; 2019.
Pluye P, Hong QN, Bush PL, Vedel I. Opening-up the definition of systematic literature review: the plurality of worldviews, methodologies and methods for reviews and syntheses. J Clin Epidemiol. 2016;73:2-5. https://doi.org/10.1016/j.jclinepi.2015.08.033.
Wilson DB, Gottfredson DC, Najaka SS. School-based prevention of problem behaviors: a meta-analysis. J Quant Criminol. 2001;17(3):247-272. https://doi.org/10.1023/A:1011050217296.
van Houwelingen HC, Arends LR, Stijnen T. Advanced methods in meta-analysis: multivariate approach and meta-regression. Stat Med. 2002;21(4):589-624. http://www.ncbi.nlm.nih.gov/pubmed/11836738. Accessed May 27, 2019.
Roberts KA, Dixon-Woods M, Fitzpatrick R, Abrams KR, Jones DR. Factors affecting uptake of childhood immunisation: a Bayesian synthesis of qualitative and quantitative evidence. Lancet. 2002;360(9345):1596-1599. https://doi.org/10.1016/S0140-6736(02)11560-1.
Voils C, Hassselblad V, Crandell J, Chang Y, Lee E, Sandelowski M. A Bayesian method for the synthesis of evidence from qualitative and quantitative reports: the example of antiretroviral medication adherence. J Health Serv Res Policy. 2009;14(4):226-233. https://doi.org/10.1258/jhsrp.2009.008186.
Buelens M, Van De Woestyne M, Mestdagh S, Bouckenooghe D. Methodological issues in negotiation research: a state-of-the-art-review. Gr Decis Negot. 2008;17(4):321-345. https://doi.org/10.1007/s10726-007-9097-3.
Morton KL, Atkin AJ, Corder K, Suhrcke M, van Sluijs EMF. The school environment and adolescent physical activity and sedentary behaviour: a mixed-studies systematic review. Obes Rev. 2016;17(2):142-158. https://doi.org/10.1111/obr.12352.
van Grootel L. Where no Reviewer Has Gone before: exploring the Potential of Mixed Studies Reviewing [dissertation]. Utrecht, Netherlands: Utrecht University; 2018.

Keywords: methods guidance; mixed research synthesis; mixed-methods research; systematic mixed studies review

Date Created: 20200725 Date Completed: 20210802 Latest Revision: 20210802

20221216

10.1002/jrsm.1436

32706175