Abstract
Background
Financial conflicts of interest in systematic reviews (e.g. funding by drug or device companies or authors’ collaboration with such companies) may impact on how the reviews are conducted and reported.
Objectives
To investigate the degree to which financial conflicts of interest related to drug and device companies are associated with results, conclusions, and methodological quality of systematic reviews.
Search methods
We searched PubMed, Embase, and the Cochrane Methodology Register for studies published up to November 2016. We also read reference lists of included studies, searched grey literature sources, and Web of Science for studies citing the included studies.
Selection criteria
Eligible studies were studies that compared systematic reviews with and without financial conflicts of interest in order to investigate differences in results (estimated treatment effect and frequency of statistically favourable results), frequency of favourable conclusions, or measures of methodological quality of the review (e.g. as evaluated on the Oxman and Guyatt index).
Data collection and analysis
Two review authors independently determined the eligibility of studies, extracted data, and assessed risk of bias. We synthesised the results of each study relevant to each of our outcomes. For meta‐analyses, we used Mantel‐Haenszel random‐effects models to estimate risk ratios (RR) with 95% confidence intervals (CIs), with RR > 1 indicating that systematic reviews with financial conflicts of interest more frequently had statistically favourable results or favourable conclusions, and had lower methodological quality. When a quantitative synthesis was considered not meaningful, results from individual studies were summarised qualitatively.
Main results
Ten studies with a total of 995 systematic reviews of drug studies and 15 systematic reviews of device studies were included. We assessed two studies as low risk of bias and eight as high risk, primarily because of risk of confounding. The estimated treatment effect was not statistically significantly different for systematic reviews with and without financial conflicts of interest (Z‐score: 0.46, P value: 0.64; based on one study of 14 systematic reviews which had a matched design, comparing otherwise similar systematic reviews). We found no statistically significant difference in frequency of statistically favourable results for systematic reviews with and without financial conflicts of interest (RR: 0.84, 95% CI: 0.62 to 1.14; based on one study of 124 systematic reviews). An analysis adjusting for confounding due to methodological quality (i.e. score on the Oxman and Guyatt index) provided a similar result. Systematic reviews with financial conflicts of interest more often had favourable conclusions compared with systematic reviews without (RR: 1.98, 95% CI: 1.26 to 3.11; based on seven studies of 411 systematic reviews). Similar results were found in two studies with a matched design, which therefore had a reduced risk of confounding. Systematic reviews with financial conflicts of interest tended to have lower methodological quality compared with systematic reviews without financial conflicts of interest (RR for 11 dimensions of methodological quality spanned from 1.00 to 1.83). Similar results were found in analyses based on two studies with matched designs.
Authors’ conclusions
Systematic reviews with financial conflicts of interest more often have favourable conclusions and tend to have lower methodological quality than systematic reviews without financial conflicts of interest. However, it is uncertain whether financial conflicts of interest are associated with the results of systematic reviews. We suggest that patients, clinicians, developers of clinical guidelines, and planners of further research could primarily use systematic reviews without financial conflicts of interest. If only systematic reviews with financial conflicts of interest are available, we suggest that users read the review conclusions with skepticism, critically appraise the methods applied, and interpret the review results with caution.
Plain language summary
available in
Financial conflicts of interests and results, conclusions, and quality of systematic reviews
Patient treatment practices are often based on clinical research. Systematic reviews are a core type of such clinical research. When several similar studies (i.e. studies investigating the same research questions using similar methods) have been conducted, these can be identified and analysed in a systematic review. Systematic reviews thereby summarise existing studies and provide an overview of a specific research field. Thus, systematic reviews may have a major influence on decisions about patient care and it is essential that such reviews are trustworthy.
Sometimes, systematic reviews are funded by companies with a financial interest in the review‘s results and conclusions, for example because they produce a drug or device investigated in the review. At other times, systematic reviews are carried out by researchers with a personal financial interest in a specific result, for example when the researcher acts as a consultant for the company producing an intervention that is assessed in the review. These financial conflicts of interest may impact on how systematic reviews are conducted and reported. Our Cochrane Methodology Review focuses on financial conflicts of interest related to drug or device companies in systematic reviews. Our primary aim was to investigate the degree to which systematic reviews with financial conflicts of interest present review results and make conclusions that are more favourable than systematic reviews without such financial conflicts of interest. Our secondary aim was to investigate the degree to which systematic reviews with financial conflicts of interest differ in methodological quality from systematic reviews without such financial conflicts of interest.
We found 10 studies comparing systematic reviews with and without financial conflicts of interest. Based on two of these studies, we found no evidence of a difference in review results between systematic reviews with and without financial conflicts of interest. Based on seven studies, we found that systematic reviews with financial conflicts of interest more often had conclusions favourable towards the experimental intervention (risk ratio (RR): 1.98, 95% confidence interval (CI): 1.26 to 3.11). Also, based on four studies, systematic reviews with financial conflicts of interest tended to have lower methodological quality (RR for 11 dimensions of methodological quality spanned from 1.00 to 1.83).
Our analyses suggest that when systematic reviews have financial conflicts of interest related to drug or device companies, they are of lower methodological quality, and have more favourable conclusions. However, it is not clear whether this derives from actual differences in the review‘s results or the over‐interpretation of those results. Based on our findings, we suggest that people who use systematic reviews, including patients, clinicians, developers of clinical guidelines, and planners of future research, could primarily use systematic reviews without financial conflicts of interest. If such reviews are not available, we suggest that users are especially cautious when they read and interpret systematic reviews with financial conflicts of interest.
Authors’ conclusions
available in
Summary of findings
| Systematic reviews with financial conflicts of interest compared with systematic reviews without | ||||||
| Sample: systematic reviews
Intervention: systematic reviews with financial conflicts of interest Comparison: systematic reviews without financial conflicts of interest |
||||||
| Outcomes | Absolute effect* (95% CI) | Relative effect | Number of studies | Certainty of the evidence (GRADE) |
Comments** | |
| Assumed risk | Corresponding risk | |||||
| Number of industry reviews with the outcome | Number of non‐industry reviews with the outcome | |||||
| Estimated treatment effect
measured as Z‐scores*** after adjustment for the number of patients |
Pooled Z‐score: 0.46 (P value: 0.64) | 1 study including 7 pairs of industry and non‐industry systematic reviews | ⊕⊝⊝⊝ very low |
Downgraded due to imprecision (only one study of 14 systematic reviews) | ||
| Frequency of statistically favourable results | 549 (405 to 745) reviews with statistically favourable results per 1000 industry reviews | 653 reviews with statistically favourable results per 1000 non‐industry reviews | RR: 0.84
(0.62 to 1.14) |
1 study including 124 systematic reviews | ⊕⊝⊝⊝ very low |
Downgraded due to limitations in design (only one study with high risk of bias) and imprecision (wide confidence intervals) |
| Frequency of favourable conclusions | 895 (569 to 1000****) reviews with favourable conclusions per 1000 industry reviews | 452 reviews with favourable conclusions per 1000 non‐industry reviews | RR: 1.98
(1.26 to 3.11) |
7 studies including 411 systematic reviews | ⊕⊕⊝⊝ low |
Fairly large effect estimate, which was substantially higher in the one study with low risk of bias. |
| GRADE Working Group grades of evidence High quality: Further research is very unlikely to change our confidence in the estimate of effect. Moderate quality: Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate. Low quality: Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate. Very low quality: We are very uncertain about the estimate. |
||||||
|
CI: confidence interval; RR: risk ratio *The assumed risk of the control group (i.e. non‐industry systematic review group) was calculated as the mean risk (i.e. number of systematic reviews with favourable conclusions divided by total number of systematic reviews). The corresponding risk (and its 95% confidence interval) is based on the assumed risk in the control group and the relative effect of the intervention (and its 95% CI). **See ‘Assessment of certainty of the evidence’ for a detailed description. ***A Z‐score expresses the number of standard deviations by which a value differs from the mean. If a Z‐score is interpreted as a standardised mean difference, a Z‐score of 0.46 would indicate a moderate effect (Kirkwood 2003). A Z‐score of 0.46 indicates that effect estimates are larger in industry reviews compared with non‐industry reviews (Jorgensen 2006). ****Upper limit of event rate truncated at 1000. |
||||||
Background
available in
Description of the problem or issue
Systematic reviews provide a rational basis for developing clinical guidelines, for therapeutic decision making, and for planning clinical trials. They have a major impact on which interventions are offered to patients (Guyatt 2008). It is therefore essential that such reviews are trustworthy and unbiased.
One area of concern is the degree to which financial conflicts of interest impact on the conduct and reporting of systematic reviews (Institute of Medicine 2011). The pharmaceutical and clinical device industries frequently fund clinical trials (Atal 2015; Chan 2005), and to a lesser extent also systematic reviews. For example, in a random sample of 300 systematic reviews, Page and colleagues found that eight (3%) were industry‐funded (Page 2016). On the other hand, another review found that a quarter of 185 meta‐analyses of trials of antidepressants were industry‐funded (Ebrahim 2016). Furthermore, systematic reviews are often produced by authors with financial conflicts of interest; a random sample of 194 systematic reviews found that 60 (31%) had at least one author with financial conflicts of interest (Hakoum 2016).
Numerous studies have investigated the relation between financial conflicts of interest and outcomes of individual research studies, mainly clinical trials. A recent update of a Cochrane Review reported clear associations between funding source and statistically significant trial efficacy results (based on 25 empirical studies) and trial conclusions (based on 29 empirical studies) (Lundh 2017). In contrast, fewer studies have investigated how financial conflicts of interest at the level of the systematic review impact on their results and conclusions.
Why it is important to do this review
The retrospective nature of a systematic review and the subjective element in selecting inclusion criteria and outcomes is likely to make such research more susceptible to influence from financial conflicts of interest than prospective clinical trials.
This concern is supported by a review of pairs of Cochrane Reviews and paper‐based reviews of the same drugs used for the same disease that reported that industry‐funded reviews had more favourable conclusions (Jorgensen 2006). However, other studies have reported a less clear association with wide confidence intervals (Yank 2007). To our knowledge, this Cochrane Methodology Review is the first systematic review of methodology to identify, analyse, and summarise such studies.
Objectives
available in
Our primary objectives were to investigate the degree to which financial conflicts of interest related to drug or device companies in systematic reviews are associated with the following.
- Results
- Estimated treatment effect
- Results statistically favourable to the experimental intervention
- Conclusions favourable to the experimental intervention
Our secondary objective was to investigate the degree to which financial conflicts of interest related to drug or device companies in systematic reviews are associated with the following.
- Methodological quality of the reviews
Terminology
We use the following definitions.
- Financial conflicts of interest: any funding of the systematic reviews by drug or device companies or any review author with financial conflicts of interest in relation to such companies.
- Industry funding: any funding of the systematic review by industry, authorship by full‐time industry employees, assistance by industry (e.g. statistical analysis by company statistician, or writing assistance by a medical writer funded by the company).
- Author financial conflicts of interest: any financial relationship of authors, apart from full‐time employment, with a drug or device company (e.g. receiving grants, owning stocks, being on an advisory board, or consultancy work).
- Industry reviews: reviews that are consistent with one or more of the above definitions.
- Non‐industry reviews: reviews that fulfil none of the above definitions.
- Drugs: medications that require approval from regulatory agencies.
- Devices: instruments used in diagnosis, treatment, or prevention of a disease. This definition follows the definition of the Food and Drug Administration (FDA) (FDA 2017) and includes imaging technologies.
Methods
available in
Criteria for considering studies for this review
Types of studies
We included studies that investigated samples of systematic reviews with and without financial conflicts of interest. We defined systematic reviews according to the definitions used by the authors of the included studies.
Eligible studies had to investigate at least one of our primary or secondary outcomes. If a study contained a mixture of systematic reviews and research of other designs (e.g. randomised trials), we included the study, but only included separate data for the systematic reviews. If this distinction was not reported in a study with a variety of research designs, we requested the data for the systematic reviews from the authors unless it contained fewer than five reviews and, therefore, was too small to be informative.
We excluded studies that investigated financial conflicts of interest related to non‐pharmaceutical or non‐device industries (e.g. tobacco and food industries). Studies of mixed domains (e.g. pharmaceuticals and nutritional supplements) were included in the review and in our analyses if separate data for the systematic reviews with drug or device industry financial conflicts of interest were obtainable. If this distinction was not reported in the study, we requested the data from the authors unless the number of systematic reviews was too small to be informative (i.e. less than five reviews).
Studies were eligible regardless of the language in which they had been reported.
Types of data
We included data on estimated treatment effect (e.g. Z‐scores and P values), frequency of statistically favourable results, and frequency of favourable conclusions (e.g. number of events and odds ratios). For methodological quality, we included both continuous and binary data for industry and non‐industry reviews (e.g. overall methodological quality score and number of events in each item of a tool such as the Oxman and Guyatt index) (Oxman 1991).
Types of methods
We included studies that investigated financial conflicts of interest related to drug and device companies. We included studies regardless of type of investigated financial conflicts of interest.
Types of outcome measures
Primary outcomes
We included the following primary outcomes.
- Results
- Estimated treatment effect (e.g. relative risks)
- Frequency of statistically favourable results (e.g. occurrence of results statistically in favour of the experimental intervention)
- Frequency of favourable conclusions (e.g. recommendation of the experimental intervention without reservations)
Secondary outcomes
We included one secondary outcome.
- Methodological quality of the systematic reviews. This included, for example, assessment using the Oxman and Guyatt index (Oxman 1991) (i.e. how many industry and non‐industry reviews fulfilled each item that was assessed for quality)
Search methods for identification of studies
Electronic searches
We searched PubMed, Embase, and the Cochrane Methodology Register (searches performed 30 November 2016) for studies. We searched Web of Science (search performed 17 January 2017) for studies citing any of the included studies. We used the strategy shown in Appendix 1 for PubMed and adapted it for the other databases. These searches were done in advance of our protocol (Hansen 2017) being published in the Cochrane Library.
Searching other resources
Grey literature
We searched proceedings from Peer Review Congresses (American Medical Association 2017) and Cochrane Colloquia (Cochrane Community 2017) for conference abstracts published up to November 2016, bearing in mind the evidence on the high proportion of research studies that are presented at conferences but not published in full (Chapman 2012; Scherer 2018). We also searched PROSPERO (search performed 01 March 2017) for registered systematic reviews and the ProQuest database (search performed 01 March 2017) for dissertations and theses. Finally, we searched Google Scholar (search performed 16 March 2017) for unpublished studies. For all searches, we adapted the search strategy shown in Appendix 1.
Reference lists
We checked the reference lists of the included studies for additional potentially eligible studies (Horsley 2011).
Data collection and analysis
Selection of studies
One review author (CH) screened titles and abstracts of all retrieved records for obvious exclusions. Two review authors (CH and KR) independently assessed potentially eligible studies based on full text. Disagreements were resolved by discussion, and arbitration by another review author was not needed.
Data extraction and management
Two review authors (CH and AL) independently extracted data from the included studies. Any difference in data extraction was resolved by discussion or with arbitration by another review author (AH). We extracted data on basic characteristics and financial conflicts of interest of the included studies. For continuous outcome data, we extracted information on difference in estimated treatment effect between industry and non‐industry reviews (reported as pooled Z‐scores and P values). For binary outcome data, we extracted the number of industry and non‐industry reviews with statistically favourable results and favourable conclusions. When reported, we also extracted risk ratios (RR) or odds ratios (OR). We extracted data for industry and non‐industry reviews based on the definitions used by the authors of the included studies, but also for reviews with industry funding only and with author financial conflicts of interest only, based on our definitions. We ensured that all numbers and effect sizes had the same directionality, e.g. a RR >1 indicated that industry reviews more often had favourable conclusions than non‐industry reviews. The full plan for data extraction is shown in Appendix 2.
Assessment of risk of bias in included studies
There are no validated criteria for assessing risk of bias in these types of studies, so we developed a set of criteria similar to criteria developed for a previous Cochrane Review by one of the authors of this review (Lundh 2017), which were influenced by items from the AMSTAR tool (Shea 2007). Two review authors (CH and AL) independently assessed risk of bias. Any disagreements were resolved by discussion, with arbitration provided by a third review author (AH) when needed. We categorised each component as high risk of bias, low risk of bias, or unclear. We used the following criteria.
- Whether there was a risk of bias in the study inclusion process (low risk of bias could, for example, include two or more assessors independently selecting studies)
- Whether there was a risk of bias in the coding of financial conflicts of interest and outcomes (low risk of bias could, for example, include an extraction done independently by two or more assessors)
- Whether there was a risk of bias in the comparability of systematic reviews (low risk of bias could, for example, imply industry and non‐industry reviews of the same intervention used for the same disease)
Our aim was primarily to differentiate between studies with higher and lower risk of bias. We coded a study as low risk of bias if all three criteria were assessed as low risk of bias. Otherwise, we coded it as high risk of bias.
Dealing with missing data
We contacted authors of the included studies in an attempt to obtain unpublished data, to clarify issues on our ‘Risk of bias’ assessments, or to receive unpublished protocols (Appendix 3) (Young 2011).
Assessment of heterogeneity
We assessed statistical heterogeneity using the I2 statistic. We defined substantial heterogeneity as I2 > 50%.
Data synthesis
Due to heterogeneity between the included studies, we used Mantel‐Haenszel random‐effects models to estimate RR with 95% confidence intervals (CIs) (RR > 1 indicated that industry reviews more often had statistically favourable results or favourable conclusions). We used a qualitative synthesis for our analysis of estimated treatment effect, and a quantitative synthesis for our analyses of frequency of statistically favourable results, frequency of favourable conclusions, and methodological quality. We also calculated prediction intervals (IntHout 2016; Riley 2011) (Appendix 4).
For our primary analyses, we complied with the definitions of financial conflicts of interest used by the authors of the included studies and analysed industry funding and author financial conflicts of interest together.
For our analysis of methodological quality, we pooled similar items across the different quality assessment tools used in the included studies. For example, we pooled items on appropriateness of search methods in the systematic reviews (Appendix 5). We did not pool items related to reporting quality (e.g. whether the inclusion criteria were reported) or statistical methods (e.g. whether a Bayesian framework was used). We used Mantel‐Haenszel random‐effects models to estimate RR with 95% CI (RR > 1 indicated that industry reviews had lower methodological quality, i.e. more often did not fulfil the item or did not provide information on the item).
Subgroup analysis and investigation of heterogeneity
We planned to conduct the following pre‐specified subgroup analyses for our primary outcomes.
- High risk of bias studies versus low risk of bias studies
- Cochrane Reviews versus non‐Cochrane systematic reviews
- Systematic reviews of drug studies versus systematic reviews of device studies
- Systematic reviews with major financial conflicts of interest versus systematic reviews with moderate financial conflicts of interest versus systematic reviews with minor financial conflicts of interest (where “major”, “moderate” and “minor” were as defined by the authors of the included studies)
We planned to conduct the following post‐hoc subgroup analyses for our primary outcomes (see Differences between protocol and review).
- Studies defining favourable conclusions as conclusions in favour of the intervention versus studies defining favourable conclusions as conclusions recommending the intervention without reservations
Sensitivity analysis
We planned to conduct the following pre‐specified sensitivity analyses for our primary outcomes.
- Re‐categorising industry reviews into systematic reviews with industry funding only (i.e. excluding systematic reviews with author financial conflicts of interest from the industry group) and comparing with non‐industry reviews (i.e. systematic reviews without industry funding or author financial conflicts of interest)
- Re‐categorising industry reviews into systematic reviews with author financial conflicts of interest only (i.e. excluding systematic reviews with industry funding from the industry group) and comparing with non‐industry reviews (i.e. systematic reviews without author financial conflicts of interest or industry funding)
- Excluding included studies with conflicts of interest
- Re‐analysing our data using fixed‐effect models
We planned to conduct the following post‐hoc sensitivity analyses for our primary outcomes (see Differences between protocol and review).
- Excluding systematic reviews with unclear or undeclared financial conflicts of interest from the non‐industry group
- Excluding one atypical study (Yank 2007) from our pooled analyses because it compared industry reviews (financial conflicts of interest with a single drug company) with a group of both industry and non‐industry reviews (multiple drug companies, no statement, both drug and non‐profit companies, and non‐profit companies)
- Restricting our analyses to studies assessed as low risk of bias in the comparability criteria or studies performing adjusted regression analyses
- Re‐categorising industry reviews into reviews with financial conflicts of interest related to any for‐profit organisation or to the manufacturer of the investigated intervention in two separate analyses
In addition, we planned to conduct sensitivity analyses for our secondary outcomes (i.e. methodological quality) to address the issue of confounding.
- Restricting our analyses to studies assessed as low risk of bias in the comparability criteria
Assessment of certainty of the evidence
We graded the certainty of the evidence for each of our primary outcomes as high, moderate, low, or very low. In the standard GRADE approach for studies of treatment effect (Goldet 2013), observational studies are initially graded as low certainty and randomised trials as high certainty (Guyatt 2011; Schünemann 2017). We followed these principles, and initially graded the included observational studies as providing low certainty.
We assessed the following criteria for downgrading: limitations in the study design, indirectness of evidence, inconsistency of results, imprecision of results, and publication bias. We assessed the following criteria for upgrading the certainty: large magnitude of effect, dose‐response gradient, and all plausible confounding would decrease the effect (Guyatt 2011).
Results
Description of studies
See: Characteristics of included studies; Characteristics of excluded studies.
Results of the search
See: Figure 1.
Study flow diagram.
We identified a total of 5227 records through our electronic database searches. After removing duplicates, we screened 3494 records. After assessing 17 full‐text article, we included 10 studies. Nine of these were identified through the database search and one study was identified through other sources (Figure 1). We did not identify any unpublished studies or protocols for planned studies.
Included studies
See: Characteristics of included studies.
The 10 included studies were published between 2006 and 2017 and investigated a total of 1010 systematic reviews. The median number of included systematic reviews per study was 48 (range 11 to 318). Three studies investigated systematic reviews of randomised trials, six investigated systematic reviews of both randomised trials and non‐randomised studies, and one investigated network meta‐analyses. One study included systematic reviews of both drug and device interventions, whereas nine studies included solely systematic reviews of drug interventions. Five studies included systematic reviews related to specific drug classes (e.g. antidepressants), one related to a specific disease (skin psoriasis), and four included various drug comparisons. One study investigated estimated treatment effects, one investigated frequency of statistically favourable results, seven investigated frequency of favourable conclusions, and six investigated methodological quality. According to the declarations in their reports, three studies had conflicts of interest in the form of using industry facilities during the study and author employment at pharmaceutical companies (Lane 2013), or receiving honoraria for research, lecturing, and consultancy from pharmaceutical companies (Gomez‐Garcia 2017; Hartog 2012).
We received unpublished data from five studies (Gomez‐Garcia 2017; Hartog 2012; Jorgensen 2008; Wang 2010; Yank 2007) and obtained published individual review data from three studies (Dunn 2014; Ebrahim 2016; Jorgensen 2006).
Risk of bias in included studies
‘Risk of bias’ graph: review authors’ judgements about each risk of bias item presented as percentages across all included studies.
‘Risk of bias’ summary: review authors’ judgements about each risk of bias item for each included study.
Overall, we assessed two studies as low risk of bias (Jorgensen 2006; Lane 2013), and the other eight included studies as high risk of bias. The majority of the studies were assessed as low risk of bias in the study inclusion process and extraction methods, whereas they were assessed as high risk of bias in the comparability of investigated systematic reviews.
We were able to obtain one publicly available protocol (through PROSPERO) (Gomez‐Garcia 2017) and four unpublished protocols (Ebrahim 2016; Jorgensen 2006; Jorgensen 2008; Lane 2013) for the included studies. We compared outcomes between these protocols and the corresponding publications of the included studies and found that the reported outcomes were prespecified in the protocols in all cases.
Effect of methods
See: Summary of findings for the main comparison
Differences in results
Estimated treatment effect
One study comparing seven pairs of Cochrane Reviews and industry‐funded reviews of the same drug used for the same disease, investigated the estimated treatment effect. The authors calculated Z‐scores for the difference in effect estimates between each pair and found that the estimated treatment effect was not statistically significantly different between systematic reviews with and without financial conflicts of interest (pooled Z‐score: 0.46, P value: 0.64) (Jorgensen 2006).
Frequency of statistically favourable results
One study, including 124 meta‐analyses, investigated frequency of statistically favourable results (Yank 2007). Based on this study, we found no statistically significant difference in frequency of statistically favourable results (risk ratio (RR): 0.84, 95% confidence interval (CI): 0.62 to 1.14, Analysis 1.1).
Differences in frequency of favourable conclusions
Seven studies, including a total of 411 systematic reviews, investigated the frequency of favourable conclusions. Industry reviews more frequently had favourable conclusions compared with non‐industry reviews (RR: 1.98, 95% CI: 1.26 to 3.11, I2: 69%, Analysis 2.1). The analysis was mainly driven by two studies that contributed 33.2% (Yank 2007) and 30.2% (Ebrahim 2016) of the weight of the analysis.
The prediction interval for the RR including all studies was 0.62 to 6.31 (Appendix 4). Thus, one can expect industry reviews to more often have favourable conclusions compared with non‐industry reviews, although this relationship is reversed in some cases.
Differences in methodological quality
Five studies investigated methodological quality using either the Oxman and Guyatt index (Jorgensen 2006; Jorgensen 2008; Yank 2007), the AMSTAR tool (Gomez‐Garcia 2017), or a tool the authors developed for their study (Lane 2013). Similar items across these tools could be pooled from four studies only because the fifth (Yank 2007) reported an overall summary score without results of individual items. Additionally, a sixth study measured methodological quality of network meta‐analyses using the ISPOR guidance (Chambers 2015), but we analysed this study separately because these criteria for methodological quality differed from the quality criteria related to conventional systematic reviews.
In total, we analysed 11 different dimensions of methodological quality. The overall trend showed that all point estimates were above 1, and the methodological quality tended to be lowest among industry reviews (i.e. they tended to fulfil the dimensions of the quality less often) (Analysis 3.1 to Analysis 3.11). For three dimensions, the difference in methodological quality was statistically significant (Analysis 3.1; Analysis 3.3; Analysis 3.8).
In two of these 11 analyses (Analysis 3.2; Analysis 3.3), we found substantial statistical heterogeneity (I2of 60% and 57%, respectively). One study was somewhat different from the other studies in these analyses, because the authors had used the AMSTAR tool to assess methodological quality and included both drug and device systematic reviews (Gomez‐Garcia 2017).
The study on methodological quality in the network meta‐analyses had similar results (Chambers 2015, Appendix 5).
Subgroup analyses
Due to lack of data, it was only meaningful to carry out three of five planned subgroup analyses (Appendix 6).
For frequency of favourable conclusions, we found no statistically significant difference between studies with high risk of bias compared with those with low risk of bias (RR: 1.81, 95% CI: 1.21 to 2.69, based on six studies of 397 systematic reviews; versus RR: 15.00, 95% CI: 1.02 to 220.92, based on one study of 14 systematic reviews, test for interaction P value: 0.13; Analysis 4.1).
Our subgroup analysis comparing frequency of favourable conclusions between Cochrane Reviews and non‐Cochrane systematic reviews found no statistically significant difference between non‐Cochrane systematic reviews (RR: 1.32, 95% CI: 1.15 to 1.52, based on six studies of 351 non‐Cochrane systematic reviews) and Cochrane Reviews (RR: 2.17, 95% CI: 0.63 to 7.44, based on three studies of 38 Cochrane Reviews, test for interaction P value: 0.43, Analysis 4.2). Seven of the observational studies we included in this methodology review provided data from 411 systematic reviews (for our primary analysis of frequency of favourable conclusions). Out of these 411 systematic reviews, 53 were Cochrane Reviews (of which 38 were included in our subgroup analysis comparing Cochrane Reviews and non‐Cochrane Reviews). The authors of the included observational studies classified 15 of the 53 Cochrane Reviews as having authors with financial conflicts of interest or as having received industry funding. One of the 15 Cochrane Reviews was partly funded by the industry (published in 2004), and two had lead authors with financial conflicts of interest (published in 2009 and 2011). In ten of the Cochrane Reviews, less than half of the non‐lead authors had financial conflicts of interest (published between 2009 and 2013), and in the remaining two Cochrane Reviews, the reason for the classification was unclear.
For differences in the definition of favourable conclusions, we found no statistically significant difference between conclusions in favour of the interventions (RR: 1.94, 95% CI: 0.93 to 4.07, based on four studies of 173 systematic reviews) and conclusions recommending the intervention without reservations (RR: 2.11, 95% CI: 1.18 to 3.79, based on three studies of 238 systematic reviews) (test for interaction P value: 0.86; Analysis 4.3).
Sensitivity analyses
We were able to carry out all of the four pre‐planned sensitivity analyses and the four post hoc sensitivity analyses for at least one of our primary outcomes (Appendix 7). In general, all sensitivity analyses showed similar findings for frequency of statistically favourable results and frequency of favourable conclusion as our primary analyses.
The statistical heterogeneity in our primary analysis of frequency of favourable conclusions was substantial (Analysis 2.1), and the RR spanned from 1.25 to 15.00. One of the studies (Yank 2007) compared systematic reviews with financial conflicts of interest to a single drug company with a group of industry and non‐industry systematic reviews. Excluding this atypical study, the statistical heterogeneity decreased substantially (I2 reduced from 69% to 0%), but the effect estimate remained approximately the same (RR: 2.03, 95% CI: 1.56 to 2.64, Analysis 5.9).
We also carried out sensitivity analyses for 10 dimensions of methodological quality. These analyses showed similar findings as our primary analyses.
Assessment of certainty of the evidence
See summary of findings Table for the main comparison.
The certainty of the evidence on estimated treatment effect was assessed as low. Only one study (Jorgensen 2006) investigated estimated treatment effect based on 14 systematic reviews (certainty downgraded for imprecision). The study may therefore be underpowered, and we find it likely that further research will have an impact on the estimate and our confidence in the estimate.
The certainty of the evidence on frequency of statistically favourable results was assessed as very low. Only one study (Yank 2007) investigated frequency of statistically favourable results, and this study was assessed as high risk of bias (certainty downgraded for limitation in design) and the confidence interval was quite wide (95% CI: 0.62 to 1.14, Analysis 1.1) (certainty downgraded for imprecision).
The certainty of the evidence on frequency of favourable conclusions was assessed as moderate. In our assessment, we did not up‐ or downgrade the certainty. The effect estimate was fairly large and based on seven studies (RR: 1.98, Analysis 2.1). Among the one study assessed as low risk of bias, the effect estimate was substantially higher, though with a wide confidence interval (RR: 15.00, 95% CI: 1.02 to 220.92, Analysis 4.1).
Discussion
available in
Summary of main results
We included 10 studies that investigated a total of 1010 systematic reviews. All studies included systematic reviews of drug interventions (995 systematic reviews), and only one study also included systematic reviews of device interventions (15 systematic reviews). We found no statistically significant difference in results (estimated treatment effect and frequency of statistically favourable results) between systematic reviews with and without financial conflicts of interest. Systematic reviews with financial conflicts of interest more frequently had favourable conclusions compared with systematic reviews without. Systematic reviews with financial conflicts of interest tended to have lower methodological quality compared with systematic reviews without these financial conflicts of interest. Our findings were robust in a number of sensitivity analyses, and analyses based on matched studies or adjusted regression analyses (which had thereby reduced the risk of confounding) had similar results. Only two of the 10 included studies were assessed as low risk of bias.
Quality of the evidence
The majority of the included studies were assessed as having high risk of bias, because the investigated systematic reviews differed in aspects other than financial conflicts of interest (e.g. investigated different interventions used for different diseases). Thus, the included studies had a risk of confounding. Two of the included studies used a matched design and thereby reduced the risk of confounding markedly, but did not eliminate it (Jorgensen 2006; Lane 2013). We also noted a less pronounced reduction of risk of confounding in a third study adjusting for confounders (Yank 2007). Our analyses of results (estimated treatment effect and statistically favourable results) had some risk of confounding and were imprecise, and our analysis of conclusions and methodological quality had some risk of confounding.
We assessed the certainty of the evidence using the GRADE approach. All included studies were of observational design, and certainties of evidence ranged from very low to low. Despite being possible in theory, randomisation is not feasible in these types of studies, and the assessment may be too conservative and should be considered tentative.
Potential biases in the review process
Our review has several strengths. We obtained unpublished data from five studies, which enabled us to conduct subgroup and sensitivity analyses. We found no unpublished studies despite doing a comprehensive search for conference abstracts and unpublished literature. We addressed selective outcome reporting by comparing the published studies with their corresponding protocols for five studies (Ebrahim 2016; Gomez‐Garcia 2017; Jorgensen 2006; Jorgensen 2008; Lane 2013) and found no signs of selective outcome reporting.
We only found one study investigating estimated treatment effect. Investigating effect sizes involves complicated statistics in measuring differences between effect estimates expressed in varying units. Surprisingly, we only found one study investigating statistically favourable results, which seems contradictory to the fact that multiple studies investigated favourable conclusions. However, based on the five protocols we had access to, we have no reason to believe that this is due to selective reporting.
Nevertheless, our review has some limitations. First, our ability to conduct subgroup and sensitivity analyses was limited by inclusion of only 10 studies. Second, we only identified one study investigating estimated treatment effect and only one study investigating frequency of statistically favourable results. Third, for the association between financial conflicts of interest and frequency of favourable conclusions we found substantial statistical heterogeneity (I2: 69%). One possible reason for this was the inclusion of one study that compared systematic reviews with financial conflicts of interest to one drug company to a comparison group comprised of both industry and non‐industry reviews (Yank 2007). When we excluded this study from the analysis, the statistical heterogeneity decreased to an I2 of 0%.
Agreements and disagreements with other studies or reviews
A Cochrane Methodology Review by Lundh and colleagues investigated the impact of industry sponsorship on outcomes in individual research studies, mainly clinical trials. It found that industry sponsored studies more often had statistically favourable results (RR: 1.27, 95% CI: 1.17 to 1.37) and favourable conclusions (RR: 1.34, 95% CI: 1.19 to 1.51) compared with non‐industry sponsored studies (Lundh 2017). Contrary to that study, we did not find an association between financial conflicts of interest and statistically favourable results in systematic reviews. This may be due, at least in part, to systematic reviews on the same topic often including the same randomised trials and thereby obtaining similar results in their meta‐analyses. However, our findings on estimated treatment effect and statistically favourable results are based on only one study each and are fairly imprecise with wide confidence intervals. Furthermore, the authors of the other Cochrane Methodology Review found no overall difference in risk of bias between industry and non‐industry funded trials (Lundh 2017). This is in contrast to the tendency that industry reviews had lower methodological quality than non‐industry reviews that we found. One potential explanation for the higher quality of non‐industry reviews could be that such reviews may more often be authored by methodologists (Gotzsche 2012).
The association between financial conflicts of interest and conclusions has also been investigated in relation to narrative reviews, editorials, and letters to the editor for drug interventions in a number of studies. Authors with financial conflicts of interest seem to recommend a drug more often compared with authors without such conflicts of interest (Stelfox 1998; Wang 2010), but we are not aware of any systematic review of this topic.
Finally, financial conflicts of interest have also been investigated in relation to non‐drug and non‐device industries. Chartres and colleagues undertook a systematic review of the association between financial conflicts of interest in relation to the food industry and study outcomes. They found a tendency that primary research studies and reviews with financial conflicts of interest more often had favourable conclusions compared with studies without (RR: 1.31, 95% CI: 0.99 to 1.72) (Chartres 2016). Similarly, Barnes and colleagues found that authors of review articles with financial conflicts of interest related to the tobacco industry more often concluded that passive smoking is not harmful compared with reviews without such conflicts of interest (odds ratio (OR): 88.4; 95% CI: 16.4 to 476.5) (Barnes 1998).
Meaning of our review
The association between financial conflicts of interest and conclusions of systematic reviews may be explained by underlying differences in results (i.e. estimated treatment effect and frequency of statistically significant results). However, we found no statistically significant difference in review results between industry and non‐industry reviews. Even though our analyses of results had some risk of confounding and were imprecise, this could suggest that results are interpreted differently in industry and non‐industry reviews, which might be associated with the lower methodological quality of industry reviews. For example, industry reviews less often interpret results in the light of risk of bias of included studies. Another reason could be a more frequent use of spin in industry review conclusions (Yavchitz 2016). Nonetheless, our finding on the association between financial conflicts of interest and review results remains uncertain, because it is based on only one study. Second, any association may be affected by confounding (e.g. if industry and non‐industry reviews investigate different types of interventions used for different diseases). However, it is unlikely that the differences in frequency of favourable conclusions is an issue of confounding, because the association was also found in one study that used a matched design (Jorgensen 2006).
