Transparency and disclosure, neutrality and balance: shared values or just shared words?
Sander Greenland
https://sci-hub.tw/10.1136/jech-2011-200459
ABSTRACT Values influence choice of methodology and thus influence every risk assessment and inference. To deal with this inescapable reality, we need to replace vague and unattainable calls for objectivity with more precise operational qualities. Among qualities that seem widely valued are transparency (openness) and neutrality (balance, fairness). Conformity of researchers to these qualities may be evaluated by considering whether their reports disclose key information desired by readers and whether their methodology encourages initial neutrality among hypotheses of concern. A case study is given in which two authors appearing to share these values and writing on ostensibly the same issues (disclosure and methodology) nonetheless appear to have very different concepts of what the values entail in practice. Thus, more precision is needed in explicating and implementing such values.
NTRODUCTION A major assumption underlying statistical methods (whether frequentist or Bayesian) is the absence of uncontrolled bias. This assumption is an ideal that is almost never satisfied in the design, data collection, and analysis of health and social science data; even randomised trials are vulnerable. Thus, by focusing on conventional statistics, epidemiologists may develop overconfident inferences about effects that are not so large as to be obvious.1 Indeed, high confidence in epidemiological inferences may often be as misplaced as the overconfident informal judgements seen in psychology experiments.2 3 This overconfidence can be mitigated by examining specific characteristics of study design, conduct and analysis potentially related to bias.1 A more uncomfortable point is that investigators and their communities are often a major source of bias.4e8 Even for honest investigators, cognitive bias is unavoidable because it is hard-wired and essential for real-world functioning. Among the reasons is that there are always too many physically possible explanations to consider. Thus, we must exclude most explanations via our initial fact and methods claims. But those claims are subjective, and one person’s accepted facts and methods may be another person’s rubbish, or may later be recognised as rubbish by everyone. For example, any given data set might have been fabricated or altered with deceptive intent. We never mention this possibility in reviews, as it usually seems implausible. Even if we found it plausible, reviewers would object to raising suspicions without airtight evidence of fraud. We thus have a methodological bias towards assuming basic data integrity. The consequences of this bias can be serious: in one incident, it was found that an investigator fabricated 21 studies favourable to his research sponsors and his non-existent studies were used to guide practice.9 Colleagues reported these studies had been ‘particularly influential’ and said they were ‘shocked by the news’. ‘Shock’ means that the fraud revelation went strongly against their bias, which was towards presuming that integrity plus fear of discovery would have prevented the fraud, or at least that routine checks would have revealed it sooner. However, not everyone was shocked, because the historical record does not support a presumption of innocence in science.10 Literature overviews sustain concerns that the biomedical literature is affected more subtly but seriously by funding bias7 and selective-publication bias.11 12 Some incidents suggest that investigator bias can be larger than any other bias and often encourages adoption and maintenance of ineffective or even deadly treatments.13 Speaking of the general medical literature, one journal editor said ‘Far too much of the medical literature sits somewhere between unregulated advertising and abject fraud as authors, motivated by a host of factors other than determining the truth, push onto the community papers that reflect neither what was done nor what is likely the truth.’ i Here, however, I would like to focus on sources of investigator bias more subtle than fraud or other intentional deception: the influence and conflicting interpretations of scientific values (ideals) on avowed methodology and disclosures.
SOURCES AND TYPES OF INVESTIGATOR BIAS Because of the sensitivity of the topic, it may be useful to classify investigator biases according to their likelihood of intentionally deceptive versus innocent (but nonetheless harmful) bias. Outright data fabrication requires deceptive intent. While there is a deceptive element in failure to publish undesirable results, such failure may arise from an unshakeable belief in the hypothesis the data contradict, making such acts more in the realm of prejudicial, unfair or biased behaviour. Similarly, manipulation and selection of statistical results and biased emphases in methodology and discussion of results may reflect unchecked prejudices rather than deceptive intent. Variants of this problem have long been discussed under headings such as wish bias4 and interpretive bias.ii 6 1 Department of Epidemiology, University of California, Los Angeles, California, USA 2 Department of Statistics, University of California, Los Angeles, California, USA Correspondence to Professor Sander Greenland, Department of Epidemiology and Statistics, University of California, Los Angeles, CA 90095-1772, USA; lesdomes@ucla.edu Accepted 16 November 2011 i D Schriger, 2011, personal communication. iiSee also numerous entries in Porta et al14 including auxiliary hypothesis bias, cognitive dissonance bias, confirmation bias, conflict of interest bias, disclosure of interests, epistemic communities, epistemic cultures, interpretive bias, knowledge construction, publication bias, rescue bias, sociology of scientific knowledge. Essay J Epidemiol Community Health 2012;66:967–970. doi:10.1136/jech-2011-200459 967 Published Online First 20 January 2012 Even more innocently, incorrect methodological choices and claims may reflect nothing more than accepted but harmful habits, such as confusing ‘statistical significance’ with presence of an effect, or confusing ‘non-significance’ with evidence of absence of an effect.1 15e18 Such errors may be encouraged by editors and reviewers, and lead to publication bias, for example, in preferential reporting and acceptance of ‘statistically significant’ results.11 19 To deal with investigator bias, we can consider signs such as selective presentation of results or their possible explanations, or arguing for directionally biased methodologies and evaluation criteria. For example, it is not unusual to see insistence on non-differentiality of classification error (eg, by forcing equal misclassification rates for cases and non-cases) as a criterion for study validity or quality, in the belief that the null bias it tends to produce is superior to bias from differential errors. In reality, the impact of non-differentiality is highly context-specific, and can be harmful when compared with certain differential alternatives.20 Typical expected predictors or sources of investigator bias are consulting and funding ties, one’s previous published conclusions, one’s ideology and wishful thinking. Even if we accept these bias sources as common and influential, that does not imply that they arise in everyone or in every setting. Nonetheless, it is often noted that we are all subject to unconscious biases due to our values, as well the presumed facts and methodologies we have accepted (often uncritically) from our social milieu. For example, unreserved preference for non-differential error can reflect knowledge that it more often leads to null bias (and thus will be favoured by those who regard false positives as more harmful than false negatives), or it may also reflect no more than uncritical acceptance of its alleged general superiority. Only by asking for a rationale for the preference might we be able to discriminate between the two, and thus determine whether the preference is rote or instead reflects conscious adoption of general values. In either case, the preference may be enforced or reinforced during the peer-review process (even when the preference reflects an error or bias in reasoning). The influence of values on these preferences is inescapable. To quote one author: ‘Values, I argue, are an essential part of scientific reasoning, including social and ethical values. . While no part of science can be held to be value free, constraints on how the values are used in scientific reasoning are crucial to preserving the integrity and reliability of science.’ 21e26 We should thus ask of any methodology and application: What values are implicit in it, what is the impact of their adoption or violation, and are they followed closely? More direct explication of values will help the reader answer these questions.
ABANDONING OBJECTIVITY FOR MORE ATTAINABLE PROPERTIES I maintain that treating ‘objectivity’ as a cure for or opposite of bias is misguided. Consider this definition of objective conduct: ‘Objective: expressing or dealing with facts or conditions as perceived, without distortion by personal feelings, prejudices or interpretations’. 27 Perceptual distortion can be negligible in physics, but is predominant in health science. It arises not only from personal prejudices, but also from cognitive biases and values built into methodologies that investigators follow and teach. Thus perceptual objectivity in the ordinary-English sense is an unrealistic goal in scientific research22 23 even though it is valued and thus claimed by most researchers. Worse, claims of ‘objectivity’ are often simply denial (to oneself as well as to others) of subjectivity and values in one’s assessments and methodology.28 29 The problem with ‘objectivity’ is that it is too complex, ill-defined and unattainable (if not pretentious) to take as a claim or goal. In response, we can replace dubious claims and goals of objectivity with more precise, operational and approachable characteristics. Two highly valued characteristics of this sort are transparency (openness) and neutrality (fairness, balance, symmetry), which appear in analytical jurisprudence and arbitration. Reports that fail to disclose facts expected by readers might be viewed as violating transparency, although the bias implications of this violation may be subtle. Methodological behaviors or guidelines that fail to treat competing hypotheses or bias directions symmetrically may be taken as violating neutrality, although some neutrality violations may be accepted by all debate participants (eg, ignoring prevention hypotheses in debates between causal and null hypotheses). Neutrality violations may manifest as an upward or downward bias, or a bias towards or away from the null. Along either dimension, the ultimate consequences depend on whether the true effect at issue is causal, absent (null) or preventive. For example, bias towards the null is harmless to validity if there is no effect, but can be quite harmful if there is an important effect; and the acceptability of null bias will depend on whether one values avoiding false positives more than false negatives, or vice-versa. Given the complexity of typical settings, violations of transparency and neutrality are bound to occur, since we cannot report every detail of study conduct given time and space limits and thus must exercise judgement about what to report. For example, a potentially twofold bias in a RR would be crucial in evaluating estimates in the ¼ to 4 range (as is typical in studies of ‘lifestyle’ factors), but could be inconsequential when evaluating estimates well beyond that range (as is typical in outbreak investigations). A CASE
STUDY OF VALUES IN DISCLOSURE In recognising the value-laden nature of research, I endorse transparency (including full disclosure of potential conflicts of interest) and neutrality, as do others. Weed30 calls transparency one of ‘two general ethical values’ (accountability being the other, which is not discussed here), and says that ‘disclosure of interests is a central concern’ and ‘absolutely essential’. 31 We diverge on objectivity, however. Weed31 states ‘.it is method that provides us with a claim to objectivity in this value-laden world’. But in the same journal issue I reject human objectivity as chimerical, stating ‘Even though I regard the idea of a singular truth as fundamental to science (just as it is to religion), I also think everything we claim to be knowledge is subjective and hence vulnerable to personal bias’ 7 ; that is, the existence of objective reality in no way implies the existence of personal objectivity. I then lament that ‘. the illusion of objectivity is buttressed by rigid statistical [and methodological] conventions that prey upon and feed human cravings for certainty [and righteousness]. these conventions have profound biases and value judgments (e.g., favoring false negatives over false positives) built into their core. These biases and values are not shared by all stakeholders in methodological, subject-matter or ethical debates.’ Values complicate a critical evaluation of disclosures. Depending on their values, different readers may find different details important to disclose. Furthermore, the authors themselves may very well have a stake in what is and isn’t revealed in their disclosures, thus raising a conflict of interest when (as usual) the decision of what to disclose rests largely with authors. Essay 968 J Epidemiol Community Health 2012;66:967–970. doi:10.1136/jech-2011-200459 These and other issues may come into sharper focus in specific examples where details relevant to some readers are omitted. Transparency of disclosures: easier said than done? Given that Weed31 and I7 both endorse disclosures, it is interesting to critically examine those given in the cited articles. Weed was a salaried employee of the US National Cancer Institute and so declares ‘Funding: For over two decades I was funded by the National Cancer Institute.’ He then declares ‘Competing interests: I have received no compensation from any source for writing this editorial’, and ‘Disclosures: I am now in the private practice of epidemiology.’ These declarations leave open the questions of current funding and what clients and topics are served in private practice, since these specifics reveal potential conflicts of interest. The disclosure in Greenland7 also neglects information that others might want. Since I also had no funding for my article, I simply omitted the funding statement and declared no competing interests. Instead, my potential conflicts were listed as rather non-specific ‘Disclosures: The author does consulting for both plaintiffs and defendants in litigation involving epidemiologic and statistical evidence, has done a number of studies that were motivated by such litigation, and has done a number of industry-sponsored studies.’ Outside observers might instead want to know specific facts. Relevant to this example, Weed has served as an industry-defence expert in cases involving asbestos, PCBs, hairsprays, pharmaceuticals, etc, while I have served as petitioner’s expert in vaccine-autism (albeit I opined against the petitioner’s causal claim), and as plaintiff expert against manufacturers of medical devices, pharmaceuticals and other products. Detailed listing of histories in every publication could be tedious as well as wasteful of space; hence, for transparency one could have registration of consults, so that disclosures could cite complete listings at a central registration website.iii Alternatively, journals could mandate complete current listings as online disclosure supplements. Nonetheless, it would be hard to determine what should be included and when the burden of supplying details becomes too onerous to outweigh any benefit (eg, stock ownership through mutual funds). Furthermore, there are no meaningful resources for enforcement, so that those who wish to conceal conflicts could easily do so. Another problem with extensive disclosure is that key conflicts of interest for a given report would likely be buried within such detail, so that casual readers would not be alerted to such problems. Further discussions of these and other disclosure issues can be found in the August 2009 issue of this journal.32e35
DOES NEUTRALITY REQUIRE SYMMETRY? Turning to implementations of ‘balanced and fair’ or ‘neutrality,’ in the sense of trying to minimise the influence of values, the following example is illuminating. Weed31 describes a ‘good epidemiologist’ as ‘balanced and fair’, which might be interpreted as a neutrality endorsement and thus a point of agreement between us. Nonetheless, Weed et al36 later stated ‘We examined examples of questionable causal claims and practices in causal inference. By questionable it is meant not consistent with good methodologic practice.’ Their statement raises the question: good practice by whose methodology? This question is important because there appears to be marked divergences between their methodology and other methodologies, including mine. Their eight examples of questionable causal claims include as item 5 ‘Causal claims in the absence of statistically significant elevated risks’. Labelling such claims as questionable seems to ignore scores of articles and many books over the past 70 years criticising ‘statistical significance’, especially arbitrary 0.05-level criteria (eg, Chapter 10 in ref. 1;15e18 37). Even the US Supreme Court has ruled unanimously that statistical significance is not the same as material significance, noting that ‘medical professionals and regulators act on the basis of evidence of causation that is not statistically significant’ and that companies and courts must consider ‘the source, content and context of the reports’. 38 Setting aside this objection to Weed et al’s 36 item 5, we may evaluate the neutrality of their list by asking whether it is symmetric in its concerns, that is, whether it addresses the mirror of each problem raised. The mirror of item 5 would be to claim no causation from the absence of statistical significance, even when the actual statistical evidence leans towards causation. This is a common methodological error.1 15 17 As Hill39 warned, too often ‘we weaken our capacity to interpret data and to take reasonable decisions whatever the value of p. And far too often we deduce “no difference” from “no significant difference”.’ A related fallacy is to assume there is no association if there is absence of information.iv These mirrors of item 5 do not, however, appear in Weed et al’s list.36 As an example, consider the following testimony abstracted from a lawsuit involving a product X and disease Y.v Applying my interpretation of neutrality, I wrote in my plaintiff report that, based on the data limitations, ‘it is incorrect to claim that the reports evaluated here demonstrate that X is not associated with or never causes Y. Conversely, and for the same reasons, these reports do not demonstrate that X is associated with or causes Y.’ For his defence report, Weed30 wrote that ‘The scientific evidence is insufficient to justify a valid and reliable claim that X causes Y’, which appears consonant with my second sentence. Nonetheless, his report does not state that the evidence is also insufficient to claim that exposure to X does not cause Y, and so is asymmetric in this sense. Similar asymmetry is seen in certain other items in Weed et al36 Their item 6 is ‘Causal claims from cherry-picked positive associations’, but no mention is made of its mirror: Claims of no causation from cherry-picked null or non-significant associations; see Greenland41 for a prominent example. Their item 8 is ‘Overgeneralization: causal claims for groups of diseases (e.g., all lymphopoietic cancers) or for a general category of exposure (e.g., pesticides)’. Unmentioned is undergeneralisation, such as failure to apply evidence to related exposures or diseases, for example, failure to consider effects of Premarin when considering effects of synthetic oestradiol, or failure to consider causes of endometrial hyperplasia when determining causes of endometrial cancer. Such examples leave unclear whether ‘balanced and fair’ 31 entails the symmetry intended in my neutrality value. To summarise: (1) the failure to apply symmetry values and consider ‘mirror’ biases is itself a form of bias, tilting methodology towards one direction of results; and (2) this methodological bias may arise from asymmetries in the values of the proponents of the methodology. Thus, methodological choices and preferences can reveal values of the proponents, including those that lead to conflict with neutrality and symmetry values.
https://sci-hub.tw/10.1136/jech-2011-200459
ABSTRACT Values influence choice of methodology and thus influence every risk assessment and inference. To deal with this inescapable reality, we need to replace vague and unattainable calls for objectivity with more precise operational qualities. Among qualities that seem widely valued are transparency (openness) and neutrality (balance, fairness). Conformity of researchers to these qualities may be evaluated by considering whether their reports disclose key information desired by readers and whether their methodology encourages initial neutrality among hypotheses of concern. A case study is given in which two authors appearing to share these values and writing on ostensibly the same issues (disclosure and methodology) nonetheless appear to have very different concepts of what the values entail in practice. Thus, more precision is needed in explicating and implementing such values.
NTRODUCTION A major assumption underlying statistical methods (whether frequentist or Bayesian) is the absence of uncontrolled bias. This assumption is an ideal that is almost never satisfied in the design, data collection, and analysis of health and social science data; even randomised trials are vulnerable. Thus, by focusing on conventional statistics, epidemiologists may develop overconfident inferences about effects that are not so large as to be obvious.1 Indeed, high confidence in epidemiological inferences may often be as misplaced as the overconfident informal judgements seen in psychology experiments.2 3 This overconfidence can be mitigated by examining specific characteristics of study design, conduct and analysis potentially related to bias.1 A more uncomfortable point is that investigators and their communities are often a major source of bias.4e8 Even for honest investigators, cognitive bias is unavoidable because it is hard-wired and essential for real-world functioning. Among the reasons is that there are always too many physically possible explanations to consider. Thus, we must exclude most explanations via our initial fact and methods claims. But those claims are subjective, and one person’s accepted facts and methods may be another person’s rubbish, or may later be recognised as rubbish by everyone. For example, any given data set might have been fabricated or altered with deceptive intent. We never mention this possibility in reviews, as it usually seems implausible. Even if we found it plausible, reviewers would object to raising suspicions without airtight evidence of fraud. We thus have a methodological bias towards assuming basic data integrity. The consequences of this bias can be serious: in one incident, it was found that an investigator fabricated 21 studies favourable to his research sponsors and his non-existent studies were used to guide practice.9 Colleagues reported these studies had been ‘particularly influential’ and said they were ‘shocked by the news’. ‘Shock’ means that the fraud revelation went strongly against their bias, which was towards presuming that integrity plus fear of discovery would have prevented the fraud, or at least that routine checks would have revealed it sooner. However, not everyone was shocked, because the historical record does not support a presumption of innocence in science.10 Literature overviews sustain concerns that the biomedical literature is affected more subtly but seriously by funding bias7 and selective-publication bias.11 12 Some incidents suggest that investigator bias can be larger than any other bias and often encourages adoption and maintenance of ineffective or even deadly treatments.13 Speaking of the general medical literature, one journal editor said ‘Far too much of the medical literature sits somewhere between unregulated advertising and abject fraud as authors, motivated by a host of factors other than determining the truth, push onto the community papers that reflect neither what was done nor what is likely the truth.’ i Here, however, I would like to focus on sources of investigator bias more subtle than fraud or other intentional deception: the influence and conflicting interpretations of scientific values (ideals) on avowed methodology and disclosures.
SOURCES AND TYPES OF INVESTIGATOR BIAS Because of the sensitivity of the topic, it may be useful to classify investigator biases according to their likelihood of intentionally deceptive versus innocent (but nonetheless harmful) bias. Outright data fabrication requires deceptive intent. While there is a deceptive element in failure to publish undesirable results, such failure may arise from an unshakeable belief in the hypothesis the data contradict, making such acts more in the realm of prejudicial, unfair or biased behaviour. Similarly, manipulation and selection of statistical results and biased emphases in methodology and discussion of results may reflect unchecked prejudices rather than deceptive intent. Variants of this problem have long been discussed under headings such as wish bias4 and interpretive bias.ii 6 1 Department of Epidemiology, University of California, Los Angeles, California, USA 2 Department of Statistics, University of California, Los Angeles, California, USA Correspondence to Professor Sander Greenland, Department of Epidemiology and Statistics, University of California, Los Angeles, CA 90095-1772, USA; lesdomes@ucla.edu Accepted 16 November 2011 i D Schriger, 2011, personal communication. iiSee also numerous entries in Porta et al14 including auxiliary hypothesis bias, cognitive dissonance bias, confirmation bias, conflict of interest bias, disclosure of interests, epistemic communities, epistemic cultures, interpretive bias, knowledge construction, publication bias, rescue bias, sociology of scientific knowledge. Essay J Epidemiol Community Health 2012;66:967–970. doi:10.1136/jech-2011-200459 967 Published Online First 20 January 2012 Even more innocently, incorrect methodological choices and claims may reflect nothing more than accepted but harmful habits, such as confusing ‘statistical significance’ with presence of an effect, or confusing ‘non-significance’ with evidence of absence of an effect.1 15e18 Such errors may be encouraged by editors and reviewers, and lead to publication bias, for example, in preferential reporting and acceptance of ‘statistically significant’ results.11 19 To deal with investigator bias, we can consider signs such as selective presentation of results or their possible explanations, or arguing for directionally biased methodologies and evaluation criteria. For example, it is not unusual to see insistence on non-differentiality of classification error (eg, by forcing equal misclassification rates for cases and non-cases) as a criterion for study validity or quality, in the belief that the null bias it tends to produce is superior to bias from differential errors. In reality, the impact of non-differentiality is highly context-specific, and can be harmful when compared with certain differential alternatives.20 Typical expected predictors or sources of investigator bias are consulting and funding ties, one’s previous published conclusions, one’s ideology and wishful thinking. Even if we accept these bias sources as common and influential, that does not imply that they arise in everyone or in every setting. Nonetheless, it is often noted that we are all subject to unconscious biases due to our values, as well the presumed facts and methodologies we have accepted (often uncritically) from our social milieu. For example, unreserved preference for non-differential error can reflect knowledge that it more often leads to null bias (and thus will be favoured by those who regard false positives as more harmful than false negatives), or it may also reflect no more than uncritical acceptance of its alleged general superiority. Only by asking for a rationale for the preference might we be able to discriminate between the two, and thus determine whether the preference is rote or instead reflects conscious adoption of general values. In either case, the preference may be enforced or reinforced during the peer-review process (even when the preference reflects an error or bias in reasoning). The influence of values on these preferences is inescapable. To quote one author: ‘Values, I argue, are an essential part of scientific reasoning, including social and ethical values. . While no part of science can be held to be value free, constraints on how the values are used in scientific reasoning are crucial to preserving the integrity and reliability of science.’ 21e26 We should thus ask of any methodology and application: What values are implicit in it, what is the impact of their adoption or violation, and are they followed closely? More direct explication of values will help the reader answer these questions.
ABANDONING OBJECTIVITY FOR MORE ATTAINABLE PROPERTIES I maintain that treating ‘objectivity’ as a cure for or opposite of bias is misguided. Consider this definition of objective conduct: ‘Objective: expressing or dealing with facts or conditions as perceived, without distortion by personal feelings, prejudices or interpretations’. 27 Perceptual distortion can be negligible in physics, but is predominant in health science. It arises not only from personal prejudices, but also from cognitive biases and values built into methodologies that investigators follow and teach. Thus perceptual objectivity in the ordinary-English sense is an unrealistic goal in scientific research22 23 even though it is valued and thus claimed by most researchers. Worse, claims of ‘objectivity’ are often simply denial (to oneself as well as to others) of subjectivity and values in one’s assessments and methodology.28 29 The problem with ‘objectivity’ is that it is too complex, ill-defined and unattainable (if not pretentious) to take as a claim or goal. In response, we can replace dubious claims and goals of objectivity with more precise, operational and approachable characteristics. Two highly valued characteristics of this sort are transparency (openness) and neutrality (fairness, balance, symmetry), which appear in analytical jurisprudence and arbitration. Reports that fail to disclose facts expected by readers might be viewed as violating transparency, although the bias implications of this violation may be subtle. Methodological behaviors or guidelines that fail to treat competing hypotheses or bias directions symmetrically may be taken as violating neutrality, although some neutrality violations may be accepted by all debate participants (eg, ignoring prevention hypotheses in debates between causal and null hypotheses). Neutrality violations may manifest as an upward or downward bias, or a bias towards or away from the null. Along either dimension, the ultimate consequences depend on whether the true effect at issue is causal, absent (null) or preventive. For example, bias towards the null is harmless to validity if there is no effect, but can be quite harmful if there is an important effect; and the acceptability of null bias will depend on whether one values avoiding false positives more than false negatives, or vice-versa. Given the complexity of typical settings, violations of transparency and neutrality are bound to occur, since we cannot report every detail of study conduct given time and space limits and thus must exercise judgement about what to report. For example, a potentially twofold bias in a RR would be crucial in evaluating estimates in the ¼ to 4 range (as is typical in studies of ‘lifestyle’ factors), but could be inconsequential when evaluating estimates well beyond that range (as is typical in outbreak investigations). A CASE
STUDY OF VALUES IN DISCLOSURE In recognising the value-laden nature of research, I endorse transparency (including full disclosure of potential conflicts of interest) and neutrality, as do others. Weed30 calls transparency one of ‘two general ethical values’ (accountability being the other, which is not discussed here), and says that ‘disclosure of interests is a central concern’ and ‘absolutely essential’. 31 We diverge on objectivity, however. Weed31 states ‘.it is method that provides us with a claim to objectivity in this value-laden world’. But in the same journal issue I reject human objectivity as chimerical, stating ‘Even though I regard the idea of a singular truth as fundamental to science (just as it is to religion), I also think everything we claim to be knowledge is subjective and hence vulnerable to personal bias’ 7 ; that is, the existence of objective reality in no way implies the existence of personal objectivity. I then lament that ‘. the illusion of objectivity is buttressed by rigid statistical [and methodological] conventions that prey upon and feed human cravings for certainty [and righteousness]. these conventions have profound biases and value judgments (e.g., favoring false negatives over false positives) built into their core. These biases and values are not shared by all stakeholders in methodological, subject-matter or ethical debates.’ Values complicate a critical evaluation of disclosures. Depending on their values, different readers may find different details important to disclose. Furthermore, the authors themselves may very well have a stake in what is and isn’t revealed in their disclosures, thus raising a conflict of interest when (as usual) the decision of what to disclose rests largely with authors. Essay 968 J Epidemiol Community Health 2012;66:967–970. doi:10.1136/jech-2011-200459 These and other issues may come into sharper focus in specific examples where details relevant to some readers are omitted. Transparency of disclosures: easier said than done? Given that Weed31 and I7 both endorse disclosures, it is interesting to critically examine those given in the cited articles. Weed was a salaried employee of the US National Cancer Institute and so declares ‘Funding: For over two decades I was funded by the National Cancer Institute.’ He then declares ‘Competing interests: I have received no compensation from any source for writing this editorial’, and ‘Disclosures: I am now in the private practice of epidemiology.’ These declarations leave open the questions of current funding and what clients and topics are served in private practice, since these specifics reveal potential conflicts of interest. The disclosure in Greenland7 also neglects information that others might want. Since I also had no funding for my article, I simply omitted the funding statement and declared no competing interests. Instead, my potential conflicts were listed as rather non-specific ‘Disclosures: The author does consulting for both plaintiffs and defendants in litigation involving epidemiologic and statistical evidence, has done a number of studies that were motivated by such litigation, and has done a number of industry-sponsored studies.’ Outside observers might instead want to know specific facts. Relevant to this example, Weed has served as an industry-defence expert in cases involving asbestos, PCBs, hairsprays, pharmaceuticals, etc, while I have served as petitioner’s expert in vaccine-autism (albeit I opined against the petitioner’s causal claim), and as plaintiff expert against manufacturers of medical devices, pharmaceuticals and other products. Detailed listing of histories in every publication could be tedious as well as wasteful of space; hence, for transparency one could have registration of consults, so that disclosures could cite complete listings at a central registration website.iii Alternatively, journals could mandate complete current listings as online disclosure supplements. Nonetheless, it would be hard to determine what should be included and when the burden of supplying details becomes too onerous to outweigh any benefit (eg, stock ownership through mutual funds). Furthermore, there are no meaningful resources for enforcement, so that those who wish to conceal conflicts could easily do so. Another problem with extensive disclosure is that key conflicts of interest for a given report would likely be buried within such detail, so that casual readers would not be alerted to such problems. Further discussions of these and other disclosure issues can be found in the August 2009 issue of this journal.32e35
DOES NEUTRALITY REQUIRE SYMMETRY? Turning to implementations of ‘balanced and fair’ or ‘neutrality,’ in the sense of trying to minimise the influence of values, the following example is illuminating. Weed31 describes a ‘good epidemiologist’ as ‘balanced and fair’, which might be interpreted as a neutrality endorsement and thus a point of agreement between us. Nonetheless, Weed et al36 later stated ‘We examined examples of questionable causal claims and practices in causal inference. By questionable it is meant not consistent with good methodologic practice.’ Their statement raises the question: good practice by whose methodology? This question is important because there appears to be marked divergences between their methodology and other methodologies, including mine. Their eight examples of questionable causal claims include as item 5 ‘Causal claims in the absence of statistically significant elevated risks’. Labelling such claims as questionable seems to ignore scores of articles and many books over the past 70 years criticising ‘statistical significance’, especially arbitrary 0.05-level criteria (eg, Chapter 10 in ref. 1;15e18 37). Even the US Supreme Court has ruled unanimously that statistical significance is not the same as material significance, noting that ‘medical professionals and regulators act on the basis of evidence of causation that is not statistically significant’ and that companies and courts must consider ‘the source, content and context of the reports’. 38 Setting aside this objection to Weed et al’s 36 item 5, we may evaluate the neutrality of their list by asking whether it is symmetric in its concerns, that is, whether it addresses the mirror of each problem raised. The mirror of item 5 would be to claim no causation from the absence of statistical significance, even when the actual statistical evidence leans towards causation. This is a common methodological error.1 15 17 As Hill39 warned, too often ‘we weaken our capacity to interpret data and to take reasonable decisions whatever the value of p. And far too often we deduce “no difference” from “no significant difference”.’ A related fallacy is to assume there is no association if there is absence of information.iv These mirrors of item 5 do not, however, appear in Weed et al’s list.36 As an example, consider the following testimony abstracted from a lawsuit involving a product X and disease Y.v Applying my interpretation of neutrality, I wrote in my plaintiff report that, based on the data limitations, ‘it is incorrect to claim that the reports evaluated here demonstrate that X is not associated with or never causes Y. Conversely, and for the same reasons, these reports do not demonstrate that X is associated with or causes Y.’ For his defence report, Weed30 wrote that ‘The scientific evidence is insufficient to justify a valid and reliable claim that X causes Y’, which appears consonant with my second sentence. Nonetheless, his report does not state that the evidence is also insufficient to claim that exposure to X does not cause Y, and so is asymmetric in this sense. Similar asymmetry is seen in certain other items in Weed et al36 Their item 6 is ‘Causal claims from cherry-picked positive associations’, but no mention is made of its mirror: Claims of no causation from cherry-picked null or non-significant associations; see Greenland41 for a prominent example. Their item 8 is ‘Overgeneralization: causal claims for groups of diseases (e.g., all lymphopoietic cancers) or for a general category of exposure (e.g., pesticides)’. Unmentioned is undergeneralisation, such as failure to apply evidence to related exposures or diseases, for example, failure to consider effects of Premarin when considering effects of synthetic oestradiol, or failure to consider causes of endometrial hyperplasia when determining causes of endometrial cancer. Such examples leave unclear whether ‘balanced and fair’ 31 entails the symmetry intended in my neutrality value. To summarise: (1) the failure to apply symmetry values and consider ‘mirror’ biases is itself a form of bias, tilting methodology towards one direction of results; and (2) this methodological bias may arise from asymmetries in the values of the proponents of the methodology. Thus, methodological choices and preferences can reveal values of the proponents, including those that lead to conflict with neutrality and symmetry values.
Comments
Post a Comment