Search databaseBooksAll DatabasesAssemblyBiocollectionsBioProjectBioSampleBioSystemsBooksClinVarConoffered DomainsdbGaPdbVarGeneGenomeGEO DataSetsGEO ProfilesGTRHomoloGeneIdentical Protein Web CatalogNucleotideOMIMPMCPopSetProteinProtein ClustersProtein Family ModelsPubChem BioAssayPubChem CompoundPubChem SubstancePubMedSNPSRAStructureTaxonomyToolKitToolKitAllToolKitBookgh Bookshelf. A business of the National Library of Medicine, National Institutes of Health.

You are watching: Which of the following strategies might be helpful in reducing the overconfidence bias?

Bojke L, Soares M, Claxton K, et al. Developing a reference protocol for structured experienced elicitation in health-treatment decision-making: a mixed-methods examine. Southampton (UK): NIHR Journals Library; 2021 Jun. (Health Technology Assessment, No. 25.37.)


Developing a referral protocol for structured expert elicitation in health-care decision-making: a mixed-techniques research.

Show details


Formal models of judgement and also decision-making hold that judgements of probcapability and energy must be assessed making use of all of the information accessible to the decision-maker, through the application of correct statistical rules.122 However before, people are not perfect information processors. The amount of indevelopment processed have the right to be affected by time press, restrictions in cognitive capacity, lack of impetus and also personal desire for a certain outcome. When it involves probabilistic thinking, specifically the faitempt to recognise once a statistical dominance should be used and also unfamiliarity through the procedures for making statistical inferences, probcapacity judgements execute not constantly concreate to normative rules.123 Experts, being huguy, are not immune to this. Certainly, even among very educated populations, awareness of how to make basic statistical inferences can be limited.124 In the conmessage of HCDM, those practitioners with the best appropriate understanding and also expertise (e.g. nurses, physiotherapists) may not necessarily have actually a high level of training in statistics or experience with elicitation.

Humans frequently make judgements utilizing basic rules of thumb (or ‘heuristics’).123,125 These methods are typically effective in accordingly guiding judgement,126 especially among experts that have actually a huge base of experience and understanding to draw on.127 However before, in some contexts they deserve to bring about methodical errors known as ‘biases’. SEE need to look for to elicit probcapability judgements in a method that minimises the effect of these systematic errors. This is significantly recognised in the literary works on HCDM, in which SEE have the right to be offered to indevelop health plan and also therapy references.12,44,60,85,128 However, although heuristics, biases and strategies for predisposition reduction have actually been extensively studied in the larger risk, judgement and decision-making literary works, tbelow is a dearth of proof for HCDM and what does exist has actually not been summarised in this conmessage.

This chapter reregarded evidence relating to the psychological biases of biggest relevance to SEE for HCDM, specifically proof on just how these have the right to be minimised. First, key cognitive and also motivational biases that have the potential to negatively influence on the top quality of experienced elicitation for HCDM are outlined (see Cognitive and also motivational biases), then potential techniques for addressing them (view Addressing emotional biases in structured experienced elicitation) with technological steps (watch Technical bias reduction strategies) and behavioral predisposition reduction techniques (check out Behavioural predisposition reduction strategies through continuous support). Reflecting the fact that some behavioral predisposition reduction techniques have actually a huge amount of evidence to assistance them whereas others are even more tentative, techniques are categorised into those for which a high level of consensus exists and also those for which proof is lacking or conflicted. Finally, the key references are summarised in Conclusions.

Cognitive and motivational biases

A difference might be drawn between cognitive biases that result from just how indevelopment is processed, and also motivational biases that come around as an outcome of choices for specific outcomes.77,129 Both have actually been implicated in organized overconfidence, which poses a hazard to calibration in SEE.

Cognitive biases

Cognitive biases aincrease when decision-equipments execute not procedure the full selection of information available to them. This may result from restrictions in cognitive capacity, time push or a absence of incentive to expfinish cognitive effort on a job. They may also aclimb as an outcome of decision-devices doing not have the normative skill to make appropriate probabilistic inferences. In the context of SEE, cognitive biases of certain prestige incorporate availcapacity and also anchoring, and inadequate adjustment, initially, because they are both implicated in overconfidence, which leads to the methodical underestimation of uncertainty in probcapacity judgements, and, second, because unchoose biases that may result from deficits in substantive knowledge of a topic area, or from a lack of understanding around exactly how to factor through statistical information, both have the potential to impact professional judgement.77,130

In making probabilistic judgements, human being may count on how easily examples of an end result come to mind as a guide to exactly how most likely it is (the availcapacity heuristic).131 Although this is regularly a great guide to frequency, it suggests that probability judgements can easily be distorted by exceptionally current or extremely prominent events.132 For instance, a clinician may focus on particularly memorable examples of treatment success or therapy faiattract once making probcapability judgements, neglecting instances that come less easily to mind. Availability bias has been linked to the methodical underestimation of uncertainty.133 Anchoring and inenough adjustment occurs as soon as world settle (‘anchor’) on an initial worth and fail to sufficiently change their approximates ameans from it to carry out a specific judgement. For instance, in judging the success of an intervention, a clinician might ‘anchor’ on a worth provided by a resource that they know to be flawed (e.g. a poor-top quality empirical study) and fail to sufficiently adjust their very own experienced-based estimate from this suggest, despite being aware of the flaws and also adjusting in the best direction.125 Anchoring has actually proved challenging to de-prejudice, via also arbitrary and also irrelevant values being found to impact judgement (check out Kahneman and also Egan123 for an overview). This deserve to decrease accuracy in judgements of place and main tendency (e.g. suppose, median).

Motivational biases

Motivational biases, periodically described as ‘self-serving’ biases, outcome from being invested in a details outcome (e.g. a specific therapy being successful) (view Bazerman and Moore129 for discussion). In cases wbelow individuals are mindful of potential problems of interemainder and also strive to make objective and hoswarm judgements, motivational biases have the right to still distort judgements through rendering some information and also experiences more salient (cognitively ‘available’) and simpler to recontact than others. Confirmation predisposition, for circumstances, leads people to emphasis on indevelopment that is regular through their existing ideas and preferences and, therefore, subject it to a much less important appraisal than inconstant information. Desirability prejudice (additionally described as ‘optimistic bias’ or ‘wishful thinking’) leads civilization to overestimate the likelihood of positive outcomes. Undesircapability bias, meanwhile, leads to an overestimation of the likelihood of negative outcomes and worst instance scenarios (e.g. owing to a emphasis on taking a preventive approach). These biases outcome from encouraged reasoning rather than a lack of knowledge or professionals.77,129 Hence, they have the potential to adversely impact the outcomes of SEE. In HCDM, those through greatest knowledge of a certain therapy or procedure may be those a lot of invested.

Overconfidence bias

As a consequence of limiting the amount of information considered by decision-devices, both availability133 and confirmation bias134 may lead to the uncertainty bordering future outcomes being underestimated. This is recognized as ‘overconfidence bias’. It leads to interval judgements and also probcapability distributions that are as well narrowhead (e.g. estimates of 80% confidence intervals containing < 50% of subsequent realisations). Overconfidence is widespread among specialists and novices,35,135 making it a critical consideration for any create of SEE.

Addressing psychological biases in structured experienced elicitation

Strategies for reducing emotional biases might be shelp to loss right into 3 categories: (1) technological (e.g. utilizing formal statistical actions to correct for organized errors in judgement); (2) directly transforming individual behaviour and also perceptions (e.g. via training, incentives, feedback); and (3) altering the framework of the judgement or decision task (e.g. exactly how questions are asked).136,137 In exercise, but, they represent 2 basic approaches: (1) post-hoc statistical methods to make corrections after the reality, most notably via calibration (disputed in Chapter 5) (technical); and also (2) interventions to adjust judgement and also behaviour (behavioural).

In reviewing approaches for reducing emotional bias (or ‘debiasing’), we minimal our search to research studies that administer empirical evidence for the efficacy of prejudice reduction in the conmessage of SEE. For this factor, we have excluded documents that suggest viewpoints but execute not existing empirical evidence to support them. We likewise exclude researches that emphasis on biases in decision from description (i.e. once selections can be made through evaluation of a complete indevelopment set), quite than elicited judgements. Relevant records that did not appear in the searcs yet that were cited in the files figured out, were examined and also consisted of once appropriate. A potential weakness of this approach is that bias reduction methods that are relevant to SEE, but that perform not cite professional elicitation straight, might have been missed if they were not cited in various other documents identified via the search. However, a full review of the heuristics and also biases literary works, which regularly focuses on novice quite than experienced judgement, is beyond the scope of this targeted search.

Technical prejudice reduction strategies

Technical bias reduction techniques are generally questioned via respect to overconfidence. These can involve statistical predisposition correction and also the weighting of professionals based on their performance on seed concerns, as is the situation in Cooke’s classic version.138–140 These viewpoints do not call for interventions at the individual or task level, as the steps are used short article hoc. However before, they do count on the availability of appropriate seed concerns from which the level of experts propensity to overconfidence have the right to measured.141 This might be fairly straightforward in conmessages in which past realisations of the very same or equivalent tarobtain variables are easily accessible (e.g. probabilistic weather forecasting). In HCDM, but, it could prove complex to implement, as contextually equivalent seed variables through correct realisations are not always readily accessible. Likewise, HTA brings together diverse sets of experts who have actually specialist knowledge of certain therapies, interventions or steps. They are not, therefore, guaranteed to have actually similar field of expertise on the subject of seed inquiries.53

Behavioural bias-reduction techniques through continuous support

Given the obstacles in using technical ideologies to predisposition reduction, which are outlined over, it is crucial for those implementing SEE in the context of HCDM to take into consideration behavioral approaches. In this area, we outline bias reduction methods for which tbelow is regular empirical support. In Behavioural bias reduction approaches with conflicting proof we briefly talk about debiasing philosophies for which tright here is conflicting evidence.

Consider even more information

It has actually been found that people with a higher prejudice in the direction of open-minded reasoning show better calibration on judgement work.142 Increasing the amount of indevelopment taken into consideration by participants may therefore be effective in countering these biases. Behavioural prejudice reduction techniques that prompt experts to take into consideration more indevelopment (enhancing the array of possibilities considered) have actually perhaps been the the majority of generally tested in the context of experienced judgement.

Early research via student samples faicaused find included value from instructing groups of participants to think about why their approximates might be wrong, or appointing one member to be a ‘devil’s advocate’.143 However before, more structured approaches have actually had much greater success.134,144,145 Soll and also Klayman134 uncovered that asking student participants to separately provide lowest plausible approximates, highest plausible approximates and median approximates for an almanac question through which students were most likely to have actually some familiarity caused reduced levels of overconfidence than ssuggest asking for a single 80% confidence interval. It was argued that making people think about lowest, highest possible and also median approximates sequentially concentrates attention on a more comprehensive variety of possibilities than asking for a solitary variety . Building on this, Haran et al.144 discovered that better enhancing the number of considerations by asking participants to make judgements about the likelihood of various local seasonal temperature intervals diminished overconfidence. Adding a fourth step to the procedure argued by Soll and Klayman134 and Speirs-Bridge et al.145 uncovered that arrays were widened better as soon as participants (epidemiologists and also ecologists) were asked exactly how most likely it was that the ‘true’ worth would fall within their stated selection and also were permitted to revise their estimates as necessary. This is continual through study arguing that people might be much better at evaluating confidence intervals than providing them.146,147 More newly, Ferreti et al.148 provided reductions in overconfidence as soon as eco-friendly science students were instructed to (1) actively think of factors why their initial greatest and also lowest estimates of sea level climb may be incorrect; and (2) take into consideration their willingness to area hypothetical bets on elicited confidence intervals.

Together, these research studies carry out solid proof that structuring jobs in a means that boosts consideration of a wider range of possibilities can alleviate predisposition and also boost calibration. They show that confidence intervals should not be elicited as a single-phase procedure. Lower and upper bounds need to be elicited individually,134,145 or multiple smaller intervals need to be taken into consideration individually.144 Likewise, they show that participants should be provided the opportunity to evaluate and readjust their confidence intervals.


Tright here is extensive proof that receiving repetitive feedearlier on one’s judgements both boosts accuracy and reduces overconfidence.35,123,141 Experts, such as weather forecasters, that get direct and also timely feedearlier on the accuracy of their judgements tfinish to be well calibrated in their doprimary of specialization,149 although this does not lead to a domain general innovation.137 One suggestion for reducing the overconfidence predisposition in expert elicitation is to carry out feedago on a collection of exercise questions.150 A difficulty in doing this is the fact that domain-particular seed variables may be more readily available in some contexts than in others (e.g. previous realisations in forecasting tasks). Hence, although this strategy might be generally efficient in enhancing the calibration of skilled judgement, it could be difficult to implement in some HTA conmessages in which identifying correct seed concerns that a diverse collection of experts will certainly be acquainted through can be complex. Nonethemuch less, in situations in which these are accessible, the existing proof suggests that giving feedearlier seed inquiries have the right to minimize overconfidence.

Avoid unessential anchors

Ensuring that elicitation materials perform not contain unvital anchor worths is a ‘common sense’ approach to reducing biases led to by anchoring and insufficient adjustment.77 For circumstances, elicitation devices should not attribute pre-set worths that participants are then asked to readjust to complement their views. However before, it might not constantly be possible to get rid of anchors completely. In the instance of ‘carryover’ impacts, for instance, specialists might use their very own judgement on a previous question as an anchor.151 Although tright here is some evidence to imply that self-created median anchors execute not thrconsumed accuracy and also calibration to the same degree as those that are externally applied,134,152 Morgan35 advises that steps of central tendency (i.e. the median) need to only be elicited after reduced and top bounds have actually been approximated. Hence, although it may not be feasible to remove all potential anchor values in an elicitation job, a clear reference to protect against unessential anchors deserve to be made. Likewise, when eliciting confidence intervals, eliciting lower and also top bounds prior to the median might reduce the tendency to anchor on the median worth.

Reduce prejudice with professional selection

Addressing biases through experienced selection means that experts are contained or excluded based upon their potential susceptibility to bias (watch Chapter 5). As detailed above, motivational biases, such as desircapability bias and also confirmation predisposition, are challenging to eliminate. Restricting participation to those without any disputes of interest is therefore one recommended technique to reducing motivational77 biases. In HCDM this might be challenging, as those via the biggest understanding about a particular treatment or modern technology may likewise be those through the greatest vested interemainder in the elicitation’s outcome.44 Rejecting those through any type of conflict of interest or solid opinions might remove those through the greatest relevant understanding. In such situations an different strategy is to ensure that a selection of ideologies are represented in the sample, with the intention of ‘balancing out’ or at leastern diluting the result of motivational biases.77

Bias warnings and also training

Within the HCDM literary works that considers heuristics and also biases, training is the most generally referenced method to behavioural debiasing.85 Simply warning professionals not to be biased (e.g. by stating that many type of people make their confidence intervals also narrow) is mostly ineffective.143,152,153 However before, thorough training on the nature of biases and techniques for avoiding them has actually been found to be even more efficient. When biases happen as a result of professionals not being familiar with rules for using and also expushing probabilities, training on exactly how to perform so have the right to mitigate errors.154 Likewise, educating participants about biases and explicitly outlining methods for combating them (i.e. with systematically considering even more information) reduced overconfidence in a research of petroleum design students.155 However, this education and learning programme was not effective for reducing anchoring, possibly bereason the student sample lacked the substantive understanding of the field to offer a much more specific value. Nonethemuch less, a study via a general population sample156 uncovered proof that interenergetic training interventions, explaining what anchoring and also confirmation predisposition were, reduced instances of these biases on post-treatment tests family member to pre-intervention tests. These tests comprised jobs from the broader literary works that were found to elicit the emotional biases.157–159 Hence, although the obtainable evidence on the performance of warnings and also training for reducing emotional biases is not always consistent, it does carry out an indication of the conditions under which bias avoidance training might be efficient. First, it need to go past straightforward warnings and admonitions not to be biased and explain the causes and also after-effects of biases. 2nd, it have to carry out instruction as to just how to stop prejudice (e.g. consider why upper and also reduced bounds might be incorrect). Third, it is useful just if participants have the substantive field of expertise to create specific responses.

Fixed value compared via resolved probcapability methods

A little number of research studies have actually examined whether the fixed-worth method (in which one must allocate probabilities to potential worths of a targain variable) or the fixed-probability strategy (in which one allocates worths of the targain variable to probabilities) affects overconfidence. In eliciting cumulative probcapacity judgements from students about forecast variables via which they were meant to have some familiarity (i.e. neighborhood temperature and also the Dow Jones), Abbas et al.160 found much less proof of overconfidence making use of the fixed-worth technique. However, Ferretti et al.148 found that this resulted in relatively little advancement in performance. Hence, although there is some proof that fixed-worth approaches may alleviate overconfidence, this is limited.

Face-to-challenge compared with digital elicitation

In one recent study161 it was discovered that face-to-face elicitation of energy demand with sectoral professionals led to reduced overconfidence than virtual elicitation. However before, this finding was not replicated in a current comparichild of face-to-confront and online SEE.60


The objective of this testimonial has actually been to synthesise existing expertise on the clinical effectiveness of various behavioral predisposition reduction techniques for expert elicitation, concentrating particularly on their potential usefulness in the context of HCDM. Although the efficacy of some of these approaches stays undertested, the complying with five referrals are sustained based on the easily accessible evidence:

Confidence intervals should not be elicited as a single-phase procedure, as doing so leads participants to focus on a narrowhead collection of salient possibilities. Instead, lower bounds, top bounds and median values have to be elicited individually. Eliciting reduced and also upper bounds prior to median worths might likewise proccasion participants from anchoring on median worths.
Participants have to be enabled to evaluate and revise their confidence intervals or probcapability distributions.
In selecting specialists, those through pronounced disputes of interemainder have to be excluded. However before, excluding all participants who may have strong feelings or vested interests in the outcome might cause the exemption of those individuals via the biggest specialization in the topic. Hence, it is vital to ensure that different approaches will be represented.
When suitable seed inquiries are accessible, these might be advantageous in giving practice feedearlier to participants on their performance and for this reason minimize overconfidence. However before, treatment have to be taken to encertain that all participants are familiar with the topic of these seed inquiries.

See more: Which Of The Following Is A Disadvantage Of Virtual Teams As Compared To Traditional Teams?

Bias training might mitigate biases, however just if this goes past straightforward warnings, and also describes what prejudice is and offers methods for staying clear of it.