Panel · Template

Estimate Elicitation.

LiveEstimate Elicitation

Ask specialists to estimate one uncertain number from the same evidence, each giving a plausible low and high before the central value, and get the estimates side by side, why they differ, the pooled range, and what would narrow it.

You getAn elicitation report of about four pages (Standard)

DepthsLight (three seats), Standard (four) or Full (five)

Written byAI specialists, each judging from the shared dossier

SourcesEvery anchor cited or marked as a gap

Add it to your library to reuse from New → From template. Edit the saved copy in your library on the service form.

When to use it

You need a number that no dataset gives you, such as a probability, a rate or a share, and you want to see the spread of informed judgment and its basis, not a single guess.

Who it is for. Leads who must plan on an uncertain quantity: development, clinical operations, regulatory, brand, access and evidence leads preparing a forecast, a plan or a business case.

Why one expert guess is not an estimate

Experts tend to give intervals that are too narrow, and opening with a single central guess anchors the range. Structured elicitation protocols address this by defining the quantity precisely, sharing an evidence dossier, asking for plausible low and high before the central value, and aggregating by a stated method. This template follows that order, reports each seat’s interval and the pooled range, and never narrows the range to look decisive.^{7Aggregating probability intervals to improve calibration. Judgment and Decision Making, 14(1).,8Hemming, V., Burgman, M. A., Hanea, A. M., McBride, M. F. and Wintle, B. C. A practical guide to structured expert elicitation using the IDEA protocol. Methods in Ecology and Evolution, 2018;9(1):169–180.}

What you get

The quantity, defined. The quantity in one sentence, with its unit, population, time horizon and the event at which the true value would become known.^{8Hemming, V., Burgman, M. A., Hanea, A. M., McBride, M. F. and Wintle, B. C. A practical guide to structured expert elicitation using the IDEA protocol. Methods in Ecology and Evolution, 2018;9(1):169–180.}

An evidence dossier. The base rates, comparators and prior results the seats may use, each with its source and weakness, so that judgment differs on what the evidence implies and not on what each seat remembers. Pharmaceutical teams have used the same practice to make their beliefs about success explicit for investment boards.^{1Holzhauer, B. et al. Eliciting judgements about dependent quantities of interest: the SHELF extension and copula methods illustrated using an asthma case study. arXiv:2102.02852.,2O’Hagan, A. SHELF extension at Novartis: Pharmaceutical Statistics paper abstract.,3Formal prior elicitation in drug development at GSK. arXiv:1708.09823.,9University of Sheffield. SHELF: the Sheffield Elicitation Framework.}

Seat judgments in a fixed order. Each seat gives its anchors, a plausible low and high, then a central value, its confidence that the interval contains the truth, and its reasoning.^{4A review of prior knowledge elicitation: biases, task design, seed variables and aggregation. arXiv:2112.01380.,7Aggregating probability intervals to improve calibration. Judgment and Decision Making, 14(1).}

Estimates side by side, and why they differ. A table of every seat’s interval and central value, then the reasons for the differences. At Full depth, revised estimates after a rebuttal round.^{8Hemming, V., Burgman, M. A., Hanea, A. M., McBride, M. F. and Wintle, B. C. A practical guide to structured expert elicitation using the IDEA protocol. Methods in Ecology and Evolution, 2018;9(1):169–180.}

The pooled range, and what would narrow it. The pooled range with the method stated, and a plain statement that it is a synthesis of AI judgments and not a calibrated distribution. Every report closes with the evidence that would narrow the range.^{5A mapping study of expert elicitation for clinical trial parameters. arXiv:2508.06288.,6A comparison of the Classical Method and SHELF for prior elicitation. arXiv:2001.11365.}

What is in it, by length

Section	Brief	Standard	Comprehensive
The quantity and how it is judgedThe exact quantity and when it resolves	●	●	●
Evidence dossierThe shared evidence and its weaknesses	–	●	●
Seat judgmentsEach seat’s anchors, interval and reasoning	●	●	●
Estimates side by sideEvery interval and central value in one table	●	●	●
Why the seats differThe reasons for the spread	–	●	●
Revised estimates after rebuttalHow the estimates moved after challenge	–	–	●
The pooled range and what it is notThe pooled range, its method and its limits	●	●	●
What would narrow the rangeThe evidence that would reduce the uncertainty	●	●	●

A Light elicitation has three seats and about two pages. Standard has four seats and adds the dossier and the reasons for the spread. Full has five seats and adds a rebuttal round and revised estimates.

Choosing a length

Choose Standard for most quantities. Choose Full when the number drives a large commitment, so that you see the estimates after challenge.

The standards and practice behind it

The structure and the checks come from published protocols for structured expert elicitation and from studies of calibration and aggregation. These are the references we drew on and what we took from each. None of their authors has reviewed the template.

What we took	From	Where it shows
Investigate, discuss, estimate privately, then aggregate; clarify what each question means before estimating	The IDEA protocol for structured expert elicitation (Hemming and colleagues, 2018)^{8Hemming, V., Burgman, M. A., Hanea, A. M., McBride, M. F. and Wintle, B. C. A practical guide to structured expert elicitation using the IDEA protocol. Methods in Ecology and Evolution, 2018;9(1):169–180.}	The quantity defined first, with a resolution criterion
Prepare an evidence dossier early and share it with the experts	Holzhauer and colleagues on expert elicitation of probability of success with dependent quantities (arXiv)^{1Holzhauer, B. et al. Eliciting judgements about dependent quantities of interest: the SHELF extension and copula methods illustrated using an asthma case study. arXiv:2102.02852.}	The evidence dossier before the judgments
Use documents, templates and a facilitator so that elicitation is competent and repeatable	The Sheffield Elicitation Framework (SHELF)^{9University of Sheffield. SHELF: the Sheffield Elicitation Framework.}	A fixed order for each seat’s judgment
Combine intervals from several judges to reduce overconfidence, without taking the envelope of the widest bounds	Aggregating probability intervals to improve calibration (Judgment and Decision Making)^{7Aggregating probability intervals to improve calibration. Judgment and Decision Making, 14(1).}	Individual intervals and a pooled range, never narrowed to look decisive
Compare aggregation methods and report which one was used	A comparison of the Classical Method and SHELF for prior elicitation (arXiv)^{6A comparison of the Classical Method and SHELF for prior elicitation. arXiv:2001.11365.}	The method named in the pooled range section
Document the elicitation so that it can be reviewed	A mapping study of expert elicitation for clinical trial parameters (arXiv), and EFSA guidance on expert knowledge elicitation^{5A mapping study of expert elicitation for clinical trial parameters. arXiv:2508.06288.,10European Food Safety Authority. Guidance on expert knowledge elicitation in food and feed safety risk assessment. EFSA Journal, 2014;12(6):3734.}	Who judged, on what evidence, with what definition
Weight experts by performance only where a track record on seed questions exists	Cooke’s Classical Model, as summarized in the comparison study^{6A comparison of the Classical Method and SHELF for prior elicitation. arXiv:2001.11365.}	Equal weighting for AI seats, because no calibration record exists

What it will not do

It is not a calibrated probability distribution. The pool is a synthesis of AI judgments, and the report says so.

It does not weight seats by performance. No calibration record exists for the AI seats.

It does not narrow the range to look decisive.

References

[1] Holzhauer, B. et al. Eliciting judgements about dependent quantities of interest: the SHELF extension and copula methods illustrated using an asthma case study. arXiv:2102.02852. Read in the template research via an ar5iv copy: evidence dossier, workshop format, and a simulated probability of success of 4 percent or 41 percent depending on whether both targets are required.
[2] O’Hagan, A. SHELF extension at Novartis: Pharmaceutical Statistics paper abstract. Abstract page read in the template research.
[3] Formal prior elicitation in drug development at GSK. arXiv:1708.09823. Abstract read; the body was unreadable.
[4] A review of prior knowledge elicitation: biases, task design, seed variables and aggregation. arXiv:2112.01380. Read in the template research via an ar5iv copy.
[5] A mapping study of expert elicitation for clinical trial parameters. arXiv:2508.06288. Read in the template research via an ar5iv copy: equal weighting and linear pooling most common, reporting inconsistent.
[6] A comparison of the Classical Method and SHELF for prior elicitation. arXiv:2001.11365. Read in the template research via an ar5iv copy.
[7] Aggregating probability intervals to improve calibration. Judgment and Decision Making, 14(1). Read in the template research.
[8] Hemming, V., Burgman, M. A., Hanea, A. M., McBride, M. F. and Wintle, B. C. A practical guide to structured expert elicitation using the IDEA protocol. Methods in Ecology and Evolution, 2018;9(1):169–180. Described on the journal’s blog, read in the template research; the paper was not read.
[9] University of Sheffield. SHELF: the Sheffield Elicitation Framework. Home page read; it did not describe the stages.
[10] European Food Safety Authority. Guidance on expert knowledge elicitation in food and feed safety risk assessment. EFSA Journal, 2014;12(6):3734. The text could not be read; the definition of elicitation as a documented and reviewable process is as reported in search results.

Links checked 9 October 2026

These are the sources the template drew on. None of their authors or issuers has reviewed it.

About the Panel service

Each panelist answers in their own voice, then the discussion is drawn together. You get the question and who was in the room, every position in full, what they agreed on, where they split, and what follows. Disagreements are written down rather than smoothed into a consensus nobody actually holds, because the split is usually the part you needed.

Use cases for the Estimate Elicitation

7 ready-to-run use cases. Each opens its own page with the question, the specialists, the standards and the references.

POLiveProbability of Success: Phase 3 EntryWhat is the probability that this phase 2 asset meets its pivotal endpoint and obtains approval, given the data to date?Estimate ElicitationCorporate Development

Open Open →

PMLivePeak Market Share RangeWhat share of treated patients will the product hold at peak, five years after launch, in its lead indication?Estimate ElicitationBrand & Marketing

Open Open →

ERLiveEnrollment Rate ForecastHow many patients per site per month will this rare-disease trial enroll once all sites are active?Estimate ElicitationClinical Operations

Open Open →

TELiveTreatment Effect RangeWhat size of treatment effect on the primary endpoint should the confirmatory trial plan for, given the phase 2 result?Estimate ElicitationMedical Affairs

Open Open →

TTLiveTime to Approval RangeHow many months from submission to approval should we plan for in the lead market?Estimate ElicitationRegulatory

Open Open →

NPLiveNet Price Erosion RangeBy what percent will the net price fall over the first three years after launch, given payer and policy conditions?Estimate ElicitationMarket Access

Open Open →

ARLiveAdoption Rate for a Digital TherapeuticWhat share of eligible clinics will prescribe the digital therapeutic within twelve months of launch?Estimate ElicitationInnovation

Open Open →

To edit a saved template, open New → Panel → From template, or the library rail on the service form.

When to use it

You need a number that no dataset gives you, such as a probability, a rate or a share, and you want to see the spread of informed judgment and its basis, not a single guess.

Who it is for. Leads who must plan on an uncertain quantity: development, clinical operations, regulatory, brand, access and evidence leads preparing a forecast, a plan or a business case.

Why one expert guess is not an estimate

What you get

What is in it, by length

Section	Brief	Standard	Comprehensive
The quantity and how it is judgedThe exact quantity and when it resolves	●	●	●
Evidence dossierThe shared evidence and its weaknesses	–	●	●
Seat judgmentsEach seat’s anchors, interval and reasoning	●	●	●
Estimates side by sideEvery interval and central value in one table	●	●	●
Why the seats differThe reasons for the spread	–	●	●
Revised estimates after rebuttalHow the estimates moved after challenge	–	–	●
The pooled range and what it is notThe pooled range, its method and its limits	●	●	●
What would narrow the rangeThe evidence that would reduce the uncertainty	●	●	●

The standards and practice behind it

What we took	From	Where it shows
Investigate, discuss, estimate privately, then aggregate; clarify what each question means before estimating	The IDEA protocol for structured expert elicitation (Hemming and colleagues, 2018)^{8Hemming, V., Burgman, M. A., Hanea, A. M., McBride, M. F. and Wintle, B. C. A practical guide to structured expert elicitation using the IDEA protocol. Methods in Ecology and Evolution, 2018;9(1):169–180.}	The quantity defined first, with a resolution criterion
Prepare an evidence dossier early and share it with the experts	Holzhauer and colleagues on expert elicitation of probability of success with dependent quantities (arXiv)^{1Holzhauer, B. et al. Eliciting judgements about dependent quantities of interest: the SHELF extension and copula methods illustrated using an asthma case study. arXiv:2102.02852.}	The evidence dossier before the judgments
Use documents, templates and a facilitator so that elicitation is competent and repeatable	The Sheffield Elicitation Framework (SHELF)^{9University of Sheffield. SHELF: the Sheffield Elicitation Framework.}	A fixed order for each seat’s judgment
Combine intervals from several judges to reduce overconfidence, without taking the envelope of the widest bounds	Aggregating probability intervals to improve calibration (Judgment and Decision Making)^{7Aggregating probability intervals to improve calibration. Judgment and Decision Making, 14(1).}	Individual intervals and a pooled range, never narrowed to look decisive
Compare aggregation methods and report which one was used	A comparison of the Classical Method and SHELF for prior elicitation (arXiv)^{6A comparison of the Classical Method and SHELF for prior elicitation. arXiv:2001.11365.}	The method named in the pooled range section
Document the elicitation so that it can be reviewed	A mapping study of expert elicitation for clinical trial parameters (arXiv), and EFSA guidance on expert knowledge elicitation^{5A mapping study of expert elicitation for clinical trial parameters. arXiv:2508.06288.,10European Food Safety Authority. Guidance on expert knowledge elicitation in food and feed safety risk assessment. EFSA Journal, 2014;12(6):3734.}	Who judged, on what evidence, with what definition
Weight experts by performance only where a track record on seed questions exists	Cooke’s Classical Model, as summarized in the comparison study^{6A comparison of the Classical Method and SHELF for prior elicitation. arXiv:2001.11365.}	Equal weighting for AI seats, because no calibration record exists