Delphi Method

Collect and refine expert judgement without allowing hierarchy, confidence or meeting dynamics to erase meaningful disagreement.

In one minute

The Delphi Method uses a sequence of questionnaires:

1. experts answer independently;
2. a neutral coordinator aggregates estimates and reasons;
3. participants receive controlled feedback;
4. experts may revise their answers in the next round;
5. the study stops under a pre-defined rule.

Anonymity between participants reduces direct status pressure. Iteration enables reflection. Statistical summaries show the distribution rather than inventing a single group voice.

The purpose is informed judgement under uncertainty, not consensus at any cost.

Best for: emerging questions where relevant expertise is distributed and direct evidence is incomplete.
Avoid when: timely empirical data can answer the question, urgent action is required or the decision owner wants only endorsement.

The problem it addresses

In ordinary expert meetings, early speakers anchor the range, senior participants influence others and a polished narrative can appear more credible than a cautious estimate.

Delphi separates initial judgement, controlled feedback and revision. It preserves the range, outliers and reasons so the decision owner can see both convergence and irreducible uncertainty.

When to use it

Use Delphi when:

• historical evidence is sparse or not transferable;
• expertise is distributed across disciplines or locations;
• emerging AI effects or risks need bounded estimates;
• power differences could distort an open meeting;
• reasons for disagreement matter as much as the median;
• a decision needs a documented uncertainty range.

The question should be specific enough that experts are judging the same event, outcome and horizon.

When not to use it

Do not use Delphi:

• to replace available measurement or user research;
• when action cannot wait for multiple rounds;
• with a panel selected only for agreement;
• to force convergence by exposing identities or pressuring outliers;
• when participants lack relevant, diverse expertise;
• to present a median forecast as established fact.

Use Scenario Planning when several coherent futures matter more than one estimate.

Inputs required

• a specific question, outcome and time horizon;
• a decision owner and intended use of the results;
• panel-selection criteria and conflict disclosures;
• a neutral coordinator;
• an anonymous response process;
• round structure and controlled-feedback rule;
• summary statistics appropriate to the question;
• stopping criteria and a plan for reporting disagreement.

Step-by-step process

1. Define the decision question

Specify the event, population, measure and horizon. “Will AI change support?” is too broad. “What proportion of standard billing contacts could be safely resolved without agent intervention by December 2027 under the stated controls?” is testable enough for structured judgement.

2. Define expertise and recruit the panel

Select relevant but diverse experience. Include operational, technical, risk and affected-user knowledge where the question crosses those boundaries. Record conflicts and missing perspectives.

3. Design round one

Ask for an estimate or ranked judgement, confidence, assumptions, evidence and the strongest reason the answer could be wrong. Pilot the questionnaire for ambiguous wording.

4. Collect independent responses

Protect participant anonymity from one another. The coordinator may know identities for administration but should not expose them in feedback.

5. Aggregate without flattening

Report median or other suitable centre, range or interquartile spread, distribution, recurring reasons and materially different assumptions. Avoid editing minority rationales into the majority view.

6. Provide controlled feedback

Return the aggregate and anonymised reasons. Ask participants to review their estimate, explain any revision and state why they retain an outlying view if they do.

7. Run the next round

Repeat only when new feedback can improve the judgement. Two or three rounds are often more useful than pursuing artificial convergence indefinitely.

8. Apply the stopping rule

Stop when the planned number of rounds is complete, the distribution stabilises, new reasoning is negligible or the decision deadline arrives.

9. Report the result honestly

Show the distribution, movement between rounds, assumptions, persistent disagreements, panel composition and limitations. The decision owner then determines how the judgement will be used.

10. Compare with later evidence

When outcomes become observable, compare them with the forecast and update the organisation’s understanding of expert calibration.

AI automation lens

AI can help code open responses, detect recurring assumptions, produce draft summaries and check whether feedback fairly represents the panel. Strong controls include:

• retaining every original response;
• linking each summary theme to source responses;
• preventing the model from revealing identities;
• human review of minority and safety-critical rationales;
• versioning prompts, summaries and corrections;
• disclosing AI assistance to participants.

Do not let AI fabricate a consensus statement, infer missing responses or rewrite cautious uncertainty into confident language.

Visual model

Text alternative: a defined question goes to a diverse expert panel. Independent anonymous responses are aggregated into a distribution and reason themes. Controlled feedback returns to the panel for revision. A stopping rule produces a report that preserves range, assumptions and disagreement.

Interactive example

Scenario

A regulated insurer asks eight experts to estimate what share of routine claims could be processed autonomously by 2028 without increasing material customer harm.

Round-one estimates range from 15% to 80%. Technical experts assume complete digital evidence; claims operators know that 28% of routine claims arrive with ambiguous documents; compliance experts disagree about contestability requirements.

Your move

Design the feedback for round two without steering the panel toward the median.

Worked answer

The coordinator reports the median and interquartile range, then separates three assumptions: evidence completeness, allowed decision scope and customer appeal. Participants receive anonymised rationales from across the distribution and are asked to revise or retain their estimates with an explanation.

The report does not state that “experts agree automation is feasible.” It shows conditional estimates for document-complete claims, persistent disagreement about contestability and the evidence needed before a live pilot.

Facilitation notes

• Publish panel-selection and stopping rules before round one.
• Ask for evidence, assumptions and counterarguments—not numbers alone.
• Protect anonymity between panel members.
• Summarise the distribution, not only the average.
• Preserve minority reasoning when consequences are material.
• Avoid language that rewards movement toward consensus.
• Budget participant time and keep later rounds focused.

Expected output

A sound application produces:

• a specific expert-judgement question;
• transparent panel criteria and limitations;
• independent round-one estimates and reasons;
• controlled aggregate feedback;
• revision records across rounds;
• a documented stopping decision;
• a final distribution, assumptions and disagreements.

Common mistakes

1. Selecting only friendly experts. Agreement is designed into the panel.
2. Using vague questions. Participants estimate different events.
3. Reporting only the mean. Uncertainty and multimodal views disappear.
4. Forcing consensus. Legitimate disagreement is treated as failure.
5. Revealing authority cues. Anonymity loses its protective effect.
6. Letting AI smooth minority arguments. Important risk information is erased.
7. Running rounds without a stopping rule. Fatigue creates superficial convergence.

Quality checklist

• The question defines outcome, scope and horizon.
• Panel selection covers relevant perspectives.
• Conflicts and missing expertise are recorded.
• Initial responses are independent and anonymous.
• Feedback shows distribution, assumptions and reasons.
• Outliers are preserved without identity cues.
• Revisions and retained positions include rationale.
• A stopping rule is applied.
• The final report distinguishes judgement from fact.

Template

Question, outcome and horizon:

How the result will be used:

Panel criteria and gaps:

Round summary: median / range / themes / minority reasons

Round-two revision prompt:

Stopping rule and final limitations:

Knowledge check

After round one, estimates range from 20% to 80%. What is the strongest feedback?

A. Reveal the senior expert’s 70% estimate.
B. Tell participants to move closer to the average.
C. Show the distribution and anonymised assumptions, then invite justified revision.
D. Remove the two most distant responses.

Answer: C. Controlled feedback supports reflection without authority pressure or forced consensus.

Related tools

• Feeds: Kepner-Tregoe Decision Analysis, Scenario Planning
• Can rank with: Pairwise Comparison
• Tests claims through: PDCA/PDSA Cycle
• Not to be confused with: a focus group, ordinary survey or majority vote

References

1. Dalkey, N. C. The Delphi Method: An Experimental Study of Group Opinion. RAND Corporation, RM-5888-PR, 1969. RAND publication (opens in a new tab).
2. Linstone, H. A., and Turoff, M., eds. The Delphi Method: Techniques and Applications. Addison-Wesley, 1975.
3. Rowe, G., and Wright, G. “The Delphi Technique as a Forecasting Tool: Issues and Analysis.” International Journal of Forecasting, 15(4), 1999, pp. 353-375. DOI (opens in a new tab).
4. National Institute of Standards and Technology. Artificial Intelligence Risk Management Framework (AI RMF 1.0). NIST AI 100-1, 2023. Official publication (opens in a new tab).

Sources reviewed 12 August 2026.