Calibrate Promotion Reviews against Shared Criteria and Evidence

How can promotion reviewers resolve differences in evidence and interpretation under shared expectations without treating votes, confidence or invented averages as proof?

Substantively revised October 23, 2026. Author interpretation, documented cases and hypothetical examples are distinguished below.

Three reviewers can agree on a promotion case while sharing the same unsupported assumption. Two reviewers can disagree because one has relevant evidence the other has not seen. Agreement alone therefore cannot establish that a review is well supported. A useful calibration asks why the judgments differ, whether the same criteria were applied and what evidence justifies the final recommendation.

Direct answer

Calibrate promotion reviews by making the applicable criteria, evidence and reasoning visible before resolving differences. Distinguish factual gaps, attribution disputes and different interpretations of the same expectation. Use the actual authorized process for recommendations and decisions, preserving material counterevidence and the reasons for changes. Do not treat majority opinion, confidence or a forced average as proof of correctness. A structured discussion can improve inspectability without guaranteeing fairness, eliminating bias or authorizing a promotion by itself.

Key takeaways

Calibration compares judgments under a shared standard; it should not quietly create a new standard after reviewers see the candidate. If the actual criterion is unclear, resolve that interpretation through the appropriate process and apply it consistently. Changing the meaning for one person while retaining the old meaning for another makes the resulting agreement difficult to interpret.

Published processes provide concrete examples without creating one universal rule. Sourcegraph's public calibration guide asks managers to connect examples with role expectations and gives participants a clarifying role. OPM's structured-interview guide separates independent review from discussion of discrepancies. Google's re:Work guidance describes shared interview criteria and recorded feedback. The interview sources concern hiring; their procedural distinctions are used here as explicit analogies, not as evidence that a particular promotion method has been validated. [S1] [S2] [S3]

The final decision needs a reason, not only a status. Preserve which evidence changed the interpretation and which uncertainty remains. A reviewer can legitimately revise a view after learning something relevant. That change should be explainable without inventing unanimity, suppressing counterevidence or claiming a numerical confidence level that the process does not establish.

Definitions and relationships

A criterion states what is being assessed. Evidence concerns the work or information used to assess it. An interpretation connects evidence to the criterion. A recommendation proposes an outcome. An authorization makes a decision under the actual process. These are different stages. A discussion can improve interpretation while leaving recommendation or approval authority elsewhere.

A calibration discrepancy is a difference that needs explanation. It can arise from different facts, different attribution, different criteria or different interpretations of the same criterion. Those causes require different work. Asking everyone to choose the middle score can conceal the cause without resolving it, just as a vote can settle a procedural outcome without establishing that the underlying factual claim is correct.

Counterevidence is relevant information that limits or challenges a claim. It is not automatically a reason to reject the candidate, and it should not be removed merely because it complicates a positive case. The useful question is how it changes what the evidence supports under the applicable expectations. Preserve its scope rather than turning one contrary episode into a verdict about every responsibility.

The problem: reviewers exchange conclusions before reasons

Imagine a hypothetical three-person review. Two reviewers recommend proceeding and one asks for clarification. The group treats the two-to-one count as sufficient proof that the case is strong. It has not yet established whether the reviewers saw the same work, used the same criterion or understood the individual's contribution in the same way. The arithmetic describes opinions, not evidence quality.

Suppose the reviewer asking for clarification knows that another colleague owned a key decision attributed to the candidate. That information could materially change the case. Alternatively, the concern could result from a misunderstanding already resolved by the work record. Neither possibility can be evaluated by counting the initial recommendations. The discussion needs the underlying claim and relevant evidence.

The panel size and count are invented. No real promotion case or reviewer behavior was tested. The example illustrates why agreement, disagreement and correctness should not be collapsed into one measure. A well-supported conclusion can be unanimous or non-unanimous under the actual process, while an unsupported one can look persuasive when everyone repeats it confidently.

Root cause: shared labels hide different mental standards

Reviewers can use the same phrase, such as independent delivery, while meaning different things. One may require the candidate to define the problem; another may mean completing assigned work without daily help. A case can appear disputed when the underlying work is agreed but the criterion differs. Identify that difference before asking the candidate to supply more examples.

Other disagreements concern facts rather than standards. The reviewers may differ about who made a decision, whether a change was accepted or what result was observed. A factual question should be checked against the relevant record. It should not be resolved by the senior person's confidence or by rewording the criterion until the uncertainty no longer appears in the summary.

Social dynamics can further compress discussion. A confident first speaker can shape the terms of the conversation before others inspect the evidence. The article does not quantify that effect or claim a particular procedure eliminates it. It uses the distinction between independent initial reasoning and later discussion to preserve information about why reviewers reached their views.

Framework: the Calibration Difference Record

The author's Calibration Difference Record has five components: Shared Expectation, Initial Reasoning, Difference Type, Evidence Resolution and Final Rationale. Shared Expectation identifies the applicable criterion. Initial Reasoning preserves each relevant view before reconciliation. Difference Type classifies the actual disagreement. Evidence Resolution records what was checked or clarified. Final Rationale explains the resulting recommendation and remaining limits under the authorized process.

Start with Shared Expectation. Identify the actual role, level and criterion being discussed without inventing a new rubric. Keep its source and scope visible. If the criterion needs interpretation, record the interpretation and the appropriate authority for it. Do not quietly change the requirement to favor or disadvantage one candidate after seeing their evidence.

Initial Reasoning records the substantive basis of a view, not merely a yes or no. A concise statement can identify the work episode and what it is thought to demonstrate. Where the actual process permits independent preparation, preserve that reasoning before group discussion. This is an author's procedural suggestion, not a universal rule or a claim that every employer's panel must operate identically.

Difference Type asks whether the issue is factual, attribution-related, criterion-related or interpretive. It can include more than one type. The point is diagnostic usefulness rather than perfect classification. If one reviewer has not seen a relevant record, obtain appropriate access rather than treating the disagreement as a personality conflict. If the standard differs, more raw material may not resolve the actual question.

Evidence Resolution states the check performed and what it establishes. A project record might clarify contribution; a relevant framework passage might clarify a responsibility; another factual question may remain unanswered. Do not rename a discussion as verification when no supporting information was obtained. Keep unknown status where the actual evidence remains insufficient.

Final Rationale records the supported interpretation, recommendation and unresolved conditions. Separate the group's view from final authorization if the actual process distinguishes them. Preserve why an earlier view changed. A useful rationale should let an authorized reader understand the decision without relying on who spoke most forcefully or how many people initially agreed.

Application: distinguish a missing fact from a different criterion

In the invented three-reviewer case, suppose all reviewers agree the candidate led an implementation but disagree about who defined the cross-team tradeoff. First identify whether the record establishes that action. If it does, the remaining discussion concerns how that action maps to the actual criterion. If it does not, preserve the factual limit rather than awarding or denying responsibility through a vote.

Now suppose two reviewers define the criterion as leading implementation while the third defines it as independently setting the technical direction. That is a standards question. Clarify the applicable expectation before interpreting the case. Asking the candidate for two more implementation examples could add material while leaving the standard mismatch exactly where it was.

The two additional examples are hypothetical, not a prescribed evidence count. The useful outcome is a diagnosis of what needs resolution. A calibration meeting can save effort by directing the next check to the real difference instead of requiring a larger packet whenever people disagree. That economy comes from a clearer question, not a lower standard.

Case study: a positive claim survives with a narrower scope

Imagine a hypothetical candidate described as owning an entire cross-team initiative. During calibration, another reviewer identifies that the candidate owned rollout planning while a colleague defined the architecture. The candidate's rollout contribution may remain substantial. The correction is to narrow the claim and assess that work against the relevant expectation, not to erase the contribution or retain an inaccurate account for convenience.

The reviewers inspect the authorized work record and update the rationale to describe the actual responsibility. They then distinguish what that episode supports from any broader expectation that remains unresolved. A favorable or unfavorable final outcome is not predetermined by the correction. The actual process decides how the supported contribution relates to the role, rather than treating any factual amendment as automatic failure.

This case is invented. No actual employee, calibration meeting or internal record was inspected. Its practical lesson is that counterevidence can improve a case's accuracy without making every contribution worthless. Calibration should preserve the strongest supported claim, not the broadest claim that can survive a brief conversation without challenge.

Failure example: averaging replaces the unresolved discussion

A separate hypothetical process produces initial ratings of two, four and six under a fictional scale. Their average is four. The arithmetic does not establish that four is the correct assessment because the disagreement's cause is unknown and the actual process may not authorize averaging. One reviewer could have used another criterion or misunderstood the evidence.

This example does not reject formal scoring systems. If the actual process defines an aggregation rule, use that rule and retain the supporting evidence. The failure addressed here is inventing a middle value to make the meeting end while leaving the material discrepancy unexplained. A numerical result can be reproducible and still fail to answer the question the review was meant to assess.

The correction is to inspect the reasons for the ratings and follow the real process for resolving them. If a factual issue remains unanswered, identify it. If reviewers legitimately disagree under a shared criterion, record the relevant reasoning and decision authority. Do not manufacture unanimity or claim that a neat mean has eliminated uncertainty.

Published review procedures illustrate structure, not guaranteed fairness

Sourcegraph's public guide connects concrete examples with role expectations and asks participants to challenge or clarify the assessment. It also describes its own recommendation and decision process. The document establishes that published approach; this article did not observe whether current internal meetings follow it or measure its effect on fairness. Do not turn a stated process objective into a verified outcome. [S1]

OPM's September 2008 structured-interview guide describes independent initial evaluation and later discussion of discrepancies with supporting examples. This is a hiring-interview method, used here to illustrate the difference between an initial view and evidence-based reconciliation. It is not a current private promotion policy or a requirement that every promotion panel adopt the same procedure. [S2]

Google's re:Work structured-interview guidance describes consistent predetermined qualifications, recorded feedback and shared rubrics. Again, the context is hiring. The article uses those mechanisms as an analogy for shared criteria, without importing Google's reported research outcomes or claiming that they validate this author's promotion record. The actual employer's process and authority remain decisive. [S3]

Counterargument: experienced reviewers should use judgment

They should use relevant judgment. The question is whether its basis can be explained and checked against the applicable expectations. A clear record need not remove professional discretion or force every contribution into a rigid formula. It asks reviewers to identify what they saw and why it supports the recommendation, making expertise useful to people who were not present for the work.

Not every minor disagreement needs extensive investigation. Focus on differences that could change the assessment or reveal inconsistent criteria. A short clarifying question may be enough. A material attribution dispute can require a more careful check. Match the effort to the consequence rather than treating either rapid consensus or exhaustive documentation as universally superior.

The tradeoff is time versus unresolved error. A meeting that ends quickly can leave the decisive claim unexamined; a meeting that explores irrelevant details can waste effort without improving the decision. The difference record keeps attention on the actual criterion, evidence and discrepancy. Its purpose is a reviewable rationale, not a longer meeting or a predetermined outcome.

Common mistakes

Do not equate majority opinion with correctness or confidence with evidence. Do not change the criterion for one candidate without acknowledging the change and applying the actual process. Do not let a title or seniority settle a factual question that relevant records could clarify.

Do not average away a discrepancy unless the actual method authorizes the operation, and even then preserve consequential evidence questions. Do not suppress counterevidence merely to keep a positive story consistent. Do not treat every correction as a reason to reject the person rather than reassess the actual supported contribution.

Do not claim that a structured meeting guarantees fairness, removes bias or establishes legal compliance. The article supplies no measured panel outcome and no legal assessment. Keep confidential evidence within authorized access and separate the recommendation from final approval where the actual process does so.

Action plan

Identify the actual criterion and role scope before comparing reviewer conclusions. Ask for concise reasoning connected to inspectable work. Preserve relevant initial views where the process permits it, then identify whether a difference concerns facts, attribution, standards or interpretation. Avoid collecting more examples until the actual missing question is clear.

Resolve material factual questions through appropriate evidence and clarify standards through the relevant authority. Record what changed and what remains unknown. Do not call an unverified discussion a completed evidence check. Keep a candidate's supported contribution even when a broader claim must be narrowed.

Write the final rationale and recommendation under the actual decision process, preserving consequential limits and any required next step. The useful outcome is a traceable assessment rather than a unanimous impression. It supports the authorized decision without pretending that agreement alone proves the result.

FAQ

Does a two-to-one vote prove a case is correct?

No. It describes initial opinions under the invented example. Reviewers may have different evidence or criteria. Examine the reasons and follow the actual process rather than treating the count as proof of the underlying work claim.

What does calibration compare?

It compares judgments under the applicable shared expectations and evidence. It should identify material discrepancies and their causes. It is not an opportunity to create an unofficial criterion after seeing one candidate's case or to substitute a group impression for the actual role process.

Should all reviewers see exactly the same raw material?

Use the actual process and authorized access. The essential point is to identify consequential information gaps rather than assume everyone has the same knowledge. A relevant record may need clarification or appropriate access before a discrepancy can be resolved.

How can disagreement be useful?

It can reveal a missing fact, attribution issue or inconsistent interpretation. The goal is to understand the difference, not reward disagreement for its own sake. Once the actual issue is identified, the next check can be narrower and more useful.

What does Sourcegraph's public guide illustrate?

It connects tangible examples to role expectations and describes clarifying participation in calibration. This article did not observe its internal implementation or measure fairness outcomes. Treat the public guide as evidence of its stated approach, not a guaranteed result. [S1]

Why use a structured-interview guide in a promotion article?

OPM's guide provides an explicit example of independent initial evaluation followed by discussion of discrepancies. The hiring context is an analogy, not a private promotion rule or validation of this author's method. Preserve that boundary. [S2]

What does Google's re:Work guidance add?

It illustrates shared predetermined criteria, recorded feedback and rubrics in structured hiring interviews. The article uses those procedural distinctions without claiming that hiring research proves a particular promotion review is fair or correct. [S3]

Can a reviewer change their mind?

Yes. A relevant factual correction or clarified interpretation can justify a change. Record the reason so the final rationale remains traceable. Revision is different from suppressing an initial concern to manufacture agreement without resolving it.

Should counterevidence automatically defeat a case?

No. It can narrow a claim while leaving a substantial supported contribution. Assess what the corrected evidence demonstrates under the actual criterion. Do not preserve an inaccurate broad claim or erase useful work merely because the narrative needs amendment.

Can ratings of two, four and six simply become four?

Their fictional average is four, but that does not resolve the discrepancy or authorize averaging. Use the actual method where scoring exists and preserve material evidence questions. An invented mean should not replace the discussion the assessment requires.

Does the method reject formal scoring?

No. It rejects unauthorized aggregation and unexplained reasoning. A real scoring system can be used according to its actual rules. The supporting evidence still needs enough clarity for the result to be interpreted responsibly.

Must every disagreement produce a long meeting?

No. Focus on differences that could change the decision or reveal inconsistent standards. A concise factual clarification can be enough. Match the investigation to the material issue rather than treating meeting length as a measure of review quality.

Were any real promotion panels tested?

No. All reviewer counts, ratings and candidate cases are hypothetical. The cited documents support scoped procedural examples. No live panel, private employee evidence or measured fairness outcome was evaluated for this guide.

Does calibration itself authorize promotion?

Only the actual process establishes decision authority. A calibration recommendation can be separate from final approval. This method does not create a role, guarantee an outcome or establish legal compliance.

What should the final rationale preserve?

State the applicable expectation, relevant evidence, material discrepancy, resolution and remaining limits. Explain any changed view and distinguish recommendation from authorization. A reader should understand why the conclusion follows without relying on majority confidence alone.

Recommended books

Thinking in Systems by Donella H. Meadows helps examine how a group's conclusion can depend on shared assumptions rather than independent evidence. Apply its attention to relationships when agreement is being treated as proof that every underlying claim has been checked.

High Output Management by Andrew S. Grove is useful for considering the purpose and discipline of managerial review. Read it to sharpen the connection between evidence, expectations and decisions, while retaining the employer's actual promotion authority and procedures.

The Making of a Manager by Julie Zhuo helps readers think about clear feedback and judgment in people decisions. Use it to improve the review conversation without treating management advice as a fairness guarantee or a substitute for checking a material factual claim.

Further exploration

Construct a hypothetical disagreement in which reviewers share the facts but use different criteria. Then construct one in which the criterion is shared but attribution differs. Identify the different next checks. Finally make the initial recommendations unanimous and ask which supporting claims still need evidence.

The durable decision is to resolve the relevant difference rather than merely collect agreement. Shared criteria, inspectable evidence and a traceable rationale make calibration useful within the actual process. They do not turn a vote into truth or remove the need for authorized judgment.

Primary evidence sources

Related concepts: systems thinking, wealth structure, and the survival loop. Read the editorial policy and citation guide.