Outgrace, home
LeadershipHow-to guide

Running a 360-degree review that changes behaviour, not rankings

A 360 built for development starts with one behaviour, enough raters and a debrief with a plan. Here is how HR and senior leaders can design and run one.

A 360-degree review that ends in a ranking teaches people to protect their scores. If the purpose is development, the design has to start from what the recipient will do differently on Monday, not from how many colleagues can be asked to fill in a form. The only study on this format in the sources our researcher reviewed is a 1998 paper by Greguras and Robie, published in the Journal of Applied Psychology. It covered 153 managers and was about whether the ratings were reliable, not whether behaviour changed afterwards. That is still useful. It shows where a review is likely to fail before it runs, and it gives HR a reason to design for measurement quality from the start.

Before you start

Three conditions must hold before anyone writes a question. None of them is a finding from the study. They are our judgement about what a development review needs, and each one is cheap to check. If any of them fails, the review will drift toward appraisal, whatever the survey says on its cover page.

  • The recipient has agreed to take part, and knows in advance exactly who will see the individual results and in what form.
  • The organisation has confirmed in writing whether the results feed pay, promotion or succession. If they do, run a different process for development and keep this one out of those decisions.
  • A named person, usually an HR partner or an experienced coach, is free to meet the recipient within two weeks of the report, and that meeting is already in both diaries.

The questions come after these conditions, not before. Write them as observable behaviours a rater could have seen in the last six months. A question about whether a manager is a strategic thinker produces opinions. A question about whether the manager asks the team to challenge a plan before it is approved produces evidence. Aim for a short instrument, perhaps twelve to twenty items, and tie every item to the one or two behaviours you chose in step one. Anything else is noise that the recipient will have to sort through alone.

  1. Step 1

    Pick one behaviour to change

    Choose one or two behaviours the recipient can practise in the next quarter, such as asking for dissent before a decision is final. Most reviews fail by listing twenty competencies, which gives the recipient twenty things to ignore. Write the behaviour in plain words, and check that a rater could describe a recent instance of it. If nobody can, the behaviour is too abstract to develop.

  2. Step 2

    Recruit more raters than feels necessary

    Greguras and Robie found that more raters per manager were needed than are typically used to reach acceptable reliability. Ask for a count the recipient's own team can supply without strain, then check it against the results. If one rater group is thin, say so in the report. Do not average three opinions and present the result as if it came from a panel.

  3. Step 3

    Hold the conditions constant

    The same study found that supervisors were most reliable when the number of raters and items was held constant, and that the gap between peer and subordinate ratings was trivial under those conditions. Under common practical conditions, it ranked peers highest, then subordinates, then supervisors. The two findings differ because the conditions differ, not because one contradicts the other. Use the same number of raters and items for each group, so that a difference in the results reflects the pe

  4. Step 4

    Show the spread, not only the average

    The study reported that a large share of error variance came from rater main effects and from the rater-by-ratee effect. Put simply, part of what looks like a finding about the manager is a finding about who was asked. Show the recipient the range of scores for each item, along with the number of raters behind each figure. A single average with no spread invites a ranking reading, and it hides the disagreement that is often the most useful part of the report.

  5. Step 5

    Debrief in person around one plan

    Give the report to the recipient first, and stay in the room while they read it. Ask which two comments surprised them, which one they think is accurate, and what they would change over the next ninety days. Finish with a short written plan that the recipient owns and a named person who will check it. Our sources contained no published debrief method, so this sequence is our judgement, not a tested procedure. Treat it as a draft to refine with your own data.

  6. Step 6

    Re-rate the same behaviour on a fixed date

    Schedule a shorter second round on the same behaviour, six to nine months later, using the same raters where you can. The interval is our suggestion; the sources do not say what gap works best. A recipient who never sees a second data point learns that the first report was a performance rather than an input. The second round also tells you whether the instrument is measuring something that moved.

Who runs the review matters as much as what it asks. In our experience, the best results come when an HR partner or external coach owns the process and the recipient's line manager is kept informed but not in charge of the report. A manager who sees the results first tends to edit the message before it reaches the person it is about. Ask the recipient whether they want their manager at the debrief at all. Some people find it helpful. Others find it hard to speak freely in front of the person who will rate their next year.

Managers rated in Greguras and Robie's 1998 reliability study, Journal of Applied Psycholo
153
Year Greguras and Robie published their study of 360 rating reliability in managerial deve
1998

The mistakes that turn development into ranking

The most common mistake is attaching the review to pay without saying so. Raters notice, even when nobody announces it. They begin to score for the outcome, and recipients begin to defend their numbers rather than learn from them. The second mistake is a long instrument. A form with forty items produces fatigue, and fatigue produces the same score for every question, which looks like consensus and is not. The third is publishing a league table of rater groups or teams. That invites comparison between people whose roles, exposure and sample sizes differ. The fourth is the debrief that never happens. A report sent by email with a link to a training course is not a development process, and it teaches the recipient that the review was an administrative step. Finally, many organisations ask for results and then never ask whether the behaviour changed. Without a second round, nobody can say whether the exercise did anything. The study we reviewed addresses the reliability of ratings and the design of systems. It does not measure change over time, so the second round is the only way to know.

Can one review serve both development and appraisal?

Our advice is no. Once a rating feeds pay or promotion, raters start scoring for the outcome, and recipients start defending their numbers. The study we reviewed is about reliability in managerial development settings, and it does not test what happens when ratings carry consequences. Keep the two processes apart. If your contracts or policies tie ratings to pay, take that question to an employment lawyer before the survey opens.

How many raters is enough?

Greguras and Robie found that more raters per manager were needed than are typically used to reach acceptable reliability. The abstract we reviewed does not give a single number, so the count has to be tested rather than assumed. A practical check is to remove one rater from a group and see whether the item scores move much. If they do, add raters before you publish.

Should we trust peer ratings more than subordinate ratings?

The study does not support a blanket answer. It found supervisors most reliable when rater and item numbers were held constant, and it found peers ranked highest under common practical conditions. The two rankings depend on the conditions set. Hold the design constant across groups and judge the results on their spread, not on the reputation of the rater group.

Should the recipient see verbatim comments?

In our view, yes, with names removed and with any comment that would identify a small rater group handled by the HR partner before it reaches the recipient. The sources we reviewed do not address anonymity or confidentiality, so treat this as a policy to agree with your legal team, not a settled rule. Decide the rule before the survey is sent, and tell raters what it is.

What if the recipient disputes the results?

Allow them to dispute the reading, not the data. If the spread shows that one rater group disagrees with the rest, say so and explain what that might mean for the behaviour being discussed. The sources do not describe a dispute process, so design one before launch: who the recipient can talk to, how long they have, and what happens to a challenged item.

This week, pick one manager who is due a review and write down the single behaviour you want to see change. Then count the raters you have for that person. If the number is below what your own team would call a panel, widen the group before the survey opens, not after the results come back. Ask your finance or legal colleague to confirm in writing whether the results will touch pay or promotion. If they will, move the review to a separate process before anything else is built. Then book the debrief and the second round now, while the calendars are still open, because a review without a date for the next conversation is unlikely to change anything.

Newsletter

The Briefing

One email a week: the leaders we profiled, the ideas worth stealing, and nothing else.

Weekly, free, one click to leave. Read our privacy policy.