In our 50 years of providing validated, customized 360 solutions, this remains one of the most common questions we receive, both from participants and administrators. And like with most topics related to 360s, there is a right way and a wrong way to do it.

In this blog post, we will provide our guidance on effective rater selection. But first, why does this matter?

The foundation of any successful 360 assessment lies in the careful selection of raters. Raters are key to the success of a 360-feedback project because they see the leaders in action day-to-day, which results in the most useful and reliable feedback. By choosing individuals who can provide accurate, honest, and insightful feedback, managers and leaders can gain valuable insights into which strengths they should leverage, and which skills they should further develop.

Inviting raters who aren’t able to give accurate, honest and insightful feedback not only wastes the raters’ time, it also can have consequential impacts on the feedback itself. The results can be diluted, misinterpreted, and in some cases, the feedback received can be flat-out wrong. And if this is happening across your 360 program, it could impact the overall success of your 360 program.

Here’s 4 tips to help ensure you’re following best practices:

Tip One: Invite feedback from various roles and multiple levels within the organization.

Raters should include the participant’s manager/supervisor, peers, direct reports, and even external stakeholders (e.g., customers, suppliers, and/or board members). To obtain a comprehensive understanding of an individual’s performance, it’s essential that feedback is gathered from people with different perspectives on their work.

Tip Two: Avoid selecting raters based on the type of feedback you expect to receive.

The objective of a 360 assessment is to acquire neutral and credible feedback that can be used to drive improvements. Avoid picking raters with personal connections to the employee or those involved in active conflicts as they may be reluctant to share honest opinions. Inviting only Raters who will give you positive feedback will not help you improve or grow.

Tip Three: Invite Raters who have consistent exposure to your work behaviors.

Select individuals who have consistent professional interactions with the participant for at least three months or more. This will ensure that they have enough background knowledge of the participant’s job and enough experience working with them to be familiar with how they work. If you invite a Peer you worked with on a one-off project 3 years ago, they aren’t going to be able to provide adequate feedback on your current work behaviors. You’ll be wasting their time and yours.

Tip Four: Communicate expectations clearly.

Once the raters have been selected, it’s crucial to communicate the organization’s expectations explicitly. This will empower raters with a clear understanding of their role in providing constructive criticism that focuses on improving performance rather than just identifying weaknesses. Encourage raters to offer specific examples related to their observed behaviors and emphasize the importance of confidentiality throughout the process. Make sure the raters know the importance of their feedback being accurate, why the organization is using the 360-feedback process, and how the results will be used.

eBook

Turn Your Managers Into Leader-Coaches

This e-book uncovers the thinking, tools, resources, and guidance that can help organizations get more results from their managers while spending less of their L&D budgets.

Download e-Book

Bonus Tip: How Many is Too Many? Consider a random sample!

We recently had a Participant who had nearly 100 Direct Reports (wow!). We suggested they use the tips above (select people with the most exposure to your behaviors who have worked with you on a consistent basis for the last 3 months). If the group was still too large, (e.g. over 25), then we suggested that they select a random sample of an acceptable number of raters, so long as all of them met the criteria mentioned above.

While rater selection may not be the most exciting topic to explore, it is important and doing it wrong can have real consequences, both in the accuracy and usability of 360 degree feedback. Fortunately, it’s relatively easy to get it right. By following the recommendations in this blog, you can help set your managers up for success in their 360 feedback journey.

Here are a few suggestions for preventing rater fatigue:

• Make sure the raters have been properly prepped before you ask them for feedback. Let them know what they can expect, why they were chosen to provide feedback, and how their feedback is going to be used.
• Only invite relevant raters. If the purpose is to receive feedback on a person’s work behaviors, make sure that only raters who have exposure to and have actually worked with the person are invited. It’s easy to invite raters from a subset of an organizational chart, but you might not be inviting the most relevant raters.
• Be aware of rater overlap. If an entire team or department is taking part in a feedback initiative, there’s a high possibility that individuals will be asked to provide feedback for several people. For example, if a team with 10 members is asked to complete a 360 feedback initiative by providing feedback for each team member and a self survey, each person is being asked to commit time to complete 10 surveys within a short time period. Consider staggering feedback invitations so that people aren’t bombarded with 10 surveys at once.
• Use a “right-sized” survey. Make sure the survey is long enough to cover the concepts to be measured, but short enough for raters to be able to make time to complete it. Using surveys that are too long will result in incomplete responses or no response at all. You must respect the time commitment of the raters to receive useful, complete feedback, and to avoid complaints from raters.
• Use “role relevant” surveys. If an individual contributor, let’s say a software engineer, invites raters to provide feedback about their people management skills, raters might feel that the survey is not relevant, since the person is not currently a manager of people. You want to ensure the concepts being measured are tied to the person’s level in the organization and responsibilities they perform. One size does not fit all. Asking raters to answer questions about concepts that have nothing to do with the day-to-day role of the person being rated is just wasting everyone’s time.
• Use high-quality surveys constructed by expert psychometricians. This helps ensure that the survey questions are easy to understand and easy to answer. Poorly written survey questions containing multiple concepts will result in rater frustration, lack of rater responses, and unclear feedback. Think bad data in, bad data out.
• Leverage feedback results. The organization must be committed to leveraging the 360 feedback results through development plans, coaching, and training. Raters put forth time and effort to complete surveys, and when they see that nothing has changed as a result of previous feedback they have given, they will be reluctant to spend more time to complete future surveys.

These are just a few points to consider when trying to avoid rater fatigue. You can ensure program success with up-front planning and thoughtfulness toward the desired outcomes and the raters’ time.