Whether you’re designing your own survey or researching a potential 360 feedback purchase, chances are good that the rating scale is not at the top of your list of critical success factors. When it comes to high-quality assessments, much of the focus tends to be on things like the reporting output, the survey platform, and of course the survey content itself.
However, finding the right rating scale can not only impact the reliability of your survey, it can also impact the overall user experience. A poorly designed rating scale has the potential to derail what might otherwise be a high-quality assessment. This article explores 360 degree feedback rating scales and the research-driven recommendations for using a frequency based, 7-point Likert scale.
Why Use Frequency vs. Quality Based Scales 360 Assessments used for leadership development will tend to embed the quality (how well someone does something) in the question item and then ask the frequency. For example, the item “Makes goals clear” contains the quality aspect, while the rating scale asks how often the person exhibits this behavior. If you word the question differently, you can make it neutral and then ask the quality (for example, goal setting– how well does this person do this?). In this scenario, it’s more reliable and efficient to ask someone a question that is easier for them to answer – an observation about frequency vs. a judgment of quality. Judgments generally require more thought, have more error, and therefore don’t predict as reliably. They also take more time to answer, which is something we are always trying to minimize. Check out this blog for more info on that topic!
Why Use 7-Point as Opposed to a 5-Point Scale? There are many variations of rating scale, but for 360s, the 5 or 7-point tend to be the most common. However, our research shows that there are 7 distinct levels of behavioral differences that most raters can easily detect when answering questions on assessments. Originally this was done with a q‐sort method, and it was found that 5 levels of differentiation was not sufficient.
A rating scale should capitalize on the number of distinctions that most people can naturally make (don’t provide too few rating points), and avoiding suggestions of distinctions that cannot be consistently made (providing too many rating points).
The following example illustrates how the 7-point scale is derived from 3 opposing scenarios with a neutral middle option. Using as an example a manager who confronts employee conflict: There are managers who:
Never step in and those who Always step in
Almost Never step in and those who Almost Always step in
Sometimes step in and those who Often step in
And finally, there is a group between Sometimes and Often.
This can be supported by asking respondents to identify managers who define each of these 7 scale points. We have found that the 5-point scale ignored the distinction raters made between extremely high and moderate levels. Therefore, using a 5-point scale would reduce descriptive power of the assessment and sacrifice insight of activity that raters can clearly differentiate.
In Summary While rating scales may not be the most exciting of topics, they are a critical component in the overall success of your assessment. They play an important factor, not only in your assessment’s reliability and validity but also in how easy and intuitive it is for raters to complete. Thanks for reading and happy surveying!