Google has published a detailed explanation of how changes to search rankings are evaluated internally before they are made available to users.
Danny Sullivan, Google's Public Liaison for Search, discusses the role of search quality evaluators and explains how their reviews are included in algorithm updates.
Thousands of changes are made to Google's search algorithms each year, and all go through a process to determine if they are useful to searchers around the world.
Here are some ways that feedback from quality assessors and ordinary people can be used when updating algorithms.
The Google research team talks to people around the world to understand how search can be more useful:
"We invite people to give us feedback on different iterations of our projects, and we conduct field research to understand how people in different communities access information online."
Ultimately, Google's mission is to make information accessible and helpful, and to help all users pursue this goal.
Search quality assessors
Google has publicly available guidelines for evaluators that describe how its algorithms should display content.
The guidelines are over 160 pages long, but if they should be narrowed down to a single phrase, Sullivan puts it like this:
"… we want to say that Search is designed to return relevant results from the most reliable sources available."
Many signals are automatically captured by the Google algorithms. But when it comes to signals such as relevance and trustworthiness, they require human judgment.
Google has a group of more than 10,000 people around the world who are referred to as "search quality assessors".
Reviewers can use Google to assess the likelihood that users will get search results.
The ratings are based on Google’s guidelines and are intended to represent real users and their likely need for information.
Study search-quality reviewers and get tested using Google’s review guidelines before they can provide feedback.
How search quality evaluators work
Google assigns a set of queries to a group of reviewers. Two versions of the results pages are displayed for these searches.
A number of results come from the current version of Google, the other from an improvement we are considering.
Each page that appears in the results is rated against the query guidelines based on the query.
To evaluate things like expertise, authority, and trustworthiness – sometimes referred to as "E-A-T" – evaluators are asked to do reputation research on the sources.
As soon as the evaluators have carried out this examination, they submit a quality rating for each page
Ratings are not used directly for rankings
"It is important to note that this rating has no direct impact on how that page or website is placed in search. Nobody decides that a particular source is" authoritative "or" trustworthy ". In particular, pages are not assigned ratings, to determine how well they should be rated. "
Search quality ratings are a data point that Google can use to measure how well its systems are working to deliver great content that complies with the guidelines.
For more information, see the Google article here.