There's an interesting twist in the way we think about indexing – and that's rendering.
When we think about page ranking, we generally think about indexing.
That said, we generally think about when a search engine does the following:
- Discovered a page through sitemaps or crawling and then visited the page for indexing.
- Gathered all content across the page source.
- Started the ranking of the page for queries.
In the past, this was the most important step in this process, as it is the trigger for rankings.
However, indexing is not the last phase of the discovery process.
I would suggest that its weight decrease over time as the final phase – rendering – increases and I suspect that the indexed version will be replaced altogether.
Indexing vs. Rendering: what's the difference?
Basically, the difference between indexing and rendering can be illustrated with these two images:
This is the indexing:
This is rendering:
This is basically the same content that is displayed when indexing (HTML) and rendering (Chrome).
Why is that important?
Rendering is more important than you might think.
In essence, the reason why it is important that rendering is the truth.
With the code, a search engine can understand what a page is about and what is going on.
When rendering, they can understand the user experience and much more about what content should be a priority.
When rendering, you can answer questions such as:
- Is the content hidden behind a click?
- Does an ad fill the page?
- Is content that is displayed at the bottom of the code actually displayed at the top or in the navigation?
- Is a page loading slow?
All of these and many other questions are answered when rendering.
These answers are important to properly understand a page and how it should be classified.
When does the rendering take place?
In 2018, the rendering took weeks.
Not surprisingly, it takes a lot less time now.
Around 6:20 p.m. in the audio here you hear how Martin Splitt from Google answers exactly this question.
How long does it take for Google to render a page?
The medium is 5 seconds long and within minutes 90% of the indexed pages are in the rendering queue.
Note that this is a queue, not necessarily a render.
That means if you are on the positive side of the medium set that starts within 5 seconds, your side will be begin Render within 5 seconds, though it may not be completed within that time.
If rendering starts in 4 seconds but takes 30 seconds, it will be considered among those counted on the positive side of the media set.
We have come a long, long way in two years, from weeks to seconds.
Bing works differently.
When I asked Frédéric Dubut, the project manager for web ranking and quality, he replied:
Same answer as before, between minutes and forever 🙂, but I can confirm that we are trying to prioritize rendering for the URLs submitted through the API.
– Frédéric Dubut (@CoperniX), August 3, 2020
The "before" to which he refers was my tweet from last September:
I would say the same thing – sometimes it is days, it can be weeks and in extreme cases it can never be. Ultimately, it's a compromise between the cost of rendering the page and the value we find when rendering.
– Frédéric Dubut (@CoperniX), September 3, 2019
Presumably they accelerated things too, although I don't have a new confirmation in time.
So the short answer to the rendering is "after indexing" and the timeline is variable, but short. This essentially means that search engines understand the content and context of a page before they get a full understanding of how it works, but in most cases the delay is controversial.
A big leap forward took place in May 2019 when the Googlebot webbot service component (WRS) was updated.
Until then, the web rendering service used Chrome version 41.
In May 2019, the web rendering service was updated to Evergreen, which means that it uses the latest version of Chrome to render (at least within a few weeks).
If your page is now rendered by Google, it will essentially render more or less the way you would see it in your browser.
What does a web rendering service do?
I wanted to quickly answer a question that did not completely cover my brain until I realized that I was completely wrong about it.
Feel free to laugh at me because the hiccups are obvious in my brain.
First, let's consider where and how a web rendering service gets its instructions.
Here's basically the rendering lifecycle:
- A page is recognized via sitemap, crawler, etc.
- The page is added to the list of pages to be crawled on a site when the crawl budget is available.
- The page content is crawled and indexed.
- The page is added to the rendering queue.
- The page is rendered.
A critical and unspoken element of the process is therefore the rendering queue.
When a page is at the top of the queue for rendering, the engine sends a headless browser to it.
This is the step that I had trouble with.
A headless browser is a browser without a graphical user interface.
For some reason, I was having trouble getting my brain to figure out how it works.
How can Google know what is there if it is not displayed graphically?
The obvious answer is:
"The bot has no eyes either, so … um … yes."
Because of this mental hiccup, I have resigned myself to "browser light" that makes the page for the search engine so that it now understands what appears where and how on a page – even though they have no eyes to see it.
If everything goes well, the rendered version will be displayed for Googlebot as well as for graphical browsers. If not, it is likely that the page is based on an unsupported feature such as a user authorization request or one of the scripts or other resources.
What about pre-rendering?
Basically, it's an ethical form of disguise, where you create a copy of the page as it appears in the DOM and provide it to search engines to ensure that they see the same content that a user does when they stop by to index the content.
"Ask and you will get an answer from Google sometimes".
And this was one of those times.
The answer was:
You generally don't need it now.
– 🍌 John 🍌 (@JohnMu) August 2, 2020
This is great news for those running Puppeteer or any other pre-rendering library.
I know I have seen cases where the pre-rendering system crashed without error notification and caused some headaches (read: Pages Removed from Index).
If we don't have to render in advance, we don't have to worry about such things.
Of course the operative word here was "general".
So if you are thinking of turning off your pre-rendering system, I recommend stopping the system from running on a handful of pages and waiting to see what happens if they are cached again.
Does Google consider the rendered content?
In this case, you may be able to stop pre-rendering entirely.
By rendering, the engines can prioritize content based on how a person would likely interact with a page.
This tells the engine how content is positioned in a browser and how visible different elements are. So when they try to judge or prioritize content or weigh usability, they work with the same product as a visitor.
The future is being rendered
The time delay and Müller's pre-rendering explanation put the writing on the wall.
Indexing, as we think, if it is likely to become a functionally irrelevant step from an SEO perspective, with rendering in the foreground when discovering web content.
In this article, we tried to generally describe what rendering is.
This can lead to a number of questions. And it should be.
To answer this, I will refer you to some important resources.
I can't recommend enough if I follow the links below:
Selected image: Adobe Stock edited by the author
Image tab: Adobe Stock, edited by the author
All screenshots made by the author