Alexis Sanders & Martin Splitt

Alexis Sanders from Merkle sat down with Martin Splitt from Google last year (before COVID) and talked about the crawling budget. It could be one of the more informative videos in the SEO Myth series so far.

The following have been covered with the timestamps if you only want to scan the video:

  • Why is the crawl budget an interesting topic? (0:00)
  • What is the crawl budget? (1:15)
  • What is the crawl rate and what is the crawl demand? (1:47)
  • How does Googlebot make the crawl rate and crawl needs decisions? (2:44)
  • ETags, HTTP headers, data last changed and the like (3:43)
  • What size websites should crawl budget take care of? (4:35)
  • Server setup vs. Crawl budget (5:00 AM)
  • Crawl frequency vs. Quality of content (6:18)
  • What can you expect in the log files when Google tests the server? (7:45)
  • Tips on how to crawl your site accurately during a site migration (8:18)
  • Crawl budget and the different levels of a site's infrastructure (9:40)
  • Does the crawl budget also affect rendering? (10:37)
  • Caching resources and crawling budget (11:46)
  • Crawl budget and certain industries like publishing (13:34)
  • What can generally be recommended to help Googlebot crawl a website? (15:03)
  • What are the common pitfalls people with the crawl budget get into? (16:52)
  • Can you tell Googlebot to crawl its website more? (17:40)

As an added bonus, here are some questions that Martin answered in connection with this talk on Twitter:

This pattern is a bit normal because Googlebot may zigzag about the maximum reasonable crawl rate.

Crawl budget issues occur when you find that we’ve discovered pages that you’ve been interested in for some time and the pages have no other issues.

– Martin Splitt on July 15, 2020 in 🏡🇨🇭 (@ g33konaut)

It is not a significant cost for us

– Martin Splitt on July 15, 2020 in 🏡🇨🇭 (@ g33konaut)

Either 404em or keep them nearby.

– Martin Splitt on July 15, 2020 in 🏡🇨🇭 (@ g33konaut)

That would qualify as dynamic rendering, but in general these setups are "foot guns" – sounds good and may work, but adds a lot of unnecessary complexity that eventually backfires.

– Martin Splitt on July 15, 2020 in 🏡🇨🇭 (@ g33konaut)

If you are concerned about this, it could make sense. I don't think it's usually necessary, tho.

– Martin Splitt on July 15, 2020 in 🏡🇨🇭 (@ g33konaut)

Correlation is not a causality 🙃
So in short: no.

– Martin Splitt on July 15, 2020 in 🏡🇨🇭 (@ g33konaut)

It depends on how this dropdown is implemented. If the links are valid links and are in the rendered HTML, the crawler can get them.

– Martin Splitt on July 15, 2020 in 🏡🇨🇭 (@ g33konaut)

Many webmasters do not give us helpful data.

– Martin Splitt on July 14, 2020 in 🏡🇨🇭 (@ g33konaut)

Forum discussion on Twitter.

LEAVE A REPLY

Please enter your comment!
Please enter your name here