1. What is web crawling?
  2. What is the crawl statistics report for the search console?
  3. How to access the crawl statistics report
  4. In total

The new Google Search Console crawl statistics report is explained in detail by Search Advocate Daniel Waisberg in a new training video.

The crawl statistics report in the search console was extensively updated a few months ago. If you haven't had a chance to look at the new report, now is a good time to familiarize yourself with all of the added insights.

The new video from Google breaks down each section of the crawl statistics report and explains how the data can be used to determine how well Googlebot can crawl a given website.

If Googlebot can crawl a website efficiently, new content can be indexed quickly in search results and Google can detect changes to existing content.

Here is a roundup of the video, starting with the absolute basics: What is crawling?

What is web crawling?

The crawling process begins with a list of URLs from previous crawls and sitemaps provided by website owners.

Google uses web crawlers to visit URLs, read the information they contain, and follow links on these pages.

The crawlers revisit pages already on the list to see if they have changed, and also crawl new pages they discover.

During this process, the crawlers must make important decisions, such as: B. Prioritize when and what to crawl while ensuring that the website can handle the server requests made by Google.

advertising

Read on below

Successfully crawled pages are processed and submitted to Google indexing to prepare the content for inclusion in Google search results.

Google wants to make sure that your servers don't get overloaded. The frequency of the crawls therefore depends on three factors:

  • Crawl rate: Maximum number of concurrent connections a crawler can use to crawl a site
  • Crawl demand: How much Google wants the content.
  • Crawling budget: Number of URLs that Google can and wants to crawl.

What is the crawl statistics report for the search console?

The crawl statistics report in the search console is used to understand and optimize Googlebot's crawling. It contains statistics on Google's crawling behavior, e.g. B. how often a website is crawled and what the answers were.

advertising

Read on below

Wasiberg said the report is relevant if you have a large website, but less important if you have a website with fewer than 1,000 pages.

Here are some questions that you can answer using the data provided in the crawl statistics report:

  • What is the general availability of your website?
  • What is the average page response for a crawl request?
  • How many requests has Google made to your website in the past 90 days?

How to access the crawl statistics report

Site owners can find the crawl statistics report by logging into the search console and going to the Settings page. There you will see the crawl statistics report.

When you open the report, you will see a summary page with a chart of crawling trends, host status details, and a breakdown of crawling requirements.

Google information on how to use the Search Console's crawl statistics report

Crawling trends chart

The crawl trends chart reports these three metrics:

advertising

Read on below

  • Total number of crawl requests for URLs on your website (whether successful or not).
  • Total size of downloads from your website during the crawl.
  • Average page response time for a crawl request to get the page content.

As you analyze this data, look for major spikes, drops, and trends over time.

For example, if you see a significant decrease in overall crawl requirements, Google suggests making sure no one is adding a new robots.txt to your website.

Or, you may find that your website is slow to respond to Googlebot. This could be a sign that your server is not able to handle all requests.

Pay attention to a steady increase in the average response time. According to Google, this may not have an immediate impact on the crawl rate. However, this is a good indicator that your servers may not be handling all of the load.

advertising

Read on below

Host status details

You can use the host status data to check the general availability of a site for the past 90 days. Errors in this section mean that Google cannot crawl your website for technical reasons.

Google information on how to use the Search Console's crawl statistics report

When you click for details on the host status, you will find three categories:

advertising

Read on below

  • Get Robots.txt: The rate of errors when crawling your robots.txt file.
  • DNS resolution: Indicates when the DNS server did not recognize your host name or did not respond while crawling.
  • Server connectivity: Indicates when your server did not respond during a crawl or did not provide the full response for your URL.

Crawl requirement cards

Crawl requirement cards show different breakdowns to help you better understand what Google crawlers found on your website.

Google information on how to use the Search Console's crawl statistics report

There are four breakdowns available:

  • Crawl response: Responses Google received while crawling your website.
  • Crawl file type: Displays the types of files returned by the request.
  • Crawling purpose: Indicates the reason your site was crawled.
  • Googlebot type: Displays the user agent that Google used to make the crawl request.

In total

These are the basics of using the Search Console crawl statistics report to ensure that Googlebot can efficiently crawl your website for searches.

advertising

Read on below

Takeaways::

  • Use the page summary chart to analyze crawl volume and trends.
  • Use the host status details to check the general availability of your site.
  • Use the crawl requirement breakdown to understand what Googlebot finds while crawling your website.

See the full video below:

LEAVE A REPLY

Please enter your comment!
Please enter your name here