Planning and executing SEO strategies for websites with hundreds of millions of pages is no easy task, but there are strategies out there to make it easier.
Programmatic pages are pages that have been automatically generated on a very large scale. SEO strategies for these pages are used to target multiple keyword variations by automatically creating landing pages of this size.
You can usually find these pages in major industries such as e-commerce, real estate, travel, and information sites. These industries rely on programmatic pages to develop their SEO strategy, and they have a separate page for each product and category. This facility can lead to hundreds of millions of pages – they are efficient, functional, and easy to use, but they come with some SEO challenges.
In my experience, the comprehensive SEO strategy covered in this post works best when tailored for a large website with programmatic pages. Many of the strategies that usually work for websites with only a few hundred pages may not produce the same results on larger websites. Small websites rely on manual and careful creation of content compared to programmatic pages which are the main traffic pages of the website.
So let's get down to business! I'm going to examine the top four SEO challenges you face when dealing with programmatic pages and how to overcome them.
1. Keyword Research and Keyword Modifiers
Well-planned keyword research is one of the greatest challenges on a programmatic scale. When you are working on a large number of pages and keywords, it is important to select and find the right keywords for all of the pages.
To be both efficient and effective, it is recommended that you divide the site's pages into a few templates before doing the research yourself. Some examples of these templates could be:
- Product Pages
- Static Pages
- Information pages
- Knowledge base / learning
Once you have all of the page templates set up, it's time to create keyword buckets and keyword modifiers.
Keyword modifiers are additional keywords that, once you combine them with your head and core keywords, help with the long-tail strategy. For example, modifiers to the main term “Amazon Stock” can be anything related to market share, statistics, insights, and so on.
Programmatic pages usually contain the majority of the pages on the site. (Take Trulia, for example, with over 30 million indexed pages, most of which are programmatic.) Therefore, on a larger website, these pages are usually the most important in terms of volume and search capabilities. Therefore, you need to make sure that the correct keyword modifiers are used on the content of each page template.
Of course, you can't go through every single page and manually change the SEO tags. Imagine a website like Pinterest trying to do this – it would never finish! . On a website with 30 to 100 million pages, it's impossible to optimize every single page individually. Because of this, the changes need to be made for a number of pages and categories. You need to find the right keyword modifiers to implement in your various page templates so that you can get the job done efficiently in bulk.
The main difference compared to typical keyword research is that it focuses on keyword modifiers. You need to find relevant keywords that can be implemented repeatedly on all relevant pages.
Let's take a look at this use case on a stock investing website:
The example above shows a website targeting users / investors with information intent and reliant on programmatic pages for SEO strategy. I found the Keyword Modifier by doing keyword research and competitor research.
I researched several relevant leading websites using Moz's Keyword Explorer and SimilarWeb's search traffic feature and found the most popular groups of keywords. After collecting the keyword groups, I found the search volume of each keyword to determine which are most popular and relevant to the target
Once you have the keyword modifiers, you need to implement them in the title tags, descriptions, heading tags, and content of the page templates for which the modifiers are intended. Even if you multiply this strategy by millions of pages, updating your programmatic pages with the right keyword modifier makes the process a lot easier and more efficient.
If you have a template with pages sorted by a specific topic, you can update and make changes to all pages with that topic, such as: For example, a website with inventory information with a certain type of inventory page or a category with stocks based on a price / industry. An update affects all pages in the same category. So if you update the SEO title tag of a stock page template, all of the pages in the same category will also be updated.
In the example above, the intent of the keywords is information. Keyword intent focuses on how search intent is matched against keyword modifiers. We are aimed at seekers who want to gain specific knowledge. You want more information on stocks or companies, market capitalizations, expert ratings, market trends, etc. In this case, it is recommended that you add additional keywords that include questions like “How?” “What?” And “Which?” ”.
As another example, transactional keywords that are better suited to ecommerce and B2C websites are extremely effective at targeting purchase-intent searches. These terms can include "buy", "receive", "buy" and "buy".
2. Internal link
Smart internal linkage plans are critical to large websites. You have the option to significantly increase the number of pages indexed and then pass the link fairness between the pages. When working on large websites, one of your top priorities should be making sure that Google recognizes and indexes your website's pages.
How should you create these internal linking functions?
When you look at the bigger picture, the goal is to have side A linked to side B and side C while side B linked to side D and side E and so on. Ideally, each page will receive at least one link from a different index page on the website. The challenge for programmatic sites is that new pages keep popping up every day. In addition to the existing pages, it is essential to calculate and project so that you can speed up the internal linking for the new pages. This way, these pages are quickly recognized and properly indexed.
Related Pages and "People Also Viewed"
One strategy to make link building easier is to add a "Related Pages" section to the site. It provides added value for the user and the crawlers as well as links to relevant pages based on affinity.
You can refer to similar content based on category, product type, content, or almost any other descriptive element. Similar content should be sorted in numerical or alphabetical order.
Yes, even large websites use HTML sitemaps to make it easier for crawlers to find new pages. They are extremely effective when you are working on large websites with millions of pages.
Let's take a look at this example from the Trulia HTML sitemap (see above): Trulia created its HTML sitemap in alphabetical order to ensure that all pages contain links. That way there are no orphaned pages which helps their goal of providing link juice for all of the pages they want to index.
In general, many ecommerce and real estate websites sort their sitemaps in alphabetical / categorical order to ensure that no page is alone.
3. Crawl budget and deindexing rules
Crawl budget is a very important issue that large websites need to consider. When you have tens of millions of programmatic pages, you need to make sure that Google is consistently finding and indexing your most valuable pages. The value of your pages should be based on content, revenue, business value, and user satisfaction.
First, select which pages should not be indexed:
- Use your favorite analytics tool to find out which pages have the lowest engagement metrics (high bounce rates, low time averages on site, no pageviews, etc.).
- Use the search console to find out which pages have high impressions and low click-through rates.
- Combine these pages into a list.
- Check for inbound links.
- Analyze the assignment of these pages to sales and business contacts.
- When you have all the relevant data and have selected the pages to be removed from the index, add a no-index tag to each and exclude them from the sitemap XML.
I work for SimilarWeb, a website with over 100 million pages, and I ran a no-index test on over 20 million pages using the checklist above. I wanted to see the effect of removing a large number of pages from our bio-channel.
The results have been amazing.
Although we lost over half a million visits over the course of a month, overall metrics of engagement on programmatic pages have improved dramatically.
If youIf you
By removing irrelevant pages, I've created more space for relevant and valuable pages that can be discovered by the google bot.
Rand Fishkin also has a really comprehensive checklist that shows you how to tell if a page is poor quality according to Google. Another good example is Britney Muller's experiment in which she deindexed 75% of Moz's pages with great results.
4. SEO split tests
Test everything! The advantage of working on a large-scale SEO campaign is that you have access to big data and can use it for your SEO efforts. Unlike regular A / B tests, which test human behavior, A / B split tests are only intended for crawlers.
The split testing process is usually based on the same or similar page templates. Divide the page into two or three groups – one group acts as a control while the other groups are activated. Test the following criteria:
- Add structured data
- Change SEO Tag Keyword Modifier (Title Tag, Description, H-Tags, etc.)
- Image ALT tags
- Content length
- Page yield
- Internal link
To measure performance, I recommend doing one experiment at a time. For example, you can customize SEO tags first and then continue testing other industries after building some trust.
Let's take a look at a split test example and let's take a look at Etsy. Etsy wanted to test which title tag would rank higher and get a better CTR, and generally improve organic traffic to the tested pages. In the image below you can see how they split test between control pages with standard title tags against test pages with different tag variations in this article.
Pinterest's dashboard also shows how the growth team relies on split-test experiments in their SEO strategy. Pinterest's goal was to create an experimentation tool that would allow them to measure the impact of SEO changes on their rankings and organic traffic.
Now it's your turn
Because programmatic pages are different from most others, it is important to properly create and optimize these pages. This requires several adjustments to your normal SEO strategy as well as the application of new and proprietary strategies. The advantage of the approach outlined above lies in the incremental scale that you can use to contribute to your business.
Programmatic page searches should match the search query, whether it's product search, address, or information. For this reason, it is important to ensure that the content is as clear as possible and that the user gets the best response for each query.
Once you understand the four tactics above, you can implement them into your SEO strategy and get better results for your programmatic pages.