I've been in a pensive mood lately.

Earlier this year (15 years after starting Distilled in 2005) we started a new company called SearchPilot to focus on our SEO A / B testing and Meta-CMS technology (formerly known as Distilled ODN) focus, and merged the consulting and conference part of the business with Brainlabs.

I am now the CEO of SearchPilot (which is primarily owned by Distilled shareholders) and an SEO partner at Brainlabs. I'm sorry, but I'm staying very much in the SEO industry.

As such, it feels more like the end of a chapter to me than the end of the book, but I've still looked back to see what has changed and what has not in the last 15 years I've been in industry.

I can't claim to be one of the first generation of SEO experts, but after building websites since around 1996 and seeing Google grow from the start, I feel like I may and may be the second generation Some interesting stories to share with those who are new to the game.

I racked my brain to remember what felt meaningful at the time, and also looked back at the big trends of my time in the industry to put together what I thought was an interesting reading list that most people online work today would be good to know.

The great epochs of search

I joked at the start of a presentation I gave in 2018 that the great epochs of search shuttled between search engine directives and search engines that quickly withdrew from those directives when they saw what webmasters actually did:

While this slide was a bit cheeky, I think there is something to think about the eras:

  1. Building websites: do you have a website? Would you like a website? It's hard to believe now, but in the early days of the internet, a lot of people had to be persuaded to even get their businesses online.
  2. Keywords: Basic information retrieval became the controversial information retrieval process when webmasters realized they could play the system with keyword filling, hidden text, and more.
  3. Links: As the size of the web grew beyond user-curated directories, link-based algorithms for search dominated.
  4. Not these links: Link-based algorithms gave way to controversial link-based algorithms when webmasters exchanged, bought, and tampered with links throughout the web graph.
  5. Long Tail Content: Alongside this era, long tail length was better understood by both webmasters and Google itself – and it was in the interests of both parties to create huge amounts of (often opaque) content, indexing it when it did was needed.
  6. Not this content: Maybe predictable (see the trend here?), The average quality of the content returned in search results has dropped dramatically, and so we see the first ranking factors for machine learning in the form of attempts to measure "quality" (alongside relevance and Website authority).
  7. Machine Learning: From that point on, everything has arguably been an adventure in machine learning and artificial intelligence, and it happened during the careers of most of the marketers working in search engine optimization today. So, while I love writing about this stuff, I'll come back to it in another day.

History of Search Engine Optimization: Crucial Moments

While I'm sure there are interesting stories to tell about the time before Google's SEO, I'm not the right person to tell (if you have a great resource please drop it in the comments) Let's start early in the google trip:

The basic technology of Google

Still, even if you step into search engine optimization in 2020 in a world of machine-learned ranking factors, I would recommend re-reading the surprisingly accessible early academic paper:

If you weren't on the internet then, it's probably hard to imagine how Google's PageRank-based algorithm has improved over the state of the art then (and it's even hard to remember) for those of us who were):

Google IPO

At the time Google went public in 2004, very few people expected Google to become one of the most profitable companies of all time. In the early days, the founders had spoken of their disdain for advertising and somewhat reluctantly experimented with keyword-based ads. Because of this attitude, even within the company, most employees did not know what kind of rocket ship they were building.

From around this time, I'd recommend reading the founders' IPO letter (see this great article by Danny Sullivan – who, ironically, is now @SearchLiaison on Google):

“Our search results are the best we can produce. They are unbiased and objective and we do not accept payments for them or for inclusion or more frequent updates. "

"Because we don't charge merchants for inclusion in Froogle (now Google Shopping), our users can browse product categories or conduct product searches with the confidence that the results we provide are relevant and unbiased." – S1 submission

Additionally, In the Plex is an enjoyable book published by Steven Levy in 2011. It tells the story of what then CEO Eric Schmidt (at the time of the IPO) called a "hiding strategy":

"Those who knew the secret … were strongly instructed to shut up."

"What Google was hiding was how it cracked the code to make money on the internet."

Fortunately for Google, for users, and even for organic search marketers, this turned out to be inconsistent with their sheer ideals from the days leading up to the IPO, as Levy was more satisfied with ads than those on repeated tests with pages they were suppressed ”. Phew!

Index everything

In April 2003, Google acquired a company called Applied Semantics and started a series of events that I think are the most underrated part of Google history.

Applied Semantics technology has been integrated with its own contextual ad technology to form AdSense. Although AdSense revenue has always been dwarfed by AdWords (now just "Google Ads"), its importance in the history of search engine optimization is difficult to underestimate.

By democratizing the monetization of content on the web and the possibility of everyone getting paid to produce obscure content, the creation of absurd amounts of that content has been funded.

Most of this content would never have been seen had it not been for a search engine that excelled in long-tail search capabilities, even if those searches were incredibly infrequent or had never been seen before.

In this way, the Google search engine (and search engine advertising business) formed a powerful flywheel with the AdSense business that provided funding for the content creation necessary to differentiate itself from the largest and most complete index on the web.

As with so many chapters in history, however, a monster emerged here in the form of inferior or even automatically generated content, which would ultimately lead to PR crises and massive correction efforts.

If you're interested in the Index era, you can read more of my thoughts on it on slide 47+ of From the Horse’s Mouth.

Web spam

The first forms of spam on the Internet were various types of messages that came into the mainstream as email spam. In the early 2000s, Google began talking about what they would ultimately call "web spam" (the earliest mention of link spam was in a 2005 presentation by Amit Singhal entitled "Challenges in Running a Commercial Web Search Engine "(PDF))).

I suspect that even people starting SEO today have heard of Matt Cutts – the first head of webspam – as he is still widely referenced despite not having worked at Google since 2014. I enjoyed this 2015 presentation that talks about his career at Google.

Era of search quality

Over time, pure web spam wasn't the only quality issue Google faced as webmasters wanted to make money unlike Google (and others) who were trying to create the best search engine possible. The cat-and-mouse game of manipulating recognition – especially the content on the page, external links, and anchor text – would be a defining feature of the next decade of search.

Following Singhal's presentation above, Eric Schmidt (then CEO of Google) said, "Brands are the solution, not the problem … Brands are the way you sort the sewer."

Those new to the industry have likely seen some Google updates (like the recent "core updates") firsthand, and have probably heard of some specific older updates. "Vince", which came after "Florida" (the first major confirmed Google update) and launched shortly after Schmidt's branding, was especially notable for favoring big brands. If you haven't followed all of the history, check out important past updates here:

A real reputational threat

As I mentioned in the AdSense section above, there has been a strong incentive for webmasters to create tons of content to target the thriving long tail of search. If you had a strong enough domain, Google would crawl and index an immense number of pages, and with enough dark queries, any matching content would potentially be ranked. This sparked the rapid growth of so-called "content farms" that mined keyword data from anywhere and brought out poor quality keyword matching content. At the same time, the websites managed to index large databases of content as very thin pages or to index a large number of pages with user-generated content.

This was a real reputational threat to Google and broke out of the search and SEO chamber. It had become such a bugbear for communities like Hacker News and StackOverflow that Matt Cutts sent the Hacker News community a personal update when Google released an update to address a specific symptom – namely, that scraper websites routinely outperformed original content Copy.

Shortly afterwards, Google released what was originally called the "Farmer Update". After launch, we learned that this was made possible by a breakthrough by an engineer named Panda. As a result, it was referred to internally by Google as the "Big Panda" update, and since then it has been mainly referred to as the Panda update by the SEO community.

Although we speculated that the internal working method of the update was one of the first real-world applications of machine learning at the core of the organic search algorithm at Google, the functions it modeled were more easily understood as human-centered quality factors, and so we began to offer our clients SEO- Recommend targeted changes based on the results of human quality surveys.

Everything goes mobile first

I gave a talk at SearchLove London in 2014 where I talked about the incredible growth and scale of cellular and how late it was realizing how seriously Google was taking it. I highlighted the surprise many heard when Google first designed the phone:

“Towards the end of last year, we made some pretty big design improvements to search on mobile and tablet devices. Today we've carried some of those changes to the desktop experience. "- Jon Wiley (lead engineer for Google Search on Google+ which means there is no link to refer to as the perfect reference for the quote, but it is referenced here and in my presentation).

This surprise came despite the fact that by the time I gave this presentation in 2014, we knew that mobile search had started cannibalizing desktop search (and we had seen the first drop in desktop search volume) :

And it came even though people were starting to say that the first year that Google got most of its sales from mobile devices was less than two years away:

By the time we write this in 2020 we will feel like we've completely internalized the size of a mobile business, but it's interesting to remember that it took a while to take hold.

Machine learning is becoming the norm

Since the Panda update, machine learning has been mentioned more and more frequently in Google's official communications about algorithm updates. We know that there has been resistance to the use of machine learning in the core algorithm from some quarters (including Singhals) in the past because it prevented human engineers from explaining the results. In 2015, Sundar Pichai took over the running of the business, pushing Singhal aside (although it may have been for other reasons) and installing AI / ML fans in key roles.

It comes full circle

Before the Florida update (until Google released an update called Fritz in the summer of 2003), search results were regularly mixed up in a process nicknamed Google Dance:

Most things have moved more in real time since then, but the recent "core updates" seem to have brought back that kind of dynamic where changes are made on Google's schedule, rather than based on the schedules of website changes. I've speculated that this is because "core updates" really are Google retraining a massive deep learning model that is very much adapted to the shape of the web at this point. Regardless of the cause, our experiences with a wide variety of customers agree with the official Google statement:

Major core updates typically happen every few months. Content that was affected by one might not be restored until improvements have been made until the next major core update is released.

Linking recent trends and discoveries like this to ancient history like the Google Dance is just one of the ways in which it is “useful” to know the history of search engine optimization.

If you are interested in all of that

I hope this journey through my memories has been interesting. What did I miss for those of you who have also worked in the industry during these years? What are the really big milestones that you remember? Drop them in the comments below or log on to Twitter.

If you enjoyed this walk back in time, you might also like my From the Horse & # 39; s Mouth presentation, where I try to extract what is really going on behind the scenes using official and unofficial Google statements, and some tips to give the same yourself:


Take the 2020 Moz Blog Reader Survey, which asks who you are, what challenges you are facing, and what you want to see more about on the Moz blog.

Take the poll


Please enter your comment!
Please enter your name here