Understanding Programmatic SEO: A Beginner’s Guide

Since the beginning of my career, I have been religiously following one rule!

If you want to thrive in business, then you have to go where your customers are. And in today’s technology-driven world, consumers are online more often than ever. 

The “Queen of the Internet” Mary Meeker and her 2019 Internet Trends Report outline some fascinating statistics, most notably about the growth in the number of internet users and digital consumer behaviors. 

Meeker highlighted digital marketing trends that will shape and affect the digital marketing landscape in the future, and one of those is the prevalence of programmatic marketing concerning advertising and SEO. 

Have you ever wondered how aggregator sites like Zillow, TripAdvisor, and Yelp have hundreds of pages of targeted content? Have you ever asked yourself how they do them? 

The answer is SEO!

If you follow me regularly, you’d know I recently reviewed some of the best SEO tools, I recommend you give this a read to familiarize yourself with these very important tools.

But today, I will outline the basics of programmatic SEO, how to create SEO-driven landing pages, and how to scale them. 

What is Programmatic SEO? 

Programmatic SEO is the creation of pages at scale using a code. For instance, creating 100s of pages for “best hotels in {city}” which is a technique used by companies like Airbnb, and Expedia.

This is a relatively new concept in SEO that involves developing landing pages on a larger scale. This method allows you to create targeted pages that aim to get more mobile, voice, and local searches on your search engine. To match the sheer number of Google search entries, websites employing this technique deploy large masses of web pages containing modifiable content. 

Its a method commonly used by aggregator sites, to create and optimize landing pages using metadata on a comprehensive and large scale. It is less about authoritative, informative, or quality content, but is more about the creation of user-friendly and high volume landing pages with transactional intent. 

This technique can also be viewed as a process where ranking factors and its changes are detected, software or scripts are updated, and changes based on the modification of ranking factors are applied and reflected on particular pages of the site. 

Sounds simple enough? It may seem easy, but as we proceed with this article, you will realize that it may not be as easy as it sounds. 

Relevant: Simple and effective tops for On-Page SEO

Elements of Programmatic SEO 

Generally, programmatic SEO has three elements: 

1. Research and Development (R&D)

R&D aims to answer questions such as: 

1. How many searches happen per month?
2. How do people search for things related to your business offering?
3. What pages are relevant to the search query?

R&D revolves around questions that quantify the intent of the searcher, the competition between the search results, and how users are searching for keywords to be used to find the page. 

2. Technical

Technical elements include answers to queries like: 

1. Can the search engine crawl all pages on the site? 
2. How do we get the search engine to crawl the pages we want? 
3. Does the engine view content on the pages?

3. Content

Questions about content include: 

1. Does the search engine think our content is relevant? 
2. Does the search engine think our website is trustworthy?

Keyword Research

SEO is all about capturing search traffic relevant to your page or business, and search intent vary based on the keywords you use. 

Keyword research is already a reasonably known SEO procedure, but applying it at scale can be quite different as it involves a modified method. You will find and list out head terms, check their search volume using a keyword research tool, identify primary and secondary identifiers, including modifiers, and put them all together using Python, a programming language. 

Find your Head Terms 

Head terms can be thought of the categories you will rank for, at a broad level. For example, Zillow will probably use words related to real estate, and TripAdvisor terms related to vacations and trips. 

Head terms already contain search volumes in themselves but are often searched with modifiers. To get some sense of the search volume, you can use a keyword research tool, such as Google Trends. 

For example, let me use terms related to vacations. 

I start with these head terms to know the overall volume related to each one. 

After that, I can flesh out the list as much as possible by adding more terms to a list. For this, you can use tools such as: 

1. Keywords Everywhere
2. Ubersuggest
3. Whatsmyserp 
4. Google AdWords
5. Moz
6. SEMRush 
7. The “People Also Search For” option in the search engine  

Also, using tools, you can take a look at the pages of your competitors in the scene to learn the keywords they are ranking in their title tags. Add anything that stands out to your list. 

For example, if your head term is about house cleaning, then here are some permutations. 

Identify Modifiers

Google searches like to help users by giving them automatic search suggestions. Using an SEO tool such as Ubersuggest, find all of those suggestions along with your head keyword and create a list with modifiers. 

At least, have a guess on what your primary and secondary keywords are. Here are some examples 

Primary Modifiers: 

1. Restaurants: Japanese restaurants, fine dining restaurants, Chinese restaurants
2. Shirts: dress shirts, floral shirts, long-sleeved shirts 
3. Hotels: Hotels in New York, Lancaster PA hotels 

Secondary Modifiers: 

1. Restaurants: cheap restaurants, romantic restaurants 
2. Shirts: comfortable shirts, affordable shirts
3. Hotels: cheap hotels, hotels with swimming pools 

Primary modifiers create a whole new category, while secondary modifiers either describe a head term or a head term including a primary modifier. 

Put It All Together

Now, we will go ahead and combine the keywords together for an extensive list. It is not unreasonable to have over 100k keywords if you are operating at a large scale, but you can get good insights for as little as 2k keywords. 

We want to get every possible version of keywords and modifiers, and for purposes of this section, we assume that we are focused on the vacation home rental niche. This Python script will let you do just that.

You can use this script as your own, and if you are focusing on a different niche, then simply replace the head terms and city-state pairs (yellow texts inside “[ ]”) to generate a different set of keywords. Additionally, there are far more head terms and city-state pairs, and you would want to use more CSV files.

When you run this script, you will get a very exhaustive list of keyword options. 

As you see, the python script results in every possible permutation of your keyword and city-state pair. View the results in Excel with the program’s text import functionality, and effectively, you have already created keyword tags. 

If not familiar with coding with Python, then there are plenty of spaces online to learn. Furthermore, Macs come preinstalled with Python coding software, and you can use the exact scripts shown above for your use, with modifications for your particular keywords. 

The inclusion of secondary modifiers can be helpful but is not necessary. If you want to though, then be sure to include another array in your script, and another column in your excel output. 

Using Excel on a large scale keyword research project is quite tricky though, as more often than not, you will find that the terms quickly reach the maximum capacity of the sheet. You can use other tools and data analytics platforms, such as Google BigQuery plus SQL. 

Get Search Volume

The Google Adwords tool does not truly give accurate data, but in programmatic SEO, we don’t need accurate. We need directional data to be used as points for comparison.  

Adwords has a pretty low keyword limit, so you will want to use a virtual assistant to divide these into blocks and chunks, or use Keyword Keg at a cost to allow you to manage large batches of data. 

Running the excel data into Adwords or other data analytics tools produces keywords with search volume, such as below. 

Retrieve the data from your columns and connect them using the VLOOKUP function in Excel. If too large for Excel, then upload to SQL tables, then join the related columns. 

Competitive Analysis at Scale

After gathering a whole string of keywords, the next step is to rank the data on a large scale to garner valuable competitive insight. Rankings can be considered as the most straightforward and useful data when sizing up your competitors. 

Top players are defined as those sites having the most keywords in the top 10 of a chosen niche. Do a quick search in search engines to see how your competitors are doing. 

Rank data for about 5,000 keywords is enough to get a picture of the competition in your industry. Pay attention to where the top sites get their links, the tools they use for the title tags, and common UX patterns. 

Examination and analysis of your competition are made a lot easier by tools such as SEMRush, Moz, Ubersuggest, and Ahrefs. 

Relevant: Read my review of WhatsMySERP

Creating Landing Pages at Scale

Once you have identified your keywords and did a competitive analysis, it is time now time to generate landing pages. This is one of the critical challenges in programmatic SEO, as content for each page needs to be unique. 

Coding the page is not the hard part; your engineering team can surely create the template and design you would want through their expertise. Here are some tips to consider when you create your page.  

Single Page, Single Intent 

There is virtually no need to create a landing page for every possible permutation of the keyword, as Google is pretty good when recognizing synonyms. You will want, though, to create one page per user search intent. 

Remember Google defines intent – not you. This intent is based on algorithms created from user data, and as with any algorithm, it is not perfect. 

To do this, spend some time or so to browse the Search Engine Results Page (SERPs), and pay specific attention to words highlighted in bold by Google, as well as the “people also searched for” section of the page. 

Check your competitors. Do they have multiple pages per intent or various pages for quite similar intents? 

At the end of the day, this will be based on your judgment. Will you follow the competitors who have multiple pages per keyword, or one page only for similar keywords? 

You can always correct it later if you recognize that your judgment is wrong.

Avoid Doorway Pages

Google defines doorway pages as “pages created to rank highly for specific search queries.” Well, that is basically SEO. 

But do not fret, for Google has also given some questions to assess whether the pages are a prohibited doorway page or not. 

1. Is the purpose for search engine optimization, or to lead visitors into the actual relevant portion of your site, or is it a vital part of the user experience? 
2. Is the content on the page specific yet the page is meant to rank for generic terms. 
3. Do the pages duplicate items that already exist on the site to capture more traffic? 
4. Do these pages not create value for users, and serve as merely a passageway for affiliate traffic? 
5. Does the page exist as a mere “island?” Are they almost impossible to navigate from a different part of your site?  

All SEO-developed pages are to an extent a doorway page, but Google is fine as long as it is vital to the site’s existence, and it delivers some sort of value. 

As you create a page, ask yourself the following questions. 

1. Is there a reason for this page to exist if not for Google? Can I arrange the page as such if it is not?
2. Is there some sort of value that the algorithm can derive from the page?  
3. How will a machine know if the page is different from others? 
4. What can I do to make a page ‘linkable’? 

At a minimum, each landing page should have enough unique content that the algorithm perceives your page to deliver unique value to its users. And ideally, the content must be useful to the users. 

You will use one, or a combination, of these elements to fill out the page. 

1. Photos
2. Reviews
3. Business listings
4. General information about the business
5. Statistics
6. Relevant data
7. About
8. FAQs

And the list goes on!

The challenge is producing and generating content unique enough for different landing pages. 

Here are strategies companies use to create enough content across hundreds of landing pages. 

• Two-Sided Marketplace 

Online marketplaces, with vendors on one side and customers on the other, are practically synonymous with SEO.  

Vendors are incentivized to contribute content as it will help them in their business, and consumers can leave reviews. Additionally, the data scraped and collected by the marketplace, such as price quotes, locations, or preferences, could be used to fill the content gap. 

Examples of these include eBay, Houzz, Care.com, Expedia, and Rover. 

• Community

Online communities naturally create limitless original content, as long as there are active users on the site.  

Q&A sites, such as Quora and StackOVerflow, have Q&A pages that frequently rank on Google. Forums, such as Pinterest and Reddit, revolve on submissions and comments for content. 

• Proprietary Data 

If your site has access to data, or it inherently focuses on generating data, then it has the potential to create great landing page content. Furthermore, proprietary data is easily linkable to other parts of your website. 

Such examples of these are Zillow and CoinMarketCap. 

• Curation 

Curator sites, such as Yellowpages and Indeed, actually have a little of their own generated content. They run their own web crawlers to scrape the content they need, which are curated listings. 

• e-Commerce 

If you want to sell products, then you must put information such as quantity, price, specifications, product descriptions, and shipping details. Reviews are another source of content in this kind of site.  

Many e-commerce SEOs write custom descriptions and take custom photos of their products, more so when the product is not unique to them. 

• Editorial 

Hand-written content is a web staple, and if all else fails, then you can always generate original content through writing them. Editorial content has a fair bit of advantages. You don’t have to rely on other people creating them, and you can always use SEO tools to include keywords. 

Handwritten content is also another way to add some bulk and flair to your page, even if you already have user-generated content.  

Building Links at Scale 

Link building has no exact process in programmatic SEO; it is much more nuanced. The companies have to figure it out on their own. 

So here are some strategies to help build links that scale. 

• Ego-Bait that Scales 

Most two-sided marketplaces use some form of “ego-bait” to get their supply and then get a link back. Often, these come in embeddable badges. 

Typically, you offer an embed code of a badge or some form of visual reward to a business on your platform. This may make the site feel special and be compelled to show off the identification on their website. 

This creates a sort of “linking back” to the site that gives out the badge. See? It is quite an effective link building scheme at a relatively low cost. 

The link building scales with your growth, so go ahead and give out those badges. 

• Viral Content 

Take Movoto as an example. They earned their place in the search engine by being the “Buzzfeed of Real Estate.” Even though they have a fraction of the budget of large competitors such as Zillow, they managed to find a way to get to the top of the rankings. 

How? 

They create goofy list-based posts that are designed to go viral. As a result, traffic comes into their site to read the posts. 

Movoto spent quite a lot to get the posts the traction they need, and as they went viral, they naturally gained a lot of backlinks as well. 

It is another way to create backlink scalability, though I have to admit that listicles and “Buzzfeedy” types of articles are not how they used to be. 

• Proprietary Data 

At the center of Thumbtacks, a website dedicated to home services is the annual Small Businesses Friendliness Survey.  

They spent the first three years of the company sourcing a supply of local businesses on their platform. Then the suppliers are surveyed about the businesses. 

With that data, they generate content. And not just one or two pages, they create original content for each and every state and metro area in the US. 

These reports earn backlinks from news outlets and websites, commerce chambers, and government sites. 

• Commissioned Surveys 

Surveys are another great way to create data that others will be interested in. And even if it is relatively more expensive than proprietary data, you can tailor your questions to predict or otherwise elicit a particular response. 

For instance, Bankrate has been publishing its commissioned surveys for an extended time, and they almost always get recognized by a media chain or outlet.  

Bankrate employs the help of professional surveys and research firms typically to conduct surveys, but there are cheaper alternatives such as Google Surveys or SurveyMonkey. Many authority sites and media, though, will not cover your story unless from sources of repute, such as a professional institute. 

• High Link Intent Content 

People write digital content every day, and are always on the lookout for sources to use. You just need to be discovered as that source. 

For example, if I am going to write a topic, I would probably use the first couple of results as my source of information. These top results will gather more and more links, solidifying their place in the SERP ranking. 

For this to work, though, you need to produce content that has an intent to be linked. You would want informational intent with a pretty high search volume. 

From there, do what you need to do to get your piece recognized, through pitching the article, or guest posting, or other ways. It may be a lot of work, but it will be worth it in the long run. 

Eventually, though, SEO compounds through time. The older some websites get, the easier it is for them to generate links out of thin air. 

You will come to a point where links and mentions are earned just by being a recognizable brand. Until then, though, keep on link building. 

Common Technical Issues in Programmatic SEO 

At this point, you will have already acquired quite a decent understanding of programmatic SEO and how to build SEO-optimized landing pages. This section is dedicated to the problems and inefficiencies a website applying programmatic SEO may encounter. 

Relevant: Also check what are White Hat SEO techniques here

Unable to Get Pages Crawled

If Google cannot find a page, then it cannot rank it. To avoid this, make sure the pages are neatly categorized, such as below. 

Categorization leads to two things: it establishes easy bread crumbs, and ensures that Google can find your page. However, you do not want Google to only find those pages, you want it to rank them. 

Internal Linking

Page rank is still central to Google’s algorithm, and page ranks are passed through internal linking. However, internal linking is one area where what is best for users versus what is best for SEO is quite different. 

Here are some of the best SEO-related practices. 

• Breadcrumbs 

Breadcrumbs with your defined hierarchy will ensure that each subcategory will be linked to parent categories. A schema markup also allows Google to know the authority of your site to make the website rank much prettier on the SERP. 

• Related Categories 

You do not want to link only up and down the hierarchy, you also want to link to pages at the same level. This is typically reflected in the “People Also Searched” or “You Might Also Like” sections of the page. 

There are typically different ways of implementing this. Categorization is one, as when sites give recommendations such as alphabetical order to decide the links to show to the user. 

• Anchor Text 

Tell Google what the page is about. Simple, easy, direct to the point. 

• Most Linked is the Money Link 

The most linked pages should be your money pages, the most valuable. Relevance matters in this type of linking, so be careful. 

This is one reason why numerous companies link to the most valuable pages and directly to the big categories.

• Minimize Clicks 

Your category pages should not be more than two clicks from the homepage, and for individual listings, aim for only three or four clicks. 

Making Sure Google Likes the Content

Indications of if Google likes your content include the indexation rate and cache date. 

Create an XML sitemap for each category in your site using software or tool. There are ones available online for free, so you can do it if you want to.  

Submit the sitemap to Google Search Console, which will then tell you which elements in the sitemap are indexed. The indexation rate should be equal or above 90%, lower than that and you may have a problem with one or some of your pages. 

When sites do their programmatic SEO well, the sites will be cached within the last week. You can also check the cache dates of your competitors to get a general benchmark; above that; then it may be a sign that the content is not very well perceived by google. 

Page Speed

You would want the landing page to deliver the most up-to-date information, primarily if it acts like a browser of sorts. Ideally, you want to have a crawling and catching script that automatically updates the content if something changes materially.  

Your engineering team will know how to do this. Talk to them to see if they could set it up. 

So that ends the discussion about basic programmatic SEO. It may be kind of technical and hard, but successful implementation will pay its dividends in the long run. 

Enjoyed this post? Read these to know more about SEO topics, tips, and tricks. 

1. Do SEO like a pro with these tips
2. What is off-page SEO?
3. Amazon SEO Guide