Eli Schwartz

Posts by

Eli

Home / Blog Archive
SEO

SEO Due Diligence for Investments and Acquisition – Lite version

It is prudent to conduct SEO due diligence when investing in (or acquiring) a business that relies on organic search traffic for a substantial amount of revenue. If the SEO foundation is somewhat tenuous the organic traffic is at risk of a decrease or even disappearing if something changes on the site, in the competitive landscape, or in a search engine algorithm update. 

It is important to keep in mind that while a site might have more ‘ownership’ of organic traffic than paid traffic –  which will just stop when the budget stops – the rankings and the traffic outcome are really at the whims of the search engines. That being said, rankings typically don’t disappear overnight IF best practices are followed and all SEO efforts are above board.

For the most part, any SEO due diligence projects I have conducted have been similar to an SEO audit, however the focus is usually on how stable the site’s traffic will be for the near future rather than what can be improved.  Anyone experienced with SEO could likely find obvious red flags in a site, but when investing a substantial amount of money into a site, you want to make sure you do this right.

Additionally, it is critical to understand what red flags are areas of opportunity versus which ones should cause the potential investors to turn and run.

Independent SEO Due Diligence

If you are in the early stages of looking at a potential website investment or acquisition, you might want to do some quick due diligence on your own.  To begin this process you are going to need access to their Google Search Console account, Google Analytics and server logs. You should also have a simulated web crawler like Oncrawl and a backlinks/traffic tool like Ahrefs on hand. 

The first step in this exercise is to see if there have been any negative SEO events in the recent past:

  • Check Google Search Console to ensure there has been a steady uptick in organic traffic
  • Check Google Analytics to see that there have never been any negatives drops in traffic since the site was launched
  • Check Ahrefs to see that there have never been any drops in visibility

If there have ever been any drops in visibility ask for clarifying details.

Technical due diligence

Next it’s time to start looking under the hood at the technical implementation of the site. Launch your crawl on Oncrawl and let it run without any filters at all. If the site is small you can use Screaming Frog’s free tool.

  • Look for large numbers of broken pages 
  • Look for chained redirects  (page → page → page) 
  • Look for duplicate content
  • Look for content that is no-indexed
  • Look for content that is indexed but shouldn’t be
  • Look for proper use of on-page meta tags
  • Drill into the internal linking architecture
  • Understand how pages link to each other and with what keywords
  • Manually look at the site for unindexable content 
  • Note the URL structure
    • Are there query parameters?
    • Are all pages connected to the root domain
    • Are there many subdirectories in the URL
    • Do they use subdomains?

Backlink Due Diligence

Moving on from the technical audit it’s time to start looking at the backlinks. Use the Ahrefs backlink checker to get a sense of the quality and quantity of backlinks.

  • Check for patterns of deliberate gray backlinking
    • Non-varied anchor text
    • Links that don’t seem to fit content
    • Links on non-relevant pages
  • Understand the quality of the links using a domain score metric
  • Dig into the links to reveal how they were obtained

Now let’s shift to the actual search results and start running some site queries.

  • Conduct a search on Google for the brand and make sure the homepage shows up first.
  • Notice how many pages are indexed in Google
  • Conduct a site: query with some common spam words appended to make sure that the site has not been hacked

Finally, we can assess the quality of the site itself

  • Analyze the quality of the content on the site. 
    • Is it in depth?
    • Are there typos?
    • Grammatical errors
    • Where is it located?
      • Blog 
      • Resource page
      • Homepage
    • What is the readership level
  • Using Ahrefs again look at what kind of pages are ranking on the site
  • Look at Google Search Console to see the kinds of terms that are ranking
  • Look at Google Analytics to make sure that time-on-site for content pages isn’t lower than the amount of time it might take to read the content.

Following the above steps should give you a good sense of the staying power for existing traffic. The next step would be to assess the conversion funnel of traffic that arrives from search engines. Regardless, the outcome of the quick SEO due diligence audit is to know whether you might want to spend more time researching the potential investment. This quick audit should in no way substitute for a full audit which must be done before any funds trade hands.

I will follow up on this post on how to do a truly in depth audit.

SEO

Podcasts of Search and Digital Marketing

I frequently get asked by people new to the world of digital marketing for some resources on learning online marketing. When I first started learning SEO, my education came from a few books I found. Those books still exist, but they wouldn’t be the best way to learn something as dynamic as search in 2020.

As an aside, I learned so much from Aaron Wall’s SEO Book in 2006 and it was a brilliant mind dump of everything he knew about SEO.

Today, learning is all about multi-tasking and learning from podcasts is a great way to learn from actual practitioners without the need to even get dressed.

In no particular order, here are the best podcasts I like on digital marketing. If you know of any that I might have missed, please let me know.

SEO

When to start SEO

Unlike paid traffic, SEO is not an instant faucet of growth; therefore an SEO foundation must be poured long before a site actually needs the traffic. At the same time, SEO is not a simple side project and an SEO focus immediately upon the launch of a company risks diverting resources from other channels that will help a company to scale.

The challenge is figuring out the precise time when to start focusing on SEO. In a perfect world, the answer is to build that SEO foundation 6 months before it will be needed. “Needed” in this case is when the marginal cost of paid marketing starts to rise. Knowing when this will occur requires a view into the future which is impossible, so we need an alternative way to plan for this eventuality.

The best way to invest in SEO without really going all in to build in best practices very early on that will enable a pivot to SEO as soon as early signs appear that it is necessary.

Most importantly, SEO efforts should not be hindered by making mistakes that will be hard to unwind later on. This means not building on a site structure that is completely unfriendly to SEO best practices. This also likely means not using hosting and a CMS that will be expensive to modify for SEO.

Even if a site does not have an SEO focus today, there should be no reason to build against it with a complex javascript implementation or not using any text on a website. There may not a be a full on content marketing effort, but there should at least be some content that can be a hook for organic visitors.

Once the marginal cost of new users/customers from paid search turns south, this is when a company should really start investing in SEO. If they have not yet built out a taxonomy for content this is where they should start.

In addition, there needs to be some time set aside for building an SEO roadmap so SEO isn’t done a haphazard way, but is managed like any other product or acquisition channel.

Most importantly, this is when effort should be allocated towards content investments that are strategic for the business and not just blog posts.

In short, SEO efforts should begin immediately on the launch of a product but this is not always realistic due to the need for immediate growth. The second best option is to prepare for SEO growth, so it is available as soon as it is necessary.

SEO

CCPA (California Consumer Privacy Act) and how it impacts marketing

In mid-2018, the EU had the greatest impact on the tech industry EVER with the implementation of their overarching privacy law: GDPR. Since the EU doesn’t have the same lobbying culture that we do in the US, tech companies and the industry overall didn’t have enough influence on the wording of the law as they would have liked.

As a result the law was written with too much room for vague interpretations of what is allowed and what isn’t, so corporate legal departments needed to exercise extreme caution. May 25, 2018 the day that the law went into effect saw many companies turn off their retargeting campaigns in Europe, stop showing Google Analytics script, pausing CRM campaigns and even blocking their websites completely.

Companies that didn’t do business in the EU breathed a sigh of relief that they didn’t have to worry about GDPR, but that might have been a false sense of complacency.

CCPA

Hiding in plain sight there was a law being worked on in California called the CCPA (the California Customer Privacy Act.) Ostensibly this law is only designed to protect California residents, but as companies don’t do business any differently between states, the law has national impact.

The initial goals of the proponents for this legislation was to have something similar to GDPR here in the US, but thanks to the power of corporate influence the final result is much lighter.

Rather than pass onerous restrictions on how companies can handle personal data and then exact large fines for breach, the actual law focuses on the right to know what data companies have and the right to delete said data.

  1. You will have the right to know what information large corporations are collecting about you…and you should. Businesses use your personal information for their own purposes, including targeting you with ads, discriminating against you based on price or service level, and compiling your information into an extensive electronic file on you.  You should be able to know what’s being collected about you.
  2. You will have the right to tell a business not to share or sell your personal information…and you should. California law has not kept pace with changing business practices. Businesses not only know where you live and how many children you have, but also how fast you drive, your personality, sleep habits, health and financial information, current location, web browsing history, to name just a few things.
  3. You will have the right to protections against businesses which do not uphold the value of your privacy…and you should. Businesses that collect your sensitive personal information should take basic steps to keep it safe. Right now there are no consequences if they don’t, and this law will introduce some consequences.

While this law might not be as scary as GDPR, it is only a first attempt. Many tools that companies have relied on for marketing like retargeting, data brokering, etc will become a thing of the past once full restrictions on trading data and allowing users to opt out become commonplace.

The real fallout from this legislation might not be known for some time as it does not come into effect until the beginning of 2020. There is still time for the trajectory of the law to become even more pro-tech company.

Retargeting and custom targeting based on brokered data has been an absolute boon for both the advertising industry as well as marketers since it enables precise targeting rather than the spray and pray objective of broad targeting. Without the ability to use this data, companies are once again going to have to fallback on advertising blindly or explore other channels.

How does CCPA effect current tools and processes?

The law is still in a to-be-decided state. How it will impact common uses of data such as retargeting, data-sharing and CRM practices is still up in the air as the attorney general of the state gathers feedback from interested parties.

One glaring area of concern is how it might effect data collection through analytics platforms like Google Analytics, Omniture and similar. Since data is collected in an aggregate form rather than personalized, it is likely that not much will need to change for analytics. To date, there has not been an EU lawsuit against the usage of Google Analytics as it relates to GDPR, so hopefully there is no issue with the far less stringent CCPA legislation.

SEO as the savior

Regardless, the existence of a law that tamps down on advertising based on personal data is a huge benefit for SEO. GDPR helped spike interest in SEO, and with the implementation of a privacy law in the US, interest in SEO will reach new highs.

SEO relies on no data whatsoever to be successful and therefore its success falls outside the purview of the CCPA. Yes, Google, a target of the CCPA, might use data to rank search results, but you don’t need to be logged into Google to conduct an effective search. Search is just as effective when the user is anonymous.

Additionally, companies can even use search to retarget churned users if they can develop content and an SEO strategy that will reach these users. Paid retargeting is useful, but there are certainly ways to recapture users without needing to pay and track users across the web.

As companies pivot towards a cleaner form of marketing, SEO will take center stage as companies that seek new channels to reach their customers.

SEO

How to hire for an in-house SEO opening

Hiring for SEO is really hard and it is about to get even harder.

After an extended period of time where many large organizations chose to not focus on SEO internally, there is suddenly renewed interest in hiring for SEO. The following is my opinion based on independent research on the sequence of events behind this sudden surge in interest in SEO.

SEO demand plummeted

The dropoff in demand for SEO talent coincided with the acceleration of paid marketing primarily on Facebook back in 2015. With the latest efforts around Facebook’s privacy pivot – and Google’s too, some granular ad targeting options will no longer be available – meaning advertisers will inevitably start paying for non-targeted impressions. The net result is the CAC (customer acquisition costs), a metric more important than ad engagement costs will continue to rise.

When this first started, marketing budgets remained fairly stagnant, but advertising outcomes – a function of the higher CAC – trended lower. The slow squeeze on what used to be a very stable and consistent ATM (the cash machine – not any sort of advertising acronym) is forced marketing leaders to look for channels that might have an opportunity.

SEO revitalized

In many instances, this internal soul searching lead to a discovery that the SEO channel has been quietly driving a significant amount of revenue with very little investment. There may be someone already responsible for SEO or this seat could be entirely empty. Either way, the executive team decides that it’s time to resource SEO, but they don’t know how to prioritize their action items.

Even if they have an SEO audit in-hand, they still don’t know what items can be backburnered and which must be a full-stop-the-presses effort. The next solution is to hire someone to oversee SEO or better manage the individuals currently responsible for organic traffic.

And, this is where they hit a wall. Others are coming to the same realization and the ratio of SEO jobs to those that can fill the roles is very lopsided.

At the moment there are 17, 241 SEO jobs listed on LinkedIn

But there are only 25k people in the United States with jobs in their title. This means that the ratio of jobs to people is 2:3.

For comparison sake, the ratio of jobs to people in software engineers is 1:2 and we all know how highly sought after engineers are.

A very logical effect of the lowered demand for SEO talent meant that less SEO talent was groomed over the preceding five years. Having participated in the marketing industry through events and conferences, I had a first hand view of the shrinking SEO community. Marketers that might have developed SEO skills with a key focus on just SEO instead were diverted into other tangential roles that did not help them build the technical SEO skills now considered to be critical.

Additionally, as SEO was not considered a key function, salaries tended to be lower than other marketing roles. Anyone working in SEO that wanted to earn more money or get promoted was inevitably forced to take on roles outside SEO.

Talent imbalance

So now in 2019, we have a truly imbalanced market for talent. There is a rising demand to fill SEO roles while at the same time the number of people that have the necessary experience have been shrinking for years.

For specialized skills such as enterprise level technical SEO, international SEO, large scale image SEO, and even enterprise level local SEO, there is even a smaller pool of potential candidates.

Many of the open roles are looking for people that have 5+ years of experience – a cohort that is really thin when it was 5 years ago that this function started to be underutilized. Those that accrued the requisite 5 years of experience either worked at an agency which means there could be a skills gap for some of the necessary cross-functional abilities required for an in-house role or they are the rare candidate that has the right mix of experiences that are now highly sought by just about everyone.

SEO leadership

For organizations seeking an SEO leader that can help with strategy, planning, internal communication, and management, many of the roles require 10+ years of experiences. Finding an individual that worked in the few companies that actually resourced SEO over the last decade is even harder to find simply because of all the forces that pulled people away from SEO over this time period.

I believe there are only two ways that I think companies can overcome the talent shortage and fill their SEO needs.

  1. Make the SEO role at their company more attractive than any other role the candidate will be interviewing for at the same time. This goes beyond salary and benefits but more into the actual job role. The position should have significantly more responsibility than their previous role, and the hiring manager should be able to demonstrate how the role will significantly impact the organization’s bottom line. This more than anything will help sell the right role for a person responsible for SEO. There are too many SEO roles that are just about closing tickets and not really driving results. The right candidate will be excited by the possibility of growing a business through organic search.
  2. Use an SEO consultant. There many people with the right amount of experience, but they are not looking for a full time role. While a company might take some time to come around to outsourcing this functional role, the impact from the right SEO consultant could be even greater than a full-time employee. A consultant brings the know-how from all of their other clients in addition to the years of experience they likely had working in-house. Most importantly, it is far easier to onboard a consultant than to find the right employee. To find a good consultant one should rely on their network on Linkedin or use a service like GetCredo.com which will
    match you with the perfect firm from their vetted network.

TL;DR

In closing, if you are an SEO professional or someone interested in working in SEO reading this, know your value. The SEO skillset is in high demand, and you have leverage in finding your next role.

For the hiring manager, be aware of how tight the market is for this role and put on your best recruiting show.

SEO

What does search query intent mean?

In the early days of SEO, the process of choosing keywords to target was quite literal. It really mattered whether you optimized for the singular vs plural version of a keyword or which specific synonym you included in a piece of content. In the same vein, ideas such as keyword density (using a keyword multiple times), keyword order and even emphasizing a keyword with bold or italics were considered best practices.

With the advancement of Google’s artificial intelligence based search algorithm, Google can go far beyond just matching strings. With years of engagement history – meaning they know what users actually click- they know what people want even if the user can’t really articulate what they are looking for! 

Here I have searched for “finals time” and I don’t need to specify that its for the NBA, and Google shows me the time for the NBA finals.


Or when I search “drinks” Google gives me a maps result of places to go have drinks.

This is called intent matching. On a simplistic level, it means that Google can skip over whether a search query is spelled correctly, contains a plural or singular, or even the language of the query. On the content side, they are also “reading” the content and determining how it might match the intent of a user.

Again, they ignore word tense, spelling and raw keyword density. Relying on AI to make the match between user and website is a closer approximation of Utopian matchmaking then just using easily gamed metrics like string and pattern matching.

To understand this in practice, here are some example queries

Say a user was looking for “Legally Blonde” but couldn’t remember the name of the actual movie. They might search something like “Blonde lawyer movie” or “Blonde harvard lawyer”. In each case, the results will show pages related to the movie: “Legally Blonde”.

Granted, the page does use those keywords; however, they are certainly not the focus of the page. To really drive the point home, this movie will appear in the results even for terms that are not on the page, but clearly exhibit the same intent! In the movie, Reese Witherspoon’s character attended Harvard Law school. Queries that use the word, “Stanford” and “Yale” are considered to have the same intent as “Harvard” and the results are therefore similar.

It should go without saying that if a user’s query were to contain “attorney” instead of “lawyer” , Google would consider the query to be lawyer regardless and would even bold the word “law” in the search snippet.

Even more interesting, for the query “Blonde Yale attorney”, the same movie still appears in the results because as before even without a keyword match the intent is the same.

One final example of how search intent works is to use an example query that describes something a user is seeking rather than keywords at all. If I were to describe a snakefly in words, I might search something like “bug long neck and wings”. In this case, the first result tells me that it is a snakefly, but that might not be so surprising since the webpage has all the words from my query in the page.

However, if instead of long I searched “bug with stretched neck and wings”, the 3rd result will be one for Snakefly and the content does not even use the words “stretched neck”.

This is intent matching at its best and is a foreshadow of the future where keywords become less and less part of how SEO works.

SEO

How to clone Amazon’s SEO best practices to dominate new shopping verticals

This post was first published on Techcrunch on May 16, 2019

Amazon dominates the top ranking positions of Google for tens of thousand of ecommerce queries, but there are plenty of products in newer shopping categories where Amazon has not yet achieved SEO supremacy. Retailers in nascent verticals have an opportunity to follow Amazon’s SEO playbook and become the default ranking ecommerce website.

Achieving this success can be done purely by focusing on on-page SEO without the need to build a brand and a backlink portfolio that rivals Amazon.

For those unfamiliar with mechanisms of SEO, there are essentially two streams of SEO tactics

  1. On-page SEO – this is anything to do with optimizing an actual page or website for maximum SEO visibility. Within this bucket will fall efforts such as the content of a page, meta data, internal links, URL/folder names,  and even things like images.
  1. Off-page SEO – A key component of Google’s algorithm is the quality and sometimes quantity of the links from external sites that point to a page or website. At a high level the better backlinks a page or website has the more authority the page has to rank in search.

On-page SEO teardown

Delving into just their on-page SEO, their tactics can be divided into four distinct areas which we will go through in detail.

  1. Content
  2. SEO site architecture
  3. Cross-linking
  4. Page layout

If you are following along with this process, make sure to logout of your Amazon account or open up an incognito window. Google only views the logged out version of the site, so all of Amazon’s SEO efforts are focused there.

PAYWALL

Off-page SEO not necessarily required

Before we go further, here’s a bit more about why off-page SEO isn’t required here.

As one of the largest brands in the world, Amazon certainly has a unique ability to garner backlinks that any smaller competitor would not have, but this is not the magical elixir some would think it is. Backlinks are only a component of the algorithm and better performance on areas like relevancy and intent matching will always beat out a better linked page.

For example, the query “Michael Jackson”, Amazon is nowhere in the top 10 results and some of the ranking results have significantly less links than Amazon.com with nearly 4 billion backlinks according Ahrefs.com. This is because for this query, a shopping site like Amazon is not considered to be relevant. As soon as a shopping intent keyword is appended to the query like “michael jackson costume” suddenly Amazon does rank.

Interestingly, Amazon is the second ranking result behind a site that specifically sells costumes likely because Google has determined that a costume site is more relevant for a searcher looking for costumes. This underscores the point of this guide that with good SEO Amazon can be beaten and Amazon does not have an innate SEO advantage because of their brand.

Content

Content on Amazon is comprised of what they choose to put on the homepage, category pages and specific product pages. From an SEO standpoint, while the rest of the ecommerce world invested and continues to invest in category pages (think the index page for all products in a category), Amazon places all of their SEO investment into creating killer product pages.

Amazon’s page not only pulls all available text from the manufacturer, but they add content unique to Amazon such as comparisons, tech specs directly on the page, and finally reviews. Again reviews is an area that Amazon has been perfecting for years while competitors neglected their own SEO efforts.

A big thing Amazon does right is that they appear to have a better flagging system that does not allow funky titles to make it in to their feeds. As a result all of their important on-page meta tags of titles, H1’s and meta description are built with minimal errors.


Conduct a search for “discontinued” or “broken” on any other competitors and you will not be as impressed.

SEO is not just focused on text search, as images are content too. Every product on Amazon has alt text (initially created for the visually impaired, but now useful for image search) with the full product title included.

This helps for a number of reasons:

  1. If someone were doing a product search in Google images (or any image search engine) they would be able to find the Amazon product.
  2. When someone finds a product on the web and they want to do a reverse image search to learn where to buy it, Amazon is eligible to be found.
  3. As more people start using visual search tools like Google Lens, Amazon is one of the top results.

SEO site Architecture

Everything about the way Amazon’s site was built was constructed for SEO. Early on in Amazon’s SEO efforts, they had links to all of their categories right on their homepage, but now they only do this within Javascript which has arguable benefits for SEO. This is the basic optimization that most sites will implement, but the real power of Amazon’s architecture is in their taxonomy. From the beginning, Amazon was creating the taxonomy for every product in the world, and it is this site structure that makes so easy for both users and of course search crawlers to always know exactly where they are and how to find a new product without needing to search.

Compare the difference between Target’s baby category page and Amazon’s.

Amazon has category links on the top of the page, popular products and then brands.

Target highlights only two products, has category links within the page and can only find brands as a part of the faceted navigation.

Amazon’s best in class site architecture is also a key driver in their very clean URL generation. Compare these 4 URL’s for the baby category.

Amazon: https://www.amazon.com/baby-car-seats-strollers-bedding/b/?node=165796011

Walmart https://www.walmart.com/cp/baby-products/5427

Target https://www.target.com/c/baby/-/N-5xtly

Overstock https://www.overstock.com/Baby/35/store.html

From Amazon’s URL it is crystal clear what to expect while the others leave something to be desired.

Cross-linking

An underpublicized secret of SEO is that internal links will do more to increase organic visibility than building more  external links. Building upon Amazon’s superstructure of taxonomy, the site uses hierarchical connections to create related links that are actually related links. A search crawler dropped on any page will be led into a near endless of stream of pages that have a connection to the previous page. This negates any risk of there ever being an orphaned page and it helps create or support the ontology driven algorithm that powers Google’s intent matching engine.

While cross-linking is a key component of any web property hardy anyone gets this right like Amazon. Compare the cross linking of the iPhone on Amazon vs Best Buy.

Best Buy

Page layout

Finally, Amazon is able to pull together the elements of content and related links and build a page that search engines are easily able to process. Category pages build the brand cross links right into the page so they seem as natural as possible and don’t need to rely on faceted side navigation like other ecommerce sites.

On product landing pages they blend the perfect mix of product description, rich content such as media, Q&A and user reviews. The immediate landing page has all of the most important content without anything hid in a collapsible box like other sites do, and the UGC which has lower search value is included at the end of the page.

Compare how Amazon adds UGC vs target.com which hides the Q&A.

Target

Summary

If you want to build SEO like Amazon, there is a very clear playbook you can follow. The most important lesson to take from Amazon is to embed SEO into the foundation of a website. Without a doubt, URL’s, titles, headlines, and product descriptions need to be generated and placed without errors.

As evidenced by Amazon’s success over the last two decades, it is worth taking a step back to build a taxonomy so clean that every product has a perfect home. Amazon created niches of niches and this is a great best practice to follow.  Using this clean taxonomy, maximize on cross linking to the best that a site’s design can handle. Show related products, related categories, and even competing categories. A broom obviously might be paired with a dustpan, but only will a clean taxonomy allow one to make the connection to a mop.

SEO

How long does SEO take?

Understanding how long it will take to generate results is one of the biggest questions anyone looking to undertake any SEO effort is going to want to answer. You could know how to build infographic links, the best usage of backlinks and keywords and still not be able to determine the time it would take for the results to show.The challenge with this and many other questions is that there is no blanket answer. Each and every site has its own unique fingerprint in Google’s index and as a result all sites will perform differently.

A site with a long history in Google’s index that is currently held back by technical issues may see near instant results as soon as the issues are fixed. On the other hand, a site that is just beginning their SEO journey might take many months until the see meaningful results.

While the idea of a Google sandbox has gone by the wayside many years ago, there is still a concept of SEO needing to marinate within the Google results. It takes time for Google to crawl and index a new page. After that, it takes even more time to acquire positive signals like backlinks, performance metrics, and on-SERP click through rates.

Regardless, in nearly all cases I am certain that within six months nearly all sites will see positive momentum that validates their SEO efforts. The growth will be slow and hardly noticeable but when one looks back after a period of time they will always see a positive change.

SEO traffic is one of the slower channels to scale, but the investment is well worth it. My experiences helping to build SEO efforts from zero to millions of dollars has always proven out that SEO is the highest ROI channel by far!

The best of way of illustrating this is to think of a hike up to the top of a mountain. Progress will be slow and painstaking, and even after some time it may feel like the summit is still far away; however, if after an hour you look back you should see progress made. After 6 hours, the summit might still be far, but the progress will be undeniable (provided of course you did not get lost!)

climb a hill SEO

One final note: don’t measure SEO progress in rankings; rather focus on the traffic and revenue that results from organic sources.

 

robots file
opinion

TurboTax’s Robots NoIndex Brings Regulatory Attention to SEO

Update May 22, 2019:  

Additional fallout from this issue includes the City of Los Angeles suing Intuit, Senators including Bernie Sanders and Elizabeth Warren launching an investigation, and the state of New York launching a probe. All of these efforts refer to the robots directive as a deceptive search practice. As of today, there is still a robots directive on the internal pages mentioned in the earlier post.

“These companies’ actions in hiding Free File [Alliance] from search engine results — and therefore from consumers — in order to artificially inflate profits and deprive low-income consumers of cheaper product merit investigation as unfair and deceptive practices,” 

Original post:

Last week, ProPublica broke the news that both Intuit and HR Block placed a Robots no index directive on their free tax filing landing pages. This discovery by ProPublica came from of a wider investigative effort that revealed how TurboTax was allegedly steering people eligible to file their taxes for free under an IRS program into a funnel where they had to pay to file.

As a result of ProPublica’s post, Intuit (the parent company of TurboTax) announced that they had updated the robots file on their free filing site and they were:


“undertaking a thorough review of our search practices to ensure we are achieving our goal of increasing eligible taxpayers’ awareness of the IRS Free File Program and its availability.”

Spoiler alert: At the time of this writing on April 30th they only removed the robots directive from the homepage of the free file site – which has very little content, but not from the support and about pages with the content that might actually rank.

Here’s the robots file on about us:

Interestingly, even if they were to remove that robots directive, they would still have the canonical in place which also might lead to Google not indexing the page.

As a result of these robots directives, here’s what this free filing subdomain looks like in search.

As you can see the homepage is now indexed thanks to the quick response by Intuit, but those other results are buried in the supplemental results and don’t have a snippet. (Supplemental means that you would need to request to see “omitted results).

Tangent aside, this incident has kicked off what could be a fascinating debate in congress, the FTC, and at the IRS on what exactly a robots directive does and why it may have been used. I am willing to give the SEO team at Intuit the benefit of the doubt that this was an oversight that stemmed from sculpting crawl budget rather than a nefarious attempt to squeeze taxpayers.

Regardless of the intent, regulatory bodies are now going to turn a spotlight on SEO. Representative Katie Porter has called for both the IRS and the FTC to investigate.

Her letter has some remarkable paragraphs as she specifically references the issues around the robots file. The letter seems to imply that without being indexed by search engines, users will be inconvenienced. This steps into a whole new argument over whether being indexed by Google is a right or just a privilege. If sites are required to be indexed by Google, shouldn’t Google be required to rank them? And, what about if they break Google’s TOS?!

She concludes the letter by stating that this “misdirection” violates the law as unfair competition.

Without weighing on whether Intuit did this deliberately and whether it was right or wrong, the fact that choosing what pages to index on a site could remotely be considered illegal is a very scary prospect for any one conducting any SEO efforts.

The reality is that Google is not required to rank or even index any site, and choosing what and how to expose pages to Google is a vital tool in an SEO toolkit.

Unless this somehow goes away, expect there to be some sort of congressional discussion on how Google interprets a robots directive and should site be required to make pages available to Google.

1 2 5 6 7 8 9 15 16