Don’t just think of a robots.txt as a file that you are going to use to block search engines from crawling pages on your site, there are creative uses for this file if you just allow yourself to think outside the box.

One of the very first technical SEO fires I ever had to extinguish was one started by a robots.txt file. The startup where I was working launched a brand-new redesigned website, and we were excited to see the results. However, instead of traffic going up it inexplicably went down. For anyone that has ever had the misfortune of screwing up a robots file (spoiler alert that’s what happened), you will know that a robots noindex directive is not an instant kiss of search death – it’s more like a slow drawn-out slide especially on a large site.

Since I had never seen anything like this before, I was frantically checking everything else that might cause this slow dive. While I turned over every rock to find the issue, I chalked it up to bad engineering, an algo update, or a penalty. Finally, I found a clue. A URL where I thought a ranking had slipped was actually no longer indexed. Then I found another URL like that and another one.

Only then did I check the robots.txt and discovered that it was set to noindex the entire site. When the brand-new site had been pushed live – everything had moved from the staging site to production including the robots file which had been set to noindex that staging site. When Googlebot revisited the homepage next, it fetched the robots.txt and saw the noindex directive. From the log data Googlebot continued to fetch pages, but the rate started declining quickly. (For thoughts around why this is, I will be doing another blogpost on this topic.)

We fixed the robots file as fast as we could and then waited for Google to love us again. If you are ever in this situation, this is your warning that this is not an instant fix. Unfortunately, it took longer to recover our positions than it did to lose them, and I have seen this be the case every time I have worked on an issue like this over the last decade. It took about five days to lose all of our rankings and weeks to recover them.

Ever since I dealt with this issue, I have had a very healthy respect for robots files. I carefully consider alternatives to using them to address issues that can be fixed in other ways, and only add folders to them if I really never want them to be indexed.

Robots.txt can have other uses

This healthy fear aside, I have also found some great uses for robots files that are completely harmless to your search visibility and could even cause some confusion with your competitors. Note that if you do any of these you should add a comment tag in front of any non-code so you don’t inadvertently break your robots file. These are just examples of what you can do with a robots file once you start being creative.

1. Recruiting: Use robots files to advertise open SEO roles. The only humans that are ever going to look at a robots file are either search engine engineers or search marketers. This would be a great place to grab their attention and encourage them to apply. For an example of this, check out TripAdvisor’s robots file here.
2. Branding: Showcase something fun about your brand like Airbnb does. They write out their brand name in ASCII code which reflects their design sense and make a casual reference to careers at the company.
3. Mystery: Anything that is in a robots file as disallowed that appears to be a traditional file (and not a reference to scripts) will inevitably be something that people will want to check out if they find it. If you produce a technical tool, this might be the place where you can offer a discount to only the people that find the link in the robots file with a URL like “secret-discount”. If someone goes to the lengths to explore your robots file, they are most definitely going to navigate to a page that references a secret discount!
4. Subterfuge: Any good marketer should always check out their competitors robots.txt files to see if there is anything that the competition is doing that you should keep track of. Assume that your competitors are going to check out your robots file and do the same to you. This where you can be sneaky by dropping links and folders that don’t exist. Would you love to do a rebrand but can’t afford it? Your competitors don’t need to know that. Put a disallow to folder called /rebrand -assets. For added fun you can even put a date on it, so the competition might mark their calendars for an event that never happens. When you start being creative along these lines, the ideas are truly endless. You can make references to events you won’t be having, job descriptions you aren’t releasing or even products with really cool names that will never exist. Just make sure you block anyone from seeing a real page with either password protection or a 404, so this does remain just a reference to an idea. Also, don’t take this too overboard into anything immoral or illegal!
5. Taxonomy: A robots file really just exists to disallow folders and files not to allow them; unless your default is disallow (like Facebook’s robots file) and you just want to allow a few pages. An exercise where you sit down with content or engineering to add folders to be allowed might be a good way to find out if there are folders that should not exist. Truthfully, the value in this is just the exercise but you can carry it forward and actually lay out all the allowed folders in the robots file as a way of detailing the taxonomy.

As you can see a robots file is not just a boring requirement – and it is a requirement, every site should have one. There is a lot you can do with it if you think of it as another asset or channel on your website. If you have other ideas on how to make robots files useful, I would love to hear them – please email me!

The Covid19 pandemic has completely upended our entire economy and lives in a way that would have seemed unimaginable just a couple months ago. Most stores are closed, the travel industry has been idled and the unemployment rate is frightening. The prospects for small businesses both on and offline seem absolutely dire.

In 2005, Google along with the other search engines that at the time still mattered launched a new attribute to describe a link termed the “nofollow”. The nofollow attribute was a way for Google to disincentivize a spam issue specifically to acquire links that was spiraling out of control. With the nofollow attribute, websites could in theory negate the SEO value of any outbound link simply by declaring them to be nofollowed.

As a result of the Coronavirus pandemic and the associated recommendations to avoid leaving the home, there are a myriad of products and services that are on a temporary pause. As a result, it would not be prudent to continue paying to market these items.

Jargon kills the internal influence and unnecessarily impacts the effectiveness of a marketing team. Jargon, abbreviations, and bullets without explanation have their utility, but they should not be used in a public setting that includes a non-marketing audience.

In the military, operations are typically given a code name to align everyone to the mission and keep the actual objective secret. As the practice took hold in World War I and II, the naming convention of missions began to take on names that connoted strength. Winston Churchill had a role in personally picking the names of missions and even set out guidelines that encouraged planners to use names of heroes of antiquity, figures from Greek and Roman mythology, and names of war heroes.

Within technology project management there are many popular methods for organizing workflow. The two most common are Waterfall and Agile. There are advantages and disadvantages to each one, but for SEO Agile is the way to go.

In this year’s Super Bowl like all the past years, Google ran a commercial. In prior years, Google highlighted search, but this year they decided to put the focus on the Google Assistant.  Based on immediate Twitter responses as well as recaps about the ads that ran this year, Google definitely hit the mark by causing an emotional response.

In their bestselling book, Blue Ocean Strategy, authors Chan Kim & Renée Mauborgne promote a creative approach to marketing. They contend that when companies compete with each other in a known space market space with defined boundaries, this should be a termed a Red Ocean.

Very often in conversations about digital marketing, I hear SEO (free search traffic) referred to as the opposite side of the spectrum of SEM (paid search marketing). In this view, SEO and SEM teams are competing with each other for resources and budget. In my opinion, this is not the right way to view these teams and is in fact counterproductive.