Eli Schwartz

Category

opinion

Home / opinion
robots file
opinion

TurboTax’s Robots NoIndex Brings Regulatory Attention to SEO

Update May 22, 2019:  

Additional fallout from this issue includes the City of Los Angeles suing Intuit, Senators including Bernie Sanders and Elizabeth Warren launching an investigation, and the state of New York launching a probe. All of these efforts refer to the robots directive as a deceptive search practice. As of today, there is still a robots directive on the internal pages mentioned in the earlier post.

“These companies’ actions in hiding Free File [Alliance] from search engine results — and therefore from consumers — in order to artificially inflate profits and deprive low-income consumers of cheaper product merit investigation as unfair and deceptive practices,” 

Original post:

Last week, ProPublica broke the news that both Intuit and HR Block placed a Robots no index directive on their free tax filing landing pages. This discovery by ProPublica came from of a wider investigative effort that revealed how TurboTax was allegedly steering people eligible to file their taxes for free under an IRS program into a funnel where they had to pay to file.

As a result of ProPublica’s post, Intuit (the parent company of TurboTax) announced that they had updated the robots file on their free filing site and they were:


“undertaking a thorough review of our search practices to ensure we are achieving our goal of increasing eligible taxpayers’ awareness of the IRS Free File Program and its availability.”

Spoiler alert: At the time of this writing on April 30th they only removed the robots directive from the homepage of the free file site – which has very little content, but not from the support and about pages with the content that might actually rank.

Here’s the robots file on about us:

Interestingly, even if they were to remove that robots directive, they would still have the canonical in place which also might lead to Google not indexing the page.

As a result of these robots directives, here’s what this free filing subdomain looks like in search.

As you can see the homepage is now indexed thanks to the quick response by Intuit, but those other results are buried in the supplemental results and don’t have a snippet. (Supplemental means that you would need to request to see “omitted results).

Tangent aside, this incident has kicked off what could be a fascinating debate in congress, the FTC, and at the IRS on what exactly a robots directive does and why it may have been used. I am willing to give the SEO team at Intuit the benefit of the doubt that this was an oversight that stemmed from sculpting crawl budget rather than a nefarious attempt to squeeze taxpayers.

Regardless of the intent, regulatory bodies are now going to turn a spotlight on SEO. Representative Katie Porter has called for both the IRS and the FTC to investigate.

Her letter has some remarkable paragraphs as she specifically references the issues around the robots file. The letter seems to imply that without being indexed by search engines, users will be inconvenienced. This steps into a whole new argument over whether being indexed by Google is a right or just a privilege. If sites are required to be indexed by Google, shouldn’t Google be required to rank them? And, what about if they break Google’s TOS?!

She concludes the letter by stating that this “misdirection” violates the law as unfair competition.

Without weighing on whether Intuit did this deliberately and whether it was right or wrong, the fact that choosing what pages to index on a site could remotely be considered illegal is a very scary prospect for any one conducting any SEO efforts.

The reality is that Google is not required to rank or even index any site, and choosing what and how to expose pages to Google is a vital tool in an SEO toolkit.

Unless this somehow goes away, expect there to be some sort of congressional discussion on how Google interprets a robots directive and should site be required to make pages available to Google.

opinion

Google is allowing users to tag movies… again

A little over a year ago, a searcher noticed that Google was requesting users to tag movies with adjectives. This week, I stumbled upon this test on one particular movie; however I was not able to replicate it on any additional movies.

Just like last year, Google is calling this feature “new”; although it clearly is not new. I find it interesting that this test has remained in limbo for a full year in that they have not rolled it out completely nor have they killed off. Either way, Google requesting user feedback around movies could be an attempt at adding additional context within search results around movie search or it could be a way of gaining deeper personalization data so they could even recommend movies like Netflix.

Have you seen this test in search?

Here are the images of the full flow from requesting the vote to tagging to the thank you message.

 

Tag movies

 

opinion

Hiring for In-House SEO

Hiring in the Bay Area is generally hard with the current state of the booming economy; however, I found it especially difficult to find the perfect candidate to fill an in-house SEO role. Thankfully, I found that ideal person to join our team, but that process of filling the role was incredibly eye opening, and worth sharing with others in the same boat.

Experience required

Since SurveyMonkey now has more than 750 employees dispersed over five offices, I was looking for someone who had experience both working internally in an SEO role and working with many stakeholders. While some of the best marketers and SEOs I know work at or run their own agency, owning SEO as a full-time employee has a completely different dynamic than at an agency.

An SEO employed by an agency is generally focused on providing value to one or maybe a few different contacts within a company, while an in-house SEO needs to satisfy a whole host of executives and cross-functional leaders. Additionally, an in-house SEO has the luxury of working on projects that could take a bit longer to bear fruit, while an agency team wouldn’t have the space to just put their heads down and not produce a tangible deliverable for multiple quarters.

For these reasons, I particularly wanted an individual who had the experience of working cross-functionally within a company on SEO projects and someone who had the creative freedom in the past to develop initiatives that have more moving parts and offered more impact to a company.

Skills required

Even within that subset of people who had experience working in-house, I discovered that skill sets were extremely diverse. As I learned, there are many companies that have internal SEO teams; however, the bulk of the SEO work is conducted by agencies. The individuals with the SEO title inside the company actually functioned as project managers for the agencies, and even for things as simple as keyword research or cross-link planning, they were lacking basic experience. Essentially, the agency owned all the knowledge of how SEO worked for the particular company, and the internal contact relied on this to get their own job done.

After in-house experience, my only other requirement was that any candidate possess the ability to be both analytical and curious. Without creativity, a SEO would only be able tofollow best practices and optimize accordingly. I was looking for someone who could help us test the limits of how Googlebot works and use this knowledge to increase our visibility. Following best practices can only take you so far; the edge comes from being creative.

Needle in a haystack

Aside from the challenge of sourcing diverse candidates with the right skills, I also found it difficult to find people with the right experience working in-house. After running into walls multiple times, I spent a significant amount of time digging through LinkedIn data to better understand what I was up against.

Here are some of my key discoveries:

  1.     Many of the top internet companies do not have anyone with the word SEO in their title. Either SEO belongs to a team or employee with other words in their title, or they outsource the channel to an agency.
  2.     Despite its reputation as the nexus of the internet, the Bay Area came in 3rd behind Los Angeles and New York City with individuals using the word SEO in their job titles.
  3.     Content marketing, which is an initiative closely tied to SEO, has 50% more individuals using those words in their title.
  4.     Growth marketing, which is another functional area that typically includes SEO, shows about 10% of the number of people who use SEO. Not surprisingly, the largest hub of growth marketers in the U.S. is in the Bay Area.
  5.     Many people seem to graduate away from SEO, and it is hard to find individuals with more than 5 years of experience strictly in SEO.

Given the data I found, my expectations of finding someone with years of experience working in technical SEO at a site with millions of visitors might have been a bit unrealistic. The people who match this description are few and far between, and most of them were likely not looking for a job right when I needed them. A more manageable expectation for anyone in a similar position is that they may have to find a curious and innovative person with a bit less experience and be willing to train them.

A company might also want to seek out the assistance of an agency or consultant as a stop-gap so their SEO efforts are not held back while they search for that perfect candidate.

In summary, SEO talent – especially in-house SEO talent – is a lot scarcer than I realized, and it’s necessary to be flexible in a candidate search to prioritize curiosity and capabilities rather than experience. The most important aspects to look for are aptitude for creativity and analytical thinking. Ideal interview questions should be crafted to identify people with the potential to succeed in SEO – even they don’t have a long track record of experience.  

Good luck recruiting!

opinion

Technology in Cuba: 3 Quick Observations

Over the holidays this year I had a unique opportunity to spend some time in Havana, Cuba on trip to support the Cuban people. Having been to many developing countries around the world, Cuba was by the far the back of the pack when it came to technology.

The internet is uncensored; however, it can only be accessed with timed prepaid access cards in specific areas known as WiFi zones or parks. Internet speed was fine for picking up email or checking Facebook, but was nowhere near good enough for a Whatsapp video call.

Additionally, anyone who has ever seen pictures of Havana knows that the streets of Havana have cars made in the 1950’s. While I expected to see many of these cars, I was surprised that these classic cars were by far the majority on the roads. As an aside it can be quite jarring to see a 1950’s American or Soviet car with a 21st century sound system.

While the trip is still fresh in my mind, here are my 3 quick technology takeaways.

  1. Having constant access to the internet isn’t really that necessary. It was actually kind of pleasant to be in a place where people were not constantly on their mobile devices. In almost everywhere I have ever been in the world, you can’t walk a few feet without seeing someone on a mobile device doing something, but not in Cuba. Except for people using their phones to take pictures, there are hardly any phones to be seen. I only heard a phone ring a few times during my trip, and I definitely didn’t see anyone browsing Facebook while waiting in the long lines to get into stores and banks.
  2. In our society, people buy or lease cars every few years. I once bought a used car with 75k miles and was told by the salesman that it had completed ¾ of its expected life. Most cars will survive well past 100k miles, but cars that make it to 300k are true rarity. From an age standpoint, there are hardly any cars on our roads older 20 years old. In Cuba, they are driving cars that are 60 years old! They are driving cars made by people who are likely no longer even alive anymore. This got me wondering whether we are too quick to throw things away; especially cars, and maybe with some tender care everything might just last longer. During my travels in Asia, I certainly saw things being sold and reused that would most likely be in a garbage heap in a developed country; however, nothing was as old and complicated as a car from the 1950’s.
  3. In many of the developing countries I have been to, it has been amazing to see the transformative powers of mobile devices and the web. Moped owners in Indonesia can now earn an income by joining an Uber like network called Go-Jek, Filipino programmers can supersize their income by broadening their client base through Upwork, and Cambodian tuk-tuk drivers can land customers through Tripadvisor even before they land in the country.

    In Cuba the impact of the Internet is even more apparent since it is hardly available and minimal progress from slight access shows the potential of the web. The Cuban economy is heavily reliant on foreigners visiting Cuba. Since there are very few actual hotels, many people stay in private homes. Airbnb is available, but it is quite challenging for hosts to run their business with limited Internet access. I can only imagine how much Airbnb-like services would impact the Cuban economy if the Internet was more prevalent.

    On the same note, Havana is noted for its music scene, but local artists don’t have the same promotional avenues available to them that artists around the world have with social media. It was fascinating to hear artists promote their social media channels with reminders to follow them when Internet access is available rather to just “like” them now. If Cuban artists could build their social media followings like other artists, who knows if might we see and hear more Cuban art.

Cuba was generally a fascinating place, but the ability to feel like I traveled back in time made it even more interesting.

 

opinion

How Whatsapp’s Snapchat Cloned Stories Feature is Doing

With its much-awaited IPO on the New York Stock Exchange, this is Snap’s biggest week in its short company history. For the longest time, Facebook, Snap’s most obvious competitor has been using every tool in its arsenal to undermine Snapchat’s success by offering the same features to Facebook users. Last week, Facebook aimed its most powerful weapon at Snap by integrating a clone of Snapchat’s most monetizable feature into Whatsapp by allowing users to post “status” updates that expire after 24 hours.

Companies use their IPO’s on public markets to not only get a massive infusion of new capital, but also to grow their user and customer base with the tremendous amount of media coverage that’s directed at them during this time. For Snap their IPO is also an opportunity to reframe how people think of their product. They would prefer to be known as an app with high user engagement with a rich potential for advertisers rather than a 1:1 messaging app for millennials.

With Whatsapp’s launch of a similar feature to Snapchat’s stories just days before the IPO, Snap’s communication about their key revenue driving features potentially got a little bit harder to share. While Facebook as a company is Snap’s largest competitor, Facebook’s Whatsapp is the specific domestic app (WeChat in China is larger) with the most users at 1.2 billion and the greatest potential to stall Snap’s growth.

Now that new Whatsapp status has been out for a week, I decided to see how well it has been received. On Saturday evening, which was the first time I saw the status feature on my Whatsapp app, I saw 5 status updates from my 250+ Whatsapp contacts. Now 4 days later, I do not see any at all.

A quick search on Twitter shows a lot of people complaining about the new status feature, and asking for the ability to do a simple text status update that all users will see underneath a specific contact. Possibly because of this feedback, Whatsapp is bringing back this feature, and it will be called “My About.”

It will certainly take time and a lot of effort until this feature from Whatsapp gains popularity, if it ever does become popular, but in the meantime, it has not had the runaway success that might have impacted Snap’s IPO. Analysts might be quite bearish on the long term prospects for Snap, but nonetheless Snap had a great first day with an opening price of 40% higher than its offering price.

As Snap’s IPO buzz fades, we can certainly expect that Facebook will continue to iterate on their products in ways that undermine Snapchat’s appeal. We may even one day see sponsored stories a.k.a status updates on Whatsapp, but first Facebook is going to need to get brands on to the Whatsapp platform like Wechat did in China.