It has been interesting week to say the least. From one standpoint I was excited, and from another completely nervous. Knowing that I perform whitehat SEO practices, I knew that the probability of being nailed by Google’s Penguin algorithm update was rather small, however, I still know that often times algorithm updates can take down the clean with the dirty.

That being said, still to this day most of my clients are not seeing anything if at all in terms of the impact of the algorithm update. It may still be too early to notice, however, from most of the articles from industry experts, many of the sites impacted have already noticed significant negative ranking changes. No, not 3-6 positions in the search result pages (SERPs), we are seeing 30-50 position adjustments at times.

For those unaware of this new algorithm update, Google has basically come out against SEO’s or webmasters who employed spammy SEO tactics to acquire links, or search marketers who decide to anchor text spam Google with exact match internal and external links. An over-optimization penalty basically focuses on low quality content with keyword rich optimizations throughout the site with external and internal links following suit. Sites who decided to mirror another site with a different domain name or even different content completely, yet internally linking the two sites throughout their  web ring, have seen some authoritative link dilution from the use of these tactics.

Therefore, in order to secure or attain the top positions within the search engines, I recommend the following:

1. Develop user rich optimized content – This is content that is truly going to help the users in one way or another. It will either offer them “how to” information, or “did you know data”, or even paint a picture of a complex topic in an easily digestible manner or simplify the purchase decision process.

2. Share your content – As more and more of these anchor text and link spam algorithm changes occur, the natural and temporal algorithm levers are going to become more and more important. Therefore, make sure you let the industry experts, bloggers, and leading authorities in your niche know about the information or helpful content you have developed for your target audience.

Both strategies will force you to constantly be chasing the user to provide the most relevant information for them in a fresh and timely manner. It will also keep you ahead of the future algorithm penalties, as the search engines are also trying to stay as relevant as possible to the users who leverage their engine to find content in order to grow advertising revenues via this traffic.

I am interested to know if anyone has experienced any specific penalties in the last few days and is brave enough to share their opinions.

Robots.txt

The robots.txt file is a simple text file that resides within the root directory of a website. The purpose of this file is to help search engines understand which content you would like them to illustrate to users within the search engine result pages (SERP’s) and which pages they should ignore. Google and Bing pay very close attention to this file, reason being, they do not want to potentially share information on websites which their webmasters have specifically requested be ignored.

Example: You are about to enter a building and on the front door read a sign saying, “Do not enter the first door on the right upon entry – this area is strictly forbidden”. Knowing that you are only wanting to reach the top of the building for a brilliant view to take a quick shot with your camera, you adhere to their request. Google is doing the same thing when their spider enters a site to do a crawl. They look for two critical files; a sitemap.xml and robots.txt. The sitemap XML is a file which illustrates all or most of the pages within your site as well as the last modified date to assist Google with a quick link to the majority of your content you want visible to users of search engines. Prior to running through all the links within the sitemap.xml file, the crawlers check to see whether or not there is a robots.txt file and which pages (or rooms of your building) they are not allowed to enter.

This file is extremely powerful and the search engines most definitely respect the directions given within it. However, many people make a few mistakes which often times prove to be extremely costly.

In order to allow Googlebot or other search engine crawlers to index your site, a simple command within a robots file could look like the following:

User-agent: *
Allow: /
or
User-agent:*
Disallow:

This, in essence, is telling all robots or crawlers that the whole building is fair game.  Crawl away and take all content you interact with back to your indices to share with your users.

On the other hand, if someone includes:

User-agent: *
Disallow: /

This tells the robots that you do not want them to enter the building at all. You are basically adding a “no-entry” sign to the front door of the building with a “Trespassers will be prosecuted…private property” comment. So why would you ever use this? Most webmasters when developing a new site, develop a pre-production environment for testing purposes and to experience the site on the web or sharing the new design and functionality with a client prior to rolling it live. That being said, they do not want the search engines viewing it or sharing it with the world quite yet…hence adding the “disallow everything” comment to the robots.

This causes problems for many site owners or developers being that it is a simple text file within the root directory and not dynamically updated via a content management system (CMS) or eCommerce platform. Developers will often times “unknowingly” publish all pre-production files live. It is at that time where the robots file (Disallow: /) moves its way to the root of their new live site. I have personally experienced this on numerous occasions by future clients who come running to us  believing their domain received a serious penalty from Google and we need to help them immediately, as they cannot find their new site in Google. As an SEO, there is always a quick checklist each of us have developed to run through while analyzing files like these to verify that nothing has been implemented incorrectly.

Unfortunately, that is not the only way in which you could block a search engine from serving up your content with the SERP’s.  You could also have written the wrong code within your meta name=”robots” code within your page source.

Meta Name=”Robots”

The meta robots field within the header of your source code serves a similar purpose to that of the robots.txt file with a few exceptions. Contrary to the robots.txt file, this file does not act as a “no-entry” sign. Search Engines are still able to crawl the page, however, you are now telling them whether or not to index the page. That being said, depending on how you communicate this to them could mean different things at times.

For example: If you wanted to tell Google that you did not want a page in their search results, you would simply add:

<meta name=”robots” content=”noindex“/>

This tells Google that I don’t care whether you like the content on the page or not (obviously it was not blocked within the robots txt file), I do not want you to show it within your SERP’s. The difference between this route vs. using the robots.txt file, is that it is just a guideline to the search engines and not a blockage. For the most part this will be obeyed, however,  there are times where Google may still include the page within the SERP’s if it has received significant inbound links. While this is rare, it still happens.

Another feature of the meta robots that webmasters use are “follow” and “nofollow”. What do these mean? These are used to tell Google and Bing whether to pass authority or relevance to the links on the page. These attributes were originally developed to help block “link juice” flowing out of your site to spammers who have tried to leverage your comments section to drive anchor text links as exact match inbound links to their sites. Adding the “nofollow” to your page or backend code of your comments will in essence tell the robots that you do not want them passing any votes from you on this page to all links within the page.

There are four types of meta robots implementations:

<meta name=”robots” content=”index,follow”>
<meta name=”robots” content=”noindex,follow”>
<meta name=”robots” content=”index,nofollow”>
<meta name=”robots” content=”noindex,nofollow”>

“Index” refers to whether or not you want the robot to include your page within the search engines. “Follow” refers to whether of not you would like the relevance or “vote” of the links mentioned within the page to pass to their link destination.

Why would you ever use “noindex, follow”? There may be a time where you have a resource that you want to keep private from the world, however, you may have given it to a few people to access the valuable links within it. You are now telling the search engines that you do not want them indexing this page within the SERP’s, however that you do want them to pass the trust and relevancy of your “link juice” to the pages linked to within the page.

Additional Meta Name=”robots” that are commonly used include:

<meta name=”robots” content=”noodp,noydir”

Each are comprehensive site directories that search engines often times use to replace a meta descriptions on a page with the descriptions of your site from each of these sites.
ODP – Open Directory Project (www.dmoz.org)
YDIR – Yahoo Directory (dir.yahoo.com)

One of the toughest aspects of any agency or search marketer is managing expectations in a quantifiable online marketing world. Let’s take a minute to remind ourselves how we bought media in the past. It probably went something like this from a billboard advertising sales representative. On a typical weekday, approximately 70,000 cars will drive by Hwy 43 and Good Hope in the direction of our banner advertisement. Therefore this $24,000 a month would be a sound investment if a large percentage of them come into the store or visit your website. What they didn’t tell you was that in fact only 503 cars would actually read your billboard ad. Regardless of the fake numbers mentioned in this example, an advertiser would never ever know the true economic impact of their billboard advertisement spend. Don’t get me wrong, I am not by any means bashing traditional marketing, I personally believe that it is still an extremely important part of the overall marketing mix. In fact when used correctly, you can leverage both online and offline in a very effective and fruitful manner.

The point that I am trying to make is that the digital world of online marketing has simply closed the gap between understanding where your audience truly is and what it costs to market directly to them. For example, one can now educate a client how much traffic they could expect on a given month from a specific keyword phrase after applying many click-through-rate probabilities. Not only will these advertisers receive traffic, but they will then understand what the traffic actually did while on their website and whether or not they enjoyed the content upon arrival and interacted with more pages, or simply left. All of this data is available within the respective analytics suite of the website, be it Google Analytics, Adobe Site Catalyst (Omniture) or WebTrends etc.

Why is this information so important? Let’s go back to my previous example where I speak about billboard advertising. What if “me” as an account manager or a traditional advertising firm could tell an advertiser…”Jim, not only did 76,518 cars drive past your billboard this month, 25,401 read your advertisement (viewed your search result on a search engine result page – SERP) after asking to see this content that your company specifically sells. In fact things were so great this month that out of the 25,401 who read your billboard ad, 8,325 decided to swing by your store to view the products in the ad (clicked on your SERP listing). In fact, you know the section of the store where you keep all of your “engines”, about 64% of the visitors decided to review and interact with only those products in the store prior to leaving. And lastly, out of the 8,325, approximately 167 of the visitors actually bought (completed an online purchase) an engine after speaking with your salesman for an average selling price of $1,595.”

Wouldn’t that be an unbelievable experience for the advertiser who is considering billboard advertising? Not to mention receiving a monthly report with that type of information? Well that is exactly what search engine marketing and other online marketing mediums are able to deliver daily. Because of this, our mediums are scrutinized and questioned more than their marketing mix counterparts mentioned above. So what’s my point? Don’t simply give a percentage of revenue to an advertising firm to throw out the window, analyze the different persona’s of your customers, identify the online demand of your products and services from the keywords used by each specific persona and create engaging useful content to give them what they are looking for in the easiest most thorough manner. Identify the marketing mix that makes the most sense online and offline for your business and continue to test different percentages of traffic in each to give you the highest return on investment.

How do I know which traffic medium is working and what isn’t?

1. Select a product or category from your website
2. Dive into your analytics suite and analyze which traffic sources are sending the majority of the traffic to these pages or sections
3. Understand how each traffic medium is interacting with content. For example – Are certain mediums providing a higher bounce rate than others or a lower time on site (T.O.S) or average page views?
4. If you are running an e-commerce site, do you receive a higher conversion rate or average order value from one medium to another? If so, can you reallocate your funds accordingly to take advantage of these findings? Do the same thing from a goal/lead conversion analysis across mediums

Go even further ⇒ Isolate the medium

1. Identify whether the medium providing the highest traffic, AOV, and conversion rate is doing it single-handedly or getting support/assist from other channels (via multi-channel funnel analysis or other attribution software)
2. Analyze the most popular pages in this medium from a visits, T.O.S, bounce rate, conversion rate, and AOV
3. Does the content that is performing have greater depth versus other areas of the site? Are their seasonal considerations I should be taking into account? Am I more price competitive in this area compared with market conditions or is the competition less limited for these products?
4. Once you gain the insight from these pages that are performing, you can take that knowledge as a solid baseline to analyze the lesser performing areas of the site in order to find gaps or quick hit opportunities. Check top exits pages, leverage the weighted sort feature in order to gain critical mass for a data driven decision. Identify which landing pages have the highest bounce rates or least amount of engagement.

I could obviously go on and on as this is a never ending process. Make sure you don’t fall into the trap of using macro-level metrics (i.e. Overall site conversion rate) to benchmark whether a specific traffic medium is working or not. You need to dive into the micro-level metrics within each medium to give you the proper visibility to make the appropriate decisions related to spend and effort spent.

What do I mean by “in it’s entirety”? Nearly all website modifications or additions will effect SEO to some degree. It is no longer about simply verifying that your on-page optimization is keyword targeted…or the inbound links anchor texts are all spot on and aligned. It is now even more about things like “recency”. How often others are mentioning your site and all the temporal factors that go along with that. How relevant the site that mentions your website is to yours and how the audience may be similar and what they may be saying about your site. What the user experience was like….meaning…did the user engage with multiple pages on the site for a decent length of time…or did they simply visit one page for 30 seconds and bounce back to the search engine. Factors like this are now far more likely to influence your rank than previous algorithms.

To get even more granular, the days of simply making sure that your content is unique and in depth, or includes many synonyms or your keyword targets, and is linked to both internally and externally with different keyword phrases…is simply NOT enough anymore. Sure the content is great…and the research you employed to develop it…unbelievable. But if it is simply a block of content without much “sharing” ability…then a site who may not have as much content on their page but developed a neat info-graphic to visually depict your same great content in pictures…may potentially have a better chance of positioning themselves in the SERP’s. Why? Because content is no longer simply about words and anchor text…it is now about traffic volume and popularity…CTR’s (click-through-rates) from the search engines….time on site (bounce rate) analysis…social mentions etc. etc.

Now is this a completely new concept to SEO? No. Experienced and successful SEO’s have been concerned about creativity, marketing, user experience for years. The difference is, the folks who simply developed content farms for the sake of having many different words and phrases spread accross hundreds of pages…are now paying the price. Just like the old cliche…quality over quantity. Google has now employed that concept very strategically and rather well in my opinion.

So you get my point. We are in a new era of search engine optimization..one that should be very exciting for you SEO’s who know you are not just a coder but rather a marketer and creative thinker. We are now not only search marketers, but also website strategists, who are responsible for optimizing how all of your websites assets are created, implemented and promoted.

UPDATE 7/12: Rand Fishkin added a great blog post which speaks directly to the additional responsibilities of us SEO’s. Check it out here http://www.seomoz.org/blog/the-responsibilities-of-seo-have-been-upgraded

Search engines rely heavily on back-links sent from other sites, as well as the text used, known as anchor text, as the hyperlink. In a nutshell, this basically provides a vote to the site being linked to for the topical relevance of the anchor text.

When JC Penney hired SearchDex, it did so to manage their search marketing efforts. In an attempt to game the search engines, SearchDex set out and acquired backlinks to JC Penney for various phrases including, “dresses”, “area rugs”, “bedding” and many others. Their efforts paid off…for over nine months to a year, Penney dominated the search engines for these respective phrases. Mostly positioned in the top 4 positions resulting in Penney receiving exponentially more traffic to their site.

An SEO specialist, Doug Pierce of Blue Fountain Media, recently stumbled across this inbound anchor text after leveraging a tool from SEOmoz called, Open Site Explorer. The tool confirmed the anchor text location as well as confirmed that the sites sending the links back to JC Penney were irrelevant to the users query…which signals to search marketers that they were likely acquired via paid means or via black hat tactics. The search engines scrutinize black hat tactics, as they do not satisfy their overall objectives, which is to provide relevant results to the user and not to be gamed by exploiting a weakness in their algorithms.

After writing the article “The Dirty Little Secrets of Search” in the New York Times, the SEO industry and the efforts of JC Penney were turned upside down. Not only did this expose Penney’s black hat SEO tactics, however, it also exposed Google and Bing’s algorithm weight given to the links from irrelevant sites…which do not align with their typical guidelines.

There have been many articles that have beaten this issue to death, so I will try and not do the same. What I want to focus on is the fact that I believe that their may NOT necessary be a direct tie to the amount spent on Paid Search (PPC) and the position a site receives in organic or SEO rankings. What I am saying is that, I believe the search engines have conveniently turned a blind eye to these black hat tactics used by larger brands even when they know that possible “shady” efforts may have been used, an effort to obviously not increase the risk of losing the large amount of revenue.

A friend of mine who works in an extremely competitive business of selling Halloween costumes has experienced a loss in transactions, revenue and traffic to the same “shady” black hat tactics. Each and every year a competitor who is nowhere to be seen for the majority of the year in the search results, all of a sudden has thousands more inbound links from sites with targeted anchor text and now ranks well above his site that is suppose to be considered an industry leader. Search engines were informed of these black hat tactics, however, chose to not do anything about it. But when they did do something about it…the season was over. Yes, so too was the amount of media spend the competitor in AdWords and Bing was spending.

It is my hope that with these recent JC Penney issues, that both the search engines and search engine optimizers put more effort into the content development and overall user-experience of their sites. It is my belief that sites who add not only a 360 degree marketing mix to their approach to acquire traffic but a 360 degree effort towards the content in each of their sites, will see their biggest “bang-for-their-buck”.

For those of you who know me, you probably would agree that there is not much that I keep inside when it comes to opinion or passion for the task at hand. For many years this has always been looked down upon as being arrogant, too confident, and what ever else they say.

I can honestly say this…with the movement of social media and entrepreneurship amongst my age demographic…it is bloody exciting. Back in the day when I had thoughts of entrepreneurial ideas, there was always this fear tactic pushed down on the idea or the large disconcerting fact of raising money, or having enough liquid capital to throw in the trash if things didn’t work out.

What I am finding is…search engine optimization and social media have empowered us to be passionate about what we love to do. We can live it, breathe it every day and we are surrounded by peers who are open, caring and share almost everything they know with confidence in their ability to differentiate via execution and their ability to build strong relationships.

That just feels phenomenal. I am honestly so proud to be an SEO and be part of this community. I never thought that at 28 years of age, I would be doing something that I love and could do for another 25-50 years. Obviously things will change within the field as they always do, however, at this point in time, I am so thankful to many of the great indirect mentors out there. You know who you are and I am very grateful for the advice and conversations we have had and look forward to many more.

For those of you who know me, I am a huge advocate for honest constructive criticism. I do not like to run around a point or try and “polish a turd” so to speak. Therefore, if I say something that you disagree with, I would love to hear about it. Now I do ask that if you do disagree, that you come out and explain your difference of opinion. I also want to state that the opinions expressed in this blog are written solely by me and do not speak for my employer, Fullhouse.

I look forward to reading all of your comments and connecting with you.