Meta robots tags are not new – they have been around for a while and many webmasters are now well versed in implementing them. However, as websites become more complex, understanding exactly what you can do with a Meta robots tag can help you optimise your site and save you time managing your website indexing and follow status simply and easily.

The HTML Meta tag tells robots how to index the content of a page, and/or how to scan it for links to follow. It is therefore most widely used for ‘Nofollow’ or ‘Noindex’ commands. However, robots can choose to ignore your Meta tag, especially if they are malware robots scanning the web for security vulnerabilities, or spammers looking to harvest email addresses. The main search engines, however, all agreed long ago that they would read and honour the Meta robots tags. So from an SEO and Search Engine indexing perspective utilising the Meta robots tag is still best practice for controlling indexing.

“Usage of Meta robots tags is very important, specifically for complex websites such as ecommerce sites where you can often end up with duplicated product pages across several URLs. While most Meta robots tags can be used on their own it’s also worth knowing that when adding multiple tags such as Nofollow, Noindex these two can be replaced with a single ‘None’ tag. As the default stance for pages is for them to be indexed and followed it’s not really required to add an index or a follow tag in most cases. So, having the choice of Nofollow, Noindex or None gives you a single tag which covers each likely requirement you would have on an average website,” explains Simon Parker, ClickThrough’s Technical SEO Lead.

Google’s Recommendations on Meta Robots Tags

Google recommends that webmasters place all content values in one Meta robots tag, for example:

<META NAME=”ROBOTS” CONTENT=”NOINDEX, NOFOLLOW”>

If your page contains multiple Meta robots tags of the same type, Google will group the content values together. For example, Google will interpret:

<META NAME=”ROBOTS” CONTENT=”NOINDEX”>

<META NAME=”ROBOTS” CONTENT=”NOFOLLOW”>

The same way as:

<META NAME=”ROBOTS” CONTENT=”NOINDEX, NOFOLLOW”>

If your content values conflict, then Google will use the most restrictive, so if your page has: <META NAME=”ROBOTS” CONTENT=”NOINDEX”> and <META NAME=”ROBOTS” CONTENT=”INDEX”> Google will follow the NOINDEX value.

Understanding The Various Meta Robots Tags and What They Do

Here’s a quick overview of the different Meta robots tags and their values:

Index – Enables search. This is a default setting.

Noindex – Prevents search engines from showing this page.

Noimageindex – Prevents search engines from spidering images on this page.

None – As mentioned, this is a shortcut for Noindex and No follow and tells the search engines: don’t do anything with this page at all

Follow – Tells search engine robots to follow the links on this page

Nofollow – Tells search engine robots not to follow any links on this page

Noarchive – Prevents search engines from showing cached copies of this page

Nocache – same as Noarchive, but only used by MSN/Live

Nosnippet – Prevents search engines from showing snippets of this page and from caching the page.

Noodp – Blocks search engines from showing this page in DMOZ (ODP).

Noydir – Prevents Yahoo! from showing the description for this page as in the Yahoo! Directory as the snippet for the page in the SERPs.

Not sure if your website is using Meta robots tags? Our team of Technical SEO Specialists can help. Contact us today.

Did you find this page useful?

Comments

About the author:

ClickThrough is a digital marketing agency, providing search engine optimisation, pay per click management, conversion optimisation, web development and content marketing services.