Sometimes you may have publicly facing website, which you don’t want it indexed by the major search engines. Easy way is to add robots.txt or meta tags on html but more global setting can be set on IIS settings from ‘Administrative Tools’ open ‘Internet Information Services (IIS) Manager’ > Select the Server > ‘HTTP Response Headers’.
Once dialog for HTTP Response Header is open
Add > Name = X-Robots-Tag > Value = noindex > and press OK.
General info about
X-Robots-Tag HTTP header
X-Robots-Tag can be used as an element of the HTTP header response for a given URL. Any directive that can be used in a robots meta tag can also be specified as an
X-Robots-Tag. Here’s an example of an HTTP response with an
X-Robots-Tag instructing crawlers not to index a page:
About Robots Tag
A meta robots tag can include the following values:
- Index: Allows search engines to add the page to their index.
- Noindex: Disallows search engines from adding a page to their index and disallows it from appearing in search results for that specific search engine.
- Follow: Instructs search engines to follow links on a page, so that the crawl can find other pages
- Nofollow: Instructs search engines to not follow links on a page.
- None: This is a shortcut for noindex, nofollow.
- All: This is a shortcut for index, follow.
- Noimageindex: Disallows search engines from indexing images on a page (images can still be indexed using the meta robots tag, though, if they are to linked to from another site).
- Noarchive: Tells search engines to not show a cached version of a page.
- Nocache: This is the same as noarchive tag, but specific to the Bingbot/MSNbot.
- Nosnippet: Instructs search engines to not display text or video snippets.
- Notranslate: Instructs search engines to not show translations of a page in SERPs.
- Unavailable_after: Tells search engines a specific day and time that they should not display a result in their index.
- Noyaca: Instructs Yandex crawler bots to not use page descriptions in results.
Read more at Google Robots meta tag, data-nosnippet, and X-Robots-Tag specifications