Robots.txt Generator

How Strong Is Your Site?
Gain Access to our Premium SEO Tools for only $14.95 per month. Try them free for 30 days!

About Robots.txt Generator

Generate a robots.txt file for your website for free. Robots.txt Generator is an SEO tool used to create a text file that is uploaded to your website's root directory. When web spiders and robots crawl your site for information, they read your robots.txt file as directives to identify parts of your website that may be blocked. Using our Robots.txt Generator allows you to customize your robots.txt file to keep web crawlers from accessing all or certain parts of your website.

What is a Robots.txt File? "The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website." Source - MOZ

See also: Wordpress Robots.txt Default File


Robots.txt Generator


Default - All Robots are:  
    
Crawl-Delay:
    
Sitemap: (leave blank if you don't have) 
     
Search Robots: Google
  Google Image
  Google Mobile
  MSN Search
  Yahoo
  Yahoo MM
  Yahoo Blogs
  Ask/Teoma
  GigaBlast
  DMOZ Checker
  Nutch
  Alexa/Wayback
  Baidu
  Naver
  MSN PicSearch
   
Restricted Directories: The path is relative to root and must contain a trailing slash "/"
 
 
 
 
 
 
   



Now, Create 'robots.txt' file at your root directory. Copy above text and paste into the text file.


Add Bookmark Go Back to Tools