WordPress robots.txt example for Better SEO

A few days ago, my friends asked me for tips on how to optimize the robots.txt file to improve SEO. Robots.txt file tells Google how to crawl your website. It is friendly to SEO.

In this post, I will teach you how to create a good robots.txt file for SEO.

What is robots.txt file?

Robots.txt is a file that tells search engine spiders to crawl certain pages or sections of a website.

Where do I put robots txt?

The robots.txt file lives at the root of your site. So, for site www.example.com, the robots.txt file lives at www.example.com/robots.txt. robots.txt is a plain text file.

Robots.txt Example

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-content/
Disallow: /wp-includes/
Disallow: /*/comment-page-*
Disallow: /*?replytocom=*
Disallow: /category/*/page/
Disallow: /tag/*/page/
Disallow: /*/trackback
Disallow: /feed
Disallow: /*/feed
Disallow: /comments/feed
Disallow: /?s=*
Disallow: /*/?s=*\
Disallow: /attachment/