LIVE Sunday, November 22, 2026 DOW ▲ +0.42% S&P 500 ▲ +0.65% NASDAQ ▲ +0.81%
BBBBFNEWS
BBBBF News – Breaking World, U.S. & Business Headlines
Real-time international reporting, market trends, technological breakthroughs, and in-depth global perspectives.
ALERT
WEBSITE TUTORIAL

How to correctly set robots.txt on WordPress website

How to correctly set robots.txt on WordPress website

How to correctly set robots.txt on WordPress website

File header rules:
The beginning of robots.txt file starts with User-agent:, which is used to specify search engine spiders. If you want to target Google search spiders, you can enter

User-agent: Googlebot

If you want to work on all search engines, enter

User-agent: *

Disallow rules:
Disallow: /abc means prohibiting access to abc.php, abc.html and all files under the abc folder.

Disallow: /abc/ means prohibiting access to all files under the abc folder, but not restricting abc.php and abc.html files.

Allow rules:
Allow rules are the same as Disallow.

Sitemap rules:
Sitemap is used to tell search engines the location of the sitemap

Sitemap: http://your domain name/sitemap.xml

Where sitemap.xml is the sitemap file of your website

To avoid including WordPress system files:
Disallow: /wp-admin/

Disallow: /wp-includes/

Disallow: /wp-content/plugins

Disallow: /wp-content/themes

To avoid including duplicate content:
Disallow: /feed

Disallow: /articles/*/feed

The full text of robots.txt is summarized as follows:
User-agent: *

Disallow: /wp-admin/

Disallow: /wp-includes/

Disallow: /wp-content/plugins

Disallow: /wp-content/themes

Disallow: /feed

Disallow: /articles/*/feed

Sitemap: http://yourdomain/sitemap.xml