What is robots.txt?
robots.txt is a text file in your website root that tells search engine crawlers which pages or files they can or cannot request. It helps control indexing and crawl budget.
Where do I put robots.txt?
Place robots.txt in the root directory of your website. It must be accessible at https://yourwebsite.com/robots.txt. Most CMS platforms have settings to manage this.
Should I block AI bots?
Many website owners block AI training bots like GPTBot and CCBot to prevent their content being used to train AI models without permission. Search bots like Googlebot should generally be allowed.
Does robots.txt affect SEO?
Yes. Blocking Googlebot from crawling important pages will prevent them from being indexed. Use Disallow carefully and never block CSS or JS files that affect rendering.