Robots.txt setup and audit for Jacksonville websites.
Your robots.txt file decides which parts of your site Google, Bing and AI crawlers may visit. One wrong line can hide an entire website from search. We write it, test it and keep it right.
What robots.txt controls, and what it doesn’t.
Robots.txt is a plain text file at the root of your domain (yourdomain.com/robots.txt). Before crawling, search engines and most AI crawlers read it to see which folders and pages they may visit, and where your XML sitemap lives. The format is an internet standard, RFC 9309.
It controls crawling, not indexing. A page blocked in robots.txt can still appear in Google if other pages link to it, just without a description. To keep a page out of search results, it needs a noindex tag and must stay crawlable so Google can see that tag. Mixing these up is one of the most common technical SEO mistakes we fix.
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://www.example.com/sitemap_index.xmlA typical WordPress robots.txt is short, like the example above. Most sites need very little in it; problems usually come from what was added later.
Robots.txt problems we fix most often.
“Disallow: /” left from development
A staging setting that blocks the entire site, carried over at launch. Rankings disappear within weeks, and nothing looks wrong to visitors.
Blocked CSS and JavaScript
Old rules that block theme or plugin folders stop Google rendering the page properly, which can hurt mobile rankings.
Blocking instead of noindexing
Thank-you pages, search results or tag archives blocked in robots.txt still get indexed as bare URLs. A noindex tag is the right tool.
AI crawlers blocked by accident
Security plugins, firewalls and CDN bot settings can block the crawlers behind ChatGPT search, Perplexity or Copilot, so those tools can’t cite you. See AI crawler access.
Missing or wrong sitemap line
No Sitemap line, or one pointing to an old sitemap that no longer exists. See XML sitemap setup.
Conflicting rules
Several plugins each adding rules, or a physical robots.txt file overriding the one WordPress generates, so the file you edit isn’t the one Google reads.
Choosing which AI crawlers to allow.
AI companies use different crawlers for different jobs, and robots.txt lets you treat them separately. Search crawlers such as OAI-SearchBot (ChatGPT search) and PerplexityBot fetch pages so those tools can cite and link to you. Training crawlers and controls such as GPTBot, Google-Extended, ClaudeBot, Applebot-Extended and CCBot govern whether your content is used to train models. Many businesses allow the search crawlers, because being cited brings visitors, and make a separate choice about training. We explain the trade-offs and set the rules you choose.
Google-Extended does not affect Google Search or AI Overviews, which use the regular Googlebot. Blocking Googlebot removes you from Google entirely.
Our robots.txt service.
Audit
We read the live file the way Google and Bing do, check Search Console’s robots.txt report, and crawl the site to see which important pages are blocked.
Rewrite
A clean file with only the rules your site needs, the correct sitemap line, and the AI crawler policy you choose.
Platform setup
WordPress through your SEO plugin or server file; Shopify through its robots.txt template; Wix and Squarespace within what each platform allows.
Firewall and CDN check
We confirm your security plugin, host and CDN aren’t blocking the crawlers your robots.txt allows.
Rated 5.0 on Google by 33 clients.
10+ years in business and 100+ websites and apps launched. Every site we build launches with a clean robots.txt, an XML sitemap, structured data and Google Search Console connected, and our own site scores in the 90s on Google’s mobile PageSpeed test.
FAQ
Where is my robots.txt file?
At the root of your domain: type your web address followed by /robots.txt. On WordPress it is usually generated by WordPress or your SEO plugin rather than stored as a file.
Can robots.txt remove a page from Google?
No. Robots.txt stops crawling, not indexing. To remove a page from results, add a noindex tag and leave the page crawlable, or remove the page and return a 404 or 410.
Should I block AI crawlers in robots.txt?
If you want to be recommended by AI assistants, don’t block their search crawlers, such as OAI-SearchBot and PerplexityBot. Blocking training crawlers is a separate business choice and doesn’t remove you from AI search results.
How quickly does Google see robots.txt changes?
Google generally caches robots.txt for up to 24 hours, and you can ask it to recrawl the file from Search Console’s robots.txt report.
Does every website need a robots.txt file?
Without one, crawlers assume everything is allowed, which is fine for many small sites. A short file with a sitemap line is still best practice.