Skip to content
English
Submit a Ticket
Contact Support
  • There are no suggestions because the search field is empty.

Robots.txt - Customize How Crawlers Access Your Website

Learn how your robots.txt file works and how to replace the Real Geeks default with a custom file.

Admin Feature

Editing the website is an account owner-level permission. If you don't see this option, contact your account owner to enable these permissions for you.

A robots.txt file tells search engine and AI crawlers which parts of your website they can access. Real Geeks automatically provides an optimized file for your website, so no action is needed for most users.

The customization tools are intended for developers and SEO professionals who need more control over crawler access. Customizing this file is discouraged if you have limited experience with robots.txt rules. Uninformed changes to this file can reduce your website’s reach.

Need to Know

  • Robots.txt gives instructions to search engines and AI Crawlers

  • No action is needed for most websites, as Real Geeks automatically provides an optimized robots.txt file for your website.

  • Creating a new robots.txt file will replace the default one generated for you.

  • Only customize this file if you understand robots.txt rules or are working with a trusted professional.

Table of Contents

What Is a Robots.txt File?

A robots.txt file provides instructions to search engine and AI crawlers visiting your website. It tells them which parts of your website they can access and which parts they should avoid.

Search engines use crawlers to discover and understand your website content. Giving these crawlers access to your public pages can help your listings, local content, and other resources appear in search results.

You can view your current file by adding /robots.txt to the end of your website address:

https://yourdomain.com/robots.txt

The file does not control what human visitors can view. It only provides instructions to crawlers.

Back to top

Your Default Real Geeks Robots.txt File

Real Geeks automatically generates and manages an optimized robots.txt file for your website. It is designed to give your public content broad visibility while keeping crawlers away from internal areas that should not appear in search results.

By default, the file:

  • Allows major search engines, including Google and Bing, to crawl your public content
  • Welcomes well-known AI crawlers from services such as OpenAI, Anthropic, Perplexity, Google, Apple, Meta, and Amazon
  • Blocks internal areas, including admin pages, member login pages, and behind-the-scenes search pages
  • Includes a link to your sitemap.xml file so crawlers can find your website content

The default works automatically and is the right option for most users. You do not need to edit this file to make your website available to search engines or AI tools.

Back to top

When and Why to Customize Your File

Most Real Geeks websites do not need a custom robots.txt file. If you are not experienced with these files, keep the default.

 

Only customize your file if you understand the impact or are following guidance from a trusted professional. Incorrect rules can block crawlers from your public content and reduce your website’s reach.

Consider adding a custom file only when:

  • You need to block or allow a specific crawler

  • A developer or SEO professional has provided a custom crawler policy

  • Your website has a technical need that is not covered by the Real Geeks default

A custom file does not add new rules to the Real Geeks file. It completely replaces the default.

Back to top

How to Add a Custom Robots.txt File

  1. Open your website editor by adding /admin to the end of your website address. For example: www.yourwebsite.com/admin

  2. Click SEO Files.

    seo-files-editor-location

  3. Select Add SEO File.

  4. Open the Name menu and select robots.txt.

    seo-files-robots-select-menu

  5. Enter (paste) your custom robots.txt text into the editor.

    robots-seo-file-content

  6. Click Save.

Your custom file will appear on your website immediately. You can view this new file by adding /robots.txt to the end of your website's URL. Example: https://yourdomain.com/robots.txt

Please Note

It can take search engines and AI crawlers up to a few days to check the file again.

You can request Google to recrawl your website manually. 

You can view your new robots.txt file by going to 

Real Geeks records each change, including who made it and when. You can use the file history to review changes or restore an earlier version.

How to Return to the Real Geeks Default

  1. Open your website editor by adding /admin to the end of your website address. For example: www.yourwebsite.com/admin

  2. Click SEO Files.

    seo-files-editor-location

  3. Open the Name menu and select robots.txt.

  4. Remove all custom content from the editor.

  5. Click Save.

Leaving the editor blank returns your website to the Real Geeks default.

Your previous custom file will remain available in the file history if you need to review or restore it later.

Back to top

Frequently Asked Questions

  • Do I need to customize my robots.txt file?
    No. Real Geeks already provides an optimized default that works for most websites. Customizing the file is discouraged unless you understand robots.txt rules or are working with a trusted professional.

  • What happens when I add a custom robots.txt file?
    Your custom content completely replaces the Real Geeks default. Your website will serve exactly what you entered until you update it or return to the default.

  • Do I need to edit robots.txt for my website to appear in AI tools?
    No. The Real Geeks default already welcomes major search engines and well-known AI crawlers.

  • What should I do if my website traffic drops after I customize the file?
    Check for an accidental Disallow: / rule. You can restore an earlier version from the file history or clear the custom content to return to the Real Geeks default.

  • Does removing my custom content delete the Real Geeks default?No. Removing your custom content returns your website to the optimized default provided by Real Geeks.

Back to top

Need Help?

Back to top

Related Articles

Back to top