Real Geeks Bot, Crawler, and Automated Traffic Policy

A crawler is a computer program that reads pages on the internet. Some crawlers help people find and share your website. Others collect content without sending visitors back.

Real Geeks manages this automated traffic to keep your website fast, safe, and available to real buyers and sellers.

Table of Contents

• Why Real Geeks Manages Crawlers

• Website Visitors and Allowed Crawlers

• Full List of Allowed, Managed, and Blocked Crawlers

• Traffic We Filter That Isn't a Crawler

• What to Do if Something Is Wrong

• Policy Review Frequency

• Frequently Asked Questions

• Need Help?

Why Real Geeks Manages Crawlers

AI crawlers now make up about 15% of traffic across websites.

Some crawlers read pages at a safe speed. Others may request thousands of pages per minute. That activity can slow down your website or take it offline. A crawler should never cost you a lead.

We consider two questions:

• Does the crawler send real visitors to your website?
• Does it read your website at a safe speed?

Crawlers that meet both standards are allowed. Crawlers that may send visitors but make too many requests are managed. Crawlers that collect content without providing value are blocked.

These rules apply automatically to every website hosted by Real Geeks.

_{Back to top}

Website Visitors and Allowed Crawlers

The crawler rules in this policy do not limit people browsing your website. Visitors can view listings, search for homes, and sign up as leads.

Some crawlers are also always allowed because they help people find or share your website:

• Search engines: Googlebot, Bingbot, and DuckDuckBot help your pages appear in search results.
• Link previews: Tools from Facebook, X, LinkedIn, and Slack create a preview when someone shares your link.
• Website services: Monitoring and analytics tools continue to work normally.

Search engine crawlers are never slowed or blocked. Your ability to appear in search results is not affected by this policy.

Separate security rules may block a person if their request appears harmful or comes from a blocked region. These rules are explained later in this policy.

_{Back to top}

Full List of Allowed, Managed, and Blocked Crawlers

The tables below show which automated tools are allowed, managed, or blocked.

• Allowed: Can access your website without limits.
• Managed: Can access your website at a controlled speed.
• Blocked: Cannot access your website.

The allowed tools shown below are examples. Any crawler not covered by a Real Geeks rule is allowed by default.

Search Engines and Link Previews

How Managed Access Works

Managed AI crawlers have a limit on how quickly they can request pages. If they reach that limit, they receive an HTTP 429 response and must return later.

No content is hidden or removed from their index. The crawler can return and finish at a safer speed.

Real Geeks finds crawlers by matching text within their user agent, which is the name a crawler gives when it visits a website. Related versions may be covered by the shortest identifier shown in the tables.

The blocked group covers 56 identifiers in total.

_{Back to top}

Traffic We Filter That Isn't a Crawler

Real Geeks also filters traffic that may pose a security risk. These rules are separate from the crawler policy.

• Attack probes: Automated scanners look for exposed admin pages, settings files, and login information. Real Geeks blocks these requests.
• Known harmful addresses: Individual internet addresses found attacking the platform are blocked. This list is updated as attacks are detected.
• High-risk regions: A small number of countries generate most of the attack traffic against the platform and very little legitimate buyer or seller traffic. Real Geeks currently blocks requests from:
  • China, Russia, Ukraine, Vietnam, Singapore, the Netherlands, Finland, and Luxembourg.
  • The regional filter can affect a real visitor. For example, a client traveling in one of these countries may be blocked.

Anyone affected by these filters sees a page explaining that the request was blocked and how to contact Support.

_{Back to top}

Something Wrong? Tell Us!

Contact Real Geeks Support if:

• A tool you pay for is blocked: Tell us which service you use and why it needs to read your website.
• A custom connection stopped working: A tool built with common scraping software may be caught by these rules. Giving it a clear identifier may fix the issue.
• A real person sees the blocked page: Send us the date, time, and their approximate location. We can identify which rule blocked the request.
• You want a crawler reviewed: A blocked crawler may become a useful source of website traffic over time. Applebot and PerplexityBot were moved after a policy review.

_{Back to top}

Policy Review Frequency

We review the policy quarterly and immediately after any incident. A crawler gets moved out of the blocked tier when it can show it sends real visitors back to the sites it reads, respects a crawl budget, and identifies itself honestly. It gets moved in when it does the opposite.

Every change is published here before or at the time it takes effect. You should never have to guess what is reaching your site.

_{Back to top}

Frequently Asked Questions

• Will this hurt my website’s Google ranking?
  No. Googlebot can freely read your website for Google Search.
• Will shared links still show a preview on social media?
  Yes. The tools that create link previews for Facebook, X, LinkedIn, and Slack are allowed.
• Do I need to manage these crawlers myself?
  No. Real Geeks manages these rules for every website on the platform.

_{Back to top}

Need Help?

• Call us at 844-311-4969 (Mon–Fri, 8 AM–8 PM CST)
• Email support@realgeeks.com
• View our Live Events page for free coaching and training.
• Join the Real Geeks Mastermind Group on Facebook for peer tips and best practices

_{Back to top}

Related Articles

• How Search Engines Find and Understand Your Website

_{Back to top}