AI Crawlers and Your Dubai Website (2026): When to Block, When to Allow, and When to Charge
Bots passed humans at 57.5% of web traffic in June 2026 and Anthropic crawled 1,917 pages per referral in July. How Dubai sites should block, signal, enforce or charge.
- PUBLISHED
- 20 OCT 2026
- READ TIME
- 12 MIN
- AUTHOR
- AQUARIUS · DUBAI
- UNIT
- REV 2026.09
In 2026 the question is no longer whether AI crawlers hit your Dubai website — it is which of the four levers you have pulled. You can block a crawler in robots.txt, state machine-readable permissions with Content Signals, enforce the decision at the edge with bot rules, or let it in and get paid when your content is used. Most Dubai businesses have pulled none of them, which means they are funding AI answers for free while their own organic clicks fall.
The traffic has already flipped, and the exchange rate is terrible
Three numbers set the stakes. First, automated traffic took the majority of the web in June 2026: bots generated 57.5% of HTML requests against 42.5% from humans, and on 23–24 August 2026 worldwide HTML requests measured 61% bot to 39% human. Second, the agentic share is the part growing fastest — AI traffic rose 82.3% between July 2025 and June 2026 to 1.7% of all web traffic, while traffic from agents and agentic browsers that act on a user behalf grew roughly 7,851% in a single year. Third, and this is the one that should bother a Dubai marketing budget, the crawl-to-refer ratio.
Cloudflare Radar tracks how many pages a platform crawls for every one visitor it sends back. For the full month of July 2026, Anthropic sat at 1,917 crawls per referral and OpenAI at 251:1; in August 2026 the figures were 984.3:1 and 501.3:1. Perplexity ran near 1,100:1. Google, by comparison, crawls about 5.1 pages per referral. The direction of travel is genuinely improving — in July 2025 Anthropic was at 38,744:1 and OpenAI at 1,104:1 — but an AI assistant that takes a thousand pages and returns one click is not a traffic channel. It is a supply agreement you never signed.
Meanwhile the click side is shrinking. Analysis of Google Search Console data across a portfolio of 64 sites found organic clicks down 42% since AI Overviews began expanding, with click-through rate down 61% on queries where an AI Overview appears. Similarweb puts zero-click searches at 69% of all queries, rising to 80–83% on AI Overview queries. UAE consumers are ahead of that curve, not behind it: 72% of UAE shoppers used AI for their most recent purchase, 70% now use AI tools to inform shopping decisions (a 44% jump on 2024), and 65% say AI is how they discover new products. ChatGPT was the tool 78% of likely Dubai Shopping Festival buyers planned to use, ahead of Gemini at 53%, Meta AI at 39%, Claude at 24% and Copilot at 19%.
The mistake most Dubai businesses are about to make
Blocking every AI bot is not protection. The crawler that trains a model and the crawler that fetches the page it is about to cite are different jobs, and a blanket Disallow stops the citation while barely slowing the training.
This is the most expensive misread in the market right now. Cloudflare data from May 2026 showed that under 10% of AI crawler requests were search-purpose — the other 90% went to training and answer generation. Those are separate user agents doing separate jobs. If a Dubai store blocks the answer-fetching agent, it disappears from the surface where 72% of local shoppers are already researching. If it blocks nothing, it hands over its product data, pricing and guides to a training corpus with no attribution and no payment.
The behaviour of the wider web reflects that split. Fewer than 1% of Cloudflare sites block search crawlers, while 17% restrict AI training. Between July 2025 and January 2026, sites using Cloudflare blocking tools refused GPTBot and ClaudeBot nearly seven times as often as Googlebot and Bingbot. Operators are not choosing block-everything or allow-everything. They are separating by purpose. That is the decision your robots file should encode too.
The four levers, and what each one actually does
These are not alternatives; they stack. Lever one states intent, lever two states licensing, lever three enforces, lever four monetises.
- robots.txt by user agent. Free, honoured voluntarily, and still the baseline. Name the agents individually — GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, Bytespider, meta-externalagent — rather than using one wildcard that also hits the crawlers feeding your citations.
- Content Signals. Cloudflare published the Content Signals Policy on 24 September 2025 and pushed it into managed robots.txt files on more than 3.8 million domains. It adds three machine-readable preferences: search (index and link), ai-input (use as input to a generated answer) and ai-train (use to train a model). The default set is search=yes, ai-train=no, ai-input left neutral until the owner decides. Important caveat for anyone planning around it: Google has not committed to honouring these signals.
- Edge enforcement. Signals are requests; a WAF rule is a decision. Bot management at the CDN layer is what actually stops an unverified scraper that ignores robots.txt, and it is also where you verify the honest agents rather than blocking them by accident. If agentic shoppers and AI browsers are part of your plan, read our note on making a Dubai website agent-ready before you tighten the rules.
- Licensing and payment. Two standards now exist for being paid rather than scraped. Really Simple Licensing (RSL) launched on 10 September 2025 with the 1.0 specification published on 10 December 2025; it is an XML licence linked from robots.txt, backed by Reddit, Yahoo, Medium and Quora. Cloudflare Pay Per Crawl entered private beta on 1 July 2025, charging crawlers via an HTTP 402 Payment Required response at a documented minimum of USD 0.001 per successful crawl, with observed pricing clustering at USD 0.001–0.005 for general content and USD 0.05–0.25 for premium news and primary research.
Then the model changed. On 1 July 2026 Cloudflare moved payment from the fetch to the citation, compensating publishers when their content appears inside an answer rather than when a bot downloads the page — partly because more than half of legitimate bot crawl traffic was re-fetching pages that had not changed since the previous visit. Launch partners included Ceramic.ai on a pay-per-query basis, with reporting on the exact queries and snippets used, and You.com letting agents pay on demand for a single premium page. No public rate card has been disclosed for either. From 15 September 2026, newly onboarded domains get Training and Agent crawlers blocked by default on pages that display ads, while Search crawlers stay allowed.
Which posture fits which Dubai business
There is no universal answer, only a defensible one per business model. Use this as the starting matrix.
| Business type | Search crawlers | AI answer input | AI training | Why |
|---|---|---|---|---|
| Dubai ecommerce store | Allow | Allow | Block | 72% of UAE shoppers research with AI; you want to be in the answer, not in the corpus |
| Lead-gen services firm | Allow | Allow | Block | Citations in AI answers are now a primary discovery path for high-ticket enquiries |
| Publisher or research house | Allow | Charge | Block or licence | Primary content is the asset; pay-per-use and RSL exist for exactly this case |
| SaaS with docs and a knowledge base | Allow | Allow | Allow selectively | Model familiarity with your product drives recommendations and cuts support load |
| Marketplace with user-generated content | Allow | Allow partial | Block | Personal data inside listings and reviews carries UAE PDPL exposure once it leaves your control |
| Regulated finance or healthcare | Allow | Allow curated pages | Block | Accuracy liability; expose only the pages you are willing to be quoted on |
One legal point specific to the UAE that rarely appears in imported SEO advice: there is no text-and-data-mining exception in UAE copyright law. Federal Decree-Law No. 38 of 2021 protects written works, software, applications and databases, and Article 7 reserves to the author the right to authorise use of the work in any manner whatsoever. Nothing in UAE law currently carves out AI training. Your robots file and licence terms are therefore evidence of the permission you did and did not grant — not merely a technical preference.
What this costs to implement in Dubai
The baseline work is cheap. The part that pays is the measurement and the citability rebuild that follows it. Typical Dubai ranges for 2026:
| Workstream | AED range | Timeline | What you get |
|---|---|---|---|
| robots.txt, Content Signals and llms.txt baseline | 2,500 – 6,000 | 2 – 4 days | Per-agent policy, three content signals set deliberately, documented rationale |
| Edge bot rules and verified-agent allowlist | 6,000 – 18,000 | 1 – 2 weeks | CDN or WAF enforcement, scraper mitigation, no accidental blocking of paying agents |
| Crawler and AI-referral analytics | 12,000 – 40,000 | 2 – 4 weeks | Log pipeline, per-agent crawl volume, your own crawl-to-refer ratio by platform |
| Citability rebuild (schema, answer blocks, clean markup) | 18,000 – 75,000 | 3 – 8 weeks | Pages structured to be quoted correctly and attributed to your brand |
| Licensing and monetisation setup (RSL, pay-per-use, terms) | 9,000 – 30,000 | 2 – 3 weeks | Machine-readable licence, payment path, legal terms aligned to UAE copyright |
Set that against the exposure. UAE e-commerce is running at roughly USD 12.3 billion (about AED 45 billion) in 2026 across more than 11 million online shoppers and 45,000+ active stores, heading for AED 67.2 billion by 2030. UAE digital ad spend is forecast to grow 15.2% to USD 2.64 billion in 2026, with Dubai alone above USD 1.2 billion. A 42% decline in organic clicks against that backdrop is not a reporting curiosity; it is the fastest-rising cost line in a UAE marketing plan, and it gets paid in paid-media budget.
How Aquarius handles it
We treat crawler policy as a revenue decision, not a hygiene task. First we measure: a week of server and CDN logs tells us which agents are actually hitting your Dubai site, how often, and what your real crawl-to-refer ratio is per platform — most clients have never seen that number. Then we separate by purpose: search and answer-input crawlers get a clean, fast, well-structured page; training crawlers get an explicit, documented refusal unless you decide otherwise. Then we enforce at the edge, because a policy that only lives in a text file is a request. Finally we rebuild the pages that matter so they are quotable and attributable, which is the only durable way to convert AI visibility into UAE enquiries — the mechanics are in our guide on getting cited by AI search and our breakdown of what AI Overviews did to UAE businesses.
Risk reversal is simple: the audit is fixed-fee and the deliverable is yours whether or not you continue — a per-agent policy, the deployed configuration, and a baseline report you can re-run each quarter. Published ranges sit on our pricing page, and structured data work usually pairs with it (see schema markup for Dubai websites).
FAQ
Will blocking AI crawlers hurt my Google rankings?
Blocking Googlebot would. Blocking Google-Extended, the signal that governs Gemini model training, is documented as separate from Search indexing. The caution worth stating plainly: Google has not committed to honouring the Content Signals Policy, so treat search=yes, ai-train=no as a stated preference with legal value rather than a guaranteed technical control.
Can a Dubai SME realistically earn money from pay-per-use?
Rarely, and you should plan as if the answer is no. Pay-per-crawl pricing clustered at USD 0.001–0.005 per fetch for general content, and the model has since moved to payment on citation with no disclosed rate card. The meaningful upside for an SME is control and evidence of permission, not revenue. Monetisation is a publisher play.
Does llms.txt do anything?
It is optional and not a ranking factor — Google has said it ignores it. It costs almost nothing to publish and can help agents navigate your content, but it is not a substitute for robots.txt directives, Content Signals or edge rules. Do not let a vendor sell it to you as compliance.
Is scraping my content for AI training legal in the UAE?
There is no text-and-data-mining exception in UAE copyright law. Under Federal Decree-Law No. 38 of 2021, written works, software and databases are protected and Article 7 reserves authorisation to the author, so training use without permission is on its face an infringement. Draft UAE generative-AI rules put the onus on providers to screen copyrighted material out of training sets. A documented refusal in robots.txt and a machine-readable licence make your position far easier to assert.
What should I do first if I only have a day?
Pull one week of access logs and count requests by user agent. You cannot make a sensible block, allow or charge decision before you know whether the crawlers taking your bandwidth are the ones feeding the answers your customers read.
The web stopped being majority-human in June 2026. Your Dubai site is already serving two audiences, and only one of them clicks. Decide what each is allowed to take — or keep paying to produce content for a channel that returns one visitor per two thousand pages. Book a crawler-policy audit and we will show you your own ratio before we change a single line.