> ## Documentation Index
> Fetch the complete documentation index at: https://docs.siftly.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Cloudflare Worker

> Deploy a Cloudflare Worker that intercepts AI bot traffic at the edge and logs it to Siftly — works in front of any website regardless of tech stack.

<Frame>
  <img src="https://mintcdn.com/siftly/4tAGLoP6M7hAzxgx/images/bot-traffic-flow.svg?fit=max&auto=format&n=4tAGLoP6M7hAzxgx&q=85&s=4b21b43aea4908955e324df7731fa918" alt="AI bot traffic detection — Cloudflare Worker intercepts AI crawler requests at the edge" className="rounded-xl" width="900" height="240" data-path="images/bot-traffic-flow.svg" />
</Frame>

## How It Works

A Cloudflare Worker sits in front of your entire website. Every request passes through it before reaching your origin server. When an AI bot is detected by User-Agent, the Worker logs it to the traffic-ingest-service as a non-blocking background task (`ctx.waitUntil`), then passes the request through to your origin unchanged.

Your real visitors never notice any difference.

## Setup

<Steps>
  <Step title="Install the Wrangler CLI">
    Wrangler is Cloudflare's official CLI for deploying Workers.

    ```bash theme={null}
    npm install -g wrangler
    ```

    Then log in to your Cloudflare account:

    ```bash theme={null}
    wrangler login
    ```
  </Step>

  <Step title="Create your Worker project">
    Create a new directory for the Worker (or use the `cloudflare-wroker` folder from the downloaded files):

    ```bash theme={null}
    mkdir siftly-bot-tracker
    cd siftly-bot-tracker
    ```
  </Step>

  <Step title="Create the Worker files">
    **`worker.js`** — Copy the code below into this file:

    ```js worker.js theme={null}
    const AI_BOT_PATTERNS = [
      // OpenAI — citations & training
      "OAI-SearchBot", "ChatGPT-User", "GPTBot",
      // Anthropic / Claude — citations & training
      "Claude-Web", "Claude-User", "ClaudeBot", "anthropic-ai",
      // Perplexity
      "PerplexityBot", "Perplexity-User",
      // Google / Gemini — citations & training
      "Googlebot", "Google-Extended", "Google-Gemini",
      // Microsoft / Bing
      "bingbot", "BingPreview",
      // Meta
      "Meta-ExternalAgent", "FacebookBot",
      // Others
      "Bytespider", "cohere-ai", "CCBot",
      // Traditional search
      "YouBot", "DuckDuckBot", "Baiduspider",
    ];

    // 👇 Replace with your traffic-ingest-service URL
    const LOG_ENDPOINT = "<TRAFFIC_INGEST_SERVICE_URL>/log/cloudflare";

    // 👇 Replace with your Organisation ID from the Siftly dashboard
    const ORGANIZATION_ID = "<YOUR_ORGANIZATION_ID>";

    export default {
      async fetch(request, env, ctx) {
        const userAgent = request.headers.get("user-agent") || "";
        const isBot = AI_BOT_PATTERNS.some(bot =>
          userAgent.toLowerCase().includes(bot.toLowerCase())
        );

        if (isBot) {
          const payload = {
            timestamp: new Date().toISOString(),
            bot_ua: userAgent,
            url: request.url,
            method: request.method,
            ip: request.headers.get("cf-connecting-ip"),
            country: request.cf?.country,
            city: request.cf?.city,
            asn: request.cf?.asn,
            asOrganization: request.cf?.asOrganization,
          };

          // Fire-and-forget — does not block the actual request
          ctx.waitUntil(
            fetch(LOG_ENDPOINT, {
              method: "POST",
              headers: {
                "Content-Type": "application/json",
                "Authorization": `Bearer ${ORGANIZATION_ID}`,
              },
              body: JSON.stringify(payload),
            }).catch(() => {}) // silently fail if the service is unreachable
          );
        }

        // Always pass the request through to the origin normally
        return fetch(request);
      },
    };
    ```

    **`wrangler.toml`** — Worker configuration:

    ```toml wrangler.toml theme={null}
    name = "siftly-bot-tracker"
    main = "worker.js"
    compatibility_date = "2025-01-01"

    routes = [
      { pattern = "yourdomain.com/*", zone_name = "yourdomain.com" }
    ]

    [vars]
    LOG_ENDPOINT = "<TRAFFIC_INGEST_SERVICE_URL>/log/cloudflare"
    # Replace with your Organisation ID from the Siftly dashboard
    ORGANIZATION_ID = "<YOUR_ORGANIZATION_ID>"
    # Comma-separated list of bot UA substrings to intercept.
    # Override here without touching worker.js.
    AI_BOT_PATTERNS = "OAI-SearchBot,ChatGPT-User,GPTBot,Claude-Web,Claude-User,ClaudeBot,anthropic-ai,PerplexityBot,Perplexity-User,Googlebot,Google-Extended,Google-Gemini,bingbot,BingPreview,Meta-ExternalAgent,FacebookBot,Bytespider,cohere-ai,CCBot,YouBot,DuckDuckBot,Baiduspider"
    ```

    <Note>
      Replace `yourdomain.com` with your actual domain. The Worker will intercept all traffic matching this pattern.
    </Note>
  </Step>

  <Step title="Fill in your credentials">
    In `wrangler.toml` under `[vars]`, replace:

    | Placeholder                    | Value                                                         |
    | ------------------------------ | ------------------------------------------------------------- |
    | `<TRAFFIC_INGEST_SERVICE_URL>` | The URL of the traffic-ingest-service (ask your Siftly admin) |
    | `<YOUR_ORGANIZATION_ID>`       | Your Organisation ID UUID from the Siftly dashboard           |

    Also replace `yourdomain.com` in the `routes` block with your actual domain.

    <Note>
      You can customise the `AI_BOT_PATTERNS` variable in `wrangler.toml` to add or remove bots without touching `worker.js`.
    </Note>
  </Step>

  <Step title="Deploy the Worker">
    From the Worker directory, run:

    ```bash theme={null}
    wrangler deploy
    ```

    Wrangler will upload the Worker to Cloudflare's edge network and activate the route. You should see output like:

    ```
    Uploaded siftly-bot-tracker (1.23 sec)
    Published siftly-bot-tracker (0.42 sec)
      https://siftly-bot-tracker.your-account.workers.dev
      yourdomain.com/*
    ```
  </Step>
</Steps>

## Verifying It Works

After deploying, simulate an AI bot request to confirm events flow through:

```bash theme={null}
curl -A "GPTBot/1.0" https://yourdomain.com
```

Check the **Traffic** section of the Siftly dashboard within a few minutes. You should see an entry for the simulated GPTBot visit.

## Updating the Worker

To change the endpoint URL or Organisation ID later:

1. Edit `worker.js` with the new values.
2. Run `wrangler deploy` again.

Changes propagate globally within seconds.

## Troubleshooting

<AccordionGroup>
  <Accordion title="Worker deployed but no events in Siftly">
    1. Confirm the route in `wrangler.toml` matches your domain exactly.
    2. Visit [dash.cloudflare.com](https://dash.cloudflare.com) → **Workers & Pages** → your worker → **Logs** to see real-time request logs.
    3. Check the `LOG_ENDPOINT` URL is correct and the service is reachable.
    4. Test with `curl -A "GPTBot/1.0" https://yourdomain.com` from a terminal.
  </Accordion>

  <Accordion title="The Worker is interfering with my site">
    The Worker calls `return fetch(request)` at the end, which proxies the request to your origin unchanged. If something looks wrong, check that you haven't modified this line.

    You can also add specific paths to an allowlist or denylist using the `matcher` pattern in `wrangler.toml`.
  </Accordion>

  <Accordion title="I need to track multiple domains">
    Add multiple route entries to `wrangler.toml`:

    ```toml theme={null}
    routes = [
      { pattern = "site1.com/*", zone_name = "site1.com" },
      { pattern = "site2.com/*", zone_name = "site2.com" }
    ]
    ```

    Each domain must be in your Cloudflare account.
  </Accordion>
</AccordionGroup>