Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.siftly.ai/llms.txt

Use this file to discover all available pages before exploring further.

AI bot traffic detection — Vercel/Netlify middleware intercepts AI crawler requests

​
How It Works

Framework middleware intercepts every incoming request before your page code runs. When an AI bot is detected, the bot’s metadata is sent to the traffic-ingest-service as a fire-and-forget background fetch — zero latency added to real user loads.

​
Setup

1

Create the middleware file

Add the following file to the root of your project (same level as package.json):
import { NextRequest, NextResponse } from 'next/server';

const AI_BOT_PATTERNS = [
  // OpenAI — citations & training
  'OAI-SearchBot', 'ChatGPT-User', 'GPTBot',
  // Anthropic / Claude — citations & training
  'Claude-Web', 'Claude-User', 'ClaudeBot', 'anthropic-ai',
  // Perplexity
  'PerplexityBot', 'Perplexity-User',
  // Google / Gemini — citations & training
  'Googlebot', 'Google-Extended', 'Google-Gemini',
  // Microsoft / Bing
  'bingbot', 'BingPreview',
  // Meta
  'Meta-ExternalAgent', 'FacebookBot',
  // Others
  'Bytespider', 'cohere-ai', 'CCBot',
  // Traditional search
  'YouBot', 'DuckDuckBot', 'Baiduspider',
];

const LOG_ENDPOINT = '<TRAFFIC_INGEST_SERVICE_URL>/log/vercel';
const ORGANIZATION_ID = '<YOUR_ORGANIZATION_ID>';

export function middleware(request: NextRequest) {
  const ua = request.headers.get('user-agent') ?? '';
  const isBot = AI_BOT_PATTERNS.some((p) =>
    ua.toLowerCase().includes(p.toLowerCase())
  );

  if (isBot) {
    const payload = {
      timestamp: new Date().toISOString(),
      bot_ua: ua,
      url: request.url,
      method: request.method,
      ip: request.headers.get('x-forwarded-for') ?? request.headers.get('x-real-ip'),
      country: request.headers.get('x-vercel-ip-country'),
      city: request.headers.get('x-vercel-ip-city'),
    };

    // Fire-and-forget — never awaited, does not block the response
    fetch(LOG_ENDPOINT, {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
        Authorization: `Bearer ${ORGANIZATION_ID}`,
      },
      body: JSON.stringify(payload),
    }).catch(() => {});
  }

  return NextResponse.next();
}

export const config = {
  // Run on all routes except Next.js internals and static files
  matcher: ['/((?!_next/static|_next/image|favicon.ico).*)'],
};
2

Fill in your credentials

Replace the two placeholder values in the file:
PlaceholderValue
<TRAFFIC_INGEST_SERVICE_URL>The URL of the traffic-ingest-service (ask your Siftly admin)
<YOUR_ORGANIZATION_ID>Your Organisation ID from the Siftly dashboard
3

Deploy

Commit and push (or run vercel deploy). The middleware activates automatically on every incoming request.No additional environment variables, build steps, or packages are required.

​
Netlify

For Netlify, use an Edge Function instead of middleware:
netlify/edge-functions/bot-tracker.js
export default async (request, context) => {
  const AI_BOT_PATTERNS = [
    // OpenAI — citations & training
    'OAI-SearchBot', 'ChatGPT-User', 'GPTBot',
    // Anthropic / Claude — citations & training
    'Claude-Web', 'Claude-User', 'ClaudeBot', 'anthropic-ai',
    // Perplexity
    'PerplexityBot', 'Perplexity-User',
    // Google / Gemini — citations & training
    'Googlebot', 'Google-Extended', 'Google-Gemini',
    // Microsoft / Bing
    'bingbot', 'BingPreview',
    // Meta
    'Meta-ExternalAgent', 'FacebookBot',
    // Others
    'Bytespider', 'cohere-ai', 'CCBot',
    // Traditional search
    'YouBot', 'DuckDuckBot', 'Baiduspider',
  ];

  const LOG_ENDPOINT = '<TRAFFIC_INGEST_SERVICE_URL>/log/vercel';
  const ORGANIZATION_ID = '<YOUR_ORGANIZATION_ID>';

  const ua = request.headers.get('user-agent') ?? '';
  const isBot = AI_BOT_PATTERNS.some((p) => ua.toLowerCase().includes(p.toLowerCase()));

  if (isBot) {
    const payload = {
      timestamp: new Date().toISOString(),
      bot_ua: ua,
      url: request.url,
      method: request.method,
      ip: request.headers.get('x-nf-client-connection-ip'),
      country: context.geo?.country?.code,
      city: context.geo?.city,
    };

    context.waitUntil(
      fetch(LOG_ENDPOINT, {
        method: 'POST',
        headers: { 'Content-Type': 'application/json', Authorization: `Bearer ${ORGANIZATION_ID}` },
        body: JSON.stringify(payload),
      }).catch(() => {})
    );
  }

  return context.next();
};
Add the edge function route in netlify.toml:
netlify.toml
[[edge_functions]]
  path = "/*"
  function = "bot-tracker"

​
Verifying It Works

After deploying, visit your site with a spoofed User-Agent to confirm events appear in Siftly: 
curl -A "GPTBot/1.0" https://yoursite.com
Check the Traffic section of the Siftly dashboard within a few minutes.