Connect a crawler log source
Connect Cloudflare, Vercel, Akamai or Netlify so Opttab can count AI crawler visits, and connect Google Analytics for visitor figures.
A crawler log source is the service that serves your website — your CDN or host. Opttab reads its request logs to see which AI crawlers fetch your pages. This guide walks you through each source, plus Google Analytics for visitor figures.
Who can do this: Owners and Admins. Members can view Website Analytics but can't connect sources.
Available on plans that include Website Analytics and Crawler Analytics. See Plans and billing.
How it works
- Cloudflare is a pull source: you give Opttab access and it fetches the figures.
- Vercel, Akamai and Netlify are push sources: you point a log stream at an endpoint Opttab gives you.
Before you start
- Know which website the source serves. Opttab matches it to your workspace's website.
- Check your plan with the provider:
| Source | Provider requirement |
|---|---|
| Cloudflare | Any plan; some figures show "Not available on this Cloudflare plan." |
| Vercel | "Drains need a Vercel Pro or Enterprise team, and Vercel bills the data a drain sends." |
| Akamai | "Needs DataStream 2 on your Akamai contract; it is included with delivery products." |
| Netlify | "Log drains are available on Netlify Enterprise plans." |
Open the connect list
Open the Crawlers tab
Go to Analytics › Website Analytics and open Crawlers.
Press Connect
On the card Connect a Crawler Log Source to See Crawler Analytics, press Connect. The Connect a crawler log source list opens.
Pick your source
Press Connect next to Cloudflare, Vercel, Akamai or Netlify, then follow the matching tab below.
You can also manage these connections in Settings › Integrations.
Connect your source
Open the drawer
Press Connect on Cloudflare. The Connect Cloudflare drawer opens.
Give Opttab access
Press Continue with Cloudflare and allow access.
Or press Use an API key instead and paste an API token. Create the token in Cloudflare under My Profile › API Tokens › Create Token, with the Zone Read and Analytics Read permissions for your websites. Add an optional Label, then press Connect Cloudflare.
Choose your websites
Choose the websites (zones) that serve this workspace, then press Save resources.
Cloudflare also powers live bot figures on the Realtime tab.
Open the drawer
Press Connect on Vercel. The Connect Vercel drawer opens.
Choose the manual setup
Press Set up the log stream yourself instead. Enter the Website whose request logs the drain sends (for example www.example.com), then press Connect Vercel.
Copy your details
The drawer shows Send Vercel logs to Opttab with a copyable Endpoint URL and Signing secret.
Add the drain in Vercel
In Vercel, open Team Settings › Drains and add a drain for Logs with a Custom endpoint. Paste the endpoint URL and the signing secret, and choose NDJSON.
Pick projects and sources
Pick the projects that serve this website, the sources Static files, Functions, Edge and Redirects, and the Production environment.
Save
Save the drain. Vercel tests the endpoint straight away.
If your drawer offers Continue with Vercel, you can sign in instead and pick your projects. Opttab then creates the log drain for you and keeps it in step with your projects. Disconnecting removes it.
Open the drawer
Press Connect on Akamai, enter the Website, and press Connect Akamai.
Copy your details
The drawer Send Akamai logs to Opttab shows an Endpoint URL, User name and Password, plus a block of User-Agent patterns. Press Copy the User-Agent patterns.
Log the user agent
In Property Manager, open the property that serves this website and turn on Log Request Details › Log User-Agent Header.
Add a rule for AI crawlers
Add a rule whose criteria is Request Header User-Agent "is one of" the patterns you copied. Put the DataStream behavior in that rule with 100% sampling.
Create the stream
In DataStream 2, create a stream on that property with a Custom HTTPS destination: the endpoint URL, Basic authentication with the user name and password, JSON format, compressed, every 30 seconds. Include Request host, Request path, Status code, Request time, Request ID and User-Agent.
Activate
Activate the stream and the new property version.
Akamai can take about an hour to start a new stream.
Open the drawer
Press Connect on Netlify, enter the Website, and press Connect Netlify.
Copy your details
The drawer shows an Endpoint URL and an Authorization header value that starts with "Bearer".
Enable a log drain
In Netlify, open the project that serves this website and go to Logs & metrics › Log drains › Enable a log drain.
Configure it
Choose General HTTP endpoint, tick Traffic logs only, and pick JSON. Paste the endpoint URL, and add a header named Authorization with the value you copied.
Save
Leave Exclude PII off — it removes the user agent — then save.
Check the connection
Push sources show a health badge in the setup drawer and in Settings › Integrations (press Setup, then Refresh).
| Badge | What it means |
|---|---|
| Waiting for the first logs | Nothing has arrived yet. Vercel and Netlify send logs on the first request; Akamai takes about an hour. |
| User-Agent logging is off | Logs arrive without the user agent, so Opttab can't spot crawlers. |
| No logs since … | Nothing for over a day. Check that the stream is still active and points at the endpoint. |
| Logs arriving · last … | Healthy. The badge also shows how many AI-crawler requests were kept. |
What "verified" means per source
| Cloudflare | Vercel | Akamai | Netlify | |
|---|---|---|---|---|
| How it connects | Sign in or API token | Log drain | DataStream 2 stream | Log drain |
| Bots verified? | Yes, by Cloudflare | No — user-agent claim | No — user-agent claim | No — user-agent claim |
| Look-alike requests | Shown as +N unverified | Counted with real ones | Counted with real ones | Counted with real ones |
| Live bots on Realtime | Yes | No | No | No |
| Counts against log allowance | No | Yes | Yes | Yes |
A user-agent claim means the request says it's GPTBot, for example, but nothing proves it. See What's the difference between verified and unverified bots?.
Connect Google Analytics
Google Analytics 4 feeds the Realtime visitor figures and the Visitors tab.
Press Connect
On the Realtime or Visitors tab, press Connect on the Google Analytics card. The Connect Google Analytics 4 drawer opens.
Sign in with Google
Press Continue with Google, then review and allow read-only Analytics access.
Choose your property
Press Choose properties, pick the GA4 property for this website, and press Save resources. The row then shows "Serving this workspace:" followed by the property name.
Wait for the first sync
The first sync usually takes under a minute and the page updates on its own. Opttab then backfills up to 13 months of history, recent days first.