Agent Pages
Create, edit and publish clean markdown versions of your web pages, written for AI agents and crawlers to read.
An agent page is a clean markdown version of one page on your website: just the content, without menus, scripts or layout. It's the version an AI crawler would read. In Agent Pages you generate these versions, edit them, and mark them published.
Publishing stores your agent page; it isn't served to AI crawlers yet. When you publish, Opttab saves the page as ready to serve. Delivering published pages to AI crawlers isn't live yet. Each published page shows: "Published pages reach AI crawlers once content serving is switched on."
Everyone in the workspace can view agent pages. Owners, Admins and Members with full access can add, generate, edit, reset, delete and publish them. Generations come from a monthly allowance that depends on your plan. See Plans and billing.
How it works
Opttab fetches each page once and converts it to markdown. Pages are written in the background, at most two at a time per site. You don't need a setup step: Opttab prepares your agent site the first time you add or generate a page.
Generations and your allowance
- Each successful generation, Regenerate or Reset to generated version uses one generation.
- The allowance is monthly and shared across all workspaces on your account.
- Only successful generations count. Queued and running generations hold their place.
- Deleting a page doesn't give the generation back.
- The counter shows "used of limit generations used this month".
Generate agent pages
Open Agent Pages
Open Optimize › Agent Pages. The left tree lists the pages in your sitemap, grouped as Your agent pages, Home, Product pages, Blog posts, Categories and tags and Other pages. Each group shows "done of count generated". Filter with Search Pages.
Add a page that isn't in your sitemap (optional)
Press Add page, enter the Page URL, then press Add page. The page must be on your website's host. Opttab keeps your sitemap for 7 days; press Refresh sitemap to read it again now.
Select pages
Tick the pages you want, or press Select All. You can select up to 50 pages at a time.
Generate
Press Generate (or Generate count). Rows move through Queued and Generating to Generated, or Generation failed. If some pages are skipped, a notice says "n pages were skipped" with the reason.
Why a page was skipped
| Reason | What it means |
|---|---|
| "Edited by hand" | You edited this page. Opttab never overwrites your edits. |
| "Already generating" | A generation for this page is already running. |
| "Generated in the last 24 hours" | Nothing would change, so Opttab skips it. |
| "Not a usable page URL" | The address isn't a page Opttab can fetch. |
| "Not on your website" | The page is on a different host. |
| "Read-only after a plan change" | The workspace is read-only after a plan change. |
Edit an agent page
Select a page to see its Agent version, with the word count, "Generated date", and Copy markdown. From the page detail you can:
- Edit markdown or Paste markdown, then Save. The page is marked as edited by hand, and Opttab won't overwrite it. The limit is 1,000,000 characters.
- Set a Category, an optional label such as "Product guide".
- Regenerate the page. This uses one generation. A page generated in the last 24 hours isn't regenerated.
- Reset to generated version to throw away your edits. This uses one generation and asks you to confirm.
- Delete page to remove it, along with its published copy.
Publish an agent page
Turn on publishing
In the page detail, turn on Publish for AI crawlers. The page's state changes to Published.
Keep it up to date
While a page is published, saving an edit or regenerating it updates the published copy automatically.
Fix an out-of-sync copy
If the state shows Out of sync, the stored copy doesn't match your page. Press Retry.
To unpublish, turn the toggle off and confirm "Unpublish this page?".
"Published" today means your copy is stored and ready. It doesn't mean AI crawlers receive it yet.
When a generation fails
| Message | What to do |
|---|---|
| "Your site's bot protection (provider) blocked our request." | Allow Opttab's fetcher in your CDN or firewall, then regenerate. After three blocks in a row, Opttab pauses that site for about 10 minutes. |
| "This page builds its content in the browser, so there was nothing to read." | Paste the markdown by hand in the page detail. |
| "We could not reach this page." | Check the page is online, then regenerate. |
| "The page answered with an error status." | Fix the page, then regenerate. |
| "This URL is not a web page." | Use the address of an HTML page. |
| "The page had no readable text." | Add text to the page, or paste markdown by hand. |
| "The page is too large to convert." | Paste a shorter version by hand. |
| "The generation stopped before it finished." | Regenerate the page. |
Notices
| Notice | What it means |
|---|---|
| "This month's generations are used up" | Your allowance resets on the date shown. Owners can press Upgrade plan; others see "Ask an account owner to change the plan." |
| "Choose a plan to generate agent pages" | The account has no plan yet. Press See plans. |
| "Add your website first" | Set your website, then come back. Press Open workspace settings. |
| "Your website address changed" | Your website changed after your agent pages were set up. Contact support to move the pages. |
| "This workspace is read-only" | After a plan change, the workspace can be viewed but not changed. Activate it again in Subscriptions. |
| "No sitemap found" / "The sitemap lists no pages" / "Could not read your sitemap" | Opttab can't list your pages. Fix your sitemap, or add pages one at a time with Add page. |