Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion proto/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ and the onboarding page's code comments name an RSL license rather than a file.

**A term can require usage reporting (additive protocol change).** `ObligationKind` gains `OBLIGATION_KIND_REPORTING`, and `Obligation` gains an optional `reporting` field, a `ReportingObligation` that states the `window` within which the agent files `ReportUsage` and the `required_fields` the report carries. The Exchange carries the obligation onto `Offer.reporting`: `required` is true, the window and fields are the obligation's where it states them and the Exchange's own where it does not, and the endpoint is the Exchange's. A term without one keeps the Exchange's own reporting policy. Four rules: `obligation.reporting.requires_reporting_kind` (the field only on that kind), `obligation.reporting.no_endpoint` (the Exchange sets the endpoint), `license_term.reporting.metering_none` (a term metered `NONE` needs no report, so it cannot carry one) and `offer.reporting.required_by_term` (an offer selling such a term requires a report). A revenue-share term now states its reporting duty on the term. The TypeScript and Python cross-field mirrors, the conformance corpus and the license-term vectors cover the four rules. Licensing Terms, Transaction Flow and the CoMP extension say the same.

**An offer carries restriction tokens in canonical form, and the party that sets a license URI supplies its digest (comments only; no wire change).** The `LicenseTerm` comment now says what an Exchange already does: the term on an offer carries each restriction token as ingest canonicalized it (trimmed, case-folded, alias-resolved), with no other translation or reformulation, so an agent compares tokens against the registry without folding them itself. The `License.uri_digest` comment no longer says the Exchange computes the digest at ingestion: the party that sets `uri` supplies the digest with it, the publisher on a term it pushes. Licensing Terms and the threat model (T-LIC-4) say the same. Proto examples use `news.example` and a generic collecting society in place of real company names.
**An offer carries restriction tokens in canonical form, and the party that sets a license URI supplies its digest (comments only; no wire change).** The `LicenseTerm` comment now says what an Exchange already does: the term on an offer carries each restriction token as ingest canonicalized it (trimmed, case-folded, alias-resolved), with no other translation or reformulation, so an agent compares tokens against the registry without folding them itself. The `License.uri_digest` comment no longer says the Exchange computes the digest at ingestion: the party that sets `uri` supplies the digest with it, the publisher on a term it pushes. Licensing Terms says the same. Proto examples use `news.example` and a generic collecting society in place of real company names.

**fora-music-v1 carries the rights data a cue sheet needs, and finds tracks by key, version,
publisher and IPI (additive; no wire change).** `music.publishers` lists the music publishers
Expand Down
10 changes: 5 additions & 5 deletions website/src/content/docs/components/broker/deployment.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -306,19 +306,19 @@ Each Exchange scopes the key to the agent, whatever the route, so two agents beh

| Threat | Attack | Countermeasure |
|---|---|---|
| Hidden orchestration fee | Broker adds markup to relayed offers | Signed offers passed through. Agent verifies signature. Any fee is disclosed separately. |
| Kickback-driven selection | Broker has side deal with an Exchange | Selection rationale logged. Agent can spot-check by querying directly. |
| Query intelligence harvesting | Broker logs queries and sells data | Data processing agreements. In sidecar mode, agent operator controls the Broker. |
| Altering offers in transit | Broker raises a price or changes the terms of a relayed offer | The Exchange signs every offer field. A changed offer fails its signature, and the agent's acceptance signs the offer it chose. |
| Steering discovery toward favored Exchanges | Broker has a side deal with an Exchange | Not stopped by the protocol. The agent can query Exchanges directly and buy there. |
| Collecting agents' queries | Broker logs queries and sells data | Not stopped by the protocol. A data-handling agreement covers it, or the agent operator runs its own Broker. |

### TLS Requirements

All Exchange communication uses TLS 1.3 minimum. The Broker's HTTP clients are configured with `tls.Config{MinVersion: tls.VersionTLS13}`.

### Exchange Public Keys

The Broker fetches the Exchange's offer-signing keys from its WBA directory (the JWK Set at `/.well-known/http-message-signatures-directory`). Public keys are cached in Redis with a 24-hour TTL. On signature verification failure, the cache is refreshed before retrying.
The Broker fetches the Exchange's offer-signing keys from its WBA directory (the JWK Set at `/.well-known/http-message-signatures-directory`). It caches each Exchange's keys in memory, per Exchange domain, for five minutes and never past a key's `not_after`.

Key rotation: Exchanges SHOULD publish both current and previous keys during a rotation window (recommended: 7 days).
Key rotation: an Exchange publishes the new key beside the old one for the grace period [Key Rotation](/protocol/authentication/#key-rotation) describes, so offers signed with either key verify.

### Metrics

Expand Down
34 changes: 15 additions & 19 deletions website/src/content/docs/components/edge-function/bot-detection.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -21,25 +21,25 @@ Known AI crawler User-Agents. This is the primary detection mechanism. The AI co
|---|---|---|
| Training and datasets | Collects pages to train models or build datasets | `403` |
| Live fetch | Loads a page because a person asked a question | `403` |
| AI search | Builds the index that the vendor's AI search and answers draw on | Let through by default; the provider may choose `403` |
| AI search | Builds the index that the vendor's AI search and answers draw on | The provider decides per crawler |

A live fetch and a training crawl are what an agent buys a license for, so both get the `403` that points to the Exchange. An AI search crawler decides whether the site appears in that vendor's AI search at all. The vendors document that blocking one removes the site from those results, so the provider chooses, and by default the edge function lets it through.
A live fetch and a training crawl are what an agent buys a license for, so both get the `403` that points to the Exchange. An AI search crawler decides whether the site appears in that vendor's AI search at all. The vendors document that blocking one removes the site from those results, so the provider chooses which to let through. The default deny list refuses most of them, including OAI-SearchBot and PerplexityBot, and its generic `bot` token catches others; a provider that wants to stay in a vendor's AI search lets that crawler through with its own lists.

<pre><code>{[
<pre style={{ overflowX: 'auto' }}><code>{[
'# Training and datasets: 403 on licensed paths',
botLines('training'),
'',
'# Live fetch: 403 on licensed paths',
botLines('live-fetch'),
`${'fora-ai-buyer'.padEnd(24)}- a FORA agent that names itself`,
'',
'# AI search: let through unless the provider chooses 403',
'# AI search: the provider decides',
botLines('ai-search'),
].join('\n')}</code></pre>

The lists come from one file, `website/src/data/ai-bots.ts`, with each bot's vendor documentation; they are also published as [`/registry/ai-bots.json`](/registry/ai-bots.json) and shown, with what refusing each one costs, on [What you can do about AI crawlers today](/resources/ai-crawlers/).

The edge function keeps these lists in configuration (KV store or inline config): the first two kinds in `bot_detection.user_agent_patterns`, the AI search crawlers in `bot_detection.ai_search_patterns`, with the provider's choice in `bot_detection.ai_search_policy` (see [Edge function deployment](/components/edge-function/deployment/)). A bot moves between lists when its vendor changes what it does, and leaves them when its vendor retires it.
The edge function has built-in lists: a deny list of AI bots, which also matches the generic tokens `bot`, `crawler` and `spider`, and an allow list of search crawlers that read for free, such as Googlebot and Bingbot. An empty User-Agent is treated as a bot. A provider replaces either list with `BOT_UA_DENY_JSON` or `BOT_UA_ALLOW_JSON` (see [Edge function deployment](/components/edge-function/deployment/)). A bot moves between lists when its vendor changes what it does, and leaves them when its vendor retires it.

#### Search crawlers that also serve AI

Expand All @@ -61,7 +61,7 @@ All major CDNs maintain their own bot classification databases:
- **Fastly**: Signal Sciences WAF -- bot detection signals.
- **CloudFront**: AWS WAF Bot Control -- managed rule group with bot categorization.

These are available as request metadata without any computation cost at the edge. The edge function can check CDN-native bot signals as a second layer.
These are available as request metadata without any computation cost at the edge. The edge function reads Cloudflare's verified-bot signal on Cloudflare, where it overrides User-Agent matching. On Fastly and CloudFront it reads none, so a crawler that claims a search crawler's User-Agent passes there.

### Layer 3: Behavioral Signals (0ms -- pattern matching)

Expand All @@ -73,7 +73,7 @@ Heuristic patterns that indicate automated access:
- **Sequential path access**: Bots tend to walk paths sequentially (`/premium/1`, `/premium/2`, ...). Humans jump around.
- **TLS fingerprint**: JA3/JA4 fingerprint mismatches (User-Agent claims to be Chrome but TLS fingerprint is Python `requests`).

Layer 3 signals are supplementary. They can reduce false positives (bot with unknown User-Agent) but should not override a valid signed URL.
Layer 3 signals are supplementary. They can reduce false positives (bot with unknown User-Agent) but should not override a valid signed URL. The edge function does not compute them; they are what the CDN's own bot product uses, which is why it stays on beside the edge function.

## Bot Classification Tiers

Expand Down Expand Up @@ -250,30 +250,26 @@ When a bot is detected on a protected path without a signed URL:
```http
HTTP/1.1 403 Forbidden
Content-Type: application/json
X-Content-Rules: https://techcrunch.com/.well-known/fora.json
X-FORA-Exchange: exchange.ssp-example.com
Cache-Control: no-store
X-Content-Rules: https://publisher.example/.well-known/fora.json
X-FORA-Exchange: exchange.example
```

```json
{
"error": "Licensed content. Negotiate access via the Exchange.",
"protocol": "FORA",
"version": "1.0",
"fora_json_url": "https://techcrunch.com/.well-known/fora.json"
"error": "Access denied. Negotiate access via the exchange.",
"reason": "ai_bot"
}
```

**Header semantics** (the FORA edge discovery headers; the normative rules are in [Discovery Paths](/protocol/discovery-paths/#edge-discovery-headers)):
- `X-Content-Rules` -- the absolute URL of the provider's own manifest, `https://<provider domain>/.well-known/fora.json`. This is the key discovery mechanism for bots that did not check `fora.json`: the manifest carries the licensing terms and, in `exchanges[]`, the Exchanges that sell this content. It never points at an Exchange.
- `X-FORA-Exchange` -- optional. The bare domain of one Exchange that sells this content directly, in the same form as `Offer.exchange` (no scheme, no path). An agent can start discovery there without fetching the manifest first, resolving the Exchange's endpoint from the Exchange's own `fora.json`. It is an optimisation, not authorization: the edge sends it only beside `X-Content-Rules` and only for an Exchange the manifest lists, and when the two disagree the manifest wins.
- `Cache-Control: no-store` -- the 403 should not be cached. Content availability may change.

**Body semantics**:
- `error` -- human/machine readable explanation.
- `protocol` -- identifies this as a FORA protocol response, not a generic 403.
- `version` -- protocol version for forward compatibility.
- `fora_json_url` -- same as the `X-Content-Rules` header, in the body for agents that parse JSON responses. The body is informative; the headers are the protocol.
- `error` -- a human-readable explanation.
- `reason` -- `ai_bot`, the same token the edge function's log record carries. Every edge refusal uses this `{error, reason}` shape.

The body is informative; the headers are the protocol.

## Classification Algorithm

Expand Down
36 changes: 7 additions & 29 deletions website/src/content/docs/components/edge-function/cdn-adapters.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -35,8 +35,6 @@ interface EdgeContext {
/** Provider configuration loaded from config source. */
config: EdgeConfig;

/** Edge KV store (optional, for single-use enforcement). */
kv?: EdgeKV;
}

interface EdgeResponse {
Expand All @@ -45,13 +43,6 @@ interface EdgeResponse {
body: string | ReadableStream | null;
}

interface EdgeKV {
/** Atomic put-if-absent. Returns true if key was inserted (first use). */
putIfAbsent(key: string, value: string, ttlSeconds: number): Promise<boolean>;
/** Read a key. */
get(key: string): Promise<string | null>;
}

interface EdgeConfig {
/** Glob patterns that require a license (carry FORA terms); all other paths
* are open. Compiled from RSL/catalog entries — a local edge type, not a wire
Expand All @@ -75,9 +66,6 @@ interface EdgeConfig {
/** Optional pinned public keys; a covering `kid` skips the directory fetch. */
verifyKeys?: JsonWebKey[];

/** Whether to enforce single-use URLs (best-effort). Requires kv. */
singleUseEnabled: boolean;

/** Whether to enforce agent identity binding. Defaults to true. */
enforceBinding: boolean;
}
Expand Down Expand Up @@ -110,17 +98,15 @@ graph LR
end

subgraph "Optional"
LAMBDA["Lambda@Edge<br/>(origin request)<br/>Single-use enforcement<br/>Agent identity binding"]
DDB["DynamoDB Global Table<br/>(single-use tracking)"]
LAMBDA["Lambda@Edge<br/>(origin request)<br/>Agent identity binding"]
end

CF_FUNC --> BEHAVIOR
BEHAVIOR --> LAMBDA
LAMBDA --> DDB
```

- **CloudFront Function** handles bot detection and signature passthrough, and hands `/.well-known/fora.json` and `/rsl.txt` back to CloudFront, which fetches them from the origin. CloudFront's `trusted_key_groups` on the cache behavior natively verifies RSA-signed URLs -- no custom code needed.
- **Lambda@Edge** (optional) handles agent identity binding and single-use enforcement. These require network access (DynamoDB lookup) which CloudFront Functions cannot do.
- **Lambda@Edge** (optional) handles agent identity binding, which needs an Ed25519 verification CloudFront Functions cannot run within their limits.

**What CloudFront does natively that you do not reimplement**:
- RSA signature verification on signed URLs via `trusted_key_groups` configured on the cache behavior. The CDN itself rejects invalid signatures before your function code runs.
Expand All @@ -131,7 +117,6 @@ graph LR
- Bot detection (User-Agent check against `botPatterns`).
- Handing `/.well-known/fora.json` and `/rsl.txt` to the origin, ahead of bot detection.
- Custom parameters validation (agent_id binding).
- Single-use enforcement (requires Lambda@Edge + DynamoDB, best-effort).
- The 403 response body and the `X-Content-Rules` and `X-FORA-Exchange` headers.

```javascript
Expand Down Expand Up @@ -190,29 +175,25 @@ function handler(event) {
| Memory | 128 MB |
| Network access | Yes (fetch) |
| KV access | Workers KV (eventually consistent, ~60s) |
| Durable Objects | Yes (strongly consistent, for single-use) |
| Native signed URL verification | No (no equivalent to CloudFront trusted_key_groups) |
| Package size | 10 MB |
| Languages | JavaScript, TypeScript, Rust (WASM) |

**Key difference from CloudFront**: Cloudflare has no native signed URL verification. The Worker verifies the Ed25519 signature in code, using **only** the Exchange's public key. Nothing secret is deployed to the edge.

**Recommended approach**: Single Worker with optional Durable Objects.
**Recommended approach**: Single Worker.

```mermaid
graph LR
subgraph "Cloudflare Edge"
WORKER["Cloudflare Worker<br/>Bot detection<br/>fora.json + rsl.txt passthrough<br/>Ed25519 signature verification<br/>Agent identity binding"]
KV["Workers KV<br/>Config cache<br/>Bot pattern updates"]
DO["Durable Object<br/>(optional)<br/>Single-use enforcement<br/>Atomic check-and-consume"]
end

WORKER --> KV
WORKER --> DO
```

- **Signature verification**: Must be done in Worker code. Use `crypto.subtle.verify` with the `Ed25519` algorithm and the imported public key. Workers support Ed25519 natively; Fastly does not, which is why the SDK's verifier accepts an injected implementation.
- **Single-use enforcement**: Workers KV is eventually consistent (up to 60 seconds propagation). For strict single-use, use Durable Objects which provide transactional consistency. For soft single-use (acceptable eventual consistency), Workers KV with a short TTL is cheaper.
- **Config**: Store `EdgeConfig` in Workers KV. Refresh on a timer or via Cron Trigger.

## Akamai
Expand Down Expand Up @@ -241,7 +222,6 @@ mean writing the Ed25519 verifier as an EdgeWorker, which the platform's 4 ms bu

**Key advantages**:
- No execution time limit (billed per request + compute duration). Complex verification logic is fine.
- Fastly KV Store is strongly consistent within a POP, making single-use enforcement reliable without the eventual-consistency problems of Workers KV or EdgeKV.
- Full WASM runtime means you can compile the same Rust/Go verification logic that runs in the Exchange.

**With the TypeScript SDK**: Fastly's JavaScript runtime has no WebCrypto Ed25519 and no `AbortController`. Pass `verifyEd25519` to the SDK's verifiers and directory readers; its document reads need no `AbortController`, and still end at their 30-second deadline.
Expand All @@ -251,8 +231,8 @@ mean writing the Ed25519 verifier as an EdgeWorker, which the platform's 4 ms bu
```mermaid
graph LR
subgraph "Fastly Edge"
COMPUTE["Compute Service (WASM)<br/>Bot detection<br/>fora.json + rsl.txt passthrough<br/>Ed25519 signature verification<br/>Agent identity binding<br/>Single-use enforcement"]
KV_STORE["KV Store<br/>Config + single-use tracking<br/>(strongly consistent in-POP)"]
COMPUTE["Compute Service (WASM)<br/>Bot detection<br/>fora.json + rsl.txt passthrough<br/>Ed25519 signature verification<br/>Agent identity binding"]
KV_STORE["KV Store<br/>Config"]
CONFIG_STORE["Config Store<br/>Signing keys, bot patterns"]
end

Expand All @@ -268,8 +248,6 @@ graph LR
| fora.json + rsl.txt passthrough | Yes | Yes | Yes | Yes |
| Native signed URL verification | Yes (trusted_key_groups) | Yes (trusted_key_groups) | No (must implement) | No (must implement) |
| Ed25519 signed-URL verification | No (1ms limit, no Ed25519) | Yes | Yes (native Web Crypto) | Yes (injected verifier) |
| Agent identity binding | No (no network) | Yes | Yes | Yes |
| Single-use enforcement | No (no KV write) | Yes (DynamoDB) | Yes (Durable Objects) | Yes (KV Store, strong) |
| Consistency model | N/A | Strong (DynamoDB) | Strong (Durable Objects) | Strong (in-POP) |
| Agent identity binding | No (no Ed25519) | Yes | Yes | Yes |
| Latency overhead | &lt;0.1ms | 1-5ms | &lt;1ms | &lt;1ms |
| Cost at 1M req/day | $0.10 | $0.70 + DDB | $0.50 | Usage-based |
| Cost at 1M req/day | $0.10 | $0.70 | $0.50 | Usage-based |
Loading
Loading