Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/content/1.getting-started/2.installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@ Besides the providers, the package has a few more subpaths for specific jobs:
| `forges/fake` | An in-memory forge for [tests](/guides/testing) |
| `forges/testing` | Fixture replay and webhook signing for [tests](/guides/testing) |
| `forges/schema` | [JSON Schemas](/reference/json-schemas) for the models, and every verb signature |
| `forges/wire` | Encode and decode dates and bytes when models cross JSON |
| `forges/kit` | Building blocks for [writing a provider](/contributing/write-a-provider) |

## Browsers
Expand Down
8 changes: 4 additions & 4 deletions docs/content/5.reference/14.json-schemas.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,8 +38,8 @@ If you need a file, `forges/schema/<Type>.json` contains one type as a standalon

```ts
import Ajv from 'ajv'
import { reviveDates } from 'forges'
import thread from 'forges/schema/Thread.json' with { type: 'json' }
import { decode } from 'forges/wire'

const validate = new Ajv({ strict: false, validateFormats: false }).compile(thread)

Expand All @@ -48,15 +48,15 @@ if (!validate(stored)) {
throw new Error('Not a Thread')
}

// JSON has no dates, so turn the ISO strings back into `Date`s.
const value = reviveDates(stored)
// Turn the tagged dates and bytes back into `Date`s and `Uint8Array`s.
const value = decode(stored)
```

## What the models say

A few conventions apply to every model:

- `Date` fields are `{ type: 'string', format: 'date-time' }`, because JSON has no dates. To turn them back into dates, use [`reviveDates()`](/reference/data-model#revivedates).
- `Date` fields are `{ "$date": "<ISO 8601>" }`; the inner string uses `format: date-time`. Use [`decode()`](/reference/wire) to turn the object back into a `Date`.
- JSON can't carry bytes, so `Uint8Array` fields, such as the content of a binary file, are `{ "$bytes": "<base64>" }`.
- A listing result is `PageOf<Type>`, the schema of `Page<Type>` with its `items` of that type.
- The `raw` and `payload` fields accept anything, because they hold the response of the forge as it was.
Expand Down
29 changes: 29 additions & 0 deletions docs/content/5.reference/15.wire.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
---
title: Wire codec
description: Encode and decode the dates and bytes in forges models when they cross JSON.
icon: i-lucide-arrow-left-right
---

JSON has no `Date` or `Uint8Array`. `forges` returns both in its model values, so anything that stores them or sends them over HTTP must encode them first.

Most code does not need this. Providers already return real `Date` and `Uint8Array` values. Use `forges/wire` only when model values cross a JSON boundary, such as storage, a worker, or a proxy.

`forges/wire` exports `encode()` and `decode()` for that boundary.

```ts
import { decode, encode } from 'forges/wire'

const wire = encode(result)
const value = decode(JSON.parse(JSON.stringify(wire)))
```

## Tags

`encode()` replaces values that JSON cannot represent:

- `Date` becomes `{ "$date": "<ISO 8601>" }`.
- `Uint8Array` becomes `{ "$bytes": "<base64>" }`.

Everything else is copied unchanged.

`decode()` is strict: a malformed `$date` or `$bytes` tag throws a `TypeError`.
2 changes: 2 additions & 0 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,7 @@
"./tangled": "./src/tangled/index.ts",
"./testing": "./src/testing/index.ts",
"./web": "./src/web/index.ts",
"./wire": "./src/wire.ts",
"./package.json": "./package.json",
"./schema/*.json": "./dist/schema/*.json"
},
Expand Down Expand Up @@ -90,6 +91,7 @@
"./tangled": "./dist/tangled/index.mjs",
"./testing": "./dist/testing/index.mjs",
"./web": "./dist/web/index.mjs",
"./wire": "./dist/wire.mjs",
"./package.json": "./package.json",
"./schema/*.json": "./dist/schema/*.json"
}
Expand Down
4 changes: 0 additions & 4 deletions scripts/api-docs/config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -146,10 +146,6 @@ const DATA_MODEL_SECTIONS: SectionConfig[] = [
title: 'Notifications and events',
names: [/^(?:Notification|BulkNotification|ForgeEvent|Event|PushCommit|SourceKind|StreamOptions|StreamItem)/],
},
{
title: 'Dates',
names: ['reviveDates', 'DATE_FIELDS'],
},
]

export const OVERVIEW_INTRO = '`forges` is one client for every forge. You create a provider for each forge that you use, call the same operations on all of them, such as `provider.threads.merge()`, and get the same objects back.'
Expand Down
5 changes: 5 additions & 0 deletions scripts/bundle-size.ts
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,11 @@ const BUNDLES: Array<{ name: string, source: string, imports: string, absent?: s
absent: 'x-github-api-version',
budget: 10_000,
},
{
name: 'forges/wire',
source: 'wire.mjs',
imports: 'decode, encode',
},
]

interface Size {
Expand Down
21 changes: 2 additions & 19 deletions scripts/generate-schemas.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ import { readFileSync, writeFileSync } from 'node:fs'
import process from 'node:process'
import { fileURLToPath, pathToFileURL } from 'node:url'
import { createGenerator } from 'ts-json-schema-generator'
import { BYTES_SCHEMA, isRecord, refs, sortByKey, tidyDescriptions } from './schema/json.ts'
import { isRecord, refs, sortByKey, tidyDescriptions, wireSchema } from './schema/json.ts'
import { generateVerbSignatures } from './schema/verbs.ts'

/** Model types a schema is generated for; their dependencies are included. */
Expand Down Expand Up @@ -80,23 +80,6 @@ function rewriteRefs(value: Json, prefix: string): Json {
]))
}

function replaceByteArrays(value: Json): Json {
if (Array.isArray(value)) {
return value.map(replaceByteArrays)
}
if (!isRecord(value)) {
return value
}
const { properties } = value
if (isRecord(properties) && 'BYTES_PER_ELEMENT' in properties) {
return BYTES_SCHEMA
}
return Object.fromEntries(Object.entries(value).map(([key, item]) => [
key,
replaceByteArrays(item),
]))
}

/** `name` and its transitive dependencies. */
function dependenciesOf(name: string, models: Record<string, Schema>): string[] {
const found = new Set<string>()
Expand Down Expand Up @@ -132,7 +115,7 @@ function generateModels(): Record<string, Schema> {
return sortByKey(Object.fromEntries(Object.entries(definitions).map(([name, definition]) => [
name,
{
...rewriteRefs(tidyDescriptions(replaceByteArrays(definition)), '#/components/schemas/') as Schema,
...rewriteRefs(tidyDescriptions(wireSchema(definition)), '#/components/schemas/') as Schema,
'x-forges-type': name,
},
])))
Expand Down
16 changes: 8 additions & 8 deletions scripts/schema/generator.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ import type { BaseType, ChainNodeParser, Definition, SubNodeParser, SubTypeForma
import type { Schema } from './json.ts'
import { BaseType as Base, Context, createFormatter, createParser, DEFAULT_CONFIG, NeverType, SchemaGenerator } from 'ts-json-schema-generator'
import ts from 'typescript'
import { BYTES_SCHEMA, STREAM_SCHEMA } from './json.ts'
import { STREAM_SCHEMA } from './json.ts'

/** Symbols the wire maps: listings, their page, and the live provider. */
export interface ForgesTypes {
Expand All @@ -13,7 +13,6 @@ export interface ForgesTypes {
}

const WIRE_SCHEMAS = {
bytes: BYTES_SCHEMA,
opaque: { type: 'object' } as Schema,
stream: STREAM_SCHEMA,
}
Expand Down Expand Up @@ -60,11 +59,9 @@ class WireTypes implements SubNodeParser, SubTypeFormatter {
return this.chain.createType(this.forges.page as never, page)
}
const name = this.lib(node)
return name === 'Uint8Array'
? new WireType('bytes')
: name === 'ReadableStream'
? new WireType('stream')
: new NeverType()
return name === 'ReadableStream'
? new WireType('stream')
: new NeverType()
}

supportsType(type: BaseType): boolean {
Expand Down Expand Up @@ -94,7 +91,10 @@ class WireTypes implements SubNodeParser, SubTypeFormatter {

private lib(node: ts.Node): string | undefined {
return ts.isTypeReferenceNode(node)
? ['Uint8Array', 'ReadableStream', 'AbortSignal'].find(name => isLib(this.program, node, name))
? [
'ReadableStream',
'AbortSignal',
].find(name => isLib(this.program, node, name))
: undefined
}
}
Expand Down
36 changes: 36 additions & 0 deletions scripts/schema/json.ts
Original file line number Diff line number Diff line change
Expand Up @@ -22,10 +22,46 @@ export const STREAM_SCHEMA: Schema = {
description: 'The bytes, as `application/octet-stream`.',
}

export const DATE_SCHEMA: Schema = {
type: 'object',
properties: {
$date: { type: 'string', format: 'date-time' },
},
required: ['$date'],
additionalProperties: false,
description: 'Date, as `{ "$date": "<ISO 8601>" }`.',
}

export function isRecord(value: Json | undefined): value is Schema {
return !!value && typeof value === 'object' && !Array.isArray(value)
}

/** Replaces the generator's `Date` and `Uint8Array` schemas with their wire tags. */
export function wireSchema(value: Json): Json {
if (Array.isArray(value)) {
return value.map(wireSchema)
}
if (!isRecord(value)) {
return value
}
if (value.type === 'string' && value.format === 'date-time') {
return {
...DATE_SCHEMA,
...(typeof value.description === 'string'
? { description: value.description }
: {}),
}
}
const { properties } = value
if (isRecord(properties) && 'BYTES_PER_ELEMENT' in properties) {
return BYTES_SCHEMA
}
return Object.fromEntries(Object.entries(value).map(([key, item]) => [
key,
wireSchema(item),
]))
}

/** Every OpenAPI component a schema references. */
export function* refs(value: Json): Generator<string> {
if (Array.isArray(value)) {
Expand Down
4 changes: 2 additions & 2 deletions scripts/schema/verbs.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ import { fileURLToPath } from 'node:url'
import ts from 'typescript'
import { CAPABILITY_TABLE } from '../../src/capability-table.ts'
import { awaited, isStreamType, schemaGenerator } from './generator.ts'
import { clean, componentName, isRecord, refs, sortByKey, tidyDescriptions } from './json.ts'
import { clean, componentName, isRecord, refs, sortByKey, tidyDescriptions, wireSchema } from './json.ts'

const root = fileURLToPath(new URL('../..', import.meta.url))

Expand Down Expand Up @@ -111,7 +111,7 @@ export function generateVerbSignatures(): { components: Record<string, Schema>,
inline(item, stack),
]))
}
const cleanInline = (value: Json): Schema => clean(tidyDescriptions(inline(value))) as Schema
const cleanInline = (value: Json): Schema => clean(tidyDescriptions(wireSchema(inline(value)))) as Schema

const classes = new Map<string, Pick<VerbSignature, 'write' | 'account'> & { listing: boolean }>()
for (const capability of CAPABILITY_TABLE) {
Expand Down
2 changes: 1 addition & 1 deletion src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ export { github } from './github/index.ts'
export type { GitHubAuth, GitHubGraphQL, GitHubOptions, GitHubProvider, GitHubTokenAuth, GitHubTokenKind, TypedGraphQLDocument } from './github/index.ts'
export { gitlab } from './gitlab/index.ts'
export type { GitLabAuth, GitLabOptions } from './gitlab/index.ts'
export { affectedKeys, commentKey, commitCommentKey, commitKey, DATE_FIELDS, isNamespaceRef, isResolvedThread, isUnsupported, namesAreCaseInsensitive, namespaceRef, normaliseRepoName, notificationKey, notificationThread, parseCommentKey, parseCommitCommentKey, parseCommitKey, parseNotificationKey, parseReleaseKey, parseRepoKey, parseThreadKey, REACTION_CONTENTS, releaseKey, repoKey, repoRef, repoSlug, reviveDates, threadKey, threadSlug } from './model.ts'
export { affectedKeys, commentKey, commitCommentKey, commitKey, isNamespaceRef, isResolvedThread, isUnsupported, namesAreCaseInsensitive, namespaceRef, normaliseRepoName, notificationKey, notificationThread, parseCommentKey, parseCommitCommentKey, parseCommitKey, parseNotificationKey, parseReleaseKey, parseRepoKey, parseThreadKey, REACTION_CONTENTS, releaseKey, repoKey, repoRef, repoSlug, threadKey, threadSlug } from './model.ts'
export type {
Actor,
AddCollaboratorResult,
Expand Down
55 changes: 0 additions & 55 deletions src/model.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2511,58 +2511,3 @@ export interface Installation extends ForgeOrigin {
repositorySelectionRaw?: string
raw: unknown
}

/** Every `Date` field in the model, by name. */
export const DATE_FIELDS: ReadonlySet<string> = new Set([
'closedAt',
'completedAt',
'createdAt',
'date',
'deliveredAt',
'dismissedAt',
'dueOn',
'expiresAt',
'lastActivityAt',
'lastReadAt',
'mergedAt',
'nextPollAt',
'occurredAt',
'publishedAt',
'pushedAt',
'requestedAt',
'resetAt',
'startedAt',
'submittedAt',
'updatedAt',
'week',
])

const UNNORMALISED_FIELDS: ReadonlySet<string> = new Set(['payload', 'raw'])

/**
* Turns the ISO strings `JSON.stringify` left in a model value back into
* `Date`s, at any depth, for every field in {@link DATE_FIELDS}, skipping
* `raw` and `payload`. Returns a copy.
*/
export function reviveDates<T>(value: T): T {
if (Array.isArray(value)) {
return value.map(item => reviveDates(item)) as T
}
if (!value || typeof value !== 'object' || value instanceof Date) {
return value
}
return Object.fromEntries(Object.entries(value).map(([key, item]) => [
key,
reviveField(key, item),
])) as T
}

function reviveField(key: string, item: unknown): unknown {
if (UNNORMALISED_FIELDS.has(key)) {
return item
}
if (DATE_FIELDS.has(key) && typeof item === 'string' && !Number.isNaN(Date.parse(item))) {
return new Date(item)
}
return reviveDates(item)
}
Loading
Loading