Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions requirements/storage.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,8 +38,7 @@
- `key` is the `userId`
- `value` is the metadata of the user
- name
- token
- proxyPassword (optional)
- token, every earlier token, and proxyPassword (optional) - never in plain text (`system.md`)
- The system stores Actors in dedicated key-value store called `__ACTORS__`:
- `key` is the id of the Actor `actorId`
- `value` is the metadata of the Actor
Expand Down
7 changes: 7 additions & 0 deletions requirements/system.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,13 @@
actor-runtime
```

- **Credentials are never stored in plain text.** Tokens, proxy passwords and the keys to Actors'
secret env vars and secret input in the data directory can be read only with a key file the runtime
creates there, readable by the runtime's own user and root alone. A data directory written before this
is rewritten on start.
- If the key file is deleted, logins keep working, but a user's schedules and standby Actors cannot
start runs until that user makes an API request again; the runtime warns. Secret env vars set before
then must be set again.
- **Docker and Podman are equally supported**, rootful or rootless: everything the system offers works
the same on either engine. The user picks the engine by mounting its Docker-compatible API socket in
place of the Docker one (e.g. `-v /run/podman/podman.sock:/var/run/docker.sock`). One exception:
Expand Down
5 changes: 5 additions & 0 deletions skills/actor-runtime/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,11 @@ Any non-empty token authenticates - `apify login --token local-dev-token` is eno
`APIFY_DISABLE_KEYRING=1` first in a sandbox with no OS keyring. To act as a second user, pass a
different token on a single call: `apify api v2/datasets -H '{"authorization": "Bearer OTHER"}'`.

Tokens and secret env vars are never readable in the data directory, so do not look for them there.
Never delete `.token-vault.key` in the data directory: it is the key to them. If it was deleted, a
schedule or standby Actor fails with "the token of user ... cannot be read" until any API call with that
user's token, and a run with "Secret env var ... cannot be decrypted" until that env var is set again.

### Other ports

When 3333 or 3000 is taken on this machine, start the runtime on other ports:
Expand Down
9 changes: 6 additions & 3 deletions src/api/auth.ts
Original file line number Diff line number Diff line change
Expand Up @@ -54,9 +54,12 @@ export function auth(): RequestHandler {
* practice (surfaces as a 500 through `h`'s error middleware, not a user-facing error) - a route that
* calls this without `auth()` having run first is a wiring bug, not a request the caller can cause.
*/
export function requireUser(req: Request): UserRecord {
if (!req.user) {
export function requireUser(req: Request): UserRecord & { token: string } {
const { user } = req;
if (!user) {
throw new Error('requireUser() called outside the authenticated router - auth() did not run first');
}
return req.user;
// `getOrCreateUserForToken` always hands back a user holding a token: the request just brought one.
if (user.token === undefined) throw new Error(`User ${user.id} has no token after authenticating`);
return { ...user, token: user.token };
}
2 changes: 1 addition & 1 deletion src/console/server.ts
Original file line number Diff line number Diff line change
Expand Up @@ -234,7 +234,7 @@ export function createConsoleServer(deps: ConsoleServerDeps): Express {
const pool = standbyPoolSnapshot(service);
return (
'<h2>Actor Standby</h2>' +
(owner
(owner?.token !== undefined
? "<p>Open or copy the standby URL with the owner's token already in it:</p>" +
standbyLink(url, owner.token)
: '') +
Expand Down
10 changes: 10 additions & 0 deletions src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,20 @@ import { startScheduler } from './services/scheduler.js';
import { buildDto, runDto } from './api/dto/actors.js';
import { gracefulShutdown } from './shutdown.js';
import { API_PORT, CONSOLE_PORT, DEFAULT_DATA_DIR } from './config.js';
import { initTokenVault } from './services/token-vault.js';
import { sealStoredCredentials } from './services/users.js';

async function main(): Promise<void> {
const vault = initTokenVault(DEFAULT_DATA_DIR);
bootstrapStorage(DEFAULT_DATA_DIR);
await openRegistries();
const { unreadableUsers: unreadable } = await sealStoredCredentials();
if (vault.created && unreadable > 0) {
console.warn(
`The runtime's key file was missing, so the tokens of ${unreadable} user(s) cannot be read. Their ` +
'schedules and standby Actors start runs again after that user makes an API request with their token.',
);
}
// No default user is created here any more - users are created ad-hoc, per previously-unseen token,
// at the first API request that carries it (`services/users.ts: getOrCreateUserForToken()`,
// `cli.md`'s User bootstrap).
Expand Down
38 changes: 38 additions & 0 deletions src/services/actor-codec.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
import type { RecordCodec } from '../storage/registry.js';
import type { ActorRecord, ActorSecretKeys } from '../storage/entities.js';
import { seal, unseal } from './token-vault.js';

/** `secretKeys` as written to disk: the private key is already protected by the passphrase, so sealing the
* passphrase alone keeps the Actor's secrets unreadable to anyone without the runtime's key file. */
interface StoredSecretKeys {
publicKey: string;
privateKey: string;
sealedPassphrase: string;
}

type StoredActor = Omit<ActorRecord, 'secretKeys'> & { secretKeys?: StoredSecretKeys | ActorSecretKeys };

export const actorCodec: RecordCodec<ActorRecord> = {
isOutdated: (stored) => {
const { secretKeys } = stored as StoredActor;
return secretKeys !== undefined && 'passphrase' in secretKeys;
},
encode(actor): StoredActor {
if (!actor.secretKeys) return actor;
const { publicKey, privateKey, passphrase } = actor.secretKeys;
return { ...actor, secretKeys: { publicKey, privateKey, sealedPassphrase: seal(passphrase, actor.id) } };
},
decode(stored): ActorRecord {
const { secretKeys, ...actor } = stored as StoredActor;
// Written before passphrases were sealed: read as is, rewritten sealed on start (`Registry.rewriteOutdated`).
if (!secretKeys || 'passphrase' in secretKeys) return stored as ActorRecord;
const passphrase = unseal<string>(secretKeys.sealedPassphrase, actor.id);
// Unreadable once the key file is deleted: a key pair that cannot open is no key pair, so a new one
// is made on the next write or run (`secrets.ts: ensureSecretKeys`).
if (typeof passphrase !== 'string') return actor;
return {
...actor,
secretKeys: { publicKey: secretKeys.publicKey, privateKey: secretKeys.privateKey, passphrase },
};
},
};
4 changes: 2 additions & 2 deletions src/services/scheduler.ts
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ import { listOwnedRuns, startRun } from './runs.js';
import { MAX_LOG_MESSAGES, nextRunAtOf } from './schedules.js';
import { standbyUrl } from './standby-config.js';
import { getOwnedTask } from './tasks.js';
import { getUserById, resolveProxyPassword } from './users.js';
import { getUserById, resolveProxyPassword, tokenOf } from './users.js';

/** `setTimeout` overflows past this; a longer wait is re-armed when it elapses. */
const MAX_TIMER_DELAY_MILLIS = 2 ** 31 - 1;
Expand Down Expand Up @@ -231,7 +231,7 @@ async function startActionRun(
standbyUrl: standbyUrl(actor, user.username),
proxyPassword: resolveProxyPassword(user),
apiBaseUrl: CONTAINER_API_BASE_URL,
token: user.token,
token: tokenOf(user),
});
}

Expand Down
21 changes: 16 additions & 5 deletions src/services/secrets.ts
Original file line number Diff line number Diff line change
Expand Up @@ -107,17 +107,28 @@ export function inputSecretsEnv(secretKeys: ActorSecretKeys): Record<string, str
export function decryptedEnvVars(actor: ActorRecord, envVars: ActorEnvVarRecord[] | undefined): ActorEnvVarRecord[] {
const sealed = (envVars ?? []).filter((envVar) => envVar.isSecret && envVar.encryptedAes256Password);
if (sealed.length === 0) return envVars ?? [];
if (!actor.secretKeys) throw new Error(`Actor ${actor.id} has an encrypted secret but no key pair`);
const cannotDecrypt = (name: string) =>
new Error(
`Secret env var "${name}" of Actor ${actor.id} cannot be decrypted: the runtime's key file was deleted. ` +
'Set it again.',
);
// No key pair: the one it was sealed with no longer opens (`actor-codec.ts`).
if (!actor.secretKeys) throw cannotDecrypt(sealed[0]!.name);
const privateKey = createPrivateKey({ key: actor.secretKeys.privateKey, passphrase: actor.secretKeys.passphrase });
return (envVars ?? []).map((envVar) => {
if (!sealed.includes(envVar)) return envVar;
const opened = JSON.parse(
privateDecrypt({
let plain: string;
try {
plain = privateDecrypt({
privateKey,
encryptedPassword: envVar.encryptedAes256Password!,
encryptedValue: envVar.value,
}),
) as { actId?: string; name?: string; value?: string };
});
} catch {
// Sealed under a key pair replaced since (`ensureSecretKeys`), after the old one stopped opening.
throw cannotDecrypt(envVar.name);
}
const opened = JSON.parse(plain) as { actId?: string; name?: string; value?: string };
if (opened.actId !== actor.id || opened.name !== envVar.name || typeof opened.value !== 'string') {
throw new Error(`Secret env var "${envVar.name}" was not encrypted for Actor ${actor.id}`);
}
Expand Down
4 changes: 2 additions & 2 deletions src/services/standby.ts
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ import { publishAborting, publishPersistState } from './events-channel.js';
import { isTerminalJobStatus } from './job-status.js';
import { resolveTaggedBuild } from './builds.js';
import { resolveBuildInput } from './input-schema.js';
import { resolveProxyPassword } from './users.js';
import { resolveProxyPassword, tokenOf } from './users.js';
import { containerServerPortFor, startRun } from './runs.js';
import { findVersion } from './actors.js';
import { markStandbyRunFinishing } from './standby-finish.js';
Expand Down Expand Up @@ -167,7 +167,7 @@ async function startStandbyRun(
standbyUrl: standbyUrl(task ?? actor, user.username),
proxyPassword: resolveProxyPassword(user),
apiBaseUrl: CONTAINER_API_BASE_URL,
token: user.token,
token: tokenOf(user),
});
}

Expand Down
79 changes: 79 additions & 0 deletions src/services/token-vault.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
import { createCipheriv, createDecipheriv, createHash, randomBytes } from 'node:crypto';
import { chmodSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs';
import { join } from 'node:path';

/**
* Keeps login tokens out of the data directory in plain text (`system.md`): a token is stored only as its
* SHA-256 (enough to recognise a login) and sealed with AES-256-GCM under a key file the runtime creates in
* the data directory, readable only by the runtime's own user. Root and that user can still open the
* tokens; any other user on the host, reading the rest of the data directory, cannot.
*/

const KEY_FILE = '.token-vault.key';
const KEY_BYTES = 32;
const IV_BYTES = 12;
const TAG_BYTES = 16;
const SEALED_PREFIX = 'v1:';

let key: Buffer | undefined;

/** Loads the key from `dataDir`, creating it on first start. `created` with users already stored means the
* previous key file was deleted, so their sealed tokens no longer open. */
export function initTokenVault(dataDir: string): { created: boolean } {
mkdirSync(dataDir, { recursive: true });
const path = join(dataDir, KEY_FILE);
try {
const existing = readFileSync(path);
if (existing.length !== KEY_BYTES) throw new Error(`${path} is not a ${KEY_BYTES}-byte key`);
// Re-tightened on every start: a copy or restore of the data directory may have loosened it.
chmodSync(path, 0o600);
key = existing;
return { created: false };
} catch (error) {
if ((error as NodeJS.ErrnoException).code !== 'ENOENT') throw error;
}
const fresh = randomBytes(KEY_BYTES);
// `wx`: never overwrite a key another process wrote in the meantime - that would orphan its tokens.
writeFileSync(path, fresh, { mode: 0o600, flag: 'wx' });
// `mode` above is narrowed by the umask only, never widened; this pins it whatever the umask.
chmodSync(path, 0o600);
key = fresh;
return { created: true };
}

function requireKey(): Buffer {
if (!key) throw new Error('initTokenVault() has not been called yet.');
return key;
}

/** Unsalted on purpose: lookups must match across restarts and even after the key is lost. API tokens
* are long random strings, so the hash cannot be reversed by guessing. */
export function hashToken(token: string): string {
return createHash('sha256').update(token, 'utf8').digest('hex');
}

/** `context` (the record id) is bound in as associated data: a sealed value copied onto another record
* does not open there. */
export function seal(value: unknown, context: string): string {
const iv = randomBytes(IV_BYTES);
const cipher = createCipheriv('aes-256-gcm', requireKey(), iv);
cipher.setAAD(Buffer.from(context, 'utf8'));
const encrypted = Buffer.concat([cipher.update(JSON.stringify(value), 'utf8'), cipher.final()]);
return SEALED_PREFIX + Buffer.concat([iv, cipher.getAuthTag(), encrypted]).toString('base64');
}

/** `undefined` when the value was sealed under another key, for another record, or was tampered with. */
export function unseal<T>(sealed: string, context: string): T | undefined {
if (!sealed.startsWith(SEALED_PREFIX)) return undefined;
const raw = Buffer.from(sealed.slice(SEALED_PREFIX.length), 'base64');
if (raw.length < IV_BYTES + TAG_BYTES) return undefined;
try {
const decipher = createDecipheriv('aes-256-gcm', requireKey(), raw.subarray(0, IV_BYTES));
decipher.setAAD(Buffer.from(context, 'utf8'));
decipher.setAuthTag(raw.subarray(IV_BYTES, IV_BYTES + TAG_BYTES));
const plain = Buffer.concat([decipher.update(raw.subarray(IV_BYTES + TAG_BYTES)), decipher.final()]);
return JSON.parse(plain.toString('utf8')) as T;
} catch {
return undefined;
}
}
55 changes: 55 additions & 0 deletions src/services/user-codec.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
import type { RecordCodec } from '../storage/registry.js';
import type { UserRecord } from '../storage/entities.js';
import { hashToken, seal, unseal } from './token-vault.js';

/** A `__USERS__` record as written to disk: the user's credentials only ever sealed (`token-vault.ts`). */
interface StoredUser {
id: string;
username: string;
createdAt: string;
tokenHashes: string[];
sealed: string;
}

interface SealedCredentials {
token?: string;
proxyPassword?: string;
}

/** A record written before tokens were sealed, rewritten sealed on start (`Registry.rewriteOutdated`). */
interface LegacyUser {
id: string;
username: string;
createdAt: string;
token: string;
tokens?: string[];
proxyPassword?: string;
}

function isLegacyUser(stored: unknown): stored is LegacyUser {
return typeof (stored as LegacyUser).token === 'string';
}

export const userCodec: RecordCodec<UserRecord> = {
isOutdated: isLegacyUser,
encode({ id, username, createdAt, tokenHashes, token, proxyPassword }): StoredUser {
const credentials: SealedCredentials = { token, proxyPassword };
return { id, username, createdAt, tokenHashes, sealed: seal(credentials, id) };
},
decode(stored): UserRecord {
if (isLegacyUser(stored)) {
const { id, username, createdAt, token, tokens, proxyPassword } = stored;
return { id, username, createdAt, token, tokenHashes: (tokens ?? [token]).map(hashToken), proxyPassword };
}
const { id, username, createdAt, tokenHashes, sealed } = stored as StoredUser;
const credentials = unseal<SealedCredentials>(sealed, id) ?? {};
return {
id,
username,
createdAt,
tokenHashes,
...(credentials.token !== undefined ? { token: credentials.token } : {}),
...(credentials.proxyPassword !== undefined ? { proxyPassword: credentials.proxyPassword } : {}),
};
},
};
Loading
Loading