From 31a1060d17dcefb585f5684925fd80dc868160d9 Mon Sep 17 00:00:00 2001 From: Rachael Graham Date: Fri, 6 Nov 2026 13:30:35 -0600 Subject: [PATCH 1/3] 1.0.0 release switchover closes https://github.com/kagent-dev/website/issues/521 Repoints the navigation and renames doc versions for the 1.0.0 GA release, planned for monday. Signed-off-by: Rachael Graham --- .github/workflows/framework-tests.yml | 20 +- .github/workflows/update-ref-docs.yaml | 25 +- DEVELOPMENT.md | 2 +- Makefile | 2 +- docs-site/.docs-test.toml | 6 +- .../versions/agent-substrate-chart.md | 2 +- .../kagent-docs/versions/agent-substrate.md | 2 +- .../assets/kagent-docs/versions/kagent.md | 2 +- .../assets/kagent-docs/versions/max-kube.md | 2 +- .../content/kagent/{1.x => latest}/_index.md | 2 +- .../kagent/{1.x => latest}/about/_index.md | 0 .../about/architecture/_index.md | 0 .../about/architecture/agent-substrate.md | 89 ++ .../about/architecture/kagent.md | 0 .../{1.x => latest}/about/core-concepts.md | 0 .../latest/about/substrate-runtime/_index.md | 8 + .../about/substrate-runtime/identity.md | 225 ++++ .../networking-and-egress.md | 125 ++ .../about/substrate-runtime/sandboxing.md | 115 ++ .../substrate-runtime/standalone-sandboxes.md | 167 +++ .../substrate-runtime/suspend-and-resume.md | 93 ++ .../{1.x => latest}/about/what-is-kagent.md | 0 .../kagent/{1.x => latest}/agents/_index.md | 0 .../{1.x => latest}/agents/agent-harness.md | 0 .../{1.x => latest}/agents/agent-memory.md | 0 .../agents/bring-your-own-agent.md | 0 .../agents/context-management.md | 0 .../agents/declarative-agents.md | 0 .../{1.x => latest}/agents/egress-policies.md | 0 .../agents/human-in-the-loop.md | 0 .../{1.x => latest}/agents/scheduled-runs.md | 0 .../{1.x => latest}/agents/share-a-session.md | 0 .../agents/structured-output.md | 0 .../{1.x => latest}/agents/system-prompts.md | 0 .../kagent/{1.x => latest}/examples/_index.md | 0 .../{1.x => latest}/examples/a2a-agents.md | 0 .../{1.x => latest}/examples/a2a-byo.md | 0 .../examples/agent-delegation.md | 0 .../examples/agent-substrate.md | 0 .../{1.x => latest}/examples/agentgateway.md | 0 .../examples/agents-via-mcp.md | 0 .../examples/documentation-agent.md | 0 .../kagent/{1.x => latest}/examples/skills.md | 0 .../{1.x => latest}/get-started/_index.md | 0 .../get-started/your-first-agent.md | 0 .../get-started/your-first-mcp-tool.md | 0 .../kagent/{1.x => latest}/not-in-version.md | 0 .../{1.x => latest}/observability/_index.md | 0 .../observability/audit-prompts.md | 0 .../observability/lightweight-otel-stack.md | 0 .../{1.x => latest}/observability/metrics.md | 0 .../observability/otel-stack.md | 0 .../observability/substrate-telemetry.md | 0 .../{1.x => latest}/observability/tracing.md | 0 .../{1.x => latest}/operations/_index.md | 0 .../{1.x => latest}/operations/debug.md | 0 .../operations/export-to-helm.md | 0 .../operations/multiple-namespaces.md | 0 .../operations/operational-considerations.md | 0 .../operations/tune-agent-substrate.md | 0 .../{1.x => latest}/operations/uninstall.md | 0 .../operations/upgrade-from-0x.md | 0 .../{1.x => latest}/operations/upgrade.md | 0 .../{1.x => latest}/reference/_index.md | 0 .../{1.x => latest}/reference/a2a-metadata.md | 0 .../{1.x => latest}/reference/api-ref.md | 0 .../{1.x => latest}/reference/cli/_index.md | 0 .../reference/cli/kagent-agent-get.md | 0 .../reference/cli/kagent-agent-invoke.md | 0 .../reference/cli/kagent-agent-list.md | 0 .../cli/kagent-agent-session-create.md | 0 .../cli/kagent-agent-session-delete.md | 0 .../reference/cli/kagent-agent-session-get.md | 0 .../cli/kagent-agent-session-list.md | 0 .../cli/kagent-agent-session-push-create.md | 0 .../cli/kagent-agent-session-push-delete.md | 0 .../cli/kagent-agent-session-push-get.md | 0 .../cli/kagent-agent-session-push-list.md | 0 .../cli/kagent-agent-session-push.md | 0 .../reference/cli/kagent-agent-session.md | 0 .../cli/kagent-agent-template-get.md | 0 .../cli/kagent-agent-template-list.md | 0 .../reference/cli/kagent-agent-template.md | 0 .../reference/cli/kagent-agent.md | 0 .../reference/cli/kagent-apply.md | 0 .../reference/cli/kagent-bug-report.md | 0 .../reference/cli/kagent-completion-bash.md | 0 .../reference/cli/kagent-completion-fish.md | 0 .../cli/kagent-completion-powershell.md | 0 .../reference/cli/kagent-completion-zsh.md | 0 .../reference/cli/kagent-completion.md | 0 .../reference/cli/kagent-config-schema.md | 0 .../reference/cli/kagent-config-validate.md | 0 .../reference/cli/kagent-config-view.md | 0 .../reference/cli/kagent-config.md | 0 .../reference/cli/kagent-dashboard.md | 0 .../reference/cli/kagent-db-migrate-down.md | 0 .../reference/cli/kagent-db-migrate-goto.md | 0 .../reference/cli/kagent-db-migrate-status.md | 0 .../reference/cli/kagent-db-migrate-up.md | 0 .../cli/kagent-db-migrate-version.md | 0 .../reference/cli/kagent-db-migrate.md | 0 .../reference/cli/kagent-db.md | 0 .../reference/cli/kagent-install.md | 0 .../reference/cli/kagent-sandbox-create.md | 0 .../reference/cli/kagent-sandbox-delete.md | 0 .../reference/cli/kagent-sandbox-download.md | 0 .../reference/cli/kagent-sandbox-exec.md | 0 .../reference/cli/kagent-sandbox-get.md | 0 .../reference/cli/kagent-sandbox-kill.md | 0 .../reference/cli/kagent-sandbox-list.md | 0 .../reference/cli/kagent-sandbox-process.md | 0 .../reference/cli/kagent-sandbox-resume.md | 0 .../reference/cli/kagent-sandbox-suspend.md | 0 .../reference/cli/kagent-sandbox-templates.md | 0 .../reference/cli/kagent-sandbox-upload.md | 0 .../reference/cli/kagent-sandbox-wait.md | 0 .../reference/cli/kagent-sandbox.md | 0 .../reference/cli/kagent-uninstall.md | 0 .../reference/cli/kagent-version.md | 0 .../{1.x => latest}/reference/community.md | 0 .../kagent/{1.x => latest}/reference/faq.md | 0 .../{1.x => latest}/reference/glossary.md | 0 .../content/kagent/latest/reference/helm.md | 281 ++++ .../reference/release-notes/1.0.md | 0 .../reference/release-notes/_index.md | 0 .../reference/tools-ecosystem.md | 0 .../{1.x => latest}/reference/versions.md | 0 .../kagent/{1.x => latest}/setup/_index.md | 0 .../{1.x => latest}/setup/installation.md | 0 .../setup/model-providers/_index.md | 0 .../model-providers/about-model-providers.md | 0 .../setup/model-providers/amazon-bedrock.md | 0 .../setup/model-providers/anthropic.md | 0 .../setup/model-providers/azure-ai-foundry.md | 0 .../setup/model-providers/azure-openai.md | 0 .../setup/model-providers/byo-agentgateway.md | 0 .../setup/model-providers/byo-openai.md | 0 .../setup/model-providers/gemini.md | 0 .../setup/model-providers/google-vertexai.md | 0 .../setup/model-providers/mistral.md | 0 .../setup/model-providers/ollama.md | 0 .../setup/model-providers/openai.md | 0 .../setup/model-providers/sap-ai-core.md | 0 .../setup/model-providers/xai.md | 0 .../{1.x => latest}/skills-and-mcp/_index.md | 0 .../skills-and-mcp/about-tools.md | 0 .../{1.x => latest}/skills-and-mcp/plugins.md | 0 .../{1.x => latest}/skills-and-mcp/skills.md | 0 .../kagent/{1.x => latest}/ui/_index.md | 0 .../content/kagent/latest/ui/launch-ui.md | 232 ++++ .../content/kagent/latest/ui/reverse-proxy.md | 141 ++ docs-site/content/kagent/main/_index.md | 9 + docs-site/content/kagent/main/about/_index.md | 8 + .../kagent/main/about/architecture/_index.md | 5 + .../about/architecture/agent-substrate.md | 2 +- .../kagent/main/about/architecture/kagent.md | 79 ++ .../kagent/main/about/core-concepts.md | 174 +++ .../about/substrate-runtime/_index.md | 2 +- .../about/substrate-runtime/identity.md | 2 +- .../networking-and-egress.md | 2 +- .../about/substrate-runtime/sandboxing.md | 2 +- .../substrate-runtime/standalone-sandboxes.md | 2 +- .../substrate-runtime/suspend-and-resume.md | 2 +- .../kagent/main/about/what-is-kagent.md | 49 + .../content/kagent/main/agents/_index.md | 6 + .../kagent/main/agents/agent-harness.md | 424 ++++++ .../kagent/main/agents/agent-memory.md | 263 ++++ .../main/agents/bring-your-own-agent.md | 285 ++++ .../kagent/main/agents/context-management.md | 110 ++ .../kagent/main/agents/declarative-agents.md | 128 ++ .../kagent/main/agents/egress-policies.md | 282 ++++ .../kagent/main/agents/human-in-the-loop.md | 275 ++++ .../kagent/main/agents/scheduled-runs.md | 237 ++++ .../kagent/main/agents/share-a-session.md | 44 + .../kagent/main/agents/structured-output.md | 381 ++++++ .../kagent/main/agents/system-prompts.md | 167 +++ .../content/kagent/main/examples/_index.md | 6 + .../kagent/main/examples/a2a-agents.md | 436 ++++++ .../content/kagent/main/examples/a2a-byo.md | 282 ++++ .../kagent/main/examples/agent-delegation.md | 202 +++ .../kagent/main/examples/agent-substrate.md | 212 +++ .../kagent/main/examples/agentgateway.md | 400 ++++++ .../kagent/main/examples/agents-via-mcp.md | 585 +++++++++ .../main/examples/documentation-agent.md | 327 +++++ .../content/kagent/main/examples/skills.md | 430 ++++++ .../content/kagent/main/get-started/_index.md | 6 + .../main/get-started/your-first-agent.md | 181 +++ .../main/get-started/your-first-mcp-tool.md | 188 +++ .../content/kagent/main/not-in-version.md | 8 + .../kagent/main/observability/_index.md | 6 + .../main/observability/audit-prompts.md | 157 +++ .../observability/lightweight-otel-stack.md | 410 ++++++ .../kagent/main/observability/metrics.md | 145 ++ .../kagent/main/observability/otel-stack.md | 590 +++++++++ .../main/observability/substrate-telemetry.md | 192 +++ .../kagent/main/observability/tracing.md | 283 ++++ .../content/kagent/main/operations/_index.md | 6 + .../content/kagent/main/operations/debug.md | 226 ++++ .../kagent/main/operations/export-to-helm.md | 84 ++ .../main/operations/multiple-namespaces.md | 307 +++++ .../operations/operational-considerations.md | 448 +++++++ .../main/operations/tune-agent-substrate.md | 453 +++++++ .../kagent/main/operations/uninstall.md | 65 + .../kagent/main/operations/upgrade-from-0x.md | 307 +++++ .../content/kagent/main/operations/upgrade.md | 240 ++++ .../content/kagent/main/reference/_index.md | 6 + .../kagent/main/reference/a2a-metadata.md | 129 ++ .../content/kagent/main/reference/api-ref.md | 1168 +++++++++++++++++ .../kagent/main/reference/cli/_index.md | 19 + .../main/reference/cli/kagent-agent-get.md | 27 + .../main/reference/cli/kagent-agent-invoke.md | 39 + .../main/reference/cli/kagent-agent-list.md | 29 + .../cli/kagent-agent-session-create.md | 29 + .../cli/kagent-agent-session-delete.md | 28 + .../reference/cli/kagent-agent-session-get.md | 27 + .../cli/kagent-agent-session-list.md | 29 + .../cli/kagent-agent-session-push-create.md | 29 + .../cli/kagent-agent-session-push-delete.md | 27 + .../cli/kagent-agent-session-push-get.md | 27 + .../cli/kagent-agent-session-push-list.md | 29 + .../cli/kagent-agent-session-push.md | 33 + .../reference/cli/kagent-agent-session.md | 34 + .../cli/kagent-agent-template-get.md | 27 + .../cli/kagent-agent-template-list.md | 29 + .../reference/cli/kagent-agent-template.md | 31 + .../kagent/main/reference/cli/kagent-agent.md | 34 + .../kagent/main/reference/cli/kagent-apply.md | 28 + .../main/reference/cli/kagent-bug-report.md | 27 + .../reference/cli/kagent-completion-bash.md | 47 + .../reference/cli/kagent-completion-fish.md | 38 + .../cli/kagent-completion-powershell.md | 35 + .../reference/cli/kagent-completion-zsh.md | 49 + .../main/reference/cli/kagent-completion.md | 34 + .../reference/cli/kagent-config-schema.md | 28 + .../reference/cli/kagent-config-validate.md | 28 + .../main/reference/cli/kagent-config-view.md | 28 + .../main/reference/cli/kagent-config.md | 33 + .../main/reference/cli/kagent-dashboard.md | 27 + .../reference/cli/kagent-db-migrate-down.md | 25 + .../reference/cli/kagent-db-migrate-goto.md | 25 + .../reference/cli/kagent-db-migrate-status.md | 26 + .../reference/cli/kagent-db-migrate-up.md | 25 + .../cli/kagent-db-migrate-version.md | 25 + .../main/reference/cli/kagent-db-migrate.md | 34 + .../kagent/main/reference/cli/kagent-db.md | 24 + .../main/reference/cli/kagent-install.md | 31 + .../reference/cli/kagent-sandbox-create.md | 30 + .../reference/cli/kagent-sandbox-delete.md | 27 + .../reference/cli/kagent-sandbox-download.md | 27 + .../main/reference/cli/kagent-sandbox-exec.md | 30 + .../main/reference/cli/kagent-sandbox-get.md | 27 + .../main/reference/cli/kagent-sandbox-kill.md | 27 + .../main/reference/cli/kagent-sandbox-list.md | 29 + .../reference/cli/kagent-sandbox-process.md | 27 + .../reference/cli/kagent-sandbox-resume.md | 27 + .../reference/cli/kagent-sandbox-suspend.md | 27 + .../reference/cli/kagent-sandbox-templates.md | 27 + .../reference/cli/kagent-sandbox-upload.md | 28 + .../main/reference/cli/kagent-sandbox-wait.md | 29 + .../main/reference/cli/kagent-sandbox.md | 42 + .../main/reference/cli/kagent-uninstall.md | 39 + .../main/reference/cli/kagent-version.md | 27 + .../kagent/main/reference/community.md | 15 + .../content/kagent/main/reference/faq.md | 45 + .../content/kagent/main/reference/glossary.md | 10 + .../kagent/{1.x => main}/reference/helm.md | 2 +- .../main/reference/release-notes/1.0.md | 87 ++ .../main/reference/release-notes/_index.md | 8 + .../kagent/main/reference/tools-ecosystem.md | 329 +++++ .../content/kagent/main/reference/versions.md | 48 + docs-site/content/kagent/main/setup/_index.md | 6 + .../content/kagent/main/setup/installation.md | 512 ++++++++ .../main/setup/model-providers/_index.md | 6 + .../model-providers/about-model-providers.md | 294 +++++ .../setup/model-providers/amazon-bedrock.md | 135 ++ .../main/setup/model-providers/anthropic.md | 119 ++ .../setup/model-providers/azure-ai-foundry.md | 142 ++ .../setup/model-providers/azure-openai.md | 130 ++ .../setup/model-providers/byo-agentgateway.md | 162 +++ .../main/setup/model-providers/byo-openai.md | 125 ++ .../main/setup/model-providers/gemini.md | 113 ++ .../setup/model-providers/google-vertexai.md | 37 + .../main/setup/model-providers/mistral.md | 78 ++ .../main/setup/model-providers/ollama.md | 186 +++ .../main/setup/model-providers/openai.md | 93 ++ .../main/setup/model-providers/sap-ai-core.md | 35 + .../kagent/main/setup/model-providers/xai.md | 65 + .../kagent/main/skills-and-mcp/_index.md | 6 + .../kagent/main/skills-and-mcp/about-tools.md | 200 +++ .../kagent/main/skills-and-mcp/plugins.md | 108 ++ .../kagent/main/skills-and-mcp/skills.md | 53 + docs-site/content/kagent/main/ui/_index.md | 6 + .../kagent/{1.x => main}/ui/launch-ui.md | 2 +- .../kagent/{1.x => main}/ui/reverse-proxy.md | 2 +- docs-site/data/glossary.yaml | 62 +- docs-site/hugo.yaml | 99 +- .../partials/utils/version-noindex.html | 18 +- docs-site/playwright/README.md | 14 +- docs-site/playwright/playwright.config.ts | 2 +- .../playwright/provisioners/kagent-kind.sh | 10 +- docs-site/playwright/tests/launch-ui.spec.ts | 2 +- next.config.mjs | 22 +- scripts/generate-cli-docs.py | 2 +- scripts/local-version-config.py | 6 +- src/app/agents/[agentId]/page.tsx | 2 +- src/app/page.tsx | 4 +- src/blogContent/crewai-byo-agent.mdx | 6 +- .../deploy-kagent-with-agent-substrate.mdx | 8 +- src/blogContent/human-in-the-loop-kagent.mdx | 4 +- .../is-pod-right-deployment-unit-ai-agent.mdx | 2 +- .../kagent-agent-substrate-sandboxes.mdx | 2 +- src/blogContent/kgateway-guardrails.mdx | 2 +- src/blogContent/reactive-agents-khook.mdx | 2 +- src/components/footer.tsx | 32 +- src/components/key-components.tsx | 2 +- 316 files changed, 17842 insertions(+), 200 deletions(-) rename docs-site/content/kagent/{1.x => latest}/_index.md (96%) rename docs-site/content/kagent/{1.x => latest}/about/_index.md (100%) rename docs-site/content/kagent/{1.x => latest}/about/architecture/_index.md (100%) create mode 100644 docs-site/content/kagent/latest/about/architecture/agent-substrate.md rename docs-site/content/kagent/{1.x => latest}/about/architecture/kagent.md (100%) rename docs-site/content/kagent/{1.x => latest}/about/core-concepts.md (100%) create mode 100644 docs-site/content/kagent/latest/about/substrate-runtime/_index.md create mode 100644 docs-site/content/kagent/latest/about/substrate-runtime/identity.md create mode 100644 docs-site/content/kagent/latest/about/substrate-runtime/networking-and-egress.md create mode 100644 docs-site/content/kagent/latest/about/substrate-runtime/sandboxing.md create mode 100644 docs-site/content/kagent/latest/about/substrate-runtime/standalone-sandboxes.md create mode 100644 docs-site/content/kagent/latest/about/substrate-runtime/suspend-and-resume.md rename docs-site/content/kagent/{1.x => latest}/about/what-is-kagent.md (100%) rename docs-site/content/kagent/{1.x => latest}/agents/_index.md (100%) rename docs-site/content/kagent/{1.x => latest}/agents/agent-harness.md (100%) rename docs-site/content/kagent/{1.x => latest}/agents/agent-memory.md (100%) rename docs-site/content/kagent/{1.x => latest}/agents/bring-your-own-agent.md (100%) rename docs-site/content/kagent/{1.x => latest}/agents/context-management.md (100%) rename docs-site/content/kagent/{1.x => latest}/agents/declarative-agents.md (100%) rename docs-site/content/kagent/{1.x => latest}/agents/egress-policies.md (100%) rename docs-site/content/kagent/{1.x => latest}/agents/human-in-the-loop.md (100%) rename docs-site/content/kagent/{1.x => latest}/agents/scheduled-runs.md (100%) rename docs-site/content/kagent/{1.x => latest}/agents/share-a-session.md (100%) rename docs-site/content/kagent/{1.x => latest}/agents/structured-output.md (100%) rename docs-site/content/kagent/{1.x => latest}/agents/system-prompts.md (100%) rename docs-site/content/kagent/{1.x => latest}/examples/_index.md (100%) rename docs-site/content/kagent/{1.x => latest}/examples/a2a-agents.md (100%) rename docs-site/content/kagent/{1.x => latest}/examples/a2a-byo.md (100%) rename docs-site/content/kagent/{1.x => latest}/examples/agent-delegation.md (100%) rename docs-site/content/kagent/{1.x => latest}/examples/agent-substrate.md (100%) rename docs-site/content/kagent/{1.x => latest}/examples/agentgateway.md (100%) rename docs-site/content/kagent/{1.x => latest}/examples/agents-via-mcp.md (100%) rename docs-site/content/kagent/{1.x => latest}/examples/documentation-agent.md (100%) rename docs-site/content/kagent/{1.x => latest}/examples/skills.md (100%) rename docs-site/content/kagent/{1.x => latest}/get-started/_index.md (100%) rename docs-site/content/kagent/{1.x => latest}/get-started/your-first-agent.md (100%) rename docs-site/content/kagent/{1.x => latest}/get-started/your-first-mcp-tool.md (100%) rename docs-site/content/kagent/{1.x => latest}/not-in-version.md (100%) rename docs-site/content/kagent/{1.x => latest}/observability/_index.md (100%) rename docs-site/content/kagent/{1.x => latest}/observability/audit-prompts.md (100%) rename docs-site/content/kagent/{1.x => latest}/observability/lightweight-otel-stack.md (100%) rename docs-site/content/kagent/{1.x => latest}/observability/metrics.md (100%) rename docs-site/content/kagent/{1.x => latest}/observability/otel-stack.md (100%) rename docs-site/content/kagent/{1.x => latest}/observability/substrate-telemetry.md (100%) rename docs-site/content/kagent/{1.x => latest}/observability/tracing.md (100%) rename docs-site/content/kagent/{1.x => latest}/operations/_index.md (100%) rename docs-site/content/kagent/{1.x => latest}/operations/debug.md (100%) rename docs-site/content/kagent/{1.x => latest}/operations/export-to-helm.md (100%) rename docs-site/content/kagent/{1.x => latest}/operations/multiple-namespaces.md (100%) rename docs-site/content/kagent/{1.x => latest}/operations/operational-considerations.md (100%) rename docs-site/content/kagent/{1.x => latest}/operations/tune-agent-substrate.md (100%) rename docs-site/content/kagent/{1.x => latest}/operations/uninstall.md (100%) rename docs-site/content/kagent/{1.x => latest}/operations/upgrade-from-0x.md (100%) rename docs-site/content/kagent/{1.x => latest}/operations/upgrade.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/_index.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/a2a-metadata.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/api-ref.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/cli/_index.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/cli/kagent-agent-get.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/cli/kagent-agent-invoke.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/cli/kagent-agent-list.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/cli/kagent-agent-session-create.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/cli/kagent-agent-session-delete.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/cli/kagent-agent-session-get.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/cli/kagent-agent-session-list.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/cli/kagent-agent-session-push-create.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/cli/kagent-agent-session-push-delete.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/cli/kagent-agent-session-push-get.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/cli/kagent-agent-session-push-list.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/cli/kagent-agent-session-push.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/cli/kagent-agent-session.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/cli/kagent-agent-template-get.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/cli/kagent-agent-template-list.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/cli/kagent-agent-template.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/cli/kagent-agent.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/cli/kagent-apply.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/cli/kagent-bug-report.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/cli/kagent-completion-bash.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/cli/kagent-completion-fish.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/cli/kagent-completion-powershell.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/cli/kagent-completion-zsh.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/cli/kagent-completion.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/cli/kagent-config-schema.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/cli/kagent-config-validate.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/cli/kagent-config-view.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/cli/kagent-config.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/cli/kagent-dashboard.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/cli/kagent-db-migrate-down.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/cli/kagent-db-migrate-goto.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/cli/kagent-db-migrate-status.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/cli/kagent-db-migrate-up.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/cli/kagent-db-migrate-version.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/cli/kagent-db-migrate.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/cli/kagent-db.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/cli/kagent-install.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/cli/kagent-sandbox-create.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/cli/kagent-sandbox-delete.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/cli/kagent-sandbox-download.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/cli/kagent-sandbox-exec.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/cli/kagent-sandbox-get.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/cli/kagent-sandbox-kill.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/cli/kagent-sandbox-list.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/cli/kagent-sandbox-process.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/cli/kagent-sandbox-resume.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/cli/kagent-sandbox-suspend.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/cli/kagent-sandbox-templates.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/cli/kagent-sandbox-upload.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/cli/kagent-sandbox-wait.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/cli/kagent-sandbox.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/cli/kagent-uninstall.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/cli/kagent-version.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/community.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/faq.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/glossary.md (100%) create mode 100644 docs-site/content/kagent/latest/reference/helm.md rename docs-site/content/kagent/{1.x => latest}/reference/release-notes/1.0.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/release-notes/_index.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/tools-ecosystem.md (100%) rename docs-site/content/kagent/{1.x => latest}/reference/versions.md (100%) rename docs-site/content/kagent/{1.x => latest}/setup/_index.md (100%) rename docs-site/content/kagent/{1.x => latest}/setup/installation.md (100%) rename docs-site/content/kagent/{1.x => latest}/setup/model-providers/_index.md (100%) rename docs-site/content/kagent/{1.x => latest}/setup/model-providers/about-model-providers.md (100%) rename docs-site/content/kagent/{1.x => latest}/setup/model-providers/amazon-bedrock.md (100%) rename docs-site/content/kagent/{1.x => latest}/setup/model-providers/anthropic.md (100%) rename docs-site/content/kagent/{1.x => latest}/setup/model-providers/azure-ai-foundry.md (100%) rename docs-site/content/kagent/{1.x => latest}/setup/model-providers/azure-openai.md (100%) rename docs-site/content/kagent/{1.x => latest}/setup/model-providers/byo-agentgateway.md (100%) rename docs-site/content/kagent/{1.x => latest}/setup/model-providers/byo-openai.md (100%) rename docs-site/content/kagent/{1.x => latest}/setup/model-providers/gemini.md (100%) rename docs-site/content/kagent/{1.x => latest}/setup/model-providers/google-vertexai.md (100%) rename docs-site/content/kagent/{1.x => latest}/setup/model-providers/mistral.md (100%) rename docs-site/content/kagent/{1.x => latest}/setup/model-providers/ollama.md (100%) rename docs-site/content/kagent/{1.x => latest}/setup/model-providers/openai.md (100%) rename docs-site/content/kagent/{1.x => latest}/setup/model-providers/sap-ai-core.md (100%) rename docs-site/content/kagent/{1.x => latest}/setup/model-providers/xai.md (100%) rename docs-site/content/kagent/{1.x => latest}/skills-and-mcp/_index.md (100%) rename docs-site/content/kagent/{1.x => latest}/skills-and-mcp/about-tools.md (100%) rename docs-site/content/kagent/{1.x => latest}/skills-and-mcp/plugins.md (100%) rename docs-site/content/kagent/{1.x => latest}/skills-and-mcp/skills.md (100%) rename docs-site/content/kagent/{1.x => latest}/ui/_index.md (100%) create mode 100644 docs-site/content/kagent/latest/ui/launch-ui.md create mode 100644 docs-site/content/kagent/latest/ui/reverse-proxy.md create mode 100644 docs-site/content/kagent/main/_index.md create mode 100644 docs-site/content/kagent/main/about/_index.md create mode 100644 docs-site/content/kagent/main/about/architecture/_index.md rename docs-site/content/kagent/{1.x => main}/about/architecture/agent-substrate.md (99%) create mode 100644 docs-site/content/kagent/main/about/architecture/kagent.md create mode 100644 docs-site/content/kagent/main/about/core-concepts.md rename docs-site/content/kagent/{1.x => main}/about/substrate-runtime/_index.md (82%) rename docs-site/content/kagent/{1.x => main}/about/substrate-runtime/identity.md (99%) rename docs-site/content/kagent/{1.x => main}/about/substrate-runtime/networking-and-egress.md (99%) rename docs-site/content/kagent/{1.x => main}/about/substrate-runtime/sandboxing.md (99%) rename docs-site/content/kagent/{1.x => main}/about/substrate-runtime/standalone-sandboxes.md (99%) rename docs-site/content/kagent/{1.x => main}/about/substrate-runtime/suspend-and-resume.md (99%) create mode 100644 docs-site/content/kagent/main/about/what-is-kagent.md create mode 100644 docs-site/content/kagent/main/agents/_index.md create mode 100644 docs-site/content/kagent/main/agents/agent-harness.md create mode 100644 docs-site/content/kagent/main/agents/agent-memory.md create mode 100644 docs-site/content/kagent/main/agents/bring-your-own-agent.md create mode 100644 docs-site/content/kagent/main/agents/context-management.md create mode 100644 docs-site/content/kagent/main/agents/declarative-agents.md create mode 100644 docs-site/content/kagent/main/agents/egress-policies.md create mode 100644 docs-site/content/kagent/main/agents/human-in-the-loop.md create mode 100644 docs-site/content/kagent/main/agents/scheduled-runs.md create mode 100644 docs-site/content/kagent/main/agents/share-a-session.md create mode 100644 docs-site/content/kagent/main/agents/structured-output.md create mode 100644 docs-site/content/kagent/main/agents/system-prompts.md create mode 100644 docs-site/content/kagent/main/examples/_index.md create mode 100644 docs-site/content/kagent/main/examples/a2a-agents.md create mode 100644 docs-site/content/kagent/main/examples/a2a-byo.md create mode 100644 docs-site/content/kagent/main/examples/agent-delegation.md create mode 100644 docs-site/content/kagent/main/examples/agent-substrate.md create mode 100644 docs-site/content/kagent/main/examples/agentgateway.md create mode 100644 docs-site/content/kagent/main/examples/agents-via-mcp.md create mode 100644 docs-site/content/kagent/main/examples/documentation-agent.md create mode 100644 docs-site/content/kagent/main/examples/skills.md create mode 100644 docs-site/content/kagent/main/get-started/_index.md create mode 100644 docs-site/content/kagent/main/get-started/your-first-agent.md create mode 100644 docs-site/content/kagent/main/get-started/your-first-mcp-tool.md create mode 100644 docs-site/content/kagent/main/not-in-version.md create mode 100644 docs-site/content/kagent/main/observability/_index.md create mode 100644 docs-site/content/kagent/main/observability/audit-prompts.md create mode 100644 docs-site/content/kagent/main/observability/lightweight-otel-stack.md create mode 100644 docs-site/content/kagent/main/observability/metrics.md create mode 100644 docs-site/content/kagent/main/observability/otel-stack.md create mode 100644 docs-site/content/kagent/main/observability/substrate-telemetry.md create mode 100644 docs-site/content/kagent/main/observability/tracing.md create mode 100644 docs-site/content/kagent/main/operations/_index.md create mode 100644 docs-site/content/kagent/main/operations/debug.md create mode 100644 docs-site/content/kagent/main/operations/export-to-helm.md create mode 100644 docs-site/content/kagent/main/operations/multiple-namespaces.md create mode 100644 docs-site/content/kagent/main/operations/operational-considerations.md create mode 100644 docs-site/content/kagent/main/operations/tune-agent-substrate.md create mode 100644 docs-site/content/kagent/main/operations/uninstall.md create mode 100644 docs-site/content/kagent/main/operations/upgrade-from-0x.md create mode 100644 docs-site/content/kagent/main/operations/upgrade.md create mode 100644 docs-site/content/kagent/main/reference/_index.md create mode 100644 docs-site/content/kagent/main/reference/a2a-metadata.md create mode 100644 docs-site/content/kagent/main/reference/api-ref.md create mode 100644 docs-site/content/kagent/main/reference/cli/_index.md create mode 100644 docs-site/content/kagent/main/reference/cli/kagent-agent-get.md create mode 100644 docs-site/content/kagent/main/reference/cli/kagent-agent-invoke.md create mode 100644 docs-site/content/kagent/main/reference/cli/kagent-agent-list.md create mode 100644 docs-site/content/kagent/main/reference/cli/kagent-agent-session-create.md create mode 100644 docs-site/content/kagent/main/reference/cli/kagent-agent-session-delete.md create mode 100644 docs-site/content/kagent/main/reference/cli/kagent-agent-session-get.md create mode 100644 docs-site/content/kagent/main/reference/cli/kagent-agent-session-list.md create mode 100644 docs-site/content/kagent/main/reference/cli/kagent-agent-session-push-create.md create mode 100644 docs-site/content/kagent/main/reference/cli/kagent-agent-session-push-delete.md create mode 100644 docs-site/content/kagent/main/reference/cli/kagent-agent-session-push-get.md create mode 100644 docs-site/content/kagent/main/reference/cli/kagent-agent-session-push-list.md create mode 100644 docs-site/content/kagent/main/reference/cli/kagent-agent-session-push.md create mode 100644 docs-site/content/kagent/main/reference/cli/kagent-agent-session.md create mode 100644 docs-site/content/kagent/main/reference/cli/kagent-agent-template-get.md create mode 100644 docs-site/content/kagent/main/reference/cli/kagent-agent-template-list.md create mode 100644 docs-site/content/kagent/main/reference/cli/kagent-agent-template.md create mode 100644 docs-site/content/kagent/main/reference/cli/kagent-agent.md create mode 100644 docs-site/content/kagent/main/reference/cli/kagent-apply.md create mode 100644 docs-site/content/kagent/main/reference/cli/kagent-bug-report.md create mode 100644 docs-site/content/kagent/main/reference/cli/kagent-completion-bash.md create mode 100644 docs-site/content/kagent/main/reference/cli/kagent-completion-fish.md create mode 100644 docs-site/content/kagent/main/reference/cli/kagent-completion-powershell.md create mode 100644 docs-site/content/kagent/main/reference/cli/kagent-completion-zsh.md create mode 100644 docs-site/content/kagent/main/reference/cli/kagent-completion.md create mode 100644 docs-site/content/kagent/main/reference/cli/kagent-config-schema.md create mode 100644 docs-site/content/kagent/main/reference/cli/kagent-config-validate.md create mode 100644 docs-site/content/kagent/main/reference/cli/kagent-config-view.md create mode 100644 docs-site/content/kagent/main/reference/cli/kagent-config.md create mode 100644 docs-site/content/kagent/main/reference/cli/kagent-dashboard.md create mode 100644 docs-site/content/kagent/main/reference/cli/kagent-db-migrate-down.md create mode 100644 docs-site/content/kagent/main/reference/cli/kagent-db-migrate-goto.md create mode 100644 docs-site/content/kagent/main/reference/cli/kagent-db-migrate-status.md create mode 100644 docs-site/content/kagent/main/reference/cli/kagent-db-migrate-up.md create mode 100644 docs-site/content/kagent/main/reference/cli/kagent-db-migrate-version.md create mode 100644 docs-site/content/kagent/main/reference/cli/kagent-db-migrate.md create mode 100644 docs-site/content/kagent/main/reference/cli/kagent-db.md create mode 100644 docs-site/content/kagent/main/reference/cli/kagent-install.md create mode 100644 docs-site/content/kagent/main/reference/cli/kagent-sandbox-create.md create mode 100644 docs-site/content/kagent/main/reference/cli/kagent-sandbox-delete.md create mode 100644 docs-site/content/kagent/main/reference/cli/kagent-sandbox-download.md create mode 100644 docs-site/content/kagent/main/reference/cli/kagent-sandbox-exec.md create mode 100644 docs-site/content/kagent/main/reference/cli/kagent-sandbox-get.md create mode 100644 docs-site/content/kagent/main/reference/cli/kagent-sandbox-kill.md create mode 100644 docs-site/content/kagent/main/reference/cli/kagent-sandbox-list.md create mode 100644 docs-site/content/kagent/main/reference/cli/kagent-sandbox-process.md create mode 100644 docs-site/content/kagent/main/reference/cli/kagent-sandbox-resume.md create mode 100644 docs-site/content/kagent/main/reference/cli/kagent-sandbox-suspend.md create mode 100644 docs-site/content/kagent/main/reference/cli/kagent-sandbox-templates.md create mode 100644 docs-site/content/kagent/main/reference/cli/kagent-sandbox-upload.md create mode 100644 docs-site/content/kagent/main/reference/cli/kagent-sandbox-wait.md create mode 100644 docs-site/content/kagent/main/reference/cli/kagent-sandbox.md create mode 100644 docs-site/content/kagent/main/reference/cli/kagent-uninstall.md create mode 100644 docs-site/content/kagent/main/reference/cli/kagent-version.md create mode 100644 docs-site/content/kagent/main/reference/community.md create mode 100644 docs-site/content/kagent/main/reference/faq.md create mode 100644 docs-site/content/kagent/main/reference/glossary.md rename docs-site/content/kagent/{1.x => main}/reference/helm.md (99%) create mode 100644 docs-site/content/kagent/main/reference/release-notes/1.0.md create mode 100644 docs-site/content/kagent/main/reference/release-notes/_index.md create mode 100644 docs-site/content/kagent/main/reference/tools-ecosystem.md create mode 100644 docs-site/content/kagent/main/reference/versions.md create mode 100644 docs-site/content/kagent/main/setup/_index.md create mode 100644 docs-site/content/kagent/main/setup/installation.md create mode 100644 docs-site/content/kagent/main/setup/model-providers/_index.md create mode 100644 docs-site/content/kagent/main/setup/model-providers/about-model-providers.md create mode 100644 docs-site/content/kagent/main/setup/model-providers/amazon-bedrock.md create mode 100644 docs-site/content/kagent/main/setup/model-providers/anthropic.md create mode 100644 docs-site/content/kagent/main/setup/model-providers/azure-ai-foundry.md create mode 100644 docs-site/content/kagent/main/setup/model-providers/azure-openai.md create mode 100644 docs-site/content/kagent/main/setup/model-providers/byo-agentgateway.md create mode 100644 docs-site/content/kagent/main/setup/model-providers/byo-openai.md create mode 100644 docs-site/content/kagent/main/setup/model-providers/gemini.md create mode 100644 docs-site/content/kagent/main/setup/model-providers/google-vertexai.md create mode 100644 docs-site/content/kagent/main/setup/model-providers/mistral.md create mode 100644 docs-site/content/kagent/main/setup/model-providers/ollama.md create mode 100644 docs-site/content/kagent/main/setup/model-providers/openai.md create mode 100644 docs-site/content/kagent/main/setup/model-providers/sap-ai-core.md create mode 100644 docs-site/content/kagent/main/setup/model-providers/xai.md create mode 100644 docs-site/content/kagent/main/skills-and-mcp/_index.md create mode 100644 docs-site/content/kagent/main/skills-and-mcp/about-tools.md create mode 100644 docs-site/content/kagent/main/skills-and-mcp/plugins.md create mode 100644 docs-site/content/kagent/main/skills-and-mcp/skills.md create mode 100644 docs-site/content/kagent/main/ui/_index.md rename docs-site/content/kagent/{1.x => main}/ui/launch-ui.md (99%) rename docs-site/content/kagent/{1.x => main}/ui/reverse-proxy.md (99%) diff --git a/.github/workflows/framework-tests.yml b/.github/workflows/framework-tests.yml index 2188a644..d08b2be6 100644 --- a/.github/workflows/framework-tests.yml +++ b/.github/workflows/framework-tests.yml @@ -43,10 +43,6 @@ on: - 'docs-site/go.mod' - 'docs-site/go.sum' - 'docs-site/hugo.yaml' - # Layered on top of hugo.yaml by the build step below, so a change to it - # changes what this job renders. Drop this line when the overlay goes - # away at the 1.0 release. - - 'docs-site/hugo.preview.yaml' - 'docs-site/.docs-test.toml' - '.github/workflows/framework-tests.yml' workflow_dispatch: @@ -122,21 +118,7 @@ jobs: npm ci # Capture the build log so hugo-warnings.spec can scan it for ERROR / # WARN lines; surface it and fail if the build itself fails. - # - # The preview overlay and -D are what put content/kagent/1.x/ in this - # build at all. That section is withheld from production by a draft - # cascade in 1.x/_index.md, so a plain `hugo --config hugo.yaml` - # renders ZERO 1.x pages and the `content` scanners below pass green - # having walked none of them — which is the whole reason this job - # includes docs-site/content/** in its path filter. The overlay also - # restores the 1.x entry in params.versions, without which every - # {{< version include-if="1.x" >}} renders empty and hugo-warnings - # sees a clean log for a build that is quietly missing a docset. - # Both are temporary: at the 1.0 release the draft keys and - # hugo.preview.yaml go away, and this line reverts to - # `hugo --config hugo.yaml --gc --minify`. links.yml and the - # Makefile's serve-docs target layer the overlay the same way. - hugo --config hugo.yaml,hugo.preview.yaml -D --gc --minify > .build.log 2>&1 || { cat .build.log; exit 1; } + hugo --config hugo.yaml --gc --minify > .build.log 2>&1 || { cat .build.log; exit 1; } - name: Install harness dependencies working-directory: docs-theme-extras diff --git a/.github/workflows/update-ref-docs.yaml b/.github/workflows/update-ref-docs.yaml index dd62e87d..3d020de2 100644 --- a/.github/workflows/update-ref-docs.yaml +++ b/.github/workflows/update-ref-docs.yaml @@ -24,9 +24,9 @@ concurrency: # The docs are a Hugo site under docs-site/ (served at /docs). This workflow # generates the reference pages directly as Hugo markdown into docs-site/content: -# - kagent CRD API ref -> docs-site/content/kagent/1.x/reference/api-ref.md +# - kagent CRD API ref -> docs-site/content/kagent/latest/reference/api-ref.md # - kmcp CRD API ref -> docs-site/content/kmcp/reference/api-ref.md -# - kagent Helm ref -> docs-site/content/kagent/1.x/reference/helm.md +# - kagent Helm ref -> docs-site/content/kagent/latest/reference/helm.md # Each page uses plain Hugo YAML frontmatter (no MDX `export const metadata`). # # It also refreshes the version conrefs under @@ -48,7 +48,11 @@ concurrency: # 0.10.x is the stable line and still takes patches -- v0.10.2 landed on # 2026-09-23 -- so it is not frozen, and the by-hand regeneration it used to # need is what let its pages drift in the first place. Publishing a docs -# release tagged v0.10.2 regenerates 0.x; v1.0.0-alphaN regenerates 1.x. +# release tagged v0.10.2 regenerates 0.x; v1.0.N regenerates latest. +# +# main/ is not generated here. It documents unreleased work, so no release tag +# describes it, and it took its reference pages from latest/ when 1.0.0 was +# cut. Its version-split conref spans are asserted unchanged on every run. # # The two must never write each other's values. Every conref a page of either # line reads is version-split, and the conref helper refuses to continue if a @@ -166,13 +170,14 @@ jobs: fi # Which documented line this release belongs to. Both are live: 0.10.x - # is the stable line and still takes patches, 1.0 is the alpha line. + # still takes patches, and 1.x is the current release line, documented under + # latest/. # Everything downstream -- page paths, API version, frontmatter, # which conrefs are in scope -- follows from this. case "$KAGENT_VERSION" in 0.*) LINE=0.x ;; - 1.*) LINE=1.x ;; - *) echo "Error: $KAGENT_VERSION belongs to no documented line. Only 0.x and 1.x exist."; exit 1 ;; + 1.*) LINE=latest ;; + *) echo "Error: $KAGENT_VERSION belongs to no documented line. Only 0.x and latest (1.x) exist."; exit 1 ;; esac echo "Documenting the $LINE line." echo "line=$LINE" >> "$GITHUB_OUTPUT" @@ -304,7 +309,7 @@ jobs: WITH_RUNTIME_IMAGES=false WITH_KMCP=false ;; - 1.x) + latest) SECTION=reference API_DIR=go/api/v1alpha3 API_TITLE="API reference" @@ -317,9 +322,9 @@ jobs: HELM_WEIGHT=20 # --weight 30 slots CLI docs after api-ref (10) and helm (20). CLI_WEIGHT=30 - # Version-root-relative path behind the `link` shortcode. 1.x + # Version-root-relative path behind the `link` shortcode. latest # pages are rebased into the enterprise docs hub, so their - # cross-links must not hardcode the 1.x segment. + # cross-links must not hardcode the version segment. KAGENT_CLI_LINK_PREFIX=reference/cli WITH_RUNTIME_IMAGES=true WITH_KMCP=true @@ -609,7 +614,7 @@ jobs: } # Version-split conrefs. Each carries one span per documented line: - # {{< version include-if="0.x" >}}A{{< /version >}}{{< version include-if="1.x" >}}B{{< /version >}} + # {{< version include-if="0.x" >}}A{{< /version >}}{{< version include-if="latest" >}}B{{< /version >}}{{< version include-if="main" >}}C{{< /version >}} # Only this run's span is written, and the helper refuses to continue # if any other span moved -- that assertion is the whole safety net # keeping a 0.10.x run from rewriting 1.x values and vice versa. diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index 751fb7ca..40504618 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -68,7 +68,7 @@ To preview the docs alone at http://localhost:1313/docs/, use one of these comma | Command | Builds | |---|---| | `make serve-docs` | Every docs version in `docs-site/content/`. | -| `make serve-docs VERSION=` | Only one kagent version, such as `VERSION=1.x` or `VERSION=latest`, for a faster preview. kmcp always builds. Links into other versions do not resolve locally. | +| `make serve-docs VERSION=` | Only one kagent version, such as `VERSION=main` or `VERSION=latest`, for a faster preview. kmcp always builds. Links into other versions do not resolve locally. | | `make serve-docs NO_SEARCH=1` | Same as `make serve-docs`, without the search index. The search box does nothing, but the build is faster. Combine with `VERSION=`. | To preview the docs and marketing site together the way they deploy (so diff --git a/Makefile b/Makefile index 33cad617..56d622a8 100644 --- a/Makefile +++ b/Makefile @@ -60,7 +60,7 @@ inject-docs: ## Copy built docs into public/docs (preserves tracked assets, e.g. # back but the stylesheets never do, and the preview degrades edit by edit # instead of failing outright. Rendering to memory removes the shared directory. # -# VERSION= (for example VERSION=1.x, or VERSION=latest) builds only +# VERSION= (for example VERSION=main, or VERSION=latest) builds only # that docs version, for a faster preview. scripts/local-version-config.py # derives the skip list from docs-site/hugo.yaml's `params.versions` and writes # docs-site/hugo-local-version.yaml (gitignored). NO_SEARCH=1 builds without the diff --git a/docs-site/.docs-test.toml b/docs-site/.docs-test.toml index 84abaf8a..5bc0f6f0 100644 --- a/docs-site/.docs-test.toml +++ b/docs-site/.docs-test.toml @@ -34,14 +34,14 @@ scanRoots = [ "./content", ] -# kagent docs are versioned (0.x frozen, 1.x in-progress rewrite), nested -# under /docs/kagent//... . kmcp, the other product built into this +# kagent docs are versioned (main in development, latest released, 0.x frozen), +# nested under /docs/kagent//... . kmcp, the other product built into this # same Hugo site, stays flat/unversioned — see the sections.kagent.versions # comment in hugo.yaml for why versions are scoped per-section rather than # site-wide. [versioning] versionFromPath = "^/docs/kagent/(?[^/]+)/" -versions = ["0.x", "1.x"] +versions = ["main", "latest", "0.x"] # All checks default to enabled. No [checks] overrides are needed: kagent runs # the full default set. (The former `smoke`/`crossBrowser` toggles were removed diff --git a/docs-site/assets/kagent-docs/versions/agent-substrate-chart.md b/docs-site/assets/kagent-docs/versions/agent-substrate-chart.md index 5825a1e0..c4712d7f 100644 --- a/docs-site/assets/kagent-docs/versions/agent-substrate-chart.md +++ b/docs-site/assets/kagent-docs/versions/agent-substrate-chart.md @@ -1 +1 @@ -{{< version include-if="0.x" >}}0.0.9{{< /version >}}{{< version include-if="1.x" >}}1.0.0{{< /version >}} \ No newline at end of file +{{< version include-if="0.x" >}}0.0.9{{< /version >}}{{< version include-if="latest" >}}1.0.0{{< /version >}}{{< version include-if="main" >}}1.0.0{{< /version >}} \ No newline at end of file diff --git a/docs-site/assets/kagent-docs/versions/agent-substrate.md b/docs-site/assets/kagent-docs/versions/agent-substrate.md index 5825a1e0..c4712d7f 100644 --- a/docs-site/assets/kagent-docs/versions/agent-substrate.md +++ b/docs-site/assets/kagent-docs/versions/agent-substrate.md @@ -1 +1 @@ -{{< version include-if="0.x" >}}0.0.9{{< /version >}}{{< version include-if="1.x" >}}1.0.0{{< /version >}} \ No newline at end of file +{{< version include-if="0.x" >}}0.0.9{{< /version >}}{{< version include-if="latest" >}}1.0.0{{< /version >}}{{< version include-if="main" >}}1.0.0{{< /version >}} \ No newline at end of file diff --git a/docs-site/assets/kagent-docs/versions/kagent.md b/docs-site/assets/kagent-docs/versions/kagent.md index b43ee542..47ff3806 100644 --- a/docs-site/assets/kagent-docs/versions/kagent.md +++ b/docs-site/assets/kagent-docs/versions/kagent.md @@ -1 +1 @@ -{{< version include-if="0.x" >}}0.10.6{{< /version >}}{{< version include-if="1.x" >}}1.0.0-rc3{{< /version >}} \ No newline at end of file +{{< version include-if="0.x" >}}0.10.6{{< /version >}}{{< version include-if="latest" >}}1.0.0-rc3{{< /version >}}{{< version include-if="main" >}}1.0.0-rc3{{< /version >}} \ No newline at end of file diff --git a/docs-site/assets/kagent-docs/versions/max-kube.md b/docs-site/assets/kagent-docs/versions/max-kube.md index 001a26d8..5184a44a 100644 --- a/docs-site/assets/kagent-docs/versions/max-kube.md +++ b/docs-site/assets/kagent-docs/versions/max-kube.md @@ -1 +1 @@ -{{< version include-if="0.x" >}}1.35{{< /version >}}{{< version include-if="1.x" >}}1.35{{< /version >}} \ No newline at end of file +{{< version include-if="0.x" >}}1.35{{< /version >}}{{< version include-if="latest" >}}1.35{{< /version >}}{{< version include-if="main" >}}1.35{{< /version >}} \ No newline at end of file diff --git a/docs-site/content/kagent/1.x/_index.md b/docs-site/content/kagent/latest/_index.md similarity index 96% rename from docs-site/content/kagent/1.x/_index.md rename to docs-site/content/kagent/latest/_index.md index ab3daec1..333a957c 100644 --- a/docs-site/content/kagent/1.x/_index.md +++ b/docs-site/content/kagent/latest/_index.md @@ -1,6 +1,6 @@ --- title: kagent -linkTitle: "kagent 1.x" +linkTitle: "kagent 1.0" description: Explore concepts, guides, and reference docs for running declarative AI agents on Kubernetes with Agent Substrate, from install through day-two operations. weight: 10 author: kagent.dev diff --git a/docs-site/content/kagent/1.x/about/_index.md b/docs-site/content/kagent/latest/about/_index.md similarity index 100% rename from docs-site/content/kagent/1.x/about/_index.md rename to docs-site/content/kagent/latest/about/_index.md diff --git a/docs-site/content/kagent/1.x/about/architecture/_index.md b/docs-site/content/kagent/latest/about/architecture/_index.md similarity index 100% rename from docs-site/content/kagent/1.x/about/architecture/_index.md rename to docs-site/content/kagent/latest/about/architecture/_index.md diff --git a/docs-site/content/kagent/latest/about/architecture/agent-substrate.md b/docs-site/content/kagent/latest/about/architecture/agent-substrate.md new file mode 100644 index 00000000..7f94608e --- /dev/null +++ b/docs-site/content/kagent/latest/about/architecture/agent-substrate.md @@ -0,0 +1,89 @@ +--- +title: Agent Substrate architecture +description: See how Agent Substrate runs, suspends, and resumes the Actors behind every Session. +weight: 40 +author: kagent.dev +aliases: + - /kagent/latest/about/agent-substrate/ +--- + +{{< reuse "kagent-docs/snippets/about/agent-substrate-intro.md" >}} + +## ActorTemplate + +An **ActorTemplate** is the definition that Agent Substrate creates an Actor from. It states the agent's container image, the command that starts the agent, the agent's environment, the sandbox class to isolate it with, and the WorkerPool to schedule it onto. The kagent controller compiles an {{< gloss "Agent" >}}Agent{{< /gloss >}} and the resources that it names into an ActorTemplate. + +Agent Substrate rejects any change to an ActorTemplate's spec after it is created, so a running Session stays on a fixed definition. The controller creates a new ActorTemplate for every compiled {{< gloss "Revision" >}}revision{{< /gloss >}} instead of editing an existing one, and it reclaims an old ActorTemplate once nothing references its revision. An Agent references one for as long as the revision is its desired or its latest successful one, and a Session references one for as long as it runs on that revision. + +## Workers and WorkerPools + +Each Actor runs on a **Worker**, a pod that Agent Substrate starts in advance. A Session that needs an Actor gets one on a Worker that is already running, so no Actor waits for a new pod. The sandbox boundary sits around the Actor, not around the Worker pod that hosts it. + +### WorkerPools + +Workers come from a **WorkerPool**, a Kubernetes custom resource that an operator provisions before any Agent can compile. A WorkerPool declares how many Workers to keep running and which sandbox technology those Workers use. + +An operator never creates a Worker directly. Agent Substrate manages them and keeps enough ready in each WorkerPool for an Actor to start or resume immediately. + +A WorkerPool exposes a Kubernetes scale subresource, so a HorizontalPodAutoscaler can size a pool from a metric instead of a fixed replica count. Agent Substrate publishes how many Workers in each pool have no free Actor slot. An autoscaler that watches this number adds Workers before the pool fills, instead of after placements fail. + + + + +### Worker capacity + +One Worker hosts many Actors at the same time. Each Worker reports its maximum Actor capacity when it starts. A Worker at that capacity refuses further Actors, and Agent Substrate places them on another Worker in the pool. The Worker's Actor capacity is a property of the Worker build, not a setting on the pool. The gVisor and micro-VM Workers that Agent Substrate ships both report a capacity of 1000 Actors. + +### Actor placement + +Agent Substrate checks two things before it places an Actor on a Worker: +* Whether the Worker has a free Actor slot. This check always runs. +* Whether the Worker has enough spare CPU and memory. This check runs only when the ActorTemplate states how much CPU and memory one Actor needs. + +Every ActorTemplate that the kagent controller compiles states a CPU and memory limit for one Actor, so both checks run for a Session's Actor. The `config.runtimeDefaults.cpu` and `config.runtimeDefaults.memory` Helm values set this limit for every agent, and default to `1` CPU and `1Gi` of memory. + +The CPU and memory check compares that limit against the capacity that the Worker reports. A WorkerPool sets no resource limits on its Workers by default, and a Worker with no limits reports its whole node as its own capacity. The check then ignores every other pod on the node, so node pressure surfaces as eviction rather than as a placement failure. To make the check reflect the Worker, [set resource limits on Workers]({{< link path="operations/tune-agent-substrate#set-resource-requests-on-workers" >}}). + +Agent Substrate first narrows the pool to the Workers that can take the Actor. It then picks two of those Workers at random and places the Actor on the less loaded of the two. A Worker's load is the higher of two shares: the share of its Actor slots that are filled, and the share of its CPU and memory that is committed. Sampling two Workers keeps the decision cheap, and it avoids the pile-up that follows when every placement aims at the Worker that looks emptiest. + + + +## Atespaces + +An **atespace** is the isolation boundary that an Actor belongs to, and the first half of its identity. Agent Substrate addresses an Actor by its atespace and its name together, so the same Actor name can exist in two atespaces without colliding. + +An atespace resembles a Kubernetes namespace, but it is not one. Agent Substrate defines it as its own cluster-wide resource. No namespace contains it, a namespace's RBAC does not reach it, and deleting a Kubernetes namespace does not delete the atespace that kagent named after it. + +Actors and Workers sit differently against this boundary. An Actor belongs to exactly one atespace for its entire life, whereas a Worker belongs to none. A Worker hosts whichever Actors Agent Substrate places on it, and those Actors can come from different atespaces. + +kagent names each atespace after the Kubernetes namespace of the Agent that owns the Actor, and creates the atespace the first time a Session in that namespace needs an Actor. The Actor's own name is `session-` followed by the Session's identifier. A Session on an Agent in the `kagent` namespace therefore runs on an Actor in the `kagent` atespace. Both halves of this identity appear in the address that traffic uses to reach the Actor, which [Sandboxing]({{< link path="about/substrate-runtime/sandboxing#how-traffic-reaches-a-sandboxed-actor" >}}) covers. + +## Sandboxing + +Because an Actor often runs a model-directed agent that calls tools and executes commands, Agent Substrate runs each Actor in an isolated sandbox rather than a plain container. A WorkerPool's `sandboxClass` field selects the sandbox technology for its Workers: [gVisor](https://gvisor.dev), or a micro-VM that runs the workload under [Cloud Hypervisor](https://www.cloudhypervisor.org) with a [Kata Containers](https://katacontainers.io) kernel and root image. Both technologies isolate an Actor from its Worker's host kernel, and both support suspend and resume operations. + +kagent compiles each ActorTemplate to the sandbox class of the WorkerPool that hosts it, so a default installation runs kagent agents in a {{< gloss "gVisor" >}}gVisor{{< /gloss >}} sandbox, and a pool on the `microvm` class runs them under the micro-VM. A WorkerPool that backs kagent Harnesses must run the Worker build that matches its class. For what each class isolates, see [Sandboxing]({{< link path="about/substrate-runtime/sandboxing" >}}). + +## Suspend, snapshot, and resume + +An Actor spends most of its time idle, waiting on a person or a large language model (LLM) to respond. Agent Substrate suspends an idle Actor, reclaims the capacity that it held, and resumes the Actor when traffic arrives. A WorkerPool can therefore run far more Actors than it has room for at any one moment. + +The following diagram traces an Actor through one suspend-and-resume cycle, and shows the second path that opens up once the resulting snapshot is tagged. +

+ +{{< reuse "kagent-docs/snippets/snapshot-cycle-diagram.md" >}} + +A **WorkerPool** keeps **Workers** running and ready, and a Worker hosts the **Actor** alongside other Actors while its Session is active. Suspending this Actor writes its durable data to an immutable **ActorSnapshot** and releases the capacity that it held on its Worker. + +The diagram forks at this snapshot, because a snapshot serves two purposes. + +- **Resume** restores the same Actor onto **any Worker in the pool with room**, which is not necessarily the Worker that it ran on before. Agent Substrate starts new containers for the Actor and restores the durable data from the snapshot. The conversation continues from where it left off. Every idle agent takes this path. +- A **{{< gloss "Tag" >}}Tag{{< /gloss >}}** pins this snapshot so that Agent Substrate does not collect it. To fork a Session from a checkpoint, kagent seeds a **New Actor** from that tag. Resuming an existing Actor never goes through a tag. + +A tag gives a snapshot a stable name, so callers do not track Agent Substrate's internal snapshot identity. The tag names one snapshot permanently, and only its visibility scope can change afterward. A tag is also a retention pin, so Agent Substrate does not delete a snapshot while a tag still names it. + +For example, an agent partway through a long incident investigation reaches a state worth keeping. Creating a [checkpoint]({{< link path="about/substrate-runtime/suspend-and-resume#checkpoints" >}}) tags the snapshot that the agent most recently suspended to, which holds that one snapshot in place while the agent carries on and writes newer ones. Without the tag, Agent Substrate collects this snapshot once a newer one supersedes it. + + + + diff --git a/docs-site/content/kagent/1.x/about/architecture/kagent.md b/docs-site/content/kagent/latest/about/architecture/kagent.md similarity index 100% rename from docs-site/content/kagent/1.x/about/architecture/kagent.md rename to docs-site/content/kagent/latest/about/architecture/kagent.md diff --git a/docs-site/content/kagent/1.x/about/core-concepts.md b/docs-site/content/kagent/latest/about/core-concepts.md similarity index 100% rename from docs-site/content/kagent/1.x/about/core-concepts.md rename to docs-site/content/kagent/latest/about/core-concepts.md diff --git a/docs-site/content/kagent/latest/about/substrate-runtime/_index.md b/docs-site/content/kagent/latest/about/substrate-runtime/_index.md new file mode 100644 index 00000000..d71682e7 --- /dev/null +++ b/docs-site/content/kagent/latest/about/substrate-runtime/_index.md @@ -0,0 +1,8 @@ +--- +title: Substrate runtime +description: Learn how Agent Substrate isolates, suspends, and resumes the Actors that run your agents. +weight: 60 +author: kagent.dev +aliases: + - /kagent/latest/substrate-runtime/ +--- \ No newline at end of file diff --git a/docs-site/content/kagent/latest/about/substrate-runtime/identity.md b/docs-site/content/kagent/latest/about/substrate-runtime/identity.md new file mode 100644 index 00000000..d406f7cc --- /dev/null +++ b/docs-site/content/kagent/latest/about/substrate-runtime/identity.md @@ -0,0 +1,225 @@ +--- +title: Identity +description: Understand how kagent identifies each caller, scopes a Session to its creator, and identifies an agent to the services that it calls. +weight: 30 +author: kagent.dev +aliases: + - /kagent/latest/substrate-runtime/identity/ +--- + +A {{< reuse "kagent-docs/snippets/name-product.md" >}} installation identifies each kind of caller with a different system. + +- An operator applying a {{< gloss "Harness" >}}Harness{{< /gloss >}} is authenticated by [Kubernetes](#the-kubernetes-plane). +- A caller that reaches [kagent's gRPC API](#the-kagent-control-plane) is assigned a principal by kagent, both for a {{< gloss "Session" >}}Session{{< /gloss >}} and for the resources that the API writes. +- A workload with no person signed in can call the API as its [Kubernetes ServiceAccount](#service-account-callers). +- An agent's runtime calls the same API with a [token that Agent Substrate issues](#agent-runtime-calls). +- A service that an agent calls can receive an [Actor token](#actor-tokens-for-other-services) that identifies the agent. +- The components inside [Agent Substrate](#the-agent-substrate-plane) authenticate each other. + +## The Kubernetes plane + +Kubernetes role-based access control (RBAC) governs who can create, read, edit, or delete Kubernetes resources with `kubectl`, including Harness, AgentTemplate, and {{< gloss "Agent" >}}Agent{{< /gloss >}} resources. A cluster's existing roles and bindings decide who authors an agent's Harness and AgentTemplate, and the pairing of the two on that path. + +Because neither a Harness nor an AgentTemplate names the other, the Agent resource carries the pairing, and RBAC on Agents governs it. Whoever can write Agents in a namespace controls which template runs on which Harness, even without edit access to the Harnesses and AgentTemplates that those Agents name. For more information about the pairing, see the [Agent core concept]({{< link path="about/core-concepts#agent" >}}). + +kagent's gRPC API reaches these same resources by a second path that Kubernetes RBAC does not evaluate. For more information, see [Resources that the API writes](#resources-that-the-api-writes). For what a namespace separates between teams, see [Namespace isolation]({{< link path="operations/multiple-namespaces#namespace-isolation" >}}). + +## The kagent control plane + +A Session is not a Kubernetes resource. kagent's gRPC API creates the Session, and kagent's database tracks it. Kubernetes RBAC therefore does not reach it. kagent resolves a principal for these calls itself. + +Every call on the Session API carries a principal. An authenticator resolves one before the request reaches the service, and a call that the authenticator declines is rejected as unauthenticated before any other check runs. + +### Controller authentication modes + +The `config.server.auth.mode` Helm value selects which authenticator the controller installs. The chart defaults to `insecure`, and enabling the bundled oauth2-proxy does not change it, because browser sign-in and controller authentication are separate boundaries. To authenticate both, enable oauth2-proxy and set `config.server.auth.mode` to `trusted-proxy`. + +| Mode | How a principal is resolved | +| ---- | --------------------------- | +| `insecure` | Every request is admitted. The caller's identity is read from the `X-User-Id` header, or from a `user_id` query parameter that takes precedence over it. A caller that supplies neither is named `admin@kagent.dev`. **The caller selects its own principal.** The authenticator verifies neither the header nor the query parameter, and the same authenticator guards the controller's `/mcp` endpoint. Treat the principal on a call as a label that the caller chose, and restrict network access to both endpoints. | +| `trusted-proxy` | The request must carry a bearer token in the `Authorization` header. The identity comes from the claim that `config.server.auth.userIdClaim` names, defaulting to `sub`. A missing custom claim falls back to `sub`, and a request with no identity is rejected. **The authenticator decodes the token without verifying it.** It checks neither the signature nor the expiry, because it assumes that an upstream boundary already validated the credential. A caller that reaches the controller directly can present a token that it wrote itself. Use this mode only where ingress validates every token before forwarding it, and network policy stops anything else from reaching port `8083`. | + +Any other value stops the controller from starting. + +Authorization is separate from both modes. For user calls, kagent installs an authorizer that permits every check regardless of the authentication mode, so neither mode constrains what an authenticated user can do. For agent runtime calls, kagent applies a fixed policy that limits each Actor to its own resources. For more information, see [Agent runtime calls](#agent-runtime-calls). + +### Service account callers + +Automation that runs with no person signed in, such as a CI pipeline, a scheduled job, or another service in the cluster, needs its own identity to create and use Sessions. Neither user authentication mode gives it a verified one. In `insecure` mode, the workload names itself. In `trusted-proxy` mode, the controller trusts a token that an upstream proxy validated, and does not verify the token itself. Service account authentication lets the workload call the controller as its Kubernetes ServiceAccount, and the controller verifies that identity with Kubernetes. + +Service account authentication is off by default and works alongside either user authentication mode. To turn it on, set `config.server.auth.serviceAccount.authority` in your Helm values. The authority is a name that you choose for the cluster that issues the tokens. The chart then grants the controller permission to create TokenReviews. + +```yaml +config: + server: + auth: + serviceAccount: + authority: prod-cluster +``` + +The workload sends a ServiceAccount token whose audience is `kagent-invocation` as a bearer token in the `Authorization` header. A workload usually mounts the token as a projected volume. To test from a terminal, create a token with `kubectl`, port-forward the controller's port `8083`, and call the API. The `grpcurl` command relies on gRPC reflection, so your installation must set `config.server.grpcReflection=true`. + +```bash +kubectl create namespace ci-bots +kubectl create serviceaccount scheduler -n ci-bots +kubectl port-forward -n kagent svc/kagent-controller 8083:8083 & +TOKEN=$(kubectl create token scheduler -n ci-bots --audience kagent-invocation) +grpcurl -plaintext -H "Authorization: Bearer $TOKEN" -d '{}' \ + localhost:8083 kagent.api.v1.SystemService/GetCurrentUser +kill %1 +``` + +Example output: + +```console +{ + "claims": { + "groups": [ + "system:serviceaccounts", + "system:serviceaccounts:ci-bots", + "system:authenticated" + ], + "username": "system:serviceaccount:ci-bots:scheduler" + } +} +``` + +The controller verifies the token through the Kubernetes TokenReview API of its own cluster. A token that has a `kubernetes.io` claim, or a subject that starts with `system:serviceaccount:`, always goes to this verifier. When verification fails, the controller rejects the call as unauthenticated and does not fall back to the user authentication mode. A ServiceAccount token whose audience is not `kagent-invocation` therefore fails, including a token for the default API server audience. Each controller caches a successful review for up to 30 seconds, so a deleted ServiceAccount or a revoked token keeps working for that interval. + +kagent builds the caller's principal from the authority and the ServiceAccount's UID, and records that principal as an opaque creator ID, such as `a3ViZXJuZXRlczpwcm9kLWNsdXN0ZXI.MTZkYjk1N2Qt...`. The [creator ownership](#creator-ownership) rules then scope the workload's Sessions in the same way as a user's, and the workload gets the same authorization as a user. Rotating the ServiceAccount's token keeps the principal so that the workload keeps its Sessions. Deleting and recreating the ServiceAccount gives it a new UID and a new principal, and the workload loses access to the Sessions that it created. + +When an agent forwards the caller's token to a model or MCP server that sets `tokenPassthrough`, that destination receives the ServiceAccount token. The token keeps its kagent authority until it expires. Set `tokenPassthrough` only on destinations that you trust with that authority. + +### Resources that the API writes + +kagent's gRPC API also writes Harness, AgentTemplate, and EgressPolicy resources. The AgentTemplate service creates, updates, and deletes AgentTemplates, the Harness service creates and deletes Harnesses, and the EgressPolicy service creates, updates, and deletes EgressPolicies, all through the kagent controller. The **Policies** page of the kagent UI calls the EgressPolicy service. The `kagent apply -f` command calls the AgentTemplate service, and any client that reaches the gRPC endpoint can call each of these services. The controller writes these resources with its own service account rather than the caller's, so Kubernetes RBAC never evaluates the caller. The controller's authenticator resolves the caller instead, as described in [Controller authentication modes](#controller-authentication-modes). + +> [!WARNING] +> By default, kagent neither authenticates nor authorizes this path. The `insecure` authenticator admits every request, and the authorizer it installs permits every check. Any caller that reaches the gRPC endpoint can author an agent's Harness and AgentTemplate, and its EgressPolicies, regardless of their Kubernetes permissions. Do not expose port `8083` outside the cluster. For the identity that this mode gives an anonymous caller, and for the mode that changes it, see [Controller authentication modes](#controller-authentication-modes). + +### Creator ownership + +kagent records a **creator** on every Session, taken from the principal on the call that created it. That creator is then part of the database query for every read, so a caller who asks for a Session that another principal created receives a not-found response rather than a permission error. Sandboxes, Checkpoints, and ScheduledRuns each carry a creator of their own, and a read on one of those is scoped the same way. + +Listing behaves the same way. A list returns the caller's own Sessions by default. A caller that sets the request's all-creators flag asks to widen that to every creator in the namespace, and kagent authorizes that request separately from an ordinary list. + +> [!IMPORTANT] +> Creator ownership separates callers without containing them. kagent calls an authorizer before every Session operation, and the authorizer it installs permits every check, so any caller can widen a list. In `insecure` mode a caller also names its own principal, so a caller that presents another creator's identifier reads that creator's Sessions. Creator scoping keeps one user's conversations out of another user's list. It is not a security boundary. + +### Shares + +A share lets a Session's owner give another account access to that one conversation. Creating a share produces a token, and a caller presenting that token reaches the shared Session without becoming its creator. + +Every share is read-only. The holder can read the conversation, but cannot send a message, answer an approval, cancel a task, or retry a turn. kagent enforces the restriction in its services and transports. A shared conversation in the UI does not show controls for those actions. + +A client that creates a share can set `CreateSessionShareRequest.ttl` to a positive lifetime. The lifetime starts when the share is created. Once that lifetime ends, the token does not grant access, and a token lookup returns not found. The owner still sees the expired share in `ListSessionShares` and can revoke it. + +The returned `SessionShare.expiresAt` field records when the token stops granting access. When `ttl` is omitted, `expiresAt` stays unset, and the share stays valid until it is revoked or the Session is deleted. An operator caps share lifetimes with `config.sessions.shareMaxTtl`. For more information, see [Expire idle Sessions]({{< link path="operations/operational-considerations#expire-idle-sessions" >}}). + +A share widens what the holder can reach to what the owner can see, and the underlying record is read as the owner rather than as the visitor. Revoking the share withdraws that access. + +### Agent runtime calls + +An agent's runtime calls the controller to store task state and memories, and to read its own Agent and Session. These calls arrive on the same port as user calls, `8083`. However, the controller authenticates agent runtime calls separately from the user authentication mode. The task state and memory calls that only an agent makes always need a valid token, even in `insecure` mode. The controller treats a call that sends no `Authorization` header as a user call, and applies the user authentication mode to it. + +The runtime, the Agent Substrate egress gateway, and the controller each take one step to authenticate a call. + +1. The runtime sends a placeholder bearer token in the `Authorization` header. +2. The Agent Substrate egress gateway replaces the placeholder with a JSON Web Token (JWT) that identifies the {{< gloss "Actor" >}}Actor{{< /gloss >}}. The token is valid for five minutes, and its audience is `kagent-runtime`. The gateway adds the token only to calls to the controller. Actor snapshots do not store the token. +3. The controller verifies the token's signature, issuer, audience, and lifetime, and allows 30 seconds of clock skew. The controller then binds the token to the Actor's current Session. The controller rejects a token that fails verification, and does not fall back to user authentication. + +An authenticated Actor can read its own Agent and use its own task state and memories. The Actor can also read, update, and send messages to its own Session. The Actor cannot create or delete a Session, use a share, reach any other Session, or act with its owner's permissions. That restriction includes other Sessions of the same agent and the same owner. + +The controller trusts the issuer in the `config.substrate.actorJwtIssuer` Helm value. This value must match the `ateApi.actorJWTIssuer` value of the Agent Substrate chart. Both values default to `https://idp.ate-system.svc`, so change them only together. The controller fetches the issuer's signing keys at startup, and a controller that cannot reach the issuer does not start. + +Upgrade the controller and the runtime images together. An Actor snapshot from an earlier runtime image still sends an identity header that the controller no longer accepts, so you must recreate that snapshot after an upgrade. + +### Actor tokens for other services + +An internal service that an agent calls, such as a large language model (LLM) gateway, an MCP server, or an API, might need to know which agent is calling so that it can authorize or audit the call. The other credentials do not identify the agent. A Secret-backed API key is the same for every agent that uses it, and `tokenPassthrough` forwards the identity of the user who started the Session. An unattended run has no such user. + +An Actor token identifies the calling {{< gloss "Actor" >}}Actor{{< /gloss >}} instead. The token is a short-lived JWT from the same Agent Substrate issuer that signs the tokens in [Agent runtime calls](#agent-runtime-calls), minted for an audience that you name. The egress gateway mints, injects, and refreshes the token, and the token never enters the runtime, so no Secret needs to be stored or rotated. + +Turn the token on for one destination in one of the following places: + +- A ModelConfig, in its `actorJwt` field. For more information, see [Authenticate as the Actor]({{< link path="setup/model-providers/about-model-providers#authenticate-as-the-actor" >}}). +- A RemoteMCPServer, in its `actorJwt` field. For more information, see [Actor token authentication]({{< link path="skills-and-mcp/about-tools#actor-token-authentication" >}}). +- An EgressPolicy destination, in its `authentication.actorJwt` field. For more information, see [Inject the Actor's token]({{< link path="agents/egress-policies#inject-the-actors-token" >}}). + +All three fields accept the same two settings, and only `audiences` is required. + +| Field | Description | +| ----- | ----------- | +| `audiences` | Required. The services that can accept the token, between 1 and 16 unique values. The API server rejects `kagent-runtime`, because that audience is reserved for calls to the controller. | +| `expirationSeconds` | Optional. The token lifetime, between 300 and 3600 seconds. Defaults to 900. | + +The receiving service gets a token that is signed with ES256 and contains the following claims: + +```json +{ + "iss": "https://idp.ate-system.svc", + "sub": "actor/kagent/session-01a18e6c-4454-7597-ba2b-e962d0f53967", + "aud": ["ticket-api"], + "actor.ate.dev": { + "atespace": "kagent", + "actorName": "session-01a18e6c-4454-7597-ba2b-e962d0f53967", + "actorUID": "3d0d0f07-40fb-4493-a7cd-3cb7f2d79c3b" + } +} +``` + +The `actor.ate.dev` claim names the Actor's {{< gloss "Atespace" >}}atespace{{< /gloss >}}, name, and UID. The atespace is the namespace of the Agent or SandboxTemplate. The token also contains the standard `exp`, `nbf`, `iat`, and `jti` claims. Each Session runs as its own Actor, so the Actor's name and UID change with every Session. To authorize a team or an environment from the token alone, base the decision on the atespace. To authorize by Agent, use the [Agent lookup](#agent-lookup). + +The receiving service must check the issuer, the signature, its own audience, and the time claims. The issuer is the `ateApi.actorJWTIssuer` value of the Agent Substrate chart. The issuer publishes its OpenID Connect discovery document at `/.well-known/openid-configuration` and its signing keys at `/openid/v1/jwks`. When `ateApi.actorJWTIssuer` is set to its default, the service reaches both over TLS inside the cluster, so it must also trust the certificate that the issuer serves. + +#### Agent lookup + +The token names the Actor and its atespace, but not the Agent. A receiving service that authorizes or audits by Agent can send the token to the controller's `kagent.api.v1.AgentIdentityService/ResolveAgentIdentity` method on port `8083`. The controller accepts only a token whose audiences include `kagent-agent-info`. Add that audience next to the service's own audience in the `actorJwt` field of the ModelConfig, RemoteMCPServer, or EgressPolicy that injects the token. + +```yaml +actorJwt: + audiences: [ticket-api, kagent-agent-info] +``` + +The service sends the token that it received as a bearer token, with an empty request. In the following command, `$ACTOR_TOKEN` holds that token. The `grpcurl` command relies on gRPC reflection, so your installation must set `config.server.grpcReflection=true`. The `-rpc-header` flag keeps the token off the reflection requests. + +```bash +grpcurl -plaintext -rpc-header "Authorization: Bearer $ACTOR_TOKEN" -d '{}' \ + kagent-controller.kagent.svc.cluster.local:8083 \ + kagent.api.v1.AgentIdentityService/ResolveAgentIdentity +``` + +Example output: + +```console +{ + "agent": { + "namespace": "kagent", + "name": "ticket-assistant", + "uid": "606e209e-e4a5-4520-929e-5a7f6f3d1926", + "labels": { + "team": "support" + }, + "annotations": { + "description": "Ticket triage" + } + }, + "sessionId": "01a18efc-83f3-7b12-a30a-73906fe617b0", + "runtimeRevision": "577c2a575fac1bf237a1fcac7674bc843feb1158b438f1c26d71bf73b735dc54" +} +``` + +The response contains the Agent's namespace, name, UID, labels, and annotations, and the ID of the Session and the revision that the Actor runs. The labels and annotations are the Agent's current values, returned unchanged. They include `kubectl.kubernetes.io/last-applied-configuration` when you applied the Agent with `kubectl apply`, so the service can read the Agent's full manifest. Do not put values in an Agent's labels or annotations that the receiving service must not see. The Session ID does not give the service access to the Session. + +A token without the `kagent-agent-info` audience fails with `Invalid Agent identity credential`. A token from a deleted Session, or from an Agent that was deleted and recreated, also does not resolve. + +## The Agent Substrate plane + +{{< gloss "Agent Substrate" >}}Agent Substrate{{< /gloss >}} authenticates its own components rather than authenticating end users. Its API server accepts Kubernetes ServiceAccount tokens issued for its audience, and the components that carry traffic to an Actor authenticate each other with mutual Transport Layer Security (mTLS). `kagent install` creates the certificate authority pools and the JWT authority pool that these identities are issued from. No Helm chart creates these pools. For an installation that you manage with Helm, see [Export the installation to Helm]({{< link path="operations/export-to-helm" >}}). + +The kagent controller sends A2A calls and Sandbox guest calls through the Agent Substrate router. When `config.substrate.routerUrl` uses `https`, the controller presents its rotating client certificate to the router and verifies the router against the Agent Substrate API certificate authority bundle. The router admits the controller only when `atenetRouter.ingressAllowedSPIFFEIDs` lists the controller's SPIFFE identity. For the Helm values, see [Agent Substrate actor ingress]({{< link path="operations/operational-considerations#agent-substrate-actor-ingress" >}}). + +Each Actor also carries an identity of its own, addressed as its {{< gloss "Atespace" >}}atespace{{< /gloss >}} and name together. [Sandboxing]({{< link path="about/substrate-runtime/sandboxing" >}}) covers how the router uses that identity to reach the right Worker over mTLS. + +> [!IMPORTANT] +> Agent Substrate authenticates callers but does not authorize them. Any provider that you configure as an authenticated caller can reach every remote procedure call, including destructive ones, so configure only providers whose users require full access to Agent Substrate. diff --git a/docs-site/content/kagent/latest/about/substrate-runtime/networking-and-egress.md b/docs-site/content/kagent/latest/about/substrate-runtime/networking-and-egress.md new file mode 100644 index 00000000..f4ddc873 --- /dev/null +++ b/docs-site/content/kagent/latest/about/substrate-runtime/networking-and-egress.md @@ -0,0 +1,125 @@ +--- +title: Egress control +description: Learn how Agent Substrate authorizes the outbound connections that your agents open, and how kagent builds the allowlist it enforces. +weight: 40 +author: kagent.dev +aliases: + - /kagent/latest/substrate-runtime/networking-and-egress/ +--- + +An agent reaches the network only through Agent Substrate's egress gateway, and the gateway authorizes every connection against a policy attached to that {{< gloss "Actor" >}}Actor{{< /gloss >}}. The policy is default-deny. An Actor can attempt any request, and the gateway denies the ones that no rule allows. kagent derives the policy from the AgentTemplate, so a documented agent works without any egress configuration of its own. An Agent can also select EgressPolicy resources, which grant destinations that the AgentTemplate does not name. For more information, see [Egress policies]({{< link path="agents/egress-policies" >}}). + +> [!NOTE] +> The gateway terminates TLS, so hostname rules constrain encrypted traffic as well as cleartext. An HTTPS request is decrypted at the gateway, matched against the policy's hostnames, and re-originated to the destination. A TLS passthrough rule is the exception, because the gateway forwards this connection without decrypting it. For details, see [Traffic types](#traffic-types). + +For the inbound half of the picture, and for the NetworkPolicy that protects Workers, see [Sandboxing]({{< link path="about/substrate-runtime/sandboxing#default-network-posture" >}}). + +## How the gateway authorizes a connection + +All outbound traffic leaves a Worker through the egress gateway, which gives Agent Substrate one place to decide what each agent can reach. The gateway takes three steps on every connection, and each step depends on the one before it, because the Actor's identity selects the policy, and the policy supplies the rules. + +1. The gateway identifies the Actor. It reads the Actor's SPIFFE ID from the client certificate on the connection from the Actor, and it verifies this certificate rather than trusting anything in the request. + +2. The gateway fetches the policy for this Actor. Each Actor has at most one Agent Substrate policy named `default`, and Agent Substrate addresses the policy by the Actor rather than by this name. This Agent Substrate policy is not the kagent EgressPolicy resource. kagent compiles the Agent Substrate policy from the destinations that the AgentTemplate needs, and from every kagent EgressPolicy that the Agent selects. The gateway denies every request from an Actor that has no policy, because only a policy allows access to a destination. + +3. The gateway picks which rule applies to the request. Rules are an unordered set rather than a list, and exactly one rule applies to a given destination name and port regardless of its protocol. When more than one rule matches, the gateway applies two criteria in order, moving to the second only when the first ties. + + 1. A matching hostname without a wildcard wins over a matching hostname with one. + 2. A matching port number wins over a rule that covers all ports. + + A request that no rule matches is denied, and only the rule that applies contributes a credential. + +Policy enforcement depends on this verified identity. The Actor's identity is established only by the certificate, so the gateway authorizes every request against the Actor that actually sent it regardless of the headers or hostnames that the request carries. The gateway refuses a connection that does not present a certificate rather than falling back to a default identity. Each Actor also gets its own connection pool, even when two of them dial the same address, so that every connection carries requests from the one Actor that opened it. + +> [!NOTE] +> Two rules that tie on the same name and port are rejected when the policy is admitted, so a policy never reaches the gateway with an ambiguous match. A policy holds at most 256 rules. + +### Traffic types + +Each rule defines the type of connection that is allowed to a destination and how the gateway determines the destination's hostname. The following connection types are allowed: + +- **`http`**: Cleartext HTTP, meaning HTTP/1.1 and the h2c that gRPC uses. The gateway reads the hostname from the `Host` header and matches it against the rule's hostnames. TLS to those same names is not allowed by this rule. +- **`https`**: HTTPS, which the gateway intercepts. It terminates the connection with a certificate minted for the destination name, reads the request inside, matches it against the rule's hostnames, and re-originates its own TLS connection onward. HTTP/1.1 and HTTP/2 work inside this connection, and HTTP/3 is not supported. +- **`tls_passthrough`**: TLS forwarded to the named servers without decryption. The gateway cannot read the request, so it decides at the TLS ClientHello instead. + +All three rule types cover TCP connections. Agent Substrate blocks UDP, and it does not support a protocol in which the server sends the first bytes once the connection opens, such as SMTP or SSH. The gateway reads the destination hostname from the bytes that the client sends first, so those protocols leave the gateway with nothing to match against. + +Intercepting HTTPS has a cost. The gateway decrypts every agent request in order to authorize it, so the cleartext exists at the gateway for the length of that check. Decrypting also lets the gateway attach credentials on the agent's behalf. For more information, see [Credential injection](#credential-injection). + +## Rule matching + +Every rule names the hosts and ports that it covers, regardless of its protocol. + +| Field | What it matches | +| ----- | --------------- | +| `hostnames` | A list of DNS name patterns. The rule matches when any pattern matches the destination. | +| `ports` | Either every port, or a list of up to 16 port numbers. | +| `effects` | Not a match condition. See [Credential injection](#credential-injection). | + +A hostname pattern is a DNS name, optionally with a `*` wildcard replacing the complete leftmost label. The wildcard matches exactly one non-empty label, so `*.example.com` matches `api.example.com` but matches neither `example.com` nor `nested.api.example.com`. A `*` wildcard on its own matches every name. No other wildcard syntax is accepted, and a pattern without one matches only the complete name. Patterns must be lowercase, must not end with a dot, and must use the punycode form for an internationalized name. + +## Policy generation + +kagent derives the allowlist from the AgentTemplate rather than taking it as configuration, so the policy describes what the agent was compiled to need. Each {{< gloss "Revision" >}}revision{{< /gloss >}} records the hosts that its agent depends on: + +- The model provider endpoint, from the ModelConfig. +- Every RemoteMCPServer, HTTP tool, and SSE tool that the AgentTemplate binds. +- Every skill artifact source, such as a git or OCI registry host. +- The OpenTelemetry collector, for each telemetry signal that is enabled. + +kagent compiles this host set into the Actor's policy when it creates the Actor. Each destination becomes one rule that covers one hostname on one port, and the scheme decides the rule type. For example, an `https` destination becomes an `https` rule, and a cleartext destination becomes an `http` rule. Keeping exactly one rule per host and port is deliberate. An overlapping rule could otherwise decide a request and skip the credential that the narrower rule attaches. A destination that uses both HTTP and HTTPS on the same host and port is rejected rather than compiled. + +kagent fixes the inferred host set when it compiles the revision. An agent also reaches the destinations of every EgressPolicy that its Agent selects, in its `spec.egressPolicyRefs` field. The gateway denies every other destination, including a web page that a model decides to fetch partway through a Session. + +When kagent applies the policy to an Actor, it merges the inferred destinations with the EgressPolicy grants. An EgressPolicy can only add destinations. For the same origin and header, an inferred credential takes precedence over a credential from an EgressPolicy. For how to grant a destination, see [Grant access to a destination]({{< link path="agents/egress-policies#grant-access-to-a-destination" >}}). + +An agent also needs a route to the OpenTelemetry collector, or its traces never leave the sandbox and the kagent UI shows nothing for this Session. kagent adds the collector host for agents on every harness, `byo` included. A `byo` image that exports to a telemetry backend of its own, through a Harness `spec.env` entry, has no route to that backend, because the AgentTemplate never named it. For more information, see [Harness coverage]({{< link path="observability/tracing#harness-coverage" >}}). + +## Policy changes + +The gateway caches each Actor's policy rather than fetching it per request, so policy changes are eventually consistent. The cache holds an entry for 10 seconds by default, so a policy that is created, updated, or deleted reaches new requests within that window. Deleting a policy produces denials rather than an absence of enforcement, because the gateway caches the absence too. + +> [!IMPORTANT] +> A Session pins the revision it was created from, so editing an AgentTemplate does not move an existing Session onto a new set of inferred destinations. To pick up a changed model or tool destination, create a new Session. EgressPolicy grants are the exception. kagent applies a change to an Agent's `egressPolicyRefs`, or to a selected EgressPolicy, to the running and suspended Actors of existing Sessions as well. + +## Diagnose a denied request + +Agent Substrate does not explain why it denied a request. The agent sees a rejected call, and its own error text names neither the destination nor the rule that refused it. The gateway records the reason in its log. + +```bash +kubectl logs -n ate-system deploy/atenet-egress -c agentgateway --tail=100 +``` + +A denial appears as an HTTP `403` alongside the Actor that made the request and the host it asked for. + +```console +error request ... http.host=kagent-tools.kagent http.status=403 + ate.actor.name=ai-01a0abbb-ad73-7dfb-a86e-f2a643dc4dc9 ate.atespace=kagent + error="actor egress policy denied: ... \"EgressPolicy not found\"" reason=Authorization +``` + +Read this entry in three parts. `reason=Authorization` means that the policy refused the request rather than the request failing for another reason. The `error` text names which check failed, such as a policy that does not exist. The `http.host` names the destination that the agent asked for. Compare this host against the AgentTemplate. + +A `503` means that the gateway could not reach the control plane or a credential provider, which is an installation problem rather than a denied request. + +An allowed request is logged the same way, with its status, the `http.host` and `http.path` it matched on, and the `substrate.connect.authority` address the Actor originally dialed. The log is therefore the quickest way to see what an agent actually reaches. + +The compiled policy is not readable through `kubectl` or `kagent`. Treat the AgentTemplate and the Agent's selected EgressPolicy resources as the source of truth for what an agent is meant to reach. A destination that is missing from the log is missing from both. To check whether kagent admitted the Agent's EgressPolicy grants, read the Agent's `EgressPoliciesApplied` condition. For the reasons that this condition reports, see [Troubleshoot a grant]({{< link path="agents/egress-policies#troubleshoot-a-grant" >}}). + +## Credential injection + +A model provider or an MCP server usually requires an API key or token. The agent sends a placeholder header in place of the real value, and the gateway replaces this header with the credential as the request passes through. The key never enters the agent's environment or its filesystem. + +kagent builds these credentials from the following resources. + +- Every ModelConfig that sets `apiKeySecret` or `actorJwt` becomes a credential on the rule for that model's host. +- Every RemoteMCPServer whose `headersFrom` reads a Secret, or that sets `actorJwt`, becomes a credential on the rule for its host. +- Every EgressPolicy destination with an `authentication` block becomes a credential on the rule for its origin. + +kagent attaches the credential to the destination's own rule rather than adding a rule for it, and it refuses to compile a credential whose host is not already an allowed destination. + +A credential is declared on a rule as a `replace_headers` entry. The entry names the header to replace, an optional prefix such as `Bearer `, and either a reference that a registered credential provider resolves or a JWT that Agent Substrate issues to assert the Actor's identity. The gateway replaces the header only when the request already sends it. A request that omits the placeholder header is forwarded unchanged, without the credential, rather than being denied. + +Replacement happens only on a connection that the gateway terminates, and only when a credential provider is configured for the reference. If either is missing, the request reaches the destination without the credential. The request fails closed when replacement is attempted and fails. `403` is returned when the provider does not hold the credential or refuses to release it, and `503` is returned when the provider is unreachable. + +Replacement also works on a cleartext `http` rule, where the gateway does not originate TLS to the destination. The credential is then readable by anything on the path between the gateway and the destination unless something outside the gateway encrypts that segment. Prefer an `https` destination for any host that takes a credential. diff --git a/docs-site/content/kagent/latest/about/substrate-runtime/sandboxing.md b/docs-site/content/kagent/latest/about/substrate-runtime/sandboxing.md new file mode 100644 index 00000000..61aa8bb1 --- /dev/null +++ b/docs-site/content/kagent/latest/about/substrate-runtime/sandboxing.md @@ -0,0 +1,115 @@ +--- +title: Sandboxing +description: Learn how Agent Substrate isolates each Actor in its own sandbox, and what that sandbox separates. +weight: 10 +author: kagent.dev +aliases: + - /kagent/latest/substrate-runtime/sandboxing/ +--- + +An agent runs an **agentic loop**. A model proposes the next step, the agent carries it out by running a command or calling a tool, and the result feeds the model's next decision. The loop repeats until the task finishes or the agent stops to wait on input from a person. + +kagent runs an agent's loop inside an **Actor**, a unit of compute that [Agent Substrate]({{< link path="about/architecture/agent-substrate" >}}) creates and manages. An Actor is not an ordinary container process. Each Actor runs inside its own **sandbox**, and one {{< gloss "Worker" >}}Worker{{< /gloss >}} hosts many of those sandboxed Actors at the same time. + +This page explains what selects a sandbox, what the sandbox separates, and how traffic reaches an Actor through it. +

+ +```mermaid +flowchart LR + subgraph worker["Worker pod"] + subgraph sandbox["Sandbox: gvisor or microvm"] + subgraph actor["Actor"] + agent["Agent"] + end + end + end + classDef boundary fill:#a78bfa26,stroke:#a78bfa,stroke-width:2px + classDef inner fill:#80808033,stroke:#9ca3af,stroke-width:1px + class sandbox boundary + class agent inner +``` + +## Sandbox classes + +A **sandbox class** is the sandbox runtime family that a Worker uses. {{< gloss "Agent Substrate" >}}Agent Substrate{{< /gloss >}} supports two. + +- **`gvisor`** (default): Runs the workload against a [gVisor](https://gvisor.dev) user-space kernel. The user-space kernel keeps the workload's system calls from reaching the host kernel. +- **`microvm`**: Runs the workload inside a lightweight virtual machine. A hypervisor boundary separates the workload from the host. + +A {{< gloss "WorkerPool" >}}WorkerPool{{< /gloss >}}'s `sandboxClasses` list sets the sandbox class for its Workers. The list takes exactly one entry. The entry names a class and, optionally, the SandboxConfig that serves it. When `kagent install` creates the default pool, the pool has one `gvisor` entry. Each class does more than name a runtime. Agent Substrate builds the pool's Worker pods to match it, mounting the virtualization devices that a micro-VM needs and placing those pods on nodes that provide them. + +> [!NOTE] +> kagent copies the class of a pool's `sandboxClasses` entry onto every {{< gloss "ActorTemplate" >}}ActorTemplate{{< /gloss >}} that it generates for this pool. The pool, not the template, resolves the SandboxConfig for that class. + +## Sandbox configuration + +A **SandboxConfig** is a cluster-scoped resource that describes the runtime files a sandbox class requires to start. The resource stores none of those files. Instead, it records the location that the node agent fetches each file from, keyed by processor architecture. A SandboxConfig stores up to 16 named versions of these files. + +A WorkerPool resolves one SandboxConfig for its `sandboxClasses` entry. When the entry sets `configRef`, the pool uses the named configuration. Otherwise, the pool uses the newest SandboxConfig of its class that has the `sandboxconfig.ate.dev/is-class-default: "true"` annotation. The pool reports the result in the `Resolved` condition under `status.sandboxClasses`. Agent Substrate blocks deletion of a SandboxConfig that a pool uses. + +A default installation creates one SandboxConfig, named `gvisor-default`, and marks it as the `gvisor` class default. When `kagent install` creates the default pool, the pool also names it in its `configRef`. The configuration looks like the following: + +```yaml +apiVersion: ate.dev/v1 +kind: SandboxConfig +metadata: + name: gvisor-default + annotations: + sandboxconfig.ate.dev/is-class-default: "true" +spec: + sandboxClass: gvisor + defaultVersion: v6 + versions: + - name: v6 + state: Enabled + assets: + amd64: + gvisor: + url: gs://gvisor/releases/release/20261026.0/x86_64/gvisor.tar.zstd + sha256: + arm64: + gvisor: + url: gs://gvisor/releases/release/20261026.0/aarch64/gvisor.tar.zstd + sha256: +``` + +To read the configuration that your own cluster installed, including the pinned digests, run `kubectl get sandboxconfig gvisor-default -o yaml`. + +| Field | Description | +| ----- | ----------- | +| `sandboxClass` | The sandbox runtime family that this configuration applies to, `gvisor` or `microvm`. Defaults to `gvisor`. A WorkerPool uses only configurations whose class matches its entry. | +| `defaultVersion` | The version that new sandboxes boot with. It must name a version that exists and is not `Disabled`. | +| `versions[].name` | The version's name, a DNS label. | +| `versions[].state` | `Enabled` or `Disabled`. Defaults to `Enabled`. Worker pods keep every enabled version available, so a snapshot that was taken on an earlier version still restores. | +| `versions[].assets` | The files that the node agent fetches, keyed first by processor architecture and then by asset name. A `gvisor` class expects one `gvisor` asset, the release archive that the node agent extracts. A `microvm` class expects `cloud-hypervisor`, `virtiofsd`, `kata-kernel`, and `kata-image`. | +| `versions[].assets...sha256` | The lowercase hex digest of the file. The node agent verifies each download against it, so changing the digest fetches the new asset instead of reusing the cached one. | + +## Sandbox boundaries {#what-the-sandbox-separates} + +The sandbox separates the Actor from the Worker node in the following places. + +- **Process and kernel**: The Actor's processes run against the sandbox runtime rather than the Worker node's kernel. A system call that the workload makes is handled by {{< gloss "gVisor" >}}gVisor{{< /gloss >}}'s user-space kernel, or by the guest kernel inside a micro-VM, instead of reaching the host directly. +- **Filesystem**: The Actor sees the filesystem assembled from its container image, plus whatever durable volume its ActorTemplate declares. Writes to the root filesystem are a layer on top of the image, captured in a `Full` {{< gloss "Snapshot" >}}snapshot{{< /gloss >}} and discarded by a `Data` one. For what each scope keeps, see [Suspend and resume]({{< link path="about/substrate-runtime/suspend-and-resume" >}}). +- **Network**: The Actor does not share the Worker pod's network position. The node agent gives the active Actor a private, point-to-point virtual network inside the Worker pod, so reaching the Actor means going through Agent Substrate's own network path rather than connecting to the Worker directly. + +## Traffic path to a sandboxed Actor {#how-traffic-reaches-a-sandboxed-actor} + +Every Actor is addressed by its {{< gloss "Atespace" >}}atespace{{< /gloss >}} and name, at `..actors.resources.substrate.ate.dev`. Reaching it involves several hops, and each one keeps a sandboxed Actor addressable without exposing the Worker that runs it. + +1. Agent Substrate runs its own Domain Name System (DNS) service that answers queries for this address pattern with the address of the router, rather than any individual Worker. +2. The router reads the Actor name and atespace from the request, asks the Agent Substrate API to resume this Actor and report which Worker it is now assigned to, and then selects this Worker as the destination. +3. The router connects to a listener on the Worker over mutual Transport Layer Security (mTLS). The listener validates that the caller is the router, and forwards traffic only to the Actor currently assigned to this Worker. + +Because the router resolves the Worker assignment on every request, an Actor keeps a stable address across suspends, resumes, and moves between Workers. + +A kagent user never uses this address. A message sent to a {{< gloss "Session" >}}Session{{< /gloss >}} through the kagent API or the web console reaches the kagent controller. The controller resolves the Session to its Actor and passes the request to Agent Substrate. Every hop after that stays inside the runtime. + +Traffic in the other direction leaves through a separate egress gateway rather than going straight out from the Worker. Routing Actor egress through one gateway provides a single place to apply outbound controls. + +## Default network posture + +Agent Substrate creates a Kubernetes NetworkPolicy for each WorkerPool that selects the pool's Worker pods. The policy restricts **ingress** to the Agent Substrate router alone. No other pod in the cluster can open a connection to a Worker, so an Actor is not reachable by anything that bypasses the routing path. + +The NetworkPolicy governs inbound traffic only. Outbound traffic is governed separately by an egress policy that the gateway enforces on every connection that an Actor opens. The egress policy is default-deny. An Actor reaches a destination only when a rule allows it, and the gateway denies the connection when no rule does. + +kagent derives the egress policy from the {{< gloss "AgentTemplate" >}}AgentTemplate{{< /gloss >}}, so an agent does not need egress configuration of its own. The destinations that the egress policy allows come from what the AgentTemplate already names, such as its model provider and its MCP servers. Agent Substrate owns the ingress policy, and kagent regenerates the egress policy whenever the AgentTemplate changes. To allow a destination that the AgentTemplate does not name, select an EgressPolicy resource on the Agent or SandboxTemplate, as described in [Grant access to a destination]({{< link path="agents/egress-policies#grant-access-to-a-destination" >}}). For what the policy holds, how a rule is matched, and how to diagnose a denied request, see [Egress control]({{< link path="about/substrate-runtime/networking-and-egress" >}}). diff --git a/docs-site/content/kagent/latest/about/substrate-runtime/standalone-sandboxes.md b/docs-site/content/kagent/latest/about/substrate-runtime/standalone-sandboxes.md new file mode 100644 index 00000000..3dc10aa8 --- /dev/null +++ b/docs-site/content/kagent/latest/about/substrate-runtime/standalone-sandboxes.md @@ -0,0 +1,167 @@ +--- +title: Standalone sandboxes +description: Run commands and transfer files in a scratch environment that is independent of any agent Session. +weight: 25 +author: kagent.dev +aliases: + - /kagent/latest/substrate-runtime/standalone-sandboxes/ +--- + +A **Sandbox** is a scratch environment for running commands and working with files. A Sandbox is scoped to the caller who created it, and it expires on a timer. You create one from a **SandboxTemplate**, run processes and move files in it, then delete it or let it expire. + +A Sandbox and a {{< gloss "Session" >}}Session{{< /gloss >}} are separate runtimes with separate configuration and separate lifetimes. Neither resource references the other. + +| What you configure | What runs | What you call it with | +| ------------------ | --------- | --------------------- | +| An {{< gloss "Agent" >}}Agent{{< /gloss >}} that pairs an AgentTemplate with a Harness | A Session | A2A interactions and tasks | +| A SandboxTemplate that defines a tools environment | A Sandbox | Guest process and file operations | + +An agent can create a Sandbox of its own through the kagent Model Context Protocol ({{< gloss "MCP" >}}MCP{{< /gloss >}}) server, by using the same service that a person calls. The Sandbox's lifetime stays independent of the agent's Session. + +## Before you begin + +1. {{< reuse "kagent-docs/snippets/sandbox-config-step.md" >}} + +2. {{< reuse "kagent-docs/snippets/install-cli.md" >}} + +3. To read the Sandbox ID from the CLI's JSON output, install [`jq`](https://jqlang.org/download/). + +## Create a SandboxTemplate + +A SandboxTemplate is a namespaced `api.kagent.dev/v1alpha3` resource that prepares a reusable runtime. Applying one does not allocate a Sandbox. + +1. Apply a SandboxTemplate. + ```yaml + kubectl apply -f - <}} For the complete schema, see the [API reference]({{< link path="reference/api-ref#sandboxtemplate" >}}). + + | Field | Required | Description | + | ----- | -------- | ----------- | + | `workload.image` | Yes | The tools image, pinned by `sha256` digest. A tag alone is rejected, because a prepared revision must be reproducible. | + | `env` | No | Environment defaults for the guest, up to 100. Each entry sets a literal `value`, which is required and can be an empty string. The schema does not define a secret-backed source, so the API server rejects a `credentialRef` entry as an unknown field. | + | `substrate.workerPoolRef.name` | Yes | The WorkerPool that this template's Actors are scheduled onto. | + | `substrate.snapshotPolicy.location` | Yes | The object storage location for Actor {{< gloss "Snapshot" >}}snapshots{{< /gloss >}}. | + | `egressPolicyRefs` | No | Up to 16 EgressPolicy resources in the template's namespace that grant the Sandbox outbound access. When this field is omitted, the Sandbox does not have outbound access. For more information, see [Egress policies]({{< link path="agents/egress-policies" >}}). | + + A SandboxTemplate does not take a startup command or a guest toggle. kagent supplies the guest entrypoint from the image that `config.sandboxes.guestImage` references. That image replaces the tools image's own entrypoint. Your tools image contributes the installed programs and nothing else. + +2. Confirm that the template prepared a revision before you create a Sandbox from it. + ```bash + kubectl get sandboxtemplate scratch -n kagent \ + -o jsonpath='{range .status.conditions[?(@.type=="Ready")]}{.status} {.reason} {.message}{end}' + ``` + +Editing a template prepares a new revision. A Sandbox that already exists keeps the revision that it was created from. Deleting the template retires preparation but does not remove the Sandboxes that pinned it. + +## Run a command + +Each `kagent sandbox` command makes one lifecycle attempt. The `create` command takes a stable `--request-id` value. To retry the same creation, reuse that value. + +> [!WARNING] +> **By default, the gateway denies every outbound connection from a Sandbox.** When the template does not select an EgressPolicy resource, kagent compiles an empty [egress policy]({{< link path="about/substrate-runtime/networking-and-egress#policy-generation" >}}) for the Sandbox. A command that fetches a package, clones a repository, or calls an API then fails. To allow a destination, select an EgressPolicy in the SandboxTemplate's `spec.egressPolicyRefs` field, as described in [Grant access to a destination]({{< link path="agents/egress-policies#grant-access-to-a-destination" >}}). Otherwise, move what a command needs into the Sandbox with `kagent sandbox upload`. + +1. List the templates that your installation prepared. + ```bash + kagent sandbox templates + ``` + + Example output: + ```console + NAMESPACE NAME IMAGE + kagent scratch registry.example.com/tools@sha256:aaaa... + ``` + +2. Create a Sandbox, and save its ID. + ```bash + export SANDBOX_ID=$(kagent sandbox create scratch --request-id my-first-sandbox -o json | jq -r '.id') + echo $SANDBOX_ID + ``` + + To see the table, run the command without `-o json`. `STATE` and `OPERATION` both matter, because lifecycle work can still be pending when a call returns. + ```console + ID TEMPLATE STATE OPERATION EXPIRES FAILURE + 0198c3f1-2a44-7c90-b5e1-9d8f3a7b2c04 kagent/scratch RUNTIME_STATE_READY RUNTIME_OPERATION_NONE 2026-10-01T16:30:00Z + ``` + +3. Run a command in the Sandbox. The default working directory is `/data/workspace`, which the guest creates before it reports ready. + ```bash + kagent sandbox exec $SANDBOX_ID -- python --version + ``` + + A timeout stops the command from waiting, but the remote process keeps running. To reconnect to a process that outlived its `exec`, pass its process ID to `kagent sandbox wait`. + + `kagent sandbox kill` sends `SIGKILL` to a process and returns when the guest delivers the signal. The command does not wait for the process to exit, so its output reports the process state and omits the exit code. To read the exit code, pass the process ID to `kagent sandbox wait`. + +4. Move a file in, run against it, and read the result back out. + ```bash + kagent sandbox upload $SANDBOX_ID ./script.py /data/workspace/script.py + kagent sandbox exec $SANDBOX_ID -- python /data/workspace/script.py + kagent sandbox download $SANDBOX_ID /data/workspace/out.txt ./out.txt + ``` + +5. When you are finished, delete the Sandbox to release its compute immediately. Deletion is optional, because the Sandbox expires on its own. + ```bash + kagent sandbox delete $SANDBOX_ID + ``` + +> [!IMPORTANT] +> **Suspending a Sandbox interrupts whatever it is doing.** `kagent sandbox suspend` does not wait for any process, output stream, or file transfer, so a command can be cut off mid-run and a file can be left partly written. Resuming restores the durable files under `/data`, but a process handle does not survive, because the guest holds it in memory. Suspend a Sandbox only when you can repeat whatever it was running. + +## Reach a Sandbox from an agent + +kagent's Helm chart installs a `RemoteMCPServer` named `kagent-api` in the controller's namespace, pointing at the controller's own `/mcp` endpoint. This server exposes the Session and checkpoint tools alongside the sandbox tools, so an AgentTemplate reaches Sandboxes through an ordinary tool binding. + +```yaml +tools: + - mcp: + server: + kind: RemoteMCPServer + name: kagent-api + tools: + - list_sandbox_templates + - create_sandbox + - list_sandboxes + - write_sandbox_file + - read_sandbox_file + - read_sandbox_outputs +``` + +An agent that creates a Sandbox owns it under the identity that the MCP connection authenticated, not under the identity of the person that the agent is talking to. A Session share token does not grant access to a Sandbox. To have an agent act for the person who invoked it, configure credential propagation by setting `config.runtimeDefaults.tokenPropagation.enabled` to `true`. The kagent runtime then passes the caller's credentials and identity to the MCP servers that you trust. + +The MCP transfer limits are tighter than the command line's. A gRPC file transfer is bounded at 64 MiB. `write_sandbox_file` and `read_sandbox_outputs` are bounded at 1 MiB and use base64. Output reads return continuation byte offsets. + +An agent that starts a process with `start_sandbox_process` polls `get_sandbox_process` to learn when the process finishes. The `state` field reads `PROCESS_STATE_RUNNING` until the process exits, then `PROCESS_STATE_EXITED`. The `exit_code` field appears only in the exited state, and reads `0` for a successful exit. `kill_sandbox_process` sends `SIGKILL` in the same way as `kagent sandbox kill`, so the agent must poll `get_sandbox_process` for the exit code. + +When an agent must inspect a result file, use `read_sandbox_file`. The tool returns content that the model reads directly, not `data_base64`. The shape of the content depends on the file type. + +Text returns as numbered lines. `offset` sets the first line to read, counting from 1. `limit` sets the number of lines, and defaults to 2000. A page also stops at 32 KiB, whichever bound it reaches first. A line longer than 2000 bytes is truncated to 2000 bytes and marked `[line cut]`. A page that stops before the end of the file reports the offset to continue from. + +A PNG, JPEG, or GIF file of at most 10 MiB returns as an image. The `claude` and `codex` harnesses resize those images for their models. A larger image returns a message asking the agent to write a smaller copy in the Sandbox. Every other file type returns a description of the file, not its bytes. + +## Next steps + +{{< cards >}} + {{< card link=`{{< link path="about/substrate-runtime/sandboxing" >}}` title="Sandboxing" subtitle="Understand the gVisor boundary that a Sandbox shares with every agent Actor." >}} + {{< card link=`{{< link path="examples/agents-via-mcp" >}}` title="Use agents from an MCP client" subtitle="Reach the same MCP server that serves the sandbox tools." >}} + {{< card link=`{{< link path="reference/cli/kagent-sandbox" >}}` title="kagent sandbox" subtitle="Look up every flag that the sandbox commands take." >}} +{{< /cards >}} diff --git a/docs-site/content/kagent/latest/about/substrate-runtime/suspend-and-resume.md b/docs-site/content/kagent/latest/about/substrate-runtime/suspend-and-resume.md new file mode 100644 index 00000000..8a0c57b0 --- /dev/null +++ b/docs-site/content/kagent/latest/about/substrate-runtime/suspend-and-resume.md @@ -0,0 +1,93 @@ +--- +title: Suspend and resume +description: Learn how Agent Substrate suspends idle Actors to snapshots and resumes them on demand. +weight: 20 +author: kagent.dev +aliases: + - /kagent/latest/substrate-runtime/suspend-and-resume/ +--- + +An agent spends most of its life waiting. It waits on a person to reply, and it waits on a large language model (LLM) to answer. [Agent Substrate]({{< link path="about/architecture/agent-substrate" >}}) runs each agent inside an **Actor**, the unit that Agent Substrate suspends and resumes. To free compute resources while an agent waits, Agent Substrate writes the idle Actor to a {{< gloss "Snapshot" >}}snapshot{{< /gloss >}} and releases the capacity that the Actor held on its {{< gloss "Worker" >}}Worker{{< /gloss >}}. The next message to the agent restores the Actor. The {{< gloss "Session" >}}Session{{< /gloss >}} stays ready the whole time, so reclaiming this capacity costs the user nothing. This page explains what a snapshot captures, when kagent suspends an Actor, and what happens when a suspended Actor is addressed again. + +## Actor lifecycle operations + +{{< gloss "Agent Substrate" >}}Agent Substrate{{< /gloss >}} provides two lifecycle operations that kagent uses. Each operation moves an Actor between states that you can observe on the Actor record. + +- **Suspend**: Writes the Actor's durable data to a snapshot in snapshot storage, then releases the capacity that the Actor held on its Worker. +- **Resume**: Restores a suspended Actor onto a Worker from its latest snapshot. + +The following diagram traces an Actor through those operations, and shows what happens once a snapshot is [pinned by a tag](#checkpoints). +

+ +{{< reuse "kagent-docs/snippets/snapshot-cycle-diagram.md" >}} + +An Actor reports its position in this cycle through its state, which is one of `RESUMING`, `RUNNING`, `SUSPENDING`, `SUSPENDED`, `CRASHED`, or `DELETING`. Over the life of a Session, its Actor alternates between `RUNNING` and `SUSPENDED`. Only a suspended Actor can be deleted. + +> [!NOTE] +> Resume restores an Actor onto whichever Worker in the pool has room, which is not necessarily the Worker that the Actor ran on before. Because a suspended Actor occupies no Worker, a {{< gloss "WorkerPool" >}}WorkerPool{{< /gloss >}} can serve far more Sessions than it could if every Actor stayed resident. + +## What a snapshot captures + +An {{< gloss "ActorTemplate" >}}ActorTemplate{{< /gloss >}}'s snapshot configuration decides how much of an Actor a given snapshot holds. Two scopes exist. + +- **`Full`**: Captures process memory, the root filesystem changes layered on top of the container image, and any attached durable volumes. A `Full` snapshot holds everything that is needed to resume the Actor hot, with its in-memory state intact. +- **`Data`**: Captures only the contents of attached durable volumes. Process memory and the rest of the root filesystem are discarded, which makes the snapshot much cheaper to write and store. + +kagent sets the `Data` scope on every ActorTemplate that it generates, for both agents and standalone Sandboxes. Each Actor gets one durable volume named `data`, mounted at `/data`. The only snapshot setting that you configure is the storage location, in the `substrate.snapshotPolicy` section of the {{< gloss "Harness" >}}Harness{{< /gloss >}}. + +A suspend keeps the durable volume and discards process memory. A resume therefore starts new containers from the image and restores the saved durable directories. The agent runtimes store what a turn needs in the durable volume, including a pending approval or question, and continue the conversation after the restart. + +The ActorTemplate's `onCommit` setting selects what a suspend uploads to snapshot storage. + +### Durable volumes and resume + +A **DurableDir volume** is the per-Actor application data surface. Its contents are preserved by the `Data` scope, so they survive a suspend and resume cycle independently of process memory. How many volumes an ActorTemplate can declare depends on its sandbox class. A `microvm` template can declare several, because they are subdirectories of a single shared filesystem. A `gvisor` template is limited to one, until {{< gloss "gVisor" >}}gVisor{{< /gloss >}} accepts more than a single durable mount. + +## Golden and per-Actor snapshots + +Two kinds of snapshot serve different purposes, and both appear in a normal installation. + +- The **golden snapshot** is shared by every Actor of one ActorTemplate. Agent Substrate captures it once by booting the workload temporarily when the template is created. Every new Actor of this template starts from this snapshot. kagent waits for the golden snapshot before an {{< gloss "AgentTemplate" >}}AgentTemplate{{< /gloss >}} becomes ready, and the controller reports `waiting for the ActorTemplate golden snapshot` while the capture runs. +- The **last snapshot** belongs to one Actor. Agent Substrate writes it on every suspend and restores this same Actor from it on the next resume. It keeps the durable files that this one Session built up, so the Session continues from where it stopped. + +Agent Substrate writes every snapshot to object storage, so Actor state is durable and portable across the cluster. The supported backends are Google Cloud Storage and Amazon Simple Storage Service (S3), and the S3 client also reaches S3-compatible stores that accept a custom endpoint and path-style addressing. For the settings that select and configure each client, see [Configure snapshot storage]({{< link path="operations/tune-agent-substrate#configure-snapshot-storage" >}}). A {{< gloss "Harness" >}}Harness{{< /gloss >}} names the location for its Actors' snapshots in its `substrate.snapshotPolicy` section. + +```yaml +spec: + substrate: + workerPoolRef: + name: kagent-default + snapshotPolicy: + # The bucket that the Agent Substrate chart creates in its bundled object store. + # If your Substrate installation uses your own object storage, use that location instead. + location: s3://ate-snapshots/kagent/ +``` + +## Suspension between turns + +kagent does not wait for an Actor to go idle for a long stretch before suspending it. It suspends the Actor at every turn boundary, as soon as nothing in the Session is running. + +A turn reaches such a boundary when its task enters a terminal state, or when the task stops to wait on a person, which is the `INPUT_REQUIRED` and `AUTH_REQUIRED` case. At that point kagent suspends the Actor durably and records the exact snapshot that the suspend produced. + +The {{< gloss "Session" >}}Session{{< /gloss >}}'s own state does not change while this happens. It stays `READY` throughout, because suspension is a property of the runtime underneath the Session rather than of the Session itself. A caller that lists Sessions sees a ready Session whether or not an Actor is running for it. + +> [!NOTE] +> Creating a Session does not start an Actor running. The Actor is created as suspended, and the first message addressed to the Session resumes it. + +## Resuming on demand + +A suspended Session resumes when the next message arrives for it. The kagent controller hands that message to Agent Substrate, which restores the Actor from its last snapshot onto a Worker with room, and then delivers the message. No caller needs to know whether the Actor was running beforehand, and nothing in kagent asks for the resume explicitly. + +Agent Substrate resolves the Actor's address on every request, so the Actor stays reachable at one name no matter which Worker it is placed on. For more information, see [Sandboxing]({{< link path="about/substrate-runtime/sandboxing#how-traffic-reaches-a-sandboxed-actor" >}}). + +Because kagent suspends at every turn boundary, every message to a Session pays for one resume. An idle Session does not consume compute, and the next message waits for its Actor to come back. + + + +## Checkpoints + +A **checkpoint** marks one point in a conversation and keeps it. You can create a checkpoint when a Session reaches a state worth returning to, such as a long investigation that is only partway through. Starting a second Session from that point is known as a {{< gloss "Fork" >}}fork{{< /gloss >}}. The fork replays the conversation through the pinned turn and then continues on its own, with its own conversation ID and its own task IDs. A fork needs the pinned snapshot to keep durable data without process memory. Agent Substrate writes that kind of snapshot on every suspend, so any checkpoint that reaches `CHECKPOINT_STATE_READY` can be forked. To create a checkpoint and fork it, follow the [Agent Substrate example]({{< link path="examples/agent-substrate" >}}). + +Create a checkpoint only after a turn finishes. The request names the task that the turn ended on, and it fails when a newer turn has already started or while that turn's snapshot is still being written. + +Underneath, a checkpoint pins a snapshot that would otherwise be discarded. Agent Substrate treats the snapshot written on each suspend as transient and collects it once a newer one supersedes it. Creating a checkpoint attaches a {{< gloss "Tag" >}}Tag{{< /gloss >}} to the snapshot that the Session most recently suspended to, and Agent Substrate keeps any snapshot that a tag still names. Deleting the checkpoint removes the tag and releases the snapshot. diff --git a/docs-site/content/kagent/1.x/about/what-is-kagent.md b/docs-site/content/kagent/latest/about/what-is-kagent.md similarity index 100% rename from docs-site/content/kagent/1.x/about/what-is-kagent.md rename to docs-site/content/kagent/latest/about/what-is-kagent.md diff --git a/docs-site/content/kagent/1.x/agents/_index.md b/docs-site/content/kagent/latest/agents/_index.md similarity index 100% rename from docs-site/content/kagent/1.x/agents/_index.md rename to docs-site/content/kagent/latest/agents/_index.md diff --git a/docs-site/content/kagent/1.x/agents/agent-harness.md b/docs-site/content/kagent/latest/agents/agent-harness.md similarity index 100% rename from docs-site/content/kagent/1.x/agents/agent-harness.md rename to docs-site/content/kagent/latest/agents/agent-harness.md diff --git a/docs-site/content/kagent/1.x/agents/agent-memory.md b/docs-site/content/kagent/latest/agents/agent-memory.md similarity index 100% rename from docs-site/content/kagent/1.x/agents/agent-memory.md rename to docs-site/content/kagent/latest/agents/agent-memory.md diff --git a/docs-site/content/kagent/1.x/agents/bring-your-own-agent.md b/docs-site/content/kagent/latest/agents/bring-your-own-agent.md similarity index 100% rename from docs-site/content/kagent/1.x/agents/bring-your-own-agent.md rename to docs-site/content/kagent/latest/agents/bring-your-own-agent.md diff --git a/docs-site/content/kagent/1.x/agents/context-management.md b/docs-site/content/kagent/latest/agents/context-management.md similarity index 100% rename from docs-site/content/kagent/1.x/agents/context-management.md rename to docs-site/content/kagent/latest/agents/context-management.md diff --git a/docs-site/content/kagent/1.x/agents/declarative-agents.md b/docs-site/content/kagent/latest/agents/declarative-agents.md similarity index 100% rename from docs-site/content/kagent/1.x/agents/declarative-agents.md rename to docs-site/content/kagent/latest/agents/declarative-agents.md diff --git a/docs-site/content/kagent/1.x/agents/egress-policies.md b/docs-site/content/kagent/latest/agents/egress-policies.md similarity index 100% rename from docs-site/content/kagent/1.x/agents/egress-policies.md rename to docs-site/content/kagent/latest/agents/egress-policies.md diff --git a/docs-site/content/kagent/1.x/agents/human-in-the-loop.md b/docs-site/content/kagent/latest/agents/human-in-the-loop.md similarity index 100% rename from docs-site/content/kagent/1.x/agents/human-in-the-loop.md rename to docs-site/content/kagent/latest/agents/human-in-the-loop.md diff --git a/docs-site/content/kagent/1.x/agents/scheduled-runs.md b/docs-site/content/kagent/latest/agents/scheduled-runs.md similarity index 100% rename from docs-site/content/kagent/1.x/agents/scheduled-runs.md rename to docs-site/content/kagent/latest/agents/scheduled-runs.md diff --git a/docs-site/content/kagent/1.x/agents/share-a-session.md b/docs-site/content/kagent/latest/agents/share-a-session.md similarity index 100% rename from docs-site/content/kagent/1.x/agents/share-a-session.md rename to docs-site/content/kagent/latest/agents/share-a-session.md diff --git a/docs-site/content/kagent/1.x/agents/structured-output.md b/docs-site/content/kagent/latest/agents/structured-output.md similarity index 100% rename from docs-site/content/kagent/1.x/agents/structured-output.md rename to docs-site/content/kagent/latest/agents/structured-output.md diff --git a/docs-site/content/kagent/1.x/agents/system-prompts.md b/docs-site/content/kagent/latest/agents/system-prompts.md similarity index 100% rename from docs-site/content/kagent/1.x/agents/system-prompts.md rename to docs-site/content/kagent/latest/agents/system-prompts.md diff --git a/docs-site/content/kagent/1.x/examples/_index.md b/docs-site/content/kagent/latest/examples/_index.md similarity index 100% rename from docs-site/content/kagent/1.x/examples/_index.md rename to docs-site/content/kagent/latest/examples/_index.md diff --git a/docs-site/content/kagent/1.x/examples/a2a-agents.md b/docs-site/content/kagent/latest/examples/a2a-agents.md similarity index 100% rename from docs-site/content/kagent/1.x/examples/a2a-agents.md rename to docs-site/content/kagent/latest/examples/a2a-agents.md diff --git a/docs-site/content/kagent/1.x/examples/a2a-byo.md b/docs-site/content/kagent/latest/examples/a2a-byo.md similarity index 100% rename from docs-site/content/kagent/1.x/examples/a2a-byo.md rename to docs-site/content/kagent/latest/examples/a2a-byo.md diff --git a/docs-site/content/kagent/1.x/examples/agent-delegation.md b/docs-site/content/kagent/latest/examples/agent-delegation.md similarity index 100% rename from docs-site/content/kagent/1.x/examples/agent-delegation.md rename to docs-site/content/kagent/latest/examples/agent-delegation.md diff --git a/docs-site/content/kagent/1.x/examples/agent-substrate.md b/docs-site/content/kagent/latest/examples/agent-substrate.md similarity index 100% rename from docs-site/content/kagent/1.x/examples/agent-substrate.md rename to docs-site/content/kagent/latest/examples/agent-substrate.md diff --git a/docs-site/content/kagent/1.x/examples/agentgateway.md b/docs-site/content/kagent/latest/examples/agentgateway.md similarity index 100% rename from docs-site/content/kagent/1.x/examples/agentgateway.md rename to docs-site/content/kagent/latest/examples/agentgateway.md diff --git a/docs-site/content/kagent/1.x/examples/agents-via-mcp.md b/docs-site/content/kagent/latest/examples/agents-via-mcp.md similarity index 100% rename from docs-site/content/kagent/1.x/examples/agents-via-mcp.md rename to docs-site/content/kagent/latest/examples/agents-via-mcp.md diff --git a/docs-site/content/kagent/1.x/examples/documentation-agent.md b/docs-site/content/kagent/latest/examples/documentation-agent.md similarity index 100% rename from docs-site/content/kagent/1.x/examples/documentation-agent.md rename to docs-site/content/kagent/latest/examples/documentation-agent.md diff --git a/docs-site/content/kagent/1.x/examples/skills.md b/docs-site/content/kagent/latest/examples/skills.md similarity index 100% rename from docs-site/content/kagent/1.x/examples/skills.md rename to docs-site/content/kagent/latest/examples/skills.md diff --git a/docs-site/content/kagent/1.x/get-started/_index.md b/docs-site/content/kagent/latest/get-started/_index.md similarity index 100% rename from docs-site/content/kagent/1.x/get-started/_index.md rename to docs-site/content/kagent/latest/get-started/_index.md diff --git a/docs-site/content/kagent/1.x/get-started/your-first-agent.md b/docs-site/content/kagent/latest/get-started/your-first-agent.md similarity index 100% rename from docs-site/content/kagent/1.x/get-started/your-first-agent.md rename to docs-site/content/kagent/latest/get-started/your-first-agent.md diff --git a/docs-site/content/kagent/1.x/get-started/your-first-mcp-tool.md b/docs-site/content/kagent/latest/get-started/your-first-mcp-tool.md similarity index 100% rename from docs-site/content/kagent/1.x/get-started/your-first-mcp-tool.md rename to docs-site/content/kagent/latest/get-started/your-first-mcp-tool.md diff --git a/docs-site/content/kagent/1.x/not-in-version.md b/docs-site/content/kagent/latest/not-in-version.md similarity index 100% rename from docs-site/content/kagent/1.x/not-in-version.md rename to docs-site/content/kagent/latest/not-in-version.md diff --git a/docs-site/content/kagent/1.x/observability/_index.md b/docs-site/content/kagent/latest/observability/_index.md similarity index 100% rename from docs-site/content/kagent/1.x/observability/_index.md rename to docs-site/content/kagent/latest/observability/_index.md diff --git a/docs-site/content/kagent/1.x/observability/audit-prompts.md b/docs-site/content/kagent/latest/observability/audit-prompts.md similarity index 100% rename from docs-site/content/kagent/1.x/observability/audit-prompts.md rename to docs-site/content/kagent/latest/observability/audit-prompts.md diff --git a/docs-site/content/kagent/1.x/observability/lightweight-otel-stack.md b/docs-site/content/kagent/latest/observability/lightweight-otel-stack.md similarity index 100% rename from docs-site/content/kagent/1.x/observability/lightweight-otel-stack.md rename to docs-site/content/kagent/latest/observability/lightweight-otel-stack.md diff --git a/docs-site/content/kagent/1.x/observability/metrics.md b/docs-site/content/kagent/latest/observability/metrics.md similarity index 100% rename from docs-site/content/kagent/1.x/observability/metrics.md rename to docs-site/content/kagent/latest/observability/metrics.md diff --git a/docs-site/content/kagent/1.x/observability/otel-stack.md b/docs-site/content/kagent/latest/observability/otel-stack.md similarity index 100% rename from docs-site/content/kagent/1.x/observability/otel-stack.md rename to docs-site/content/kagent/latest/observability/otel-stack.md diff --git a/docs-site/content/kagent/1.x/observability/substrate-telemetry.md b/docs-site/content/kagent/latest/observability/substrate-telemetry.md similarity index 100% rename from docs-site/content/kagent/1.x/observability/substrate-telemetry.md rename to docs-site/content/kagent/latest/observability/substrate-telemetry.md diff --git a/docs-site/content/kagent/1.x/observability/tracing.md b/docs-site/content/kagent/latest/observability/tracing.md similarity index 100% rename from docs-site/content/kagent/1.x/observability/tracing.md rename to docs-site/content/kagent/latest/observability/tracing.md diff --git a/docs-site/content/kagent/1.x/operations/_index.md b/docs-site/content/kagent/latest/operations/_index.md similarity index 100% rename from docs-site/content/kagent/1.x/operations/_index.md rename to docs-site/content/kagent/latest/operations/_index.md diff --git a/docs-site/content/kagent/1.x/operations/debug.md b/docs-site/content/kagent/latest/operations/debug.md similarity index 100% rename from docs-site/content/kagent/1.x/operations/debug.md rename to docs-site/content/kagent/latest/operations/debug.md diff --git a/docs-site/content/kagent/1.x/operations/export-to-helm.md b/docs-site/content/kagent/latest/operations/export-to-helm.md similarity index 100% rename from docs-site/content/kagent/1.x/operations/export-to-helm.md rename to docs-site/content/kagent/latest/operations/export-to-helm.md diff --git a/docs-site/content/kagent/1.x/operations/multiple-namespaces.md b/docs-site/content/kagent/latest/operations/multiple-namespaces.md similarity index 100% rename from docs-site/content/kagent/1.x/operations/multiple-namespaces.md rename to docs-site/content/kagent/latest/operations/multiple-namespaces.md diff --git a/docs-site/content/kagent/1.x/operations/operational-considerations.md b/docs-site/content/kagent/latest/operations/operational-considerations.md similarity index 100% rename from docs-site/content/kagent/1.x/operations/operational-considerations.md rename to docs-site/content/kagent/latest/operations/operational-considerations.md diff --git a/docs-site/content/kagent/1.x/operations/tune-agent-substrate.md b/docs-site/content/kagent/latest/operations/tune-agent-substrate.md similarity index 100% rename from docs-site/content/kagent/1.x/operations/tune-agent-substrate.md rename to docs-site/content/kagent/latest/operations/tune-agent-substrate.md diff --git a/docs-site/content/kagent/1.x/operations/uninstall.md b/docs-site/content/kagent/latest/operations/uninstall.md similarity index 100% rename from docs-site/content/kagent/1.x/operations/uninstall.md rename to docs-site/content/kagent/latest/operations/uninstall.md diff --git a/docs-site/content/kagent/1.x/operations/upgrade-from-0x.md b/docs-site/content/kagent/latest/operations/upgrade-from-0x.md similarity index 100% rename from docs-site/content/kagent/1.x/operations/upgrade-from-0x.md rename to docs-site/content/kagent/latest/operations/upgrade-from-0x.md diff --git a/docs-site/content/kagent/1.x/operations/upgrade.md b/docs-site/content/kagent/latest/operations/upgrade.md similarity index 100% rename from docs-site/content/kagent/1.x/operations/upgrade.md rename to docs-site/content/kagent/latest/operations/upgrade.md diff --git a/docs-site/content/kagent/1.x/reference/_index.md b/docs-site/content/kagent/latest/reference/_index.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/_index.md rename to docs-site/content/kagent/latest/reference/_index.md diff --git a/docs-site/content/kagent/1.x/reference/a2a-metadata.md b/docs-site/content/kagent/latest/reference/a2a-metadata.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/a2a-metadata.md rename to docs-site/content/kagent/latest/reference/a2a-metadata.md diff --git a/docs-site/content/kagent/1.x/reference/api-ref.md b/docs-site/content/kagent/latest/reference/api-ref.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/api-ref.md rename to docs-site/content/kagent/latest/reference/api-ref.md diff --git a/docs-site/content/kagent/1.x/reference/cli/_index.md b/docs-site/content/kagent/latest/reference/cli/_index.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/cli/_index.md rename to docs-site/content/kagent/latest/reference/cli/_index.md diff --git a/docs-site/content/kagent/1.x/reference/cli/kagent-agent-get.md b/docs-site/content/kagent/latest/reference/cli/kagent-agent-get.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/cli/kagent-agent-get.md rename to docs-site/content/kagent/latest/reference/cli/kagent-agent-get.md diff --git a/docs-site/content/kagent/1.x/reference/cli/kagent-agent-invoke.md b/docs-site/content/kagent/latest/reference/cli/kagent-agent-invoke.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/cli/kagent-agent-invoke.md rename to docs-site/content/kagent/latest/reference/cli/kagent-agent-invoke.md diff --git a/docs-site/content/kagent/1.x/reference/cli/kagent-agent-list.md b/docs-site/content/kagent/latest/reference/cli/kagent-agent-list.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/cli/kagent-agent-list.md rename to docs-site/content/kagent/latest/reference/cli/kagent-agent-list.md diff --git a/docs-site/content/kagent/1.x/reference/cli/kagent-agent-session-create.md b/docs-site/content/kagent/latest/reference/cli/kagent-agent-session-create.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/cli/kagent-agent-session-create.md rename to docs-site/content/kagent/latest/reference/cli/kagent-agent-session-create.md diff --git a/docs-site/content/kagent/1.x/reference/cli/kagent-agent-session-delete.md b/docs-site/content/kagent/latest/reference/cli/kagent-agent-session-delete.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/cli/kagent-agent-session-delete.md rename to docs-site/content/kagent/latest/reference/cli/kagent-agent-session-delete.md diff --git a/docs-site/content/kagent/1.x/reference/cli/kagent-agent-session-get.md b/docs-site/content/kagent/latest/reference/cli/kagent-agent-session-get.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/cli/kagent-agent-session-get.md rename to docs-site/content/kagent/latest/reference/cli/kagent-agent-session-get.md diff --git a/docs-site/content/kagent/1.x/reference/cli/kagent-agent-session-list.md b/docs-site/content/kagent/latest/reference/cli/kagent-agent-session-list.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/cli/kagent-agent-session-list.md rename to docs-site/content/kagent/latest/reference/cli/kagent-agent-session-list.md diff --git a/docs-site/content/kagent/1.x/reference/cli/kagent-agent-session-push-create.md b/docs-site/content/kagent/latest/reference/cli/kagent-agent-session-push-create.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/cli/kagent-agent-session-push-create.md rename to docs-site/content/kagent/latest/reference/cli/kagent-agent-session-push-create.md diff --git a/docs-site/content/kagent/1.x/reference/cli/kagent-agent-session-push-delete.md b/docs-site/content/kagent/latest/reference/cli/kagent-agent-session-push-delete.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/cli/kagent-agent-session-push-delete.md rename to docs-site/content/kagent/latest/reference/cli/kagent-agent-session-push-delete.md diff --git a/docs-site/content/kagent/1.x/reference/cli/kagent-agent-session-push-get.md b/docs-site/content/kagent/latest/reference/cli/kagent-agent-session-push-get.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/cli/kagent-agent-session-push-get.md rename to docs-site/content/kagent/latest/reference/cli/kagent-agent-session-push-get.md diff --git a/docs-site/content/kagent/1.x/reference/cli/kagent-agent-session-push-list.md b/docs-site/content/kagent/latest/reference/cli/kagent-agent-session-push-list.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/cli/kagent-agent-session-push-list.md rename to docs-site/content/kagent/latest/reference/cli/kagent-agent-session-push-list.md diff --git a/docs-site/content/kagent/1.x/reference/cli/kagent-agent-session-push.md b/docs-site/content/kagent/latest/reference/cli/kagent-agent-session-push.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/cli/kagent-agent-session-push.md rename to docs-site/content/kagent/latest/reference/cli/kagent-agent-session-push.md diff --git a/docs-site/content/kagent/1.x/reference/cli/kagent-agent-session.md b/docs-site/content/kagent/latest/reference/cli/kagent-agent-session.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/cli/kagent-agent-session.md rename to docs-site/content/kagent/latest/reference/cli/kagent-agent-session.md diff --git a/docs-site/content/kagent/1.x/reference/cli/kagent-agent-template-get.md b/docs-site/content/kagent/latest/reference/cli/kagent-agent-template-get.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/cli/kagent-agent-template-get.md rename to docs-site/content/kagent/latest/reference/cli/kagent-agent-template-get.md diff --git a/docs-site/content/kagent/1.x/reference/cli/kagent-agent-template-list.md b/docs-site/content/kagent/latest/reference/cli/kagent-agent-template-list.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/cli/kagent-agent-template-list.md rename to docs-site/content/kagent/latest/reference/cli/kagent-agent-template-list.md diff --git a/docs-site/content/kagent/1.x/reference/cli/kagent-agent-template.md b/docs-site/content/kagent/latest/reference/cli/kagent-agent-template.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/cli/kagent-agent-template.md rename to docs-site/content/kagent/latest/reference/cli/kagent-agent-template.md diff --git a/docs-site/content/kagent/1.x/reference/cli/kagent-agent.md b/docs-site/content/kagent/latest/reference/cli/kagent-agent.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/cli/kagent-agent.md rename to docs-site/content/kagent/latest/reference/cli/kagent-agent.md diff --git a/docs-site/content/kagent/1.x/reference/cli/kagent-apply.md b/docs-site/content/kagent/latest/reference/cli/kagent-apply.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/cli/kagent-apply.md rename to docs-site/content/kagent/latest/reference/cli/kagent-apply.md diff --git a/docs-site/content/kagent/1.x/reference/cli/kagent-bug-report.md b/docs-site/content/kagent/latest/reference/cli/kagent-bug-report.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/cli/kagent-bug-report.md rename to docs-site/content/kagent/latest/reference/cli/kagent-bug-report.md diff --git a/docs-site/content/kagent/1.x/reference/cli/kagent-completion-bash.md b/docs-site/content/kagent/latest/reference/cli/kagent-completion-bash.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/cli/kagent-completion-bash.md rename to docs-site/content/kagent/latest/reference/cli/kagent-completion-bash.md diff --git a/docs-site/content/kagent/1.x/reference/cli/kagent-completion-fish.md b/docs-site/content/kagent/latest/reference/cli/kagent-completion-fish.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/cli/kagent-completion-fish.md rename to docs-site/content/kagent/latest/reference/cli/kagent-completion-fish.md diff --git a/docs-site/content/kagent/1.x/reference/cli/kagent-completion-powershell.md b/docs-site/content/kagent/latest/reference/cli/kagent-completion-powershell.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/cli/kagent-completion-powershell.md rename to docs-site/content/kagent/latest/reference/cli/kagent-completion-powershell.md diff --git a/docs-site/content/kagent/1.x/reference/cli/kagent-completion-zsh.md b/docs-site/content/kagent/latest/reference/cli/kagent-completion-zsh.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/cli/kagent-completion-zsh.md rename to docs-site/content/kagent/latest/reference/cli/kagent-completion-zsh.md diff --git a/docs-site/content/kagent/1.x/reference/cli/kagent-completion.md b/docs-site/content/kagent/latest/reference/cli/kagent-completion.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/cli/kagent-completion.md rename to docs-site/content/kagent/latest/reference/cli/kagent-completion.md diff --git a/docs-site/content/kagent/1.x/reference/cli/kagent-config-schema.md b/docs-site/content/kagent/latest/reference/cli/kagent-config-schema.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/cli/kagent-config-schema.md rename to docs-site/content/kagent/latest/reference/cli/kagent-config-schema.md diff --git a/docs-site/content/kagent/1.x/reference/cli/kagent-config-validate.md b/docs-site/content/kagent/latest/reference/cli/kagent-config-validate.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/cli/kagent-config-validate.md rename to docs-site/content/kagent/latest/reference/cli/kagent-config-validate.md diff --git a/docs-site/content/kagent/1.x/reference/cli/kagent-config-view.md b/docs-site/content/kagent/latest/reference/cli/kagent-config-view.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/cli/kagent-config-view.md rename to docs-site/content/kagent/latest/reference/cli/kagent-config-view.md diff --git a/docs-site/content/kagent/1.x/reference/cli/kagent-config.md b/docs-site/content/kagent/latest/reference/cli/kagent-config.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/cli/kagent-config.md rename to docs-site/content/kagent/latest/reference/cli/kagent-config.md diff --git a/docs-site/content/kagent/1.x/reference/cli/kagent-dashboard.md b/docs-site/content/kagent/latest/reference/cli/kagent-dashboard.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/cli/kagent-dashboard.md rename to docs-site/content/kagent/latest/reference/cli/kagent-dashboard.md diff --git a/docs-site/content/kagent/1.x/reference/cli/kagent-db-migrate-down.md b/docs-site/content/kagent/latest/reference/cli/kagent-db-migrate-down.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/cli/kagent-db-migrate-down.md rename to docs-site/content/kagent/latest/reference/cli/kagent-db-migrate-down.md diff --git a/docs-site/content/kagent/1.x/reference/cli/kagent-db-migrate-goto.md b/docs-site/content/kagent/latest/reference/cli/kagent-db-migrate-goto.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/cli/kagent-db-migrate-goto.md rename to docs-site/content/kagent/latest/reference/cli/kagent-db-migrate-goto.md diff --git a/docs-site/content/kagent/1.x/reference/cli/kagent-db-migrate-status.md b/docs-site/content/kagent/latest/reference/cli/kagent-db-migrate-status.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/cli/kagent-db-migrate-status.md rename to docs-site/content/kagent/latest/reference/cli/kagent-db-migrate-status.md diff --git a/docs-site/content/kagent/1.x/reference/cli/kagent-db-migrate-up.md b/docs-site/content/kagent/latest/reference/cli/kagent-db-migrate-up.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/cli/kagent-db-migrate-up.md rename to docs-site/content/kagent/latest/reference/cli/kagent-db-migrate-up.md diff --git a/docs-site/content/kagent/1.x/reference/cli/kagent-db-migrate-version.md b/docs-site/content/kagent/latest/reference/cli/kagent-db-migrate-version.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/cli/kagent-db-migrate-version.md rename to docs-site/content/kagent/latest/reference/cli/kagent-db-migrate-version.md diff --git a/docs-site/content/kagent/1.x/reference/cli/kagent-db-migrate.md b/docs-site/content/kagent/latest/reference/cli/kagent-db-migrate.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/cli/kagent-db-migrate.md rename to docs-site/content/kagent/latest/reference/cli/kagent-db-migrate.md diff --git a/docs-site/content/kagent/1.x/reference/cli/kagent-db.md b/docs-site/content/kagent/latest/reference/cli/kagent-db.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/cli/kagent-db.md rename to docs-site/content/kagent/latest/reference/cli/kagent-db.md diff --git a/docs-site/content/kagent/1.x/reference/cli/kagent-install.md b/docs-site/content/kagent/latest/reference/cli/kagent-install.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/cli/kagent-install.md rename to docs-site/content/kagent/latest/reference/cli/kagent-install.md diff --git a/docs-site/content/kagent/1.x/reference/cli/kagent-sandbox-create.md b/docs-site/content/kagent/latest/reference/cli/kagent-sandbox-create.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/cli/kagent-sandbox-create.md rename to docs-site/content/kagent/latest/reference/cli/kagent-sandbox-create.md diff --git a/docs-site/content/kagent/1.x/reference/cli/kagent-sandbox-delete.md b/docs-site/content/kagent/latest/reference/cli/kagent-sandbox-delete.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/cli/kagent-sandbox-delete.md rename to docs-site/content/kagent/latest/reference/cli/kagent-sandbox-delete.md diff --git a/docs-site/content/kagent/1.x/reference/cli/kagent-sandbox-download.md b/docs-site/content/kagent/latest/reference/cli/kagent-sandbox-download.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/cli/kagent-sandbox-download.md rename to docs-site/content/kagent/latest/reference/cli/kagent-sandbox-download.md diff --git a/docs-site/content/kagent/1.x/reference/cli/kagent-sandbox-exec.md b/docs-site/content/kagent/latest/reference/cli/kagent-sandbox-exec.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/cli/kagent-sandbox-exec.md rename to docs-site/content/kagent/latest/reference/cli/kagent-sandbox-exec.md diff --git a/docs-site/content/kagent/1.x/reference/cli/kagent-sandbox-get.md b/docs-site/content/kagent/latest/reference/cli/kagent-sandbox-get.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/cli/kagent-sandbox-get.md rename to docs-site/content/kagent/latest/reference/cli/kagent-sandbox-get.md diff --git a/docs-site/content/kagent/1.x/reference/cli/kagent-sandbox-kill.md b/docs-site/content/kagent/latest/reference/cli/kagent-sandbox-kill.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/cli/kagent-sandbox-kill.md rename to docs-site/content/kagent/latest/reference/cli/kagent-sandbox-kill.md diff --git a/docs-site/content/kagent/1.x/reference/cli/kagent-sandbox-list.md b/docs-site/content/kagent/latest/reference/cli/kagent-sandbox-list.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/cli/kagent-sandbox-list.md rename to docs-site/content/kagent/latest/reference/cli/kagent-sandbox-list.md diff --git a/docs-site/content/kagent/1.x/reference/cli/kagent-sandbox-process.md b/docs-site/content/kagent/latest/reference/cli/kagent-sandbox-process.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/cli/kagent-sandbox-process.md rename to docs-site/content/kagent/latest/reference/cli/kagent-sandbox-process.md diff --git a/docs-site/content/kagent/1.x/reference/cli/kagent-sandbox-resume.md b/docs-site/content/kagent/latest/reference/cli/kagent-sandbox-resume.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/cli/kagent-sandbox-resume.md rename to docs-site/content/kagent/latest/reference/cli/kagent-sandbox-resume.md diff --git a/docs-site/content/kagent/1.x/reference/cli/kagent-sandbox-suspend.md b/docs-site/content/kagent/latest/reference/cli/kagent-sandbox-suspend.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/cli/kagent-sandbox-suspend.md rename to docs-site/content/kagent/latest/reference/cli/kagent-sandbox-suspend.md diff --git a/docs-site/content/kagent/1.x/reference/cli/kagent-sandbox-templates.md b/docs-site/content/kagent/latest/reference/cli/kagent-sandbox-templates.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/cli/kagent-sandbox-templates.md rename to docs-site/content/kagent/latest/reference/cli/kagent-sandbox-templates.md diff --git a/docs-site/content/kagent/1.x/reference/cli/kagent-sandbox-upload.md b/docs-site/content/kagent/latest/reference/cli/kagent-sandbox-upload.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/cli/kagent-sandbox-upload.md rename to docs-site/content/kagent/latest/reference/cli/kagent-sandbox-upload.md diff --git a/docs-site/content/kagent/1.x/reference/cli/kagent-sandbox-wait.md b/docs-site/content/kagent/latest/reference/cli/kagent-sandbox-wait.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/cli/kagent-sandbox-wait.md rename to docs-site/content/kagent/latest/reference/cli/kagent-sandbox-wait.md diff --git a/docs-site/content/kagent/1.x/reference/cli/kagent-sandbox.md b/docs-site/content/kagent/latest/reference/cli/kagent-sandbox.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/cli/kagent-sandbox.md rename to docs-site/content/kagent/latest/reference/cli/kagent-sandbox.md diff --git a/docs-site/content/kagent/1.x/reference/cli/kagent-uninstall.md b/docs-site/content/kagent/latest/reference/cli/kagent-uninstall.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/cli/kagent-uninstall.md rename to docs-site/content/kagent/latest/reference/cli/kagent-uninstall.md diff --git a/docs-site/content/kagent/1.x/reference/cli/kagent-version.md b/docs-site/content/kagent/latest/reference/cli/kagent-version.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/cli/kagent-version.md rename to docs-site/content/kagent/latest/reference/cli/kagent-version.md diff --git a/docs-site/content/kagent/1.x/reference/community.md b/docs-site/content/kagent/latest/reference/community.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/community.md rename to docs-site/content/kagent/latest/reference/community.md diff --git a/docs-site/content/kagent/1.x/reference/faq.md b/docs-site/content/kagent/latest/reference/faq.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/faq.md rename to docs-site/content/kagent/latest/reference/faq.md diff --git a/docs-site/content/kagent/1.x/reference/glossary.md b/docs-site/content/kagent/latest/reference/glossary.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/glossary.md rename to docs-site/content/kagent/latest/reference/glossary.md diff --git a/docs-site/content/kagent/latest/reference/helm.md b/docs-site/content/kagent/latest/reference/helm.md new file mode 100644 index 00000000..723c37e2 --- /dev/null +++ b/docs-site/content/kagent/latest/reference/helm.md @@ -0,0 +1,281 @@ +--- +title: Helm reference +description: Look up the configurable values in the kagent Helm chart, including their defaults and types. +weight: 20 +author: kagent.dev +aliases: + - /kagent/latest/reference/env-vars/ +--- + +A Helm chart for kagent, built with Google ADK + +## Requirements + +| Repository | Name | Version | +|------------|------|---------| +| https://oauth2-proxy.github.io/manifests | oauth2-proxy | ~10.7.0 | +| oci://ghcr.io/kagent-dev/tools/helm | kagent-tools | 0.3.0 | + +## Values + +| Key | Type | Default | Description | +|-----|------|---------|-------------| +| annotations | object | `{}` | Additional annotations to add to all Kubernetes deployment resources | +| config.database.owner.role | string | `"kagent_owner"` | | +| config.database.readWrite.role | string | `"kagent_readwrite"` | | +| config.database.schema | string | `"kagent"` | | +| config.database.skipMigrations | bool | `false` | | +| config.database.vectorEnabled | bool | `false` | | +| config.database.vectorSchema | string | `"extensions"` | | +| config.harnessImages | object | `{}` | | +| config.kubernetes | object | `{}` | | +| config.logging.level | string | `"info"` | | +| config.metrics.bindAddress | string | `":8443"` | | +| config.metrics.secure | bool | `true` | | +| config.push.allowHttp | bool | `false` | | +| config.push.allowPrivateNetworks | bool | `false` | | +| config.push.maxConcurrency | int | `10` | | +| config.push.recoveryInterval | string | `"60s"` | | +| config.push.sendTimeout | string | `"5s"` | | +| config.runtimeDefaults.builtinTools.bashVenvPath | string | `""` | | +| config.runtimeDefaults.builtinTools.disableAskUser | bool | `false` | | +| config.runtimeDefaults.builtinTools.disableBash | bool | `false` | | +| config.runtimeDefaults.builtinTools.disableFileSearch | bool | `false` | | +| config.runtimeDefaults.cpu | string | `"1"` | | +| config.runtimeDefaults.logging.level | string | `"info"` | | +| config.runtimeDefaults.memory | string | `"1Gi"` | | +| config.runtimeDefaults.telemetry | object | `{}` | | +| config.sandboxes.cpu | string | `"1"` | | +| config.sandboxes.defaultTtl | string | `"3600s"` | | +| config.sandboxes.guestImage | string | `""` | | +| config.sandboxes.maxTtl | string | `"86400s"` | | +| config.sandboxes.memory | string | `"1Gi"` | | +| config.server.auth.mode | string | `"insecure"` | | +| config.server.grpcReflection | bool | `false` | | +| config.sessions.idleTtl | string | `"604800s"` | | +| config.sessions.shareMaxTtl | string | `"0s"` | | +| config.substrate.actorJwtIssuer | string | `"https://idp.ate-system.svc"` | | +| config.substrate.apiEndpoint | string | `"dns:///api.ate-system.svc:443"` | | +| config.substrate.routerUrl | string | `"https://atenet-router.ate-system.svc:443"` | | +| config.telemetry | object | `{}` | | +| controller.affinity | object | `{}` | [Affinity](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity) rules for the controller pod. | +| controller.annotations | object | `{}` | Additional annotations to add to the controller Deployment metadata | +| controller.env | list | `[]` | | +| controller.envFrom | list | `[]` | | +| controller.image.pullPolicy | string | `""` | | +| controller.image.registry | string | `""` | | +| controller.image.repository | string | `"kagent-dev/kagent/controller"` | | +| controller.image.tag | string | `""` | | +| controller.metrics.enabled | bool | `false` | | +| controller.metrics.service.port | int | `8443` | | +| controller.metrics.service.type | string | `"ClusterIP"` | | +| controller.metrics.serviceMonitor | object | disabled | Prometheus Operator `ServiceMonitor` for the metrics `Service`. Requires `controller.metrics.enabled` and the `monitoring.coreos.com/v1` CRDs; the chart only renders it when the target cluster serves that API, so enabling it on a cluster without the Prometheus Operator is a no-op rather than a failed install (`helm template` needs `--api-versions monitoring.coreos.com/v1`). The endpoint follows `secureServing`: port name, scheme, bearer token and TLS settings are all derived from it, so flipping `secureServing` alone keeps the scrape working. With `secureServing` enabled the scrape is also authorized only once `-metrics-reader` is bound to the Prometheus ServiceAccount; set `prometheusServiceAccount` below and the chart renders that binding for you. | +| controller.metrics.serviceMonitor.annotations | object | `{}` | Annotations for the `ServiceMonitor`. | +| controller.metrics.serviceMonitor.bearerTokenFile | string | `"/var/run/secrets/kubernetes.io/serviceaccount/token"` | Token presented to the authenticated metrics endpoint. Only used when `secureServing` is enabled; set to `""` to omit it. | +| controller.metrics.serviceMonitor.honorLabels | bool | `false` | Keep the scraped labels when they collide with server-side ones. | +| controller.metrics.serviceMonitor.interval | string | `""` | Scrape interval. Prometheus' global default when empty. | +| controller.metrics.serviceMonitor.labels | object | `{}` | Extra labels for the `ServiceMonitor` (merged with the chart labels). Set whatever label your Prometheus `serviceMonitorSelector` matches on. | +| controller.metrics.serviceMonitor.metricRelabelings | list | `[]` | `metricRelabelings` applied to the scraped samples. | +| controller.metrics.serviceMonitor.namespace | string | the release namespace | Namespace to create the `ServiceMonitor` in. The scrape target stays the release namespace either way. | +| controller.metrics.serviceMonitor.prometheusServiceAccount | object | `{"name":"","namespace":""}` | ServiceAccount Prometheus scrapes with. When `name` is set and `secureServing` is enabled, the chart binds `-metrics-reader` to it with a `ClusterRoleBinding` so the scrape does not fail with 403. Leave `name` empty to manage the binding yourself. | +| controller.metrics.serviceMonitor.prometheusServiceAccount.namespace | string | `serviceMonitor.namespace`, then the release namespace | Namespace of the Prometheus ServiceAccount. | +| controller.metrics.serviceMonitor.relabelings | list | `[]` | `relabelings` applied to the scrape targets. | +| controller.metrics.serviceMonitor.scrapeTimeout | string | `""` | Scrape timeout. Prometheus' global default when empty. | +| controller.metrics.serviceMonitor.tlsConfig | object | insecureSkipVerify: true | `tlsConfig` for the scrape. Only used when `secureServing` is enabled, where the controller serves a self-signed certificate. | +| controller.nodeSelector | object | `{}` | Node labels to match for `Pod` [scheduling](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/). | +| controller.pdb | object | `{"annotations":{},"enabled":false,"labels":{},"maxUnavailable":1,"minAvailable":null,"unhealthyPodEvictionPolicy":""}` | [PodDisruptionBudget](https://kubernetes.io/docs/tasks/run-application/configure-pdb/) for the controller pods. Disabled by default: `controller.replicas` is 1, and a `minAvailable: 1` budget on a single-replica Deployment blocks every voluntary eviction, so node drains and cluster upgrades hang indefinitely. Raise `controller.replicas` before switching to `minAvailable`. | +| controller.pdb.annotations | object | `{}` | Annotations for the controller PodDisruptionBudget. | +| controller.pdb.enabled | bool | `false` | Set to true to create the PodDisruptionBudget. | +| controller.pdb.labels | object | `{}` | Additional labels for the controller PodDisruptionBudget. | +| controller.pdb.maxUnavailable | int | `1` | Maximum number of pods that may be unavailable. Int or percentage string (e.g. `1` or `"50%"`). Mutually exclusive with `minAvailable`. | +| controller.pdb.minAvailable | string | unset (`maxUnavailable` is used instead) | Minimum number of pods that must remain available. Int or percentage string (e.g. `1` or `"50%"`). Mutually exclusive with `maxUnavailable`. | +| controller.pdb.unhealthyPodEvictionPolicy | string | "" (defer to the Kubernetes default) | `spec.unhealthyPodEvictionPolicy`, one of `IfHealthyBudget` (the Kubernetes default) or `AlwaysAllow`. `AlwaysAllow` lets unhealthy pods be evicted even when the budget is exhausted, which avoids drains wedging on a crash-looping pod. Requires Kubernetes >= 1.27. Omitted from the manifest when empty. | +| controller.podAnnotations | object | `{}` | | +| controller.podLabels | object | `{}` | Additional labels for the controller pod template, merged over the global `podLabels` (per-key; component keys win). Selector labels can never be overridden. | +| controller.push.signing.secretName | string | `""` | | +| controller.readinessProbe | object | httpGet /health on port http, periodSeconds=30 | Custom readiness probe for the controller container. Setting a value replaces the default probe entirely — include a handler (httpGet / exec / tcpSocket / grpc) when overriding. | +| controller.replicas | int | `1` | | +| controller.resources.limits.cpu | int | `2` | | +| controller.resources.limits.memory | string | `"512Mi"` | | +| controller.resources.requests.cpu | string | `"100m"` | | +| controller.resources.requests.memory | string | `"128Mi"` | | +| controller.service.annotations | object | `{}` | | +| controller.service.ports.port | int | `8083` | | +| controller.service.ports.targetPort | int | `8083` | | +| controller.service.type | string | `"ClusterIP"` | | +| controller.serviceAccount | object | `{"annotations":{},"create":true,"name":""}` | ServiceAccount settings for the controller pod | +| controller.serviceAccount.annotations | object | {} (no extra annotations) | Annotations to add to the controller ServiceAccount. Useful for GCP Workload Identity, AWS IRSA, or Azure Workload Identity. | +| controller.serviceAccount.create | bool | `true` | Create the controller ServiceAccount. Set to false to use one managed outside the chart. | +| controller.serviceAccount.name | string | `-controller` | Name of the controller ServiceAccount. | +| controller.startupProbe | object | httpGet /health on port http, periodSeconds=15, initialDelaySeconds=15 | Custom startup probe for the controller container. Setting a value replaces the default probe entirely — include a handler (httpGet / exec / tcpSocket / grpc) when overriding. | +| controller.substrateClientCertificate.enabled | bool | `false` | | +| controller.tolerations | list | `[]` | Node taints which will be tolerated for `Pod` [scheduling](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/). | +| controller.topologySpreadConstraints | list | `[]` | [Topology spread constraints](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#pod-topology-spread-constraints) for the controller pod. | +| controller.volumeMounts | list | `[]` | | +| controller.volumes | list | `[]` | | +| database.postgres.clientCertificate | object | `{"enabled":false}` | Project the bundled PostgreSQL client certificates for kagent_readwrite_user and, unless config.database.skipMigrations is true, kagent_owner_user. The connection strings must reference the files under /run/postgres.podcert.ate.dev. | +| database.postgres.ownerConnectionStringSecretRef | object | `{"key":"ownerConnectionString","name":"kagent-postgres-owner"}` | Secret holding the owner login's connection string. The owner creates the schema and runs migrations; it is not used when config.database.skipMigrations is true. Secret changes require restarting the controller. | +| database.postgres.readWriteConnectionStringSecretRef | object | `{"key":"readWriteConnectionString","name":"kagent-postgres-readwrite"}` | Secret holding the read/write login's connection string, used for every runtime query. Required. Secret changes require restarting the controller. | +| extraObjects | list | [] | Additional arbitrary Kubernetes manifests to deploy alongside the chart. Each list entry is rendered through `tpl`, so values may reference the release context (e.g. `{{ include "kagent.fullname" . }}`, `{{ .Release.Namespace }}`). Both map and multi-line string entries are supported. Use this to manage resources such as ExternalSecret, HTTPRoute, or NetworkPolicy within the same chart lifecycle without maintaining a separate chart. To use, replace the empty list below with your manifests, e.g.: extraObjects: - apiVersion: external-secrets.io/v1 kind: ExternalSecret metadata: name: '`{{ include "kagent.fullname" . }}`-openai' namespace: '`{{ .Release.Namespace }}`' spec: secretStoreRef: name: aws-secretsmanager kind: ClusterSecretStore target: name: kagent-openai data: - secretKey: OPENAI_API_KEY remoteRef: key: prod/kagent/openai property: api_key | +| fullnameOverride | string | `""` | | +| global.clusterDomain | string | `"cluster.local"` | Kubernetes DNS domain used in runtime-facing service URLs. | +| global.imagePullPolicy | string | `""` | Fallback imagePullPolicy where neither a component nor the top-level imagePullPolicy sets one. | +| global.imagePullSecrets | list | `[]` | Pull secrets for every pod. The chart merges this list (union) into each pod's own imagePullSecrets, so a local secret is never removed. Coverage matches imageRegistry above. | +| global.imageRegistry | string | `""` | Registry that overrides every per-image registry when set. This is the air-gap mirror knob: one value redirects all images. Repository and tag stay per-image. For per-image control, leave this unset and set the per-image registry keys. Covers this chart's own images: controller, sandbox guest, ui, substrateWorkerPool.workerImage, and built-in Harness images (both packaged defaults and config.harnessImages overrides). The vendored subcharts kagent-tools adopts it when its pinned version bumps. | +| global.watchNamespaces | list | `[]` | Namespace scope for the whole install. A non-empty list replaces ClusterRoles with Roles and scopes the controller's watch. Both derive from this one value, so they cannot disagree. Prefer this value over the per-surface keys. rbac.namespaces overrides it when the key is present. An explicit empty rbac.namespaces restores cluster-scoped RBAC. Mixing in config.kubernetes.watchNamespaces is validated: the watch must stay inside the RBAC scope. A watched namespace without a Role is a permanent Forbidden loop at runtime. | +| imagePullPolicy | string | `""` | Pull policy for all containers. Empty falls back to global.imagePullPolicy, then IfNotPresent. | +| imagePullSecrets | list | `[]` | | +| ipv6 | object | false | Enable IPv6/dual-stack support. When true, configures all components for dual-stack (IPv4+IPv6) networking: - nginx listens on both IPv4 and IPv6 (adds `listen [::]:8080`) - Next.js binds to `::` instead of `0.0.0.0` - Agent pods bind to `::` for dual-stack reachability Leave disabled on clusters where IPv6 is disabled at the kernel level. | +| kagent-tools.enabled | bool | `true` | | +| kagent-tools.nameOverride | string | `"tools"` | | +| kagent-tools.nodeSelector | object | `{}` | Node labels to match for `Pod` [scheduling](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/). | +| kagent-tools.podSecurityContext.runAsNonRoot | bool | `true` | | +| kagent-tools.podSecurityContext.seccompProfile.type | string | `"RuntimeDefault"` | | +| kagent-tools.remoteMCPServer.allowedNamespaces | object | `{}` | Namespaces whose Agents may reference the tool server (spec.allowedNamespaces). Unset by default, which allows only the release namespace. Example: allowedNamespaces: from: Selector # All | Same | Selector selector: matchLabels: kagent-tools: allowed | +| kagent-tools.replicaCount | int | `1` | | +| kagent-tools.resources.limits.memory | string | `"256Mi"` | | +| kagent-tools.resources.requests.cpu | string | `"50m"` | | +| kagent-tools.resources.requests.memory | string | `"128Mi"` | | +| kagent-tools.securityContext.allowPrivilegeEscalation | bool | `false` | | +| kagent-tools.securityContext.capabilities.drop[0] | string | `"ALL"` | | +| kagent-tools.securityContext.readOnlyRootFilesystem | bool | `true` | | +| kagent-tools.tolerations | list | `[]` | Node taints which will be tolerated for `Pod` [scheduling](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/). | +| kagent-tools.tools.loglevel | string | `"debug"` | | +| kagent-tools.tools.metrics.port | int | `8085` | | +| labels | object | `{}` | Additional labels to add to all Kubernetes resources | +| nameOverride | string | `""` | | +| namespaceOverride | string | `.Release.Namespace` | Override the namespace | +| nodeSelector | object | `{}` | Node labels to match for `Pod` [scheduling](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/). | +| oauth2-proxy.config.clientID | string | `""` | | +| oauth2-proxy.config.clientSecret | string | `""` | | +| oauth2-proxy.config.cookieSecret | string | `""` | | +| oauth2-proxy.config.existingSecret | string | `""` | | +| oauth2-proxy.enabled | bool | `false` | | +| oauth2-proxy.extraArgs.approval-prompt | string | `"auto"` | | +| oauth2-proxy.extraArgs.cookie-samesite | string | `"lax"` | | +| oauth2-proxy.extraArgs.cookie-secure | bool | `true` | | +| oauth2-proxy.extraArgs.custom-templates-dir | string | `"/templates"` | | +| oauth2-proxy.extraArgs.email-domain | string | `"*"` | | +| oauth2-proxy.extraArgs.oidc-issuer-url | string | `"$(OIDC_ISSUER_URL)"` | | +| oauth2-proxy.extraArgs.pass-authorization-header | bool | `true` | | +| oauth2-proxy.extraArgs.provider | string | `"oidc"` | | +| oauth2-proxy.extraArgs.redirect-url | string | `"$(OIDC_REDIRECT_URL)"` | | +| oauth2-proxy.extraArgs.scope | string | `"openid profile email groups"` | | +| oauth2-proxy.extraArgs.set-authorization-header | bool | `true` | | +| oauth2-proxy.extraArgs.skip-auth-regex | string | `"^/(login|assets/|env-config\\.js|logo-.*\\.png|favicon\\.ico).*$"` | | +| oauth2-proxy.extraArgs.skip-auth-route | string | `"^/(health|login)$"` | | +| oauth2-proxy.extraArgs.skip-jwt-bearer-tokens | bool | `true` | | +| oauth2-proxy.extraArgs.upstream | string | `"$(UPSTREAM_URL)"` | | +| oauth2-proxy.extraEnv[0].name | string | `"KAGENT_OAUTH2_PROXY_SIGNIN_TEMPLATE_CHECKSUM"` | | +| oauth2-proxy.extraEnv[0].value | string | `"{{ include \"kagent.oauth2ProxySignInHTML\" . | sha256sum }}"` | | +| oauth2-proxy.extraEnv[1].name | string | `"OIDC_ISSUER_URL"` | | +| oauth2-proxy.extraEnv[1].value | string | `""` | | +| oauth2-proxy.extraEnv[2].name | string | `"OIDC_REDIRECT_URL"` | | +| oauth2-proxy.extraEnv[2].value | string | `""` | | +| oauth2-proxy.extraEnv[3].name | string | `"UPSTREAM_URL"` | | +| oauth2-proxy.extraEnv[3].value | string | `"http://kagent-ui:8080"` | | +| oauth2-proxy.extraVolumeMounts[0].mountPath | string | `"/templates"` | | +| oauth2-proxy.extraVolumeMounts[0].name | string | `"custom-templates"` | | +| oauth2-proxy.extraVolumeMounts[0].readOnly | bool | `true` | | +| oauth2-proxy.extraVolumes[0].configMap.name | string | `"kagent-oauth2-proxy-templates"` | | +| oauth2-proxy.extraVolumes[0].name | string | `"custom-templates"` | | +| oauth2-proxy.service.portNumber | int | `4180` | | +| oauth2-proxy.service.type | string | `"ClusterIP"` | | +| oauth2-proxy.sessionStorage.type | string | `"cookie"` | | +| podAnnotations | object | `{}` | | +| podLabels | object | `{}` | Additional labels to add to all pod templates (merged into pod labels of the controller and UI Deployments; can be overridden per component). Useful for admission policies that require specific labels on pods. | +| podSecurityContext | object | `{"runAsNonRoot":true,"seccompProfile":{"type":"RuntimeDefault"}}` | Security context for all pods | +| providers.annotations | object | `{}` | Annotations added to the metadata of the generated default ModelConfig (the one derived from `providers.default`). Omitted from the resource when empty. | +| providers.anthropic.apiKeySecretKey | string | `"ANTHROPIC_API_KEY"` | | +| providers.anthropic.apiKeySecretRef | string | `"kagent-anthropic"` | | +| providers.anthropic.model | string | `"claude-haiku-4-5"` | | +| providers.anthropic.provider | string | `"Anthropic"` | | +| providers.azureOpenAI.apiKeySecretKey | string | `"AZUREOPENAI_API_KEY"` | | +| providers.azureOpenAI.apiKeySecretRef | string | `"kagent-azure-openai"` | | +| providers.azureOpenAI.config.apiVersion | string | `"2023-05-15"` | | +| providers.azureOpenAI.config.azureAdToken | string | `""` | | +| providers.azureOpenAI.config.azureDeployment | string | `""` | | +| providers.azureOpenAI.config.azureEndpoint | string | `""` | | +| providers.azureOpenAI.model | string | `"gpt-4.1-mini"` | | +| providers.azureOpenAI.provider | string | `"AzureOpenAI"` | | +| providers.default | string | `"openAI"` | | +| providers.gemini.apiKeySecretKey | string | `"GOOGLE_API_KEY"` | | +| providers.gemini.apiKeySecretRef | string | `"kagent-gemini"` | | +| providers.gemini.model | string | `"gemini-3.5-flash-lite"` | | +| providers.gemini.provider | string | `"Gemini"` | | +| providers.mistral.apiKeySecretKey | string | `"MISTRAL_API_KEY"` | | +| providers.mistral.apiKeySecretRef | string | `"kagent-mistral"` | | +| providers.mistral.model | string | `"mistral-large-latest"` | | +| providers.mistral.provider | string | `"Mistral"` | | +| providers.ollama.config.host | string | `"host.docker.internal:11434"` | | +| providers.ollama.config.options.num_ctx | string | `"64000"` | | +| providers.ollama.model | string | `"llama3.2"` | | +| providers.ollama.provider | string | `"Ollama"` | | +| providers.openAI.apiKeySecretKey | string | `"OPENAI_API_KEY"` | | +| providers.openAI.apiKeySecretRef | string | `"kagent-openai"` | | +| providers.openAI.model | string | `"gpt-4.1-mini"` | | +| providers.openAI.provider | string | `"OpenAI"` | | +| proxy.url | string | `""` | | +| rbac | object | `{}` | | +| registry | string | `"ghcr.io"` | | +| securityContext | object | `{"allowPrivilegeEscalation":false,"capabilities":{"drop":["ALL"]},"readOnlyRootFilesystem":true}` | Security context for all containers | +| substrateWorkerPool | object | `{"create":false,"labels":{},"name":"kagent-default","replicas":1,"sandboxClass":"gvisor","sandboxConfigRef":"","serviceAccount":{"create":true,"name":""},"template":{},"workerImage":""}` | Optional Agent Substrate WorkerPool installed by this chart. This is platform capacity and is not owned by individual agents. | +| substrateWorkerPool.serviceAccount | object | `{"create":true,"name":""}` | ServiceAccount settings for the worker pods | +| substrateWorkerPool.serviceAccount.create | bool | `true` | Create the worker ServiceAccount. Set to false to use one managed outside the chart. | +| substrateWorkerPool.serviceAccount.name | string | `` | Name of the worker ServiceAccount. | +| tag | string | `""` | | +| tolerations | list | `[]` | Node taints which will be tolerated for `Pod` [scheduling](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/). | +| ui.additionalForwardedHeaders | list | `[]` | Identity headers the UI's nginx proxy will forward to the backend on /api/, /a2a/, and /mcp. Names are case-insensitive. Authorization is always forwarded; the auth-proxy identity headers (x-auth-request-*, x-forwarded-user, x-forwarded-email, x-forwarded-groups, x-forwarded-preferred-username) are stripped from client requests unless listed here, so a caller cannot spoof an identity the backend trusts. Headers outside that set are forwarded by nginx as normal. In trusted-proxy mode, X-Agent-Name and X-User-Id are always stripped, even if listed here. | +| ui.affinity | object | `{}` | [Affinity](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity) rules for the UI pod. | +| ui.annotations | object | `{}` | Additional annotations to add to the UI Deployment metadata | +| ui.auth.ssoRedirectPath | string | `"/oauth2/start"` | | +| ui.basePath | string | `""` | Prefix a reverse proxy strips before forwarding, e.g. `/ui`; root-relative URLs such as `publicBackendUrl` get it too. With oauth2-proxy, set `OIDC_REDIRECT_URL` under it and restart oauth2-proxy after changing it. | +| ui.env | list | `[]` | Extra environment variables for the UI container, applied after ConfigMap defaults. Entries stay in the Deployment to support valueFrom and $(VAR_NAME) expansion. An installed app extension's settings go here, named `KAGENT_UI_EXTENSION_*`; the container's entrypoint copies those onto `window.environmentVariables` for the browser to read. | +| ui.externalUrl | string | "" (share tools return paths only) | Public-facing base URL of the UI (e.g. https://kagent.example.com). When set, the controller injects KAGENT_UI_URL into agent pods so that share link tools return full clickable URLs instead of relative paths. | +| ui.httpRoute | object | `{"annotations":{},"enabled":false,"hostnames":[],"labels":{},"parentRefs":[],"rules":[]}` | Gateway API `HTTPRoute` for the UI. Requires the Gateway API CRDs (`gateway.networking.k8s.io/v1`) and an existing `Gateway` to attach to via `parentRefs`. Disabled by default; enable to front the UI with a Gateway API implementation (kgateway, Istio, Envoy Gateway, etc.) instead of the OpenShift Route or bundled oauth2-proxy. | +| ui.httpRoute.annotations | object | `{}` | Annotations to add to the `HTTPRoute`. | +| ui.httpRoute.hostnames | list | `[]` | Hostnames matched by the route. | +| ui.httpRoute.labels | object | `{}` | Extra labels to add to the `HTTPRoute` (merged with the chart labels). | +| ui.httpRoute.parentRefs | list | `[]` | Gateways this route attaches to. Required when `enabled` is `true`. | +| ui.httpRoute.rules | list | `[]` | Routing rules. When a rule omits `backendRefs`, it defaults to the UI `Service` on `ui.service.ports.port`. Each rule may also set `matches`, `filters`, and `timeouts`. | +| ui.image.pullPolicy | string | `""` | | +| ui.image.registry | string | `""` | | +| ui.image.repository | string | `"kagent-dev/kagent/ui"` | | +| ui.image.tag | string | `""` | | +| ui.nginx | object | `{"clientMaxBodySize":"20m","proxyReadTimeout":"1800s","proxySendTimeout":"1800s"}` | Nginx configuration for the UI sidecar (timeout values are passed directly to the corresponding nginx directives, e.g. "1800s"). | +| ui.nginx.clientMaxBodySize | string | `"20m"` | client_max_body_size: max request body size (0 disables it). Just above the gateway's 16 MiB gRPC cap, which chat attachments hit first. | +| ui.nginx.proxyReadTimeout | string | `"1800s"` | proxy_read_timeout: max time between two successive reads from the upstream. | +| ui.nginx.proxySendTimeout | string | `"1800s"` | proxy_send_timeout: max time between two successive writes to the upstream. | +| ui.nodeSelector | object | `{}` | Node labels to match for `Pod` [scheduling](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/). | +| ui.openshiftRoute.annotations."haproxy.router.openshift.io/timeout" | string | `"120m"` | | +| ui.pdb | object | `{"annotations":{},"enabled":false,"labels":{},"maxUnavailable":1,"minAvailable":null,"unhealthyPodEvictionPolicy":""}` | [PodDisruptionBudget](https://kubernetes.io/docs/tasks/run-application/configure-pdb/) for the UI pods. Disabled by default: `ui.replicas` is 1, and a `minAvailable: 1` budget on a single-replica Deployment blocks every voluntary eviction, so node drains and cluster upgrades hang indefinitely. Raise `ui.replicas` before switching to `minAvailable`. | +| ui.pdb.annotations | object | `{}` | Annotations for the UI PodDisruptionBudget. | +| ui.pdb.enabled | bool | `false` | Set to true to create the PodDisruptionBudget. | +| ui.pdb.labels | object | `{}` | Additional labels for the UI PodDisruptionBudget. | +| ui.pdb.maxUnavailable | int | `1` | Maximum number of pods that may be unavailable. Int or percentage string (e.g. `1` or `"50%"`). Mutually exclusive with `minAvailable`. | +| ui.pdb.minAvailable | string | unset (`maxUnavailable` is used instead) | Minimum number of pods that must remain available. Int or percentage string (e.g. `1` or `"50%"`). Mutually exclusive with `maxUnavailable`. | +| ui.pdb.unhealthyPodEvictionPolicy | string | "" (defer to the Kubernetes default) | `spec.unhealthyPodEvictionPolicy`, one of `IfHealthyBudget` (the Kubernetes default) or `AlwaysAllow`. `AlwaysAllow` lets unhealthy pods be evicted even when the budget is exhausted, which avoids drains wedging on a crash-looping pod. Requires Kubernetes >= 1.27. Omitted from the manifest when empty. | +| ui.podAnnotations | object | `{}` | | +| ui.podLabels | object | `{}` | Additional labels for the UI pod template, merged over the global `podLabels` (per-key; component keys win). Selector labels can never be overridden. | +| ui.podSecurityContext | object | (uses global podSecurityContext) | Pod-level security context for the UI pod. Overrides the global podSecurityContext. | +| ui.publicBackendUrl | string | `"/api"` | Base URL the browser calls the controller API on. Reaches the browser at runtime as the `apiBaseUrl` key of /config.json, which the app fetches on startup — it is deliberately not baked into the bundle, so changing it here takes effect on pod restart rather than requiring an image rebuild. The default is a path on the UI's own hostname, which nginx proxies to the controller; set an absolute URL only if the browser must reach the API somewhere other than the UI origin. | +| ui.readinessProbe | object | httpGet /health on port http, periodSeconds=30 | Custom readiness probe for the UI container. Override to adjust thresholds or change the health path. The image has no shell or curl, so use httpGet or tcpSocket. | +| ui.replicas | int | `1` | | +| ui.resources.limits.cpu | string | `"1000m"` | | +| ui.resources.limits.memory | string | `"1Gi"` | | +| ui.resources.requests.cpu | string | `"100m"` | | +| ui.resources.requests.memory | string | `"256Mi"` | | +| ui.route | object | `{"enabled":true}` | Gates the OpenShift `Route` for the UI. Additionally conditional on the `route.openshift.io/v1` API being present, so it is a no-op off-OpenShift. Set to `false` to front the UI with your own Route/ingress or the bundled oauth2-proxy instead of the chart's edge-terminated Route. | +| ui.securityContext | object | (uses global securityContext) | Container-level security context for the UI container. Overrides the global securityContext. | +| ui.service.annotations | object | `{}` | | +| ui.service.ports.port | int | `8080` | | +| ui.service.ports.targetPort | int | `8080` | | +| ui.service.type | string | `"ClusterIP"` | | +| ui.serviceAccount | object | `{"annotations":{}}` | ServiceAccount settings for the UI pod | +| ui.serviceAccount.annotations | object | {} (no extra annotations) | Annotations to add to the UI ServiceAccount. Useful for GCP Workload Identity, AWS IRSA, or Azure Workload Identity. | +| ui.startupProbe | object | httpGet /health on port http, periodSeconds=1, initialDelaySeconds=1 | Custom startup probe for the UI container. Override to adjust thresholds or change the health path. The image has no shell or curl, so use httpGet or tcpSocket. | +| ui.streamTimeoutSeconds | int | `1800` | Client-side chat stream inactivity timeout (seconds). The browser aborts a streaming response if no event is received within this window. Should be >= ui.nginx.proxyReadTimeout so nginx isn't the silent limit. Default 1800 (30m). | +| ui.tolerations | list | `[]` | Node taints which will be tolerated for `Pod` [scheduling](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/). | +| ui.topologySpreadConstraints | list | `[]` | [Topology spread constraints](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#pod-topology-spread-constraints) for the UI pod. | +| ui.volumes | object | `{"tmp":"50Mi"}` | EmptyDir volume sizes for the UI workload (typically used when enabling readOnlyRootFilesystem) | +| ui.volumes.tmp | string | `"50Mi"` | Size limit for temporary files (/tmp). Holds the nginx temp directories and the generated config.json. Default 50Mi is ample for both. | diff --git a/docs-site/content/kagent/1.x/reference/release-notes/1.0.md b/docs-site/content/kagent/latest/reference/release-notes/1.0.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/release-notes/1.0.md rename to docs-site/content/kagent/latest/reference/release-notes/1.0.md diff --git a/docs-site/content/kagent/1.x/reference/release-notes/_index.md b/docs-site/content/kagent/latest/reference/release-notes/_index.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/release-notes/_index.md rename to docs-site/content/kagent/latest/reference/release-notes/_index.md diff --git a/docs-site/content/kagent/1.x/reference/tools-ecosystem.md b/docs-site/content/kagent/latest/reference/tools-ecosystem.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/tools-ecosystem.md rename to docs-site/content/kagent/latest/reference/tools-ecosystem.md diff --git a/docs-site/content/kagent/1.x/reference/versions.md b/docs-site/content/kagent/latest/reference/versions.md similarity index 100% rename from docs-site/content/kagent/1.x/reference/versions.md rename to docs-site/content/kagent/latest/reference/versions.md diff --git a/docs-site/content/kagent/1.x/setup/_index.md b/docs-site/content/kagent/latest/setup/_index.md similarity index 100% rename from docs-site/content/kagent/1.x/setup/_index.md rename to docs-site/content/kagent/latest/setup/_index.md diff --git a/docs-site/content/kagent/1.x/setup/installation.md b/docs-site/content/kagent/latest/setup/installation.md similarity index 100% rename from docs-site/content/kagent/1.x/setup/installation.md rename to docs-site/content/kagent/latest/setup/installation.md diff --git a/docs-site/content/kagent/1.x/setup/model-providers/_index.md b/docs-site/content/kagent/latest/setup/model-providers/_index.md similarity index 100% rename from docs-site/content/kagent/1.x/setup/model-providers/_index.md rename to docs-site/content/kagent/latest/setup/model-providers/_index.md diff --git a/docs-site/content/kagent/1.x/setup/model-providers/about-model-providers.md b/docs-site/content/kagent/latest/setup/model-providers/about-model-providers.md similarity index 100% rename from docs-site/content/kagent/1.x/setup/model-providers/about-model-providers.md rename to docs-site/content/kagent/latest/setup/model-providers/about-model-providers.md diff --git a/docs-site/content/kagent/1.x/setup/model-providers/amazon-bedrock.md b/docs-site/content/kagent/latest/setup/model-providers/amazon-bedrock.md similarity index 100% rename from docs-site/content/kagent/1.x/setup/model-providers/amazon-bedrock.md rename to docs-site/content/kagent/latest/setup/model-providers/amazon-bedrock.md diff --git a/docs-site/content/kagent/1.x/setup/model-providers/anthropic.md b/docs-site/content/kagent/latest/setup/model-providers/anthropic.md similarity index 100% rename from docs-site/content/kagent/1.x/setup/model-providers/anthropic.md rename to docs-site/content/kagent/latest/setup/model-providers/anthropic.md diff --git a/docs-site/content/kagent/1.x/setup/model-providers/azure-ai-foundry.md b/docs-site/content/kagent/latest/setup/model-providers/azure-ai-foundry.md similarity index 100% rename from docs-site/content/kagent/1.x/setup/model-providers/azure-ai-foundry.md rename to docs-site/content/kagent/latest/setup/model-providers/azure-ai-foundry.md diff --git a/docs-site/content/kagent/1.x/setup/model-providers/azure-openai.md b/docs-site/content/kagent/latest/setup/model-providers/azure-openai.md similarity index 100% rename from docs-site/content/kagent/1.x/setup/model-providers/azure-openai.md rename to docs-site/content/kagent/latest/setup/model-providers/azure-openai.md diff --git a/docs-site/content/kagent/1.x/setup/model-providers/byo-agentgateway.md b/docs-site/content/kagent/latest/setup/model-providers/byo-agentgateway.md similarity index 100% rename from docs-site/content/kagent/1.x/setup/model-providers/byo-agentgateway.md rename to docs-site/content/kagent/latest/setup/model-providers/byo-agentgateway.md diff --git a/docs-site/content/kagent/1.x/setup/model-providers/byo-openai.md b/docs-site/content/kagent/latest/setup/model-providers/byo-openai.md similarity index 100% rename from docs-site/content/kagent/1.x/setup/model-providers/byo-openai.md rename to docs-site/content/kagent/latest/setup/model-providers/byo-openai.md diff --git a/docs-site/content/kagent/1.x/setup/model-providers/gemini.md b/docs-site/content/kagent/latest/setup/model-providers/gemini.md similarity index 100% rename from docs-site/content/kagent/1.x/setup/model-providers/gemini.md rename to docs-site/content/kagent/latest/setup/model-providers/gemini.md diff --git a/docs-site/content/kagent/1.x/setup/model-providers/google-vertexai.md b/docs-site/content/kagent/latest/setup/model-providers/google-vertexai.md similarity index 100% rename from docs-site/content/kagent/1.x/setup/model-providers/google-vertexai.md rename to docs-site/content/kagent/latest/setup/model-providers/google-vertexai.md diff --git a/docs-site/content/kagent/1.x/setup/model-providers/mistral.md b/docs-site/content/kagent/latest/setup/model-providers/mistral.md similarity index 100% rename from docs-site/content/kagent/1.x/setup/model-providers/mistral.md rename to docs-site/content/kagent/latest/setup/model-providers/mistral.md diff --git a/docs-site/content/kagent/1.x/setup/model-providers/ollama.md b/docs-site/content/kagent/latest/setup/model-providers/ollama.md similarity index 100% rename from docs-site/content/kagent/1.x/setup/model-providers/ollama.md rename to docs-site/content/kagent/latest/setup/model-providers/ollama.md diff --git a/docs-site/content/kagent/1.x/setup/model-providers/openai.md b/docs-site/content/kagent/latest/setup/model-providers/openai.md similarity index 100% rename from docs-site/content/kagent/1.x/setup/model-providers/openai.md rename to docs-site/content/kagent/latest/setup/model-providers/openai.md diff --git a/docs-site/content/kagent/1.x/setup/model-providers/sap-ai-core.md b/docs-site/content/kagent/latest/setup/model-providers/sap-ai-core.md similarity index 100% rename from docs-site/content/kagent/1.x/setup/model-providers/sap-ai-core.md rename to docs-site/content/kagent/latest/setup/model-providers/sap-ai-core.md diff --git a/docs-site/content/kagent/1.x/setup/model-providers/xai.md b/docs-site/content/kagent/latest/setup/model-providers/xai.md similarity index 100% rename from docs-site/content/kagent/1.x/setup/model-providers/xai.md rename to docs-site/content/kagent/latest/setup/model-providers/xai.md diff --git a/docs-site/content/kagent/1.x/skills-and-mcp/_index.md b/docs-site/content/kagent/latest/skills-and-mcp/_index.md similarity index 100% rename from docs-site/content/kagent/1.x/skills-and-mcp/_index.md rename to docs-site/content/kagent/latest/skills-and-mcp/_index.md diff --git a/docs-site/content/kagent/1.x/skills-and-mcp/about-tools.md b/docs-site/content/kagent/latest/skills-and-mcp/about-tools.md similarity index 100% rename from docs-site/content/kagent/1.x/skills-and-mcp/about-tools.md rename to docs-site/content/kagent/latest/skills-and-mcp/about-tools.md diff --git a/docs-site/content/kagent/1.x/skills-and-mcp/plugins.md b/docs-site/content/kagent/latest/skills-and-mcp/plugins.md similarity index 100% rename from docs-site/content/kagent/1.x/skills-and-mcp/plugins.md rename to docs-site/content/kagent/latest/skills-and-mcp/plugins.md diff --git a/docs-site/content/kagent/1.x/skills-and-mcp/skills.md b/docs-site/content/kagent/latest/skills-and-mcp/skills.md similarity index 100% rename from docs-site/content/kagent/1.x/skills-and-mcp/skills.md rename to docs-site/content/kagent/latest/skills-and-mcp/skills.md diff --git a/docs-site/content/kagent/1.x/ui/_index.md b/docs-site/content/kagent/latest/ui/_index.md similarity index 100% rename from docs-site/content/kagent/1.x/ui/_index.md rename to docs-site/content/kagent/latest/ui/_index.md diff --git a/docs-site/content/kagent/latest/ui/launch-ui.md b/docs-site/content/kagent/latest/ui/launch-ui.md new file mode 100644 index 00000000..b8b5e06f --- /dev/null +++ b/docs-site/content/kagent/latest/ui/launch-ui.md @@ -0,0 +1,232 @@ +--- +title: Launch the UI +description: Open the kagent dashboard, then read the agents, conversations, and Agent Substrate capacity that it reports. +weight: 10 +author: kagent.dev +aliases: + - /kagent/latest/observability/launch-ui/ +--- + +The {{< reuse "kagent-docs/snippets/name-product.md" >}} UI is a read-and-write console for everything that the controller knows about, including the agents that you defined, the conversations that the agents hold, and the Agent Substrate capacity that those conversations run on. The kagent chart installs the UI alongside the controller, so a cluster that follows [Install kagent]({{< link path="setup/installation" >}}) already runs a UI instance. + +## Before you begin + +1. [Install kagent]({{< link path="setup/installation" >}}) in your cluster. + +2. Confirm that the UI service exists. + + ```bash + kubectl get svc -n kagent {{< reuse "kagent-docs/snippets/name-ui.md" >}} + ``` + + Example output: + + ```console + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + {{< reuse "kagent-docs/snippets/name-ui.md" >}} ClusterIP 10.96.174.203 8080/TCP 3m + ``` + +## Open the dashboard + +The UI service is a `ClusterIP` by default, so it is reachable only from inside the cluster. Forward it to your machine with the kagent CLI or with `kubectl`. Both routes end at the same address, `http://localhost:8082`. + +{{< tabs >}} +{{% tab name="kagent CLI" %}} +1. Run the dashboard command to forward the service and open the dashboard in your default browser. + + ```bash + kagent dashboard + ``` + + Example output: + + ```console + kagent dashboard is available at http://localhost:8082 + Press the Enter Key to stop the port-forward... + ``` + +2. When you are done, press `Enter` in the terminal to stop the port-forward. + +> [!NOTE] +> `kagent dashboard` opens a browser only on macOS. On Linux and Windows the command prints the `kubectl` port-forward command and exits, so follow the port-forward steps instead. +{{% /tab %}} +{{% tab name="Port-forward for local testing" %}} +1. Forward the `{{< reuse "kagent-docs/snippets/name-ui.md" >}}` service to your machine, and leave the command running. + + ```bash + kubectl port-forward -n kagent service/{{< reuse "kagent-docs/snippets/name-ui.md" >}} 8082:8080 + ``` + +2. Open [http://localhost:8082](http://localhost:8082) in your browser. + +3. When you are done, press `Ctrl+C` in the terminal that runs the port-forward. +{{% /tab %}} +{{< /tabs >}} + +The dashboard opens on a summary of what the controller has loaded. On a fresh installation, the dashboard counts zero agents and zero conversations. The dashboard also counts the model configuration and the {{< gloss "Model Context Protocol" >}}Model Context Protocol{{< /gloss >}} (MCP) servers that the chart installed. + +{{< reuse-image-light src="img/kagent-ui-dashboard.png" alt="The kagent dashboard after a fresh installation" caption="Figure: The kagent dashboard after a fresh installation" >}} +{{< reuse-image-dark srcDark="img/kagent-ui-dashboard-dark.png" alt="The kagent dashboard after a fresh installation" caption="Figure: The kagent dashboard after a fresh installation" >}} + +## Explore the UI + +The left menu groups the console pages by the resource that each page reads. + +| Page | Shows | +| ---- | ----- | +| **Dashboard** | Counts of agents, model configurations, MCP servers, and discovered tools, plus recent conversations. | +| **Agents** | Every {{< gloss "Agent" >}}Agent{{< /gloss >}}, {{< gloss "AgentTemplate" >}}AgentTemplate{{< /gloss >}}, and {{< gloss "Harness" >}}Harness{{< /gloss >}}, and the conversations held with each Agent. | +| **Schedules** | Agents that receive a fixed prompt on a cron schedule. Each execution starts a new conversation. For more information, see [Scheduled runs]({{< link path="agents/scheduled-runs" >}}). | +| **Models** | The {{< gloss "ModelConfig" >}}ModelConfig{{< /gloss >}} resources that agents reference, and the providers behind them. | +| **MCP Servers** | Connected {{< gloss "RemoteMCPServer" >}}RemoteMCPServer{{< /gloss >}} resources and the tools discovered from each. | +| **Prompts** | Prompt libraries, which store reusable fragments that an AgentTemplate includes in its instructions. | +| **Substrate** | WorkerPools, Actors, and Workers, read from both Kubernetes and the Agent Substrate API. | + +Start at the **Agents** page to review your agents and the resources behind them. The page lists each resource kind on its own tab: **Agents**, **Templates**, and **Harnesses**. + +{{< reuse-image-light src="img/kagent-ui-agents.png" alt="The Agents page, listing agents, templates, and harnesses" caption="Figure: The Agents page" >}} +{{< reuse-image-dark srcDark="img/kagent-ui-agents-dark.png" alt="The Agents page, listing agents, templates, and harnesses" caption="Figure: The Agents page" >}} + +The **Agents** tab authors the Agent custom resource directly. Create one to pair a template with a Harness, edit one to change either side, and delete one to retire the pairing. Deleting an Agent leaves its Sessions in place, along with the AgentTemplate and Harness that it referenced. For CLI steps, see [Create your first agent]({{< link path="get-started/your-first-agent" >}}). + +## Read a conversation + +Opening an agent and sending a message creates a {{< gloss "Session" >}}Session{{< /gloss >}}, which is one conversation scheduled onto an Agent Substrate {{< gloss "Actor" >}}Actor{{< /gloss >}}. The conversation view shows the transcript, the tool calls that the agent made along the way, and the checkpoint controls for the conversation. + +{{< reuse-image-light src="img/kagent-ui-chat.png" alt="A conversation with an agent, showing a named snapshot and the controls on its mark" caption="Figure: A conversation with an agent" >}} +{{< reuse-image-dark srcDark="img/kagent-ui-chat-dark.png" alt="A conversation with an agent, showing a named snapshot and the controls on its mark" caption="Figure: A conversation with an agent" >}} + +The UI labels a {{< gloss "Checkpoint" >}}checkpoint{{< /gloss >}} **Snapshot**. The label names the pin, not the Agent Substrate {{< gloss "Snapshot" >}}snapshot{{< /gloss >}} beneath it. Agent Substrate writes a snapshot each time an Actor suspends, and a checkpoint pins one of those snapshots so that Agent Substrate does not collect it. + +To take a checkpoint, select the save icon beside the message box. kagent pins the snapshot that the Session most recently suspended to, and records how far the transcript advanced. To understand how pinning works, and why a turn must be complete first, see [Checkpoints]({{< link path="about/substrate-runtime/suspend-and-resume#checkpoints" >}}). + +A checkpoint appears in the transcript as a mark that shows the snapshot name and the following controls. + +- **Fork** starts a new Session from the checkpoint. The new Session replays the conversation through the pinned turn and then continues on its own. For what a fork keeps from its source, see [Checkpoints]({{< link path="about/substrate-runtime/suspend-and-resume#checkpoints" >}}). +- **Rename** replaces the default snapshot name, which kagent generates from the Session ID and the pinned turn when it creates the checkpoint. Clearing the name restores that generated default. +- **Delete** removes the checkpoint and releases the snapshot that it pinned. Agent Substrate is then free to collect that snapshot. + +Select the mark to open the record behind it. The record shows the checkpoint's ID, turn, state, and age. + +> [!NOTE] +> A checkpoint is always taken at the latest turn boundary, so the UI offers checkpoint controls on your own most recent message and not on earlier ones. + +The bell icon on the latest message of a task opens **Task notifications**. Use it to add, edit, or remove the callback URLs that kagent posts task updates to. A new callback receives only the updates that its task sends after you register it, and only when the task needs input or reaches a final state. After the task finishes, you can view and remove its callbacks, but you cannot add or edit them. For how kagent delivers and signs a callback, see [Register task callbacks]({{< link path="examples/a2a-agents#task-callbacks" >}}). + +## Attach files to a message + +In the chat interface for agents on the `kagent` runtime, you can attach files to send with a message. An agent reads any files that you attach, so you can ask the agent about a log, a manifest, or a screenshot without pasting the contents into the message box. + +Stage files in any of the following ways. + +- Select the paperclip beside the message box, then choose the files. +- Drop files anywhere on the chat page. +- Paste a file or an image from your clipboard. A clipboard that also contains plain text pastes the text instead, so that a copy from a word processor keeps its text rather than its image rendering. + +Sent files stay in the transcript as download chips, and reloading the page redraws them. How a file reaches the model depends on its type and on the agent's model provider. + +- **Text files**, such as Markdown, CSV, JSON, YAML, and HTML, reach the model as text, regardless of the provider. +- **PDF, Word (`.docx`), Excel (`.xlsx`), and PowerPoint (`.pptx`) files** depend on the API format of the model. When the ModelConfig sets `apiFormat: responses` in its `openAI` or `azureOpenAI` block, the model receives the file and reads the document itself. Every other model receives the note `[Uploaded file "report.pdf" (application/pdf) could not be read as text.]` instead of the contents. When the files in a Session total more than 50 MB, the model receives only the newest files, and a note replaces each older file. +- **Images** reach the providers that accept them. Amazon Bedrock, Ollama, and SAP AI Core receive a note in place of the image. + +When the message text includes a link to a file of an accepted type other than HTML, such as `https://example.com/q3-report.pdf`, the UI also sends the link as a file link. When the ModelConfig sets `apiFormat: responses`, or the provider is Gemini, the model fetches the file itself. The provider must therefore be able to reach the link. The UI does not send a link to an IP address or to a local hostname, such as one that ends in `.svc` or `.local`, because the provider cannot reach such a host. Every other model receives the link as text, and an agent with a fetch tool can read the file from it. Only the message that includes a link sends it as a file. In later turns, the model receives the bare URL as text. + +An empty file, or one that contains only whitespace, reaches the model as the note `[Uploaded file "notes.txt" (text/plain) contained no extractable text.]`, so the agent reports the gap. + +Different components between the browser and the model enforce the attachment limits: + +| Limit | Value | Enforced by | +| ----- | ----- | ----------- | +| Accepted file types | Plain text, Markdown, CSV, JSON, XML, YAML, HTML, PDF, Word (`.docx`), Excel (`.xlsx`), and PowerPoint (`.pptx`) files, and PNG, JPEG, GIF, and WebP images | The UI, as you stage each file. The UI reports a rejected file by name. | +| Total size of the files in one message | 10 MB | The UI, as you stage each file. | +| Text read from one file | 200,000 characters | The runtime, which appends `[truncated]` to a longer file. | +| gRPC message that the kagent API accepts | 16 MiB | The controller's gRPC server, and every kagent gRPC client. The limit is a compiled-in constant that a Helm value cannot change. | +| Request body that the UI's nginx accepts | `ui.nginx.clientMaxBodySize`, default `20m` | The UI pod's nginx. The default sits just above the 16 MiB gRPC message limit. | + +The UI falls back to the file extension when the browser does not report a media type for a file, or reports the wrong one. For example, browsers do not report a media type for `.md` files. + +If you host a reverse proxy in front of the UI, the proxy applies its own body-size limit. A default nginx allows only `1m`. To raise the limit, see [Serve the UI under a sub-path]({{< link path="ui/reverse-proxy#proxy-shapes" >}}). + +## Check Agent Substrate capacity + +The **Substrate** page shows whether Agent Substrate has capacity to run an agent. It reads WorkerPools and ActorTemplates from Kubernetes, and live Actors and Worker assignments from the Agent Substrate API. + +{{< reuse-image-light src="img/kagent-ui-substrate.png" alt="The Substrate page, showing one worker pool and no running actors" caption="Figure: The Substrate page" >}} +{{< reuse-image-dark srcDark="img/kagent-ui-substrate-dark.png" alt="The Substrate page, showing one worker pool and no running actors" caption="Figure: The Substrate page" >}} + +Read the page from the top tiles down. + +| Tile | Means | +| ---- | ----- | +| **Worker pools** | {{< gloss "WorkerPool" >}}WorkerPools{{< /gloss >}} that an operator has provisioned. A Harness cannot run an agent until at least one WorkerPool exists. | +| **Templates ready** | {{< gloss "ActorTemplate" >}}ActorTemplates{{< /gloss >}} that have compiled and are ready to be instantiated. | +| **Actors running** | Live Actors. Each Actor runs one Session. | +| **Workers busy** | {{< gloss "Worker" >}}Workers{{< /gloss >}} currently assigned to an Actor, against the total provisioned. | +| **Requested filters** | The filters this page asked for, not what the API returned. Reads `none` until you set a filter, then `K8s: ; ATE: `, with `any` in place of a filter that you leave empty. | + +On a fresh installation, the page reports one WorkerPool, zero ActorTemplates, and zero Actors, because an agent does not exist yet. The **Workers busy** tile reads `0/1`, and the Workers table lists the pool's single worker with the pod that backs it. That unassigned worker confirms that capacity is provisioned and waiting for the first agent. The Actors table reads `No actors visible in this scope.` + +When you [restrict kagent to specific namespaces]({{< link path="operations/multiple-namespaces#restrict-kagent-to-specific-namespaces" >}}), the Substrate page reads only the watched namespaces and the atespaces whose names kagent derives from those namespaces. Empty filters cover every watched namespace. A filter that specifies any other namespace or atespace fails with a `PermissionDenied` error. An installation that watches every namespace reads the entire Agent Substrate. + +> [!NOTE] +> When the Substrate page cannot reach the Agent Substrate API, it reports the failure rather than an empty result. A warning above the tables shows the error, and the Actors table reads `This read could not reach ate-api, so there may be actors it did not see.` On a cluster that does have agents, `No actors visible in this scope.` means that the read succeeded and matched nothing, so check the **Kubernetes namespace** and **ATE atespace** filters. + +## Expose the UI outside the cluster + +Port-forwarding suits local access and stops when you close the terminal. For a deployment that a team reaches, change how the chart publishes the service. Add any of the following values to the `helm upgrade --install kagent` command from [Install kagent with Helm]({{< link path="setup/installation#install-kagent-helm" >}}). + +### LoadBalancer service + +To provision a cloud load balancer for the UI, set the service type. + +```yaml +ui: + service: + type: LoadBalancer +``` + +Then read the external address from the service. + +```bash +kubectl get svc -n kagent {{< reuse "kagent-docs/snippets/name-ui.md" >}} +``` + +### OpenShift route + +On OpenShift clusters, kagent creates an edge-terminated `Route` for the UI whenever the `route.openshift.io/v1` API is present. The route is enabled by default through `ui.route.enabled`. + +The chart raises the default HAProxy timeout to 120 minutes, because agent conversations keep long-lived streaming connections open, and a shorter timeout terminates them mid-reply. To change the timeout, set the annotation. + +```yaml +ui: + openshiftRoute: + annotations: + haproxy.router.openshift.io/timeout: 60m +``` + +To front the UI with your own ingress instead, set `ui.route.enabled: false` so that kagent does not create the route. + +### Gateway API HTTPRoute + +On a cluster that runs a Kubernetes Gateway API implementation such as kgateway, Istio, or Envoy Gateway, publish the UI through an `HTTPRoute`. + +```yaml +ui: + httpRoute: + enabled: true + parentRefs: + - name: my-gateway + namespace: gateway-system + hostnames: + - kagent.example.com +``` + +The `parentRefs` field is required when `enabled` is `true`, and it must reference a `Gateway` that already exists. The `HTTPRoute` resource also requires the Gateway API custom resource definitions (`gateway.networking.k8s.io/v1`) in the cluster. + +## Next steps + +{{< cards >}} + {{< card link=`{{< link path="get-started/your-first-agent" >}}` title="Create your first agent" subtitle="Pair an AgentTemplate with a Harness, then hold a conversation with the result." >}} + {{< card link=`{{< link path="observability/tracing" >}}` title="Tracing" subtitle="Follow one agent request from the controller through to the Actor that ran it." >}} + {{< card link=`{{< link path="observability/audit-prompts" >}}` title="Audit prompts" subtitle="Capture every prompt and reply in your traces for security and compliance review." >}} +{{< /cards >}} diff --git a/docs-site/content/kagent/latest/ui/reverse-proxy.md b/docs-site/content/kagent/latest/ui/reverse-proxy.md new file mode 100644 index 00000000..536fcb22 --- /dev/null +++ b/docs-site/content/kagent/latest/ui/reverse-proxy.md @@ -0,0 +1,141 @@ +--- +title: Serve the UI under a sub-path +description: Serve the kagent UI behind a reverse proxy using ui.basePath, including both proxy shapes, validation, and oauth2-proxy configuration. +weight: 20 +author: kagent.dev +aliases: + - /kagent/latest/setup/reverse-proxy/ +--- + +The kagent UI is a static bundle served by nginx, which answers on port `8080` inside the pod. By default it serves at the root (`/`). When a reverse proxy fronts the UI and you want it reachable under a prefix such as `/ui`, set `ui.basePath` in the Helm values. + +## The `ui.basePath` value + +One Helm value controls the whole sub-path, and it is purely additive: leaving it unset leaves an existing installation serving at the root exactly as before. Set it, and the prefix reaches every URL the UI emits, not only the pages a reader navigates to. + +| Setting | Default | Effect | +|---------|---------|--------| +| `ui.basePath` | `""` (empty) | UI served at `/`. All existing behaviour unchanged. | +| `ui.basePath` | e.g. `/ui` | UI served under the prefix. One value moves every root-relative URL the UI emits. | + +When set, a single value controls the `` tag, the client-side router basename, and the root-relative API, SSO, userinfo, and share-link URLs that the browser uses. `publicBackendUrl` and the other root-relative URLs inherit the prefix, so do not adjust them by hand. + +## Setting the value + +Put `ui.basePath` in the Helm values at install time, rather than in a follow-up `helm upgrade`. The install steps live on the [Install kagent]({{< link path="setup/installation" >}}) page; the value is the only non-default setting this page needs. + +```bash +helm upgrade --install kagent \ + {{< reuse "kagent-docs/snippets/helm-path.md" >}}/{{< reuse "kagent-docs/snippets/helm-kagent.md" >}} \ + --version {{< reuse "kagent-docs/versions/kagent.md" >}} \ + --namespace kagent --create-namespace --timeout 10m \ + --set ui.basePath=/ui +``` + +## Proxy shapes + +Both of the following proxy configurations work. Choose the one that matches your infrastructure. If you enable oauth2-proxy for SSO, the proxy must strip the prefix: with the prefix forwarded, oauth2-proxy cannot match its own sign-in and skip-authentication paths, and sign-in fails. + +Both examples set `client_max_body_size`. A default nginx allows `1m`, so a larger attachment is rejected with a `413` before it reaches the UI pod. Keep the limit above the 10 MB that one chat message can carry. For the attachment limits, see [Attach files to a message]({{< link path="ui/launch-ui#attach-files-to-a-message" >}}). + +### Proxy strips the prefix + +The proxy removes the `/ui` prefix before forwarding to the UI service on port `8080`. The UI then sees requests at `/` and serves them normally. + +Example nginx configuration: + +```nginx +location /ui/ { + proxy_pass http://{{< reuse "kagent-docs/snippets/name-ui.md" >}}.kagent:8080/; + proxy_http_version 1.1; + client_max_body_size 20m; + proxy_set_header Host $host; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; +} +``` + +A trailing slash on `proxy_pass` makes nginx drop the `/ui` prefix. + +### Proxy forwards the prefix unchanged + +The proxy passes `/ui/...` through to the UI service without modification. In this case nginx inside the pod strips the prefix itself using the rewrite rules rendered from `ui.basePath`. + +Example nginx configuration: + +```nginx +location /ui/ { + proxy_pass http://{{< reuse "kagent-docs/snippets/name-ui.md" >}}.kagent:8080; + proxy_http_version 1.1; + client_max_body_size 20m; + proxy_set_header Host $host; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; +} +``` + +Do not put a trailing slash on `proxy_pass`. The full path including `/ui` reaches the UI pod, and its nginx rewrites it to `/` before serving. + +## Helm render validation + +Two validations run at Helm render time and fail the install rather than producing a broken UI. + +| Condition | Helm error message | +|-----------|-------------------| +| Not a path like `/ui`, or containing a `.` or `..` segment | `ui.basePath must be a path like /ui` | +| Starting with a path nginx serves | `ui.basePath cannot start with a path nginx serves, such as /api or /assets` | + +The reserved set that triggers the second rejection is larger than the message names. The full set of paths that must not appear as the first segment of `ui.basePath`: + +- `api` +- `a2a` +- `assets` +- `health` +- `env-config.js` +- `index.html` +- `mockServiceWorker.js` + +Choosing any of these as the base path collides with a path nginx already owns inside the UI pod. + +## oauth2-proxy + +If you use oauth2-proxy for SSO, a base-path change requires a second, manual step. + +1. Set `OIDC_REDIRECT_URL` under `oauth2-proxy.extraEnv` in your Helm values so that it points under the base path, for example `https://example.com/ui/oauth2/callback` rather than `https://example.com/oauth2/callback`. This becomes oauth2-proxy's `--redirect-url`, which is the OAuth callback, not the path that starts sign-in; the chart leaves its `--proxy-prefix` at `/oauth2`, so the callback is `/oauth2/callback`. +2. Register that same URL as a redirect URI with your identity provider. +3. Restart the oauth2-proxy pod after changing the value. + +The restart is required because the chart's sign-in HTML redirects to `{basePath}/login`, but the checksum that would roll the oauth2-proxy pod is computed where the `ui` values key is not in scope. The base path is absent from the hash, so the hash does not change and the pod keeps the old sign-in template and redirect URL until you restart it. + +## Verifying the installation + +After installing with `--set ui.basePath=/ui`, run each check and look for the named result. + +1. Open `/ui/` in a browser. Expected: the kagent UI loads, and the `` in the page source is `/ui/`. +2. Navigate to an agent list or detail page. Expected: the URL stays under `/ui/...`, for example `/ui/agents`. +3. Open the browser network tab while the page loads. Expected: API calls go to `/ui/api/...`, not `/api/...`. +4. Reload the page. Expected: the SPA router keeps the `/ui` prefix in the URL. +5. If oauth2-proxy is enabled, sign in. Expected: the OAuth redirect lands under `/ui/oauth2/...`, then returns you to the UI under `/ui/`. This holds when the proxy strips the prefix; when the proxy forwards the prefix unchanged, the OAuth paths behave as described under [Proxy shapes](#proxy-shapes). + +To confirm the Helm-side validation, install with a rejected value and expect the render to fail: + +```bash +helm upgrade --install kagent \ + {{< reuse "kagent-docs/snippets/helm-path.md" >}}/{{< reuse "kagent-docs/snippets/helm-kagent.md" >}} \ + --version {{< reuse "kagent-docs/versions/kagent.md" >}} \ + --namespace kagent --create-namespace \ + --set ui.basePath=/api +``` + +Expected: + +``` +Error: execution error at (kagent/templates/ui-deployment.yaml:21:31): ui.basePath cannot start with a path nginx serves, such as /api or /assets +``` + +## Next steps + +{{< cards >}} + {{< card link=`{{< link path="setup/installation" >}}` title="Install kagent" subtitle="The full install steps this page's `ui.basePath` value plugs into." >}} + {{< card link=`{{< link path="about/substrate-runtime/identity" >}}` title="Identity" subtitle="How kagent resolves a caller's identity and scopes a Session to its creator." >}} +{{< /cards >}} diff --git a/docs-site/content/kagent/main/_index.md b/docs-site/content/kagent/main/_index.md new file mode 100644 index 00000000..b0f8bea7 --- /dev/null +++ b/docs-site/content/kagent/main/_index.md @@ -0,0 +1,9 @@ +--- +title: kagent +linkTitle: "kagent main" +description: Explore concepts, guides, and reference docs for running declarative AI agents on Kubernetes with Agent Substrate, from install through day-two operations. +weight: 10 +author: kagent.dev +--- + +{{< reuse "kagent-docs/snippets/name-product.md" >}} 1.0 moves the runtime from Kubernetes Deployments to [Agent Substrate](https://github.com/agent-substrate/substrate), introducing Harness, AgentTemplate, and Agent as the new API surface, with a Session for each running conversation. For a summary of what changed in 1.0, see the [Release notes]({{< link path="reference/release-notes/1.0#100" >}}). \ No newline at end of file diff --git a/docs-site/content/kagent/main/about/_index.md b/docs-site/content/kagent/main/about/_index.md new file mode 100644 index 00000000..0dbc1bab --- /dev/null +++ b/docs-site/content/kagent/main/about/_index.md @@ -0,0 +1,8 @@ +--- +title: About +description: Learn the core concepts and architecture behind kagent 1.0. +weight: 20 +author: kagent.dev +--- + +Learn what changed in {{< reuse "kagent-docs/snippets/name-product.md" >}} 1.0 and the vocabulary the rest of these docs build on. diff --git a/docs-site/content/kagent/main/about/architecture/_index.md b/docs-site/content/kagent/main/about/architecture/_index.md new file mode 100644 index 00000000..29d5ce2a --- /dev/null +++ b/docs-site/content/kagent/main/about/architecture/_index.md @@ -0,0 +1,5 @@ +--- +title: Architecture +weight: 30 +description: Understand how kagent connects a Harness and AgentTemplate into a running conversation, and what Agent Substrate runs it on. +--- diff --git a/docs-site/content/kagent/1.x/about/architecture/agent-substrate.md b/docs-site/content/kagent/main/about/architecture/agent-substrate.md similarity index 99% rename from docs-site/content/kagent/1.x/about/architecture/agent-substrate.md rename to docs-site/content/kagent/main/about/architecture/agent-substrate.md index e5eb942e..2eaff919 100644 --- a/docs-site/content/kagent/1.x/about/architecture/agent-substrate.md +++ b/docs-site/content/kagent/main/about/architecture/agent-substrate.md @@ -4,7 +4,7 @@ description: See how Agent Substrate runs, suspends, and resumes the Actors behi weight: 40 author: kagent.dev aliases: - - /kagent/1.x/about/agent-substrate/ + - /kagent/main/about/agent-substrate/ --- {{< reuse "kagent-docs/snippets/about/agent-substrate-intro.md" >}} diff --git a/docs-site/content/kagent/main/about/architecture/kagent.md b/docs-site/content/kagent/main/about/architecture/kagent.md new file mode 100644 index 00000000..f90133c5 --- /dev/null +++ b/docs-site/content/kagent/main/about/architecture/kagent.md @@ -0,0 +1,79 @@ +--- +title: kagent architecture +description: See how a Harness and AgentTemplate become a running conversation, across kagent's two authorization planes. +weight: 30 +author: kagent.dev +--- + +The previous page defined the [core concepts]({{< link path="about/core-concepts" >}}) of Harness, AgentTemplate, Agent, Session, and Actor. This page connects them into one system. It traces how applying those three custom resources leads to a running conversation. It also marks which parts of that path Kubernetes governs, and which parts kagent governs itself. + +## Two authorization planes + +kagent 1.0 splits authorization across two planes: + +- The **Kubernetes plane** governs the {{< gloss "Harness" >}}Harness{{< /gloss >}} and {{< gloss "AgentTemplate" >}}AgentTemplate{{< /gloss >}} custom resources. Kubernetes Role-Based Access Control (RBAC) decides who can create, read, or edit the resources with `kubectl`, exactly as it would for any other Custom Resource Definition (CRD). +- The **kagent control plane** governs any interactions involving {{< gloss "Session" >}}Sessions{{< /gloss >}}, such as creating, sharing, or deleting a Session, or holding a conversation with one. kagent's own gRPC authentication and authorization decide who can complete these interactions, independent of Kubernetes RBAC. + +Someone with Kubernetes RBAC access to apply an Agent and the resources it names does not automatically have access to create or talk to Sessions on it. The separation runs in one direction only. kagent's gRPC API also writes those Kubernetes resources, so a caller on the kagent control plane reaches both planes. For more information on that second path, see [Identity]({{< link path="about/substrate-runtime/identity#the-kubernetes-plane" >}}). + +The following diagram shows where the boundary between the two planes falls. +

+ +```mermaid +flowchart TB + + subgraph k8s["Kubernetes plane (RBAC)"] + operator["Operator
kubectl apply"] + harness["Harness"] + template["AgentTemplate"] + agent["Agent"] + controller["kagent controller"] + operator --> harness + operator --> template + operator --> agent + harness --> agent + template --> agent + agent --> controller + end + + %% Declared outside both subgraphs on purpose. An ActorTemplate is a Substrate + %% resource reached over gRPC, not a Kubernetes object, so it belongs to + %% neither plane. A node joins whichever subgraph first references it, so both + %% of its edges have to live out here too. + actortemplate["ActorTemplate (Substrate)"] + controller -->|compiles the Agent into| actortemplate + + subgraph kagentplane["kagent control plane (gRPC auth)"] + caller["Caller"] + gateway["A2A gateway"] + session["Session"] + actor["Actor (Substrate)"] + caller -->|A2A conversation| gateway + caller -->|CreateSession| session + gateway -->|routes to| actor + session -->|runs on| actor + end + + actortemplate -->|instantiated as| session + %% Invisible link: forces the kagent control plane to sit fully below the ActorTemplate, + %% and the ActorTemplate below the Kubernetes plane. Without it the layout engine + %% staggers the two planes diagonally, which both wastes width and scrambles the + %% reading order. Anchor it to actortemplate, not controller: anchoring higher + %% loses the stacking. Verified by rendering. + actortemplate ~~~ caller + + classDef crd stroke:#a78bfa,stroke-width:2px + class harness,template crd +``` + +Follow the **Kubernetes plane** first. An operator applies a Harness, an AgentTemplate, and an Agent that pairs them with `kubectl`, governed by Kubernetes RBAC. The diagram shows this path because RBAC governs it, and kagent's gRPC API reaches the same resources instead. The kagent controller watches each Agent, resolves the template and harness that it names, and compiles the result into an {{< gloss "ActorTemplate" >}}ActorTemplate{{< /gloss >}} on Substrate. The ActorTemplate sits outside both planes in the diagram. It is a Substrate resource that the controller creates over gRPC, rather than a Kubernetes object. No Kubernetes role grants access to it. + +The **kagent control plane** starts once that ActorTemplate exists. A caller, who might be the same person as the operator, calls `CreateSession` through kagent's gRPC API. This call is governed by kagent's own authentication and authorization, not by Kubernetes RBAC. kagent creates the Session from the Agent's latest successful revision. That Session runs on an {{< gloss "Actor" >}}Actor{{< /gloss >}}. + +From there, the caller holds a conversation over the {{< gloss "A2A" >}}A2A{{< /gloss >}} (Agent2Agent) protocol. A caller addresses the Agent, and names the conversation with the Session's ID as the A2A `contextId`. The A2A gateway routes each request to the Actor running behind that Session. This means that the caller only ever needs to know the Agent's name and the Session's ID, never which Actor or {{< gloss "Worker" >}}Worker{{< /gloss >}} is behind it. + +## Why two planes + +Kubernetes RBAC is designed to authorize configuration changes: who can create a Deployment, edit a ConfigMap, or in this case, apply a Harness, an AgentTemplate, or an Agent. It is not designed to authorize a running conversation, share access to it with another user, or scope who can read, rename, or delete it. kagent's gRPC plane exists to authorize exactly those actions, at the granularity of a single Session rather than a namespace or a resource kind. + +This split also keeps the two lifecycles independent. Editing an Agent, or either resource it names, does not affect Sessions already running against the ActorTemplate that they were created from. It only affects new Sessions, created after the edit is compiled. diff --git a/docs-site/content/kagent/main/about/core-concepts.md b/docs-site/content/kagent/main/about/core-concepts.md new file mode 100644 index 00000000..95c9711e --- /dev/null +++ b/docs-site/content/kagent/main/about/core-concepts.md @@ -0,0 +1,174 @@ +--- +title: Core concepts +description: Learn the Harness, AgentTemplate, Agent, Session, and Actor model that kagent 1.0 is built on. +weight: 20 +author: kagent.dev +--- + +## kagent 1.0 + +{{< reuse "kagent-docs/snippets/name-product.md" >}} 1.0 replaces 0.x's Deployment-based `Agent` custom resource with a new model built around **Harness**, **AgentTemplate**, **Agent**, and **Session**. The 1.0 `Agent` kind keeps the 0.x name but not its meaning, because it pairs an AgentTemplate with a Harness rather than describing a Deployment. This page defines the vocabulary that the rest of the 1.0 documentation uses. + +> [!NOTE] +> Moving a 0.x installation to 1.0 is a manual migration, and no upgrade path exists. No controller converts a 0.x resource, and the two API groups share no conversion, so you author the 1.0 resources yourself and retire the 0.x ones. [Migrate from 0.x]({{< link path="operations/upgrade-from-0x#recreate-your-resources" >}}) maps each 0.x resource onto its 1.0 replacement. + +The new model separates what an agent can do from how it is allowed to run, and then names the pairing explicitly: + +- A [**Harness**](#harness) resource defines how an agent is allowed to run. It names the harness that executes the agent, and the infrastructure policy around it. +- An [**AgentTemplate**](#agenttemplate) defines what an agent can do: its model, prompt, and tools. +- An [**Agent**](#agent) pairs one AgentTemplate with one Harness. Callers address the Agent, and the controller compiles it. +- A [**Session**](#session) is a running conversation with an Agent. +- An [**Actor**](#actor) is the sandboxed process, provided by Substrate, that a Session runs on. + +Agents do not run as Kubernetes Deployments. kagent runs every Session on [Agent Substrate]({{< link path="about/architecture/agent-substrate" >}}), a compute layer that starts capacity in advance and places sandboxed workloads onto it. An idle Actor is suspended to a {{< gloss "Snapshot" >}}snapshot{{< /gloss >}} and restored when the next message arrives, and a caller never sees the pause. + +Agent Substrate needs its own definition of the workload to run, so the kagent controller compiles each Agent into an **{{< gloss "ActorTemplate" >}}ActorTemplate{{< /gloss >}}**, the Substrate resource that an Actor is created from. + +The following diagram shows how an Agent becomes a running conversation. +

+ +```mermaid +flowchart LR + template["AgentTemplate
(CRD)"] + harness["Harness
(CRD)"] + agent["Agent
(CRD)"] + controller["kagent controller"] + actortemplate["ActorTemplate
(compiled, immutable)"] + session["Session
(gRPC + database)"] + actor["Actor
(Substrate)"] + + template --> agent + harness --> agent + agent --> controller + controller -->|compiles into| actortemplate + actortemplate -->|instantiated as| session + session -->|runs on| actor + + classDef crd stroke:#a78bfa,stroke-width:2px + class template,harness,agent crd +``` + +An operator applies the three custom resources directly. The kagent controller watches each Agent, resolves the template and harness that it names, and compiles the result into an ActorTemplate. From there, each Session created against that Agent gets its own Actor to run on. + +> [!IMPORTANT] +> All three resources belong to the `api.kagent.dev` API group, which keeps them separate from the `kagent.dev` resources that 0.x serves, including 0.x's own `Agent` kind. The two groups share no conversion. On a cluster that serves both, qualify the resource name as `kubectl get agents.api.kagent.dev` to select the 1.0 API. + +## Harness + +An agent harness is the software around a model that turns the model into an agent. LangChain defines a harness as "every piece of code, configuration, and execution logic that isn't the model itself," in [The Anatomy of an Agent Harness](https://www.langchain.com/blog/the-anatomy-of-an-agent-harness). A harness typically assembles context, invokes the model, executes tool calls, loads skills, feeds results back into the agent loop, and manages session state. A harness can also provide memory, subagents, permissions, approvals, sandboxing, and event streaming. + +Examples include [Claude Code](https://code.claude.com/docs/en/how-claude-code-works), [Codex CLI](https://learn.chatgpt.com/docs/codex/cli), [Gemini CLI](https://geminicli.com/docs/), [OpenClaw](https://docs.openclaw.ai/concepts/agent-runtimes), and custom in-house applications. Each of those runs on a local machine with your own shell and files. + +A **Harness** in kagent is a Kubernetes custom resource that applies that idea to a cluster. It defines _how an agent is allowed to run_, and specifies: + +- **Execution**: What runs the agent. A Harness selects exactly one of `kagent`, `codex`, `claude`, or `byo`. `kagent` runs kagent's own agent engine, `codex` runs Codex, `claude` runs Claude, and `byo` runs any image that exposes kagent's A2A contract. Each accepts different model providers and different AgentTemplate features. +- **Workload**: The container image, command, and arguments that the agent process runs as. +- **Environment**: Literal environment values for the agent process, set in `spec.env`. +- **Substrate policy**: The capacity that the agent runs on, and where its state is kept when it goes idle. The Harness names a [WorkerPool]({{< link path="about/architecture/agent-substrate#workers-and-workerpools" >}}) to schedule onto and the storage location that snapshots write to. + +A Harness and an AgentTemplate do not reference each other. An [Agent](#agent) references both, and nothing pairs them implicitly. Applying a Harness runs nothing on its own. It registers a runtime and policy that an Agent can select. + +Each harness accepts a different subset of configuration, and an Agent that asks for something its harness cannot do reports a compatibility condition rather than running. For the per-harness restrictions and the complete Harness schema, see the [API reference]({{< link path="reference/api-ref#harness" >}}). + +> [!IMPORTANT] +> This `Harness` is unrelated to 0.x's `AgentHarness` resource, which provisions OpenClaw or Hermes coding-agent sandboxes. `Harness` is a different resource that covers how any agent is allowed to run, not a renamed or expanded version of `AgentHarness`. + +## AgentTemplate + +An **AgentTemplate** is a Kubernetes custom resource that defines _what an agent does_. It specifies: + +- **Model configuration**: The large language model (LLM) provider and model the agent uses. +- **System prompt**: A literal prompt, or a Go-templated one that can include shared ConfigMaps. +- **Tools**: A list of {{< gloss "Tool binding" >}}tool bindings{{< /gloss >}} that the agent can call. Each binding is either a {{< gloss "Model Context Protocol" >}}Model Context Protocol{{< /gloss >}} (MCP) server, or another AgentTemplate used as a subagent tool (see [Subagent tools](#subagent-tools)). +- **Skills** and **plugins**: Reusable capability packages, sourced from an Open Container Initiative (OCI) registry, Git, or S3. + +An AgentTemplate does nothing on its own. It becomes runnable once an Agent pairs it with a Harness. One AgentTemplate can serve many Agents, so expect fewer templates than Agents where several runtimes run the same behavior. + +For the complete AgentTemplate schema, see the [API reference]({{< link path="reference/api-ref#agenttemplate" >}}). + +## Agent + +An **Agent** is a Kubernetes custom resource that pairs _one AgentTemplate with one Harness_. Both sides are required. Under `spec.template`, the Agent sets exactly one of `ref`, which names an existing AgentTemplate, or `inline`, which carries a complete template spec. Under `spec.harnessRef`, the Agent names an existing Harness. A Harness has no inline form. + +An inline template is a complete definition rather than an override of a named AgentTemplate, and kagent does not create a Kubernetes resource to back one. Every name resolves in the Agent's own namespace, including a name nested inside an inline template. For the fields that express each choice, see the [API reference]({{< link path="reference/api-ref#agent" >}}). + +The following Agent resource references both sides: + +```yaml +apiVersion: api.kagent.dev/v1alpha3 +kind: Agent +metadata: + name: assistant + namespace: kagent +spec: + template: + ref: + name: shared-context + harnessRef: + name: kagent +``` + +The Agent owns readiness. Its status reports the desired revision, the latest revision that was built and prepared successfully, any non-blocking compatibility warnings, and the `Accepted`, `ResolvedRefs`, `Compatible`, and `Ready` conditions. An AgentTemplate and a Harness are shared configuration and do not report runtime status of their own. + +Deleting an Agent retires its definition. Existing Sessions keep the revisions that they pinned. Recreating an Agent under the same name creates a new identity, which cannot inherit the deleted Agent's last successful revision. + +For the complete Agent schema, see the [API reference]({{< link path="reference/api-ref#agent" >}}). + +## Session + +A **Session** is a _running conversation with one Agent_. Unlike the Harness, AgentTemplate, and Agent that it is built from, a Session is not a Kubernetes custom resource and does not live in etcd. kagent's own gRPC API creates it, and kagent's PostgreSQL database tracks it. + +{{< reuse "kagent-docs/snippets/about/core-concepts-session-authorization.md" >}} + +Each compile produces one **{{< gloss "Revision" >}}revision{{< /gloss >}}**, identified by a digest, which is a SHA-256 hash of the compiled configuration. Because that digest is derived from the configuration itself, editing an Agent or either resource it references compiles to a different digest, and therefore becomes a separate ActorTemplate. kagent never rewrites an existing one. + +That immutability keeps running Sessions stable. When you create a Session, kagent looks up the Agent's latest successful revision and creates an Actor from it. Editing the Agent afterward does not move that Session, so only Sessions created after the edit use the new revision. + +You can create a Session with the following command: + +```bash +kagent agent session create --agent assistant -n kagent +``` + +After it is created, a Session talks to callers over the {{< gloss "A2A" >}}A2A{{< /gloss >}} (Agent2Agent) protocol, through kagent's A2A gateway. Callers address the Agent rather than the Session. The HTTP endpoint is `/agents/{namespace}/{name}` and gRPC carries the same `namespace/name` in the standard A2A `tenant` field. The Session's ID is the A2A `contextId`, so a message that does not include a context identifier starts a new conversation, and a message that repeats one continues that conversation. + +An expiration worker deletes a Session that does not record task activity for seven days. The `config.sessions.idleTtl` Helm value sets that window. A value of `0s` turns the expiration worker off. For more information about what the deletion retains, see [Expire idle Sessions]({{< link path="operations/operational-considerations#expire-idle-sessions" >}}). + +To delete a Session yourself, run `kagent agent session delete `. kagent deletes a Session only when no turn or other lifecycle work, such as a resume, is in progress. To delete a Session during an active turn, add the `--force` flag. A forced deletion cancels the active turn, and records its task as canceled with the message `Session deleted while the turn was active.` + +## Actor + +An **Actor** is the sandboxed unit of compute, provided by [Agent Substrate]({{< link path="about/architecture/agent-substrate" >}}), that _runs a Session's conversation loop_. Every Session is backed by an Actor. + +An Actor lets a Session suspend and resume cheaply instead of staying resident. To understand the {{< gloss "Worker" >}}Workers{{< /gloss >}} and {{< gloss "WorkerPool" >}}WorkerPools{{< /gloss >}} that Actors run on, see [Agent Substrate architecture]({{< link path="about/architecture/agent-substrate" >}}). + +## Compute, identity, and isolation + +The Harness, AgentTemplate, and Agent resources describe what an agent is. The Actor behind each Session decides how that agent runs. An Actor supplies the agent's compute, its network identity, and the sandbox that confines it. + +### Compute + +An Actor occupies a slot on a Worker, which is a pre-started pod in a WorkerPool. A Worker hosts many Actors at once, and an Actor that goes idle is suspended to a snapshot so that its slot returns to the pool. A pool therefore serves far more Sessions than it has Workers. Agent Substrate also pauses an Actor to a node-local snapshot, and resumes an Actor onto whichever Worker has room. For these operations and what each snapshot keeps, see [Suspend and resume]({{< link path="about/substrate-runtime/suspend-and-resume" >}}). + +Starting a second Session from a {{< gloss "Checkpoint" >}}checkpoint{{< /gloss >}} is known as a fork. For what a fork replays and where it diverges, see [Checkpoints]({{< link path="about/substrate-runtime/suspend-and-resume#checkpoints" >}}). + +### Identity + +Each Actor has its own cryptographic identity, issued as a certificate and scoped to the {{< gloss "Atespace" >}}atespace{{< /gloss >}} that it belongs to. Agent Substrate's egress gateway authorizes every outbound connection against the identity on the certificate rather than against anything in the request. Because of this, an agent cannot reach a destination by claiming to be another agent. + +This identity also keeps API keys out of the agent. An agent sends a placeholder header, and the gateway replaces it with the real credential as the request passes through, so the key never enters the agent's environment or its filesystem. For the allowlist that the gateway enforces and how credentials are attached to it, see [Egress control]({{< link path="about/substrate-runtime/networking-and-egress" >}}). + +### Isolation + +Each Actor runs in its own sandbox, with its own filesystem and its own durable volume for application data. Two Sessions on the same Agent share no files and cannot see any of each other's state, even when they run on the same Worker. The sandbox also isolates the workload's system calls from the node's kernel, which matters because an agent runs commands that a model chose. For what each sandbox class separates, see [Sandboxing]({{< link path="about/substrate-runtime/sandboxing" >}}). + +## Subagent tools + +An agent tool binding can point at another AgentTemplate instead of an MCP server, so a broad-scope agent can hand part of a task to a specialist. For example, a release agent delegates a database question to another agent, gets the answer, and carries on. The specialist keeps its own prompt and its own tools, and the conversation stays with the agent that the caller addressed. + +Each subagent binding sets `tools[].subAgent.templateRef` to reference an AgentTemplate in the same namespace. The named template compiles under the parent Agent's Harness and runs inside the parent's Actor, so the two agents share one sandbox and the nesting does not create a second Actor. A subagent does not need an Agent of its own or a matching Harness reference. + +Because a subagent runs inside its parent's runtime boundary, nesting stops at one level, and a bound template cannot bind a third. Every agent therefore runs either in its own Actor or in the Actor of the agent that bound it, never deeper. For the remaining rules that a tree must meet, see [Subagent tree rules]({{< link path="skills-and-mcp/about-tools#subagent-tree-rules" >}}). + +> [!NOTE] +> Dedicated subagents, which would give a bound agent its own Harness, Session, and Actor and reach it over A2A, are not part of the served API. The `subAgent.agentRef` field that would select one is unavailable. diff --git a/docs-site/content/kagent/1.x/about/substrate-runtime/_index.md b/docs-site/content/kagent/main/about/substrate-runtime/_index.md similarity index 82% rename from docs-site/content/kagent/1.x/about/substrate-runtime/_index.md rename to docs-site/content/kagent/main/about/substrate-runtime/_index.md index fd10cbe4..0d5cb402 100644 --- a/docs-site/content/kagent/1.x/about/substrate-runtime/_index.md +++ b/docs-site/content/kagent/main/about/substrate-runtime/_index.md @@ -4,5 +4,5 @@ description: Learn how Agent Substrate isolates, suspends, and resumes the Actor weight: 60 author: kagent.dev aliases: - - /kagent/1.x/substrate-runtime/ + - /kagent/main/substrate-runtime/ --- \ No newline at end of file diff --git a/docs-site/content/kagent/1.x/about/substrate-runtime/identity.md b/docs-site/content/kagent/main/about/substrate-runtime/identity.md similarity index 99% rename from docs-site/content/kagent/1.x/about/substrate-runtime/identity.md rename to docs-site/content/kagent/main/about/substrate-runtime/identity.md index d893c970..6dfc0cd9 100644 --- a/docs-site/content/kagent/1.x/about/substrate-runtime/identity.md +++ b/docs-site/content/kagent/main/about/substrate-runtime/identity.md @@ -4,7 +4,7 @@ description: Understand how kagent identifies each caller, scopes a Session to i weight: 30 author: kagent.dev aliases: - - /kagent/1.x/substrate-runtime/identity/ + - /kagent/main/substrate-runtime/identity/ --- A {{< reuse "kagent-docs/snippets/name-product.md" >}} installation identifies each kind of caller with a different system. diff --git a/docs-site/content/kagent/1.x/about/substrate-runtime/networking-and-egress.md b/docs-site/content/kagent/main/about/substrate-runtime/networking-and-egress.md similarity index 99% rename from docs-site/content/kagent/1.x/about/substrate-runtime/networking-and-egress.md rename to docs-site/content/kagent/main/about/substrate-runtime/networking-and-egress.md index 33905e9c..5133f152 100644 --- a/docs-site/content/kagent/1.x/about/substrate-runtime/networking-and-egress.md +++ b/docs-site/content/kagent/main/about/substrate-runtime/networking-and-egress.md @@ -4,7 +4,7 @@ description: Learn how Agent Substrate authorizes the outbound connections that weight: 40 author: kagent.dev aliases: - - /kagent/1.x/substrate-runtime/networking-and-egress/ + - /kagent/main/substrate-runtime/networking-and-egress/ --- An agent reaches the network only through Agent Substrate's egress gateway, and the gateway authorizes every connection against a policy attached to that {{< gloss "Actor" >}}Actor{{< /gloss >}}. The policy is default-deny. An Actor can attempt any request, and the gateway denies the ones that no rule allows. kagent derives the policy from the AgentTemplate, so a documented agent works without any egress configuration of its own. An Agent can also select EgressPolicy resources, which grant destinations that the AgentTemplate does not name. For more information, see [Egress policies]({{< link path="agents/egress-policies" >}}). diff --git a/docs-site/content/kagent/1.x/about/substrate-runtime/sandboxing.md b/docs-site/content/kagent/main/about/substrate-runtime/sandboxing.md similarity index 99% rename from docs-site/content/kagent/1.x/about/substrate-runtime/sandboxing.md rename to docs-site/content/kagent/main/about/substrate-runtime/sandboxing.md index 1e98c4df..e1547383 100644 --- a/docs-site/content/kagent/1.x/about/substrate-runtime/sandboxing.md +++ b/docs-site/content/kagent/main/about/substrate-runtime/sandboxing.md @@ -4,7 +4,7 @@ description: Learn how Agent Substrate isolates each Actor in its own sandbox, a weight: 10 author: kagent.dev aliases: - - /kagent/1.x/substrate-runtime/sandboxing/ + - /kagent/main/substrate-runtime/sandboxing/ --- An agent runs an **agentic loop**. A model proposes the next step, the agent carries it out by running a command or calling a tool, and the result feeds the model's next decision. The loop repeats until the task finishes or the agent stops to wait on input from a person. diff --git a/docs-site/content/kagent/1.x/about/substrate-runtime/standalone-sandboxes.md b/docs-site/content/kagent/main/about/substrate-runtime/standalone-sandboxes.md similarity index 99% rename from docs-site/content/kagent/1.x/about/substrate-runtime/standalone-sandboxes.md rename to docs-site/content/kagent/main/about/substrate-runtime/standalone-sandboxes.md index 3218ab39..6a4527a2 100644 --- a/docs-site/content/kagent/1.x/about/substrate-runtime/standalone-sandboxes.md +++ b/docs-site/content/kagent/main/about/substrate-runtime/standalone-sandboxes.md @@ -4,7 +4,7 @@ description: Run commands and transfer files in a scratch environment that is in weight: 25 author: kagent.dev aliases: - - /kagent/1.x/substrate-runtime/standalone-sandboxes/ + - /kagent/main/substrate-runtime/standalone-sandboxes/ --- A **Sandbox** is a scratch environment for running commands and working with files. A Sandbox is scoped to the caller who created it, and it expires on a timer. You create one from a **SandboxTemplate**, run processes and move files in it, then delete it or let it expire. diff --git a/docs-site/content/kagent/1.x/about/substrate-runtime/suspend-and-resume.md b/docs-site/content/kagent/main/about/substrate-runtime/suspend-and-resume.md similarity index 99% rename from docs-site/content/kagent/1.x/about/substrate-runtime/suspend-and-resume.md rename to docs-site/content/kagent/main/about/substrate-runtime/suspend-and-resume.md index 31a78a64..ec88146c 100644 --- a/docs-site/content/kagent/1.x/about/substrate-runtime/suspend-and-resume.md +++ b/docs-site/content/kagent/main/about/substrate-runtime/suspend-and-resume.md @@ -4,7 +4,7 @@ description: Learn how Agent Substrate suspends idle Actors to snapshots and res weight: 20 author: kagent.dev aliases: - - /kagent/1.x/substrate-runtime/suspend-and-resume/ + - /kagent/main/substrate-runtime/suspend-and-resume/ --- An agent spends most of its life waiting. It waits on a person to reply, and it waits on a large language model (LLM) to answer. [Agent Substrate]({{< link path="about/architecture/agent-substrate" >}}) runs each agent inside an **Actor**, the unit that Agent Substrate suspends and resumes. To free compute resources while an agent waits, Agent Substrate writes the idle Actor to a {{< gloss "Snapshot" >}}snapshot{{< /gloss >}} and releases the capacity that the Actor held on its {{< gloss "Worker" >}}Worker{{< /gloss >}}. The next message to the agent restores the Actor. The {{< gloss "Session" >}}Session{{< /gloss >}} stays ready the whole time, so reclaiming this capacity costs the user nothing. This page explains what a snapshot captures, when kagent suspends an Actor, and what happens when a suspended Actor is addressed again. diff --git a/docs-site/content/kagent/main/about/what-is-kagent.md b/docs-site/content/kagent/main/about/what-is-kagent.md new file mode 100644 index 00000000..e1342092 --- /dev/null +++ b/docs-site/content/kagent/main/about/what-is-kagent.md @@ -0,0 +1,49 @@ +--- +title: What is kagent? +description: Understand what the kagent platform is and its core purpose. +weight: 10 +author: kagent.dev +--- + +{{< reuse "kagent-docs/snippets/about/what-is-kagent-intro.md" >}} + +## How kagent works + +{{< reuse "kagent-docs/snippets/about/what-is-kagent-how-it-works.md" >}} + +## Core model + +kagent 1.0 separates an agent's capabilities from its runtime, then runs the two together as a conversation: + +- A **Harness** and an **AgentTemplate** are the Kubernetes custom resources you author. Together they say how an agent is allowed to run and what it can do. +- An **Agent** pairs one of each, and a **Session** is a running conversation with that Agent, backed by an **Actor** on Agent Substrate. + +[Core concepts]({{< link path="about/core-concepts" >}}) define each of these in detail, and the [architecture]({{< link path="about/architecture/kagent" >}}) walks through how they connect end to end. + +{{< reuse "kagent-docs/snippets/about/what-is-kagent-authorization.md" >}} + +## Benefits + +{{< reuse "kagent-docs/snippets/about/what-is-kagent-benefits.md" >}} + +## Platform features + +{{< feature-cards >}} +{{< feature-card title="Agent lifecycle via CRDs" desc="Define, version, and roll out Harnesses and AgentTemplates with kubectl and GitOps, the same workflow as every other workload." >}} +{{< feature-card title="Sandboxed by default" desc="Every Session runs on a Substrate Actor, isolated from the host kernel by a gVisor sandbox. Run untrusted, model-directed code safely." >}} +{{< feature-card title="Suspend and resume" desc="Idle Sessions suspend and free their compute, then resume on demand. Run far more agents than you have capacity for at any one moment." >}} +{{< feature-card title="Checkpoints" desc="Pin a snapshot of a conversation at a turn boundary so that Agent Substrate keeps the state for inspection and retention." >}} +{{< feature-card title="Pluggable agent runtimes" desc="A Harness selects the engine behind an agent: kagent's own Go engine, Codex, Claude, or any image of your own that speaks kagent's A2A contract." >}} +{{< feature-card title="Tools over MCP" desc="Bind an agent to any Model Context Protocol (MCP) server with a RemoteMCPServer resource. An installation registers two servers already, including 124 tools for Kubernetes, Helm, Istio, and Argo Rollouts." >}} +{{< feature-card title="Agent tools" desc="Bind another AgentTemplate as a tool, so an agent can hand work to a specialist. A Shared binding nests that agent inside its parent's Actor, one level deep." >}} +{{< feature-card title="Long-term memory" desc="Persistent, vector-backed memory across sessions. Agents remember context, not just the last prompt." >}} +{{< feature-card title="Human-in-the-loop" desc="Tool approval gates and agent-initiated questions keep a person in control of consequential actions." >}} +{{< feature-card title="Agent2Agent (A2A)" desc="Agents talk to callers, and to each other, over the A2A protocol." >}} +{{< feature-card title="Skills and plugins" desc="Load skills and plugins from an Open Container Initiative (OCI) registry, Git, or S3 at startup." >}} +{{< feature-card title="Prompt templates" desc="Reusable prompt fragments stored as ConfigMaps. Keep system prompts consistent across agents." >}} +{{< feature-card title="Observability" desc="OpenTelemetry tracing, structured logs, and an optional Prometheus metrics endpoint, with control plane traces carrying the Actor that they belong to." >}} +{{< /feature-cards >}} + +{{< reuse "kagent-docs/snippets/about/what-is-kagent-enterprise.md" >}} + +{{< reuse "kagent-docs/snippets/about/what-is-kagent-community.md" >}} diff --git a/docs-site/content/kagent/main/agents/_index.md b/docs-site/content/kagent/main/agents/_index.md new file mode 100644 index 00000000..f37a52a6 --- /dev/null +++ b/docs-site/content/kagent/main/agents/_index.md @@ -0,0 +1,6 @@ +--- +title: Agents +description: Configure how an agent runs and what it knows, from the harness behind it to its prompt and memory. +weight: 40 +author: kagent.dev +--- diff --git a/docs-site/content/kagent/main/agents/agent-harness.md b/docs-site/content/kagent/main/agents/agent-harness.md new file mode 100644 index 00000000..00ec5152 --- /dev/null +++ b/docs-site/content/kagent/main/agents/agent-harness.md @@ -0,0 +1,424 @@ +--- +title: Agent harness +description: Configure a Harness, the resource that defines what executes an agent and what infrastructure the agent runs on. +weight: 10 +author: kagent.dev +--- + +A Harness defines how an agent is allowed to run. It names what executes the agent, the container image to start from, and the Agent Substrate capacity and snapshot storage that the agent runs on. A Harness runs nothing by itself. To run an agent, create an Agent that references a Harness and an AgentTemplate, which supplies the system prompt, the model, and the tools. + +This page covers the Harness schema, the four harnesses, and what each one supports. For the concept and how it relates to the other resources, see the [core concepts]({{< link path="about/core-concepts#harness" >}}). + +## Configure a Harness + +The following configuration shows a complete Harness resource. Only `substrate` and one harness block are required. The `byo` harness also requires `workload.image` and `workload.command`. + +```yaml +kubectl apply -f - <}} + env: + - name: OTEL_BSP_SCHEDULE_DELAY + value: "2000" + substrate: + workerPoolRef: + name: kagent-default + snapshotPolicy: + location: s3://ate-snapshots/kagent/ +EOF +``` + +{{< reuse "kagent-docs/snippets/review-table.md" >}} For more information, see the [API reference]({{< link path="reference/api-ref#harness" >}}). + +| Field | Required | Description | +| ----- | -------- | ----------- | +| One of `kagent`, `codex`, `claude`, `byo` | Yes | The harness that executes the agent. Naming none, or more than one, is rejected. To compare the four harnesses, see [Choose a harness](#choose-a-harness). | +| `workload.image` | For `byo` | The container image to run, pinned by `sha256` digest. A tag with no digest is rejected, because a revision must be reproducible. Omit it on a `kagent`, `codex`, or `claude` Harness to run the built-in image of the installed release. For more information, see [Built-in runtime images](#built-in-runtime-images). | +| `workload.command` | For `byo` | Overrides the image entrypoint, up to 32 entries. Required for the `byo` harness, optional otherwise. Every harness honors an explicit value, the `kagent` harness included, regardless of the language its image is written in. | +| `workload.args` | No | Overrides the image arguments, up to 64 entries. An override that you omit stays unset rather than taking a default. | +| `env` | No | Environment variables for the agent process. Each entry sets a literal `value`, which is required and can be an empty string. The schema does not define a secret-backed source, so the API server rejects a `credentialRef` entry as an unknown field. Put credentials in a ModelConfig or a RemoteMCPServer instead. For more information, see [About model providers]({{< link path="setup/model-providers/about-model-providers#credentials-that-kagent-rejects" >}}). Limits and reserved names both apply. For more information, see [Environment variables on an Actor](#environment-variables-on-an-actor). | +| `substrate.workerPoolRef.name` | Yes | The {{< gloss "WorkerPool" >}}WorkerPool{{< /gloss >}} that this Harness's Actors are scheduled onto. An operator must provision one before any agent can run. | +| `substrate.snapshotPolicy.location` | Yes | The object storage location for Actor {{< gloss "Snapshot" >}}snapshots{{< /gloss >}}. | + +A Harness and an AgentTemplate do not reference each other. An {{< gloss "Agent" >}}Agent{{< /gloss >}} references both, through `spec.harnessRef` and `spec.template.ref`, so whoever writes the Agent decides which template runs on which Harness. Both fields take a reference only. The Harness must therefore exist before an Agent can name it. For more information about the pairing, see [Core concepts]({{< link path="about/core-concepts#agent" >}}). + +The image, command, arguments, environment, and resolved AgentTemplate together form a {{< gloss "Revision" >}}revision{{< /gloss >}}, which kagent identifies by a digest of that content. Editing any of them produces a new revision at the next reconcile. A {{< gloss "Session" >}}Session{{< /gloss >}} runs the revision that it was created from until that Session ends, so only Sessions that start after the edit use the new revision. + +### Built-in runtime images + +Each kagent release ships a runtime image for the `kagent`, `codex`, and `claude` harnesses, pinned by digest. If a Harness of one of these types omits `workload.image`, it runs the image of the installed release. If the Harness explicitly sets `workload.image`, this image overrides the release image. After the Agent compiles, `status.workloadImage` reports the image that the Agent's desired revision runs. Existing Sessions keep both the revision and the image that they were created from. + +To read the image that an Agent runs, check its status. + +```bash +kubectl get agent my-agent -n kagent -o jsonpath='{.status.workloadImage}' +``` + +To replace the release defaults for the whole installation, set `config.harnessImages` in your Helm values, such as in the following example. Your values merge over the image catalog that the chart packages. `global.imageRegistry` rewrites both the packaged defaults and your overrides, so a mirrored registry needs no further configuration. + +```yaml +config: + harnessImages: + kagent: registry.example.com/kagent/golang-adk@sha256: + codex: registry.example.com/kagent/codex-harness@sha256: + claude: registry.example.com/kagent/claude-harness@sha256: +``` + +The `kagent`, `codex`, and `claude` fields set the digest-pinned default image for each harness. Omit a key to keep the release default for that harness. An empty string disables the default, and a Harness of that type must then set `workload.image`. + +The chart also sets `config.harnessImages.release` to the release that the packaged images belong to. When that value does not match the controller's release, the controller fails at startup. + +## Choose a harness + +A Harness resource names exactly one of the following four harnesses, and that choice decides what executes an agent and how much of kagent's feature set the agent can use. + +| Harness | What it runs | When to use it | +| ------- | ------------ | ----------- | +| `kagent` | kagent's built-in Go engine | You want to define the agentic loop through configuration, with a system prompt, skills, plugins, and MCP servers. This harness also supports the widest set of features. For more information, see [Declarative agents]({{< link path="agents/declarative-agents" >}}). | +| `codex` | Codex | You want Codex to do the work, and your model is OpenAI or an OpenAI-compatible Bedrock deployment. | +| `claude` | Claude | You want Claude to do the work, with Anthropic or Bedrock as the model. | +| `byo` | A container image of your own that implements kagent's A2A contract | You already have an agent that is built with an agent framework, and you want to run that agent in kagent. For more information, see [Bring your own agent]({{< link path="agents/bring-your-own-agent" >}}). | + +The harnesses accept different model providers, which [Model provider support](#model-provider-support) lists, and they accept different AgentTemplate features, which [Tool and skill support](#tool-and-skill-support) lists. + +The agent chat in the kagent UI offers file attachments only on the `kagent` harness. An agent on a `byo`, `codex`, or `claude` Harness cannot attach files in the chat. For the file types and size limits that the UI enforces, see [Attach files to a message]({{< link path="ui/launch-ui#attach-files-to-a-message" >}}). + +### kagent harness + +The `kagent` harness runs kagent's built-in Go engine, which reads the agent configuration and runs the agentic loop from it. Choose the `kagent` harness to build an agent from a system prompt, tools, skills, and plugins. Its own settings live in `spec.kagent`, which takes a long-term memory block and a context compaction block. For those settings, a complete example, and how this route compares with a BYO image, see [Declarative agents]({{< link path="agents/declarative-agents" >}}). + +### Codex harness + +The `codex` harness runs Codex against an OpenAI or Bedrock model. The harness block is empty. Codex accepts a narrower set of AgentTemplate features than the built-in engine. To understand the restrictions, see [Tool and skill support](#tool-and-skill-support). + +The ModelConfig must set `openAI.apiFormat: responses`, and the API key must be entitled to the model that you name. A key without access to that model fails the turn with `Codex execution failed`. The ModelConfig in the following example reads its API key from the `kagent-openai` Secret that the [OpenAI]({{< link path="setup/model-providers/openai#create-the-modelconfig" >}}) guide creates. + +```yaml +kubectl apply -f - <}} + substrate: + workerPoolRef: + name: kagent-default + snapshotPolicy: + location: s3://ate-snapshots/kagent/ +--- +apiVersion: api.kagent.dev/v1alpha3 +kind: AgentTemplate +metadata: + name: codex-template + namespace: kagent +spec: + modelConfig: + name: codex-model-config + systemPrompt: You are a concise, helpful coding assistant. +--- +apiVersion: api.kagent.dev/v1alpha3 +kind: Agent +metadata: + name: codex-agent + namespace: kagent +spec: + template: + ref: + name: codex-template + harnessRef: + name: codex-harness +EOF +``` + +### Claude harness + +The `claude` harness runs Claude against an Anthropic or Bedrock model. The harness block is empty. Claude accepts a narrower set of AgentTemplate features than the built-in engine. To understand the restrictions, see [Tool and skill support](#tool-and-skill-support). + +The ModelConfig in the following example reads its API key from the `kagent-anthropic` Secret that the [Anthropic]({{< link path="setup/model-providers/anthropic#create-the-modelconfig" >}}) guide creates. + +```yaml +kubectl apply -f - <}} + substrate: + workerPoolRef: + name: kagent-default + snapshotPolicy: + location: s3://ate-snapshots/kagent/ +--- +apiVersion: api.kagent.dev/v1alpha3 +kind: AgentTemplate +metadata: + name: claude-template + namespace: kagent +spec: + modelConfig: + name: claude-model-config + systemPrompt: You are a concise, helpful coding assistant. +--- +apiVersion: api.kagent.dev/v1alpha3 +kind: Agent +metadata: + name: claude-agent + namespace: kagent +spec: + template: + ref: + name: claude-template + harnessRef: + name: claude-harness +EOF +``` + +### BYO harness + +The `byo` harness runs an agent that you built with an agent framework of your own, packaged as a container image. The harness block is empty, and `workload.command` is required, because kagent cannot infer the entrypoint of an image that it does not build. + +An image that reads the agent configuration accepts the same model providers as the `kagent` harness, and kagent rejects structured output on a `byo` Harness. An image that ignores the configuration decides everything itself, and its AgentTemplate needs only a name. For the contract that the image must implement, how to read the configuration, and the known limitations, see [Bring your own agent]({{< link path="agents/bring-your-own-agent" >}}). + +```yaml +kubectl apply -f - </my-agent@sha256: + command: ["/my-agent"] + substrate: + workerPoolRef: + name: kagent-default + snapshotPolicy: + location: s3://ate-snapshots/kagent/ +--- +apiVersion: api.kagent.dev/v1alpha3 +kind: AgentTemplate +metadata: + name: byo-template + namespace: kagent +spec: + description: An agent that runs from my own image. +--- +apiVersion: api.kagent.dev/v1alpha3 +kind: Agent +metadata: + name: byo-agent + namespace: kagent +spec: + template: + ref: + name: byo-template + harnessRef: + name: byo-harness +EOF +``` + +## Model provider support + +The harness that an agent runs on decides which ModelConfig it can use. The following table covers every value that the ModelConfig `provider` field accepts, including the three that no harness accepts with a credential Secret. + +| Provider | `kagent` | `byo` | `codex` | `claude` | +| -------- | :------: | :---: | :-----: | :------: | +| `OpenAI` | ✅ | ✅ | ✅ | ❌ | +| `Anthropic` | ✅ | ✅ | ❌ | ✅ | +| `Bedrock` | ✅ | ✅ | ✅ | ✅ | +| `AzureOpenAI` | ✅ | ✅ | ❌ | ❌ | +| `Gemini` | ✅ | ✅ | ❌ | ❌ | +| `Mistral` | ✅ | ✅ | ❌ | ❌ | +| `Ollama` | ✅ | ✅ | ❌ | ❌ | +| `Foundry` | ✅ | ✅ | ❌ | ❌ | +| `AnthropicVertexAI` | ❌ | ❌ | ❌ | ❌ | +| `GeminiVertexAI` | ❌ | ❌ | ❌ | ❌ | +| `SAPAICore` | ❌ | ❌ | ❌ | ❌ | + +kagent reaches a model provider through the egress gateway, which holds the credential and adds it to each request as an HTTP header. However, the following providers authenticate in a way that this design cannot serve: + +* `AnthropicVertexAI` and `GeminiVertexAI` authenticate with a Google service account key, which kagent would have to mount as a file. An Actor does not take volumes, so a ModelConfig that names a credential Secret is rejected with `ModelConfig requires volume mounts unsupported by Substrate ActorTemplate`. +* `SAPAICore` authenticates with an OAuth client ID and secret, which the runtime exchanges for a token itself. A ModelConfig that names a credential Secret is rejected with `environment credential "" cannot use gateway header injection`. + +Some supported combinations still carry restrictions. + +| Combination | Restriction | +| ----------- | ----------- | +| `codex` with `OpenAI` | Requires `openAI.apiFormat: responses`, and does not accept any other `openAI` setting beyond `baseUrl`. | +| `codex` with `Bedrock` | Accepts only OpenAI `gpt-*` model IDs, and does not accept any `bedrock` setting beyond `region`. | +| `claude` with `Anthropic` | Does not accept any `anthropic` setting beyond `baseUrl`. | +| `claude` with `Bedrock` | Does not accept any `bedrock` setting beyond `region`. | +| `Bedrock` on any harness | The Secret must hold an `AWS_BEARER_TOKEN_BEDROCK` key. A Secret of IAM access keys is rejected with `cannot use gateway header injection`, because IAM signs each request locally. | +| `codex` and `claude` with any provider | Neither harness accepts a ModelConfig that sets `defaultHeaders`, `tls`, or `tokenPassthrough`. | + +## Tool and skill support + +The `kagent` harness accepts every AgentTemplate feature. Codex and Claude run agentic loops of their own, so each constrains what an AgentTemplate can ask for. The `byo` harness takes the same set as the `kagent` harness, apart from structured output. + +| Constraint | Applies to | +| ---------- | ---------- | +| A subagent binding cannot itself carry tools, skills, plugins, or nested agents, and must use the same provider and credentials as the agent that binds it. | `codex`, `claude` | +| The agent sees every tool that an {{< gloss "MCP" >}}MCP{{< /gloss >}} server offers, because Claude does not support partial tool selection. A binding that names a subset of tools binds the whole server instead. The Agent still reconciles, and reports the decision in `status.warnings` as `Claude RemoteMCPServer "" does not support partial tool selection`. | `claude` | +| A RemoteMCPServer must use the `STREAMABLE_HTTP` protocol. `SSE` is rejected. | `codex` | +| A RemoteMCPServer that sets `tokenPassthrough` is rejected. | `codex`, `claude` | +| Structured output is rejected, with `Harness runtime "" does not support structured output`. | `codex`, `claude`, `byo` | + +For more information about what an AgentTemplate can bind, see [About tools]({{< link path="skills-and-mcp/about-tools" >}}). + +## Telemetry content settings + +By default, kagent's telemetry records that a model call happened, without the prompt or the reply. Two settings in the kagent Helm chart add that content, and both default to `false`. `captureContent` adds the prompts and assistant replies. `captureRawApiBodies` adds the complete provider request and response bodies. Neither takes effect until you enable tracing or log export, and anyone who can read your telemetry backend can read whatever you capture. + +```yaml +config: + runtimeDefaults: + telemetry: + captureContent: false + captureRawApiBodies: false + maxCaptureBytes: 16384 +``` + +Each setting reaches a different set of harnesses, and `maxCaptureBytes` bounds the size of what either capture setting records. + +| Setting | What it includes | Applies to | +| ------- | ---------------- | ---------- | +| `config.runtimeDefaults.telemetry.captureContent` | Prompts and assistant replies. On the `kagent` runtime, the content appears in the span for each model call. That runtime records a tool call's arguments and reply on the span for the tool call instead, regardless of this setting. On the `codex` and `claude` runtimes, prompts and replies appear on the `invoke_agent` span, and tool arguments and successful results appear on the `execute_tool` span. Claude Code and Codex also record content in their own telemetry. For each location, see [Content capture by harness]({{< link path="observability/audit-prompts#content-capture-by-harness" >}}). | `kagent`, `codex`, `claude` | +| `config.runtimeDefaults.telemetry.captureRawApiBodies` | The complete provider API request and response bodies. This setting returns more than `captureContent` does, and it takes effect only when you set the logs endpoint. | `claude` | +| `config.runtimeDefaults.telemetry.maxCaptureBytes` | The budget for each captured prompt, reply, tool argument, or tool result, in bytes. Defaults to `16384`, with a maximum of `65536`. | `codex`, `claude` | + +These settings are the installation's default. To change the capture decision for one Harness, set `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` in that Harness's `spec.env` field. The `kagent`, `codex`, and `claude` harnesses all read the variable and apply it in place of the installation default. Set `SPAN_ONLY` to put prompts and replies on spans, or `NO_CONTENT` to turn capture off. + +A `byo` image receives the same variable whenever any signal is enabled, but whether the image acts on it depends on its own instrumentation. An image that exports to its own backend continues to do so. For more information, see [Tracing]({{< link path="observability/tracing#about-trace-coverage" >}}). + +## Environment variables on an Actor + +kagent adds variables of its own to every revision, alongside the Harness `spec.env` entries. These include the `KAGENT_BOOTSTRAP_JSON` descriptor of the revision's bootstrap bundle, a certificate-authority setting for each TLS client that must trust the egress gateway, and `OTEL_RESOURCE_ATTRIBUTES`, which records the revision's service version. Enabling a telemetry signal adds more. All added variables count toward the limits that Agent Substrate enforces, so a Harness cannot spend the whole allowance on `spec.env`. + +Two limits bound the result, and kagent checks both when it builds the Actor template. A Harness that breaks one is admitted, and then reports the failure on its Agents. + +* Agent Substrate accepts at most 32 environment variables on an Actor. A revision that exceeds that count reports `runtime revision has environment variables; Substrate supports at most 32`. +* A single value is capped at 32768 characters. A revision that exceeds that length reports `environment variable "" is characters; Substrate supports at most 32768`. + +The agent configuration and the agent card are not environment variables. kagent publishes them in the revision's bootstrap bundle, and limits the two documents together to 256 KiB. A long system prompt or a large tool list counts against that limit. + +A Harness tunes telemetry for its own agents by setting `OTEL_*` variables in its `spec.env` field. Each harness reads its environment first and falls back to the installation's telemetry settings, so a value in `spec.env` wins for that Harness only and leaves every other Harness unchanged. Some variable names are reserved, and a Harness that sets one is rejected. + +Every harness reserves the following certificate trust variables. The egress gateway intercepts HTTPS, so an agent that does not trust the gateway's certificate authority cannot reach a model. + +* `SSL_CERT_FILE` and `SSL_CERT_DIR` +* `REQUESTS_CA_BUNDLE`, `AWS_CA_BUNDLE`, and `CURL_CA_BUNDLE` +* `NODE_EXTRA_CA_CERTS` +* `GIT_SSL_CAINFO` + +A Harness that sets one of these variables reports `runtime environment "" conflicts with gateway trust`. + +The `codex` and `claude` harnesses each reserve a second set of variables, which describe the credentials and runtime state that their adapters write. + +| Harness | Additional reserved variables | Effect if set by a Harness | +| ------- | ----------------------------- | -------------------------- | +| `claude` | The Anthropic, AWS, and Vertex AI credential variables, `CLAUDE_CONFIG_DIR`, `DISABLE_UPDATES`, `IS_SANDBOX`, `CLAUDE_CODE_ENABLE_TELEMETRY`, `CLAUDE_CODE_ENHANCED_TELEMETRY_BETA`, `OTEL_TRACES_EXPORTER`, `OTEL_METRICS_EXPORTER`, `OTEL_LOGS_EXPORTER`, Claude's `OTEL_LOG_*` capture flags, `TRACEPARENT`, `TRACESTATE`, and the Prometheus exporter host and port | Every Agent on the Harness reports `Compatible` as `False`, and the controller reports `Harness env "" conflicts with Claude-owned runtime configuration`. | +| `codex` | `CODEX_HOME`, `OPENAI_API_KEY`, `AWS_REGION`, `AWS_BEARER_TOKEN_BEDROCK`, `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and `AWS_SESSION_TOKEN` | Every Agent on the Harness reports `Compatible` as `False`, and the controller reports `Harness env "" conflicts with Codex's compiled configuration`. | + +Two variables take a value from a Harness without replacing the controller's. + +* `OTEL_RESOURCE_ATTRIBUTES` is merged. The value in `spec.env` is kept, and the agent identity attributes that the controller adds win where the two name the same attribute. +* `OTEL_EXPORTER_OTLP_HEADERS` has no controller value to replace. No Helm setting produces it and the controller never adds it, but a value in `spec.env` reaches every harness unchanged. To attach headers without holding them in an Actor environment, export to an in-cluster collector that adds them. + +### Baggage propagation + +Baggage is a W3C header that travels with a request alongside its trace context. Each entry in that header is a member, which pairs a name with a string value. Every service that the request reaches can read the members that it carries. + +By default, kagent sets `OTEL_PROPAGATORS` to `tracecontext` on every harness. That value propagates trace context and drops baggage. A caller's baggage therefore does not reach the tools that an agent calls or the agent's model provider. To carry baggage as well, set `OTEL_PROPAGATORS` to `tracecontext,baggage` in the Harness `spec.env` field. The kagent controller reads its own value from the `controller.env` Helm value, so enabling baggage on a runtime does not enable it on the controller. + +`OTEL_PROPAGATORS` is a list of wire formats that apply to both incoming and outgoing requests. If an incoming request has a format that is not in the list, the format is not read. Similarly, the runtime does not write an omitted format to an outgoing request. When the list omits `baggage`, the runtime's own members are dropped too. + +When `baggage` is listed, the `kagent` harness adds the request's identity to the baggage of each model call and each MCP tool call. + +| Member | Model calls | MCP tool calls | +| ------ | ----------- | -------------- | +| `gen_ai.agent.name` | Yes | Yes | +| `gen_ai.conversation.id` | Yes | Yes | +| `a2a.task.id` | Yes | Yes | +| `gen_ai.tool.call.id` | No | Yes | + +These members serve as identifiers only, and the runtime can replace a caller's member of the same name. + +> [!WARNING] +> **Enabling baggage sends the caller's own members to your model provider.** A caller baggage member that kagent does not replace is forwarded unchanged in the `baggage` header of each model API request and each MCP request. This means that a caller controls what reaches those external services in the header. Enable baggage only where you trust what your callers attach. + +Review and plan for the following considerations. +* The `claude` and `codex` harnesses receive `OTEL_PROPAGATORS`, but kagent does not add identity to their model and tool calls. The four members reach only the `kagent` harness. +* kagent never copies a baggage member onto a span. To make these values searchable, add a baggage-to-attributes span processor to the service that receives the request or to its OpenTelemetry Collector. + +When an agent requires an MCP server, the first tool listing or tool call opens a session with the MCP server, and every later request to that server shares it. The session makes three requests: the event stream, the replies to server requests, and the close. These requests do not carry trace context or baggage, because these name a request that has already ended. + +## Check that an agent can run + +To check the status of a Harness, AgentTemplate, or Agent, check the status of only the Agent. The Agent's status reports for all three resources, because neither a Harness nor an AgentTemplate reports a status of its own. + +An Agent that is not `Ready` states the reason in its conditions. + +```bash +kubectl describe agent -n kagent +``` + +The following conditions report the three stages that an Agent passes through. + +| Condition | Report | +| --------- | ------ | +| `ResolvedRefs` | Whether the resources that the Agent depends on exist. The controller resolves the Harness, the AgentTemplate, and the ModelConfig, and also the WorkerPool that the Harness references. A missing WorkerPool is the most common failure, and the Agent then reports the reason as `WorkerPoolNotFound`. | +| `Compatible` | Whether the harness accepts the resolved configuration. A rejected model provider, a reserved environment variable, and an unsupported AgentTemplate feature all fail here. | +| `Ready` | Whether Agent Substrate finished preparing the agent's capacity. | + +An Agent can also reconcile and still report a non-blocking decision in `status.warnings`, such as Claude binding every tool on an MCP server. For the conditions in full, see [Your first agent]({{< link path="get-started/your-first-agent" >}}). + +## Next steps + +{{< cards >}} + {{< card link=`{{< link path="get-started/your-first-agent" >}}` title="Your first agent" subtitle="Apply a Harness, an AgentTemplate, and an Agent, then start a conversation with it." >}} + {{< card link=`{{< link path="agents/agent-memory" >}}` title="Agent memory" subtitle="Give agents on this Harness memory that outlasts a single conversation." >}} + {{< card link=`{{< link path="setup/model-providers/about-model-providers" >}}` title="About model providers" subtitle="Understand how a ModelConfig reaches a running agent." >}} +{{< /cards >}} diff --git a/docs-site/content/kagent/main/agents/agent-memory.md b/docs-site/content/kagent/main/agents/agent-memory.md new file mode 100644 index 00000000..3d30cdc8 --- /dev/null +++ b/docs-site/content/kagent/main/agents/agent-memory.md @@ -0,0 +1,263 @@ +--- +title: Agent memory +description: Give agents long-term memory that persists across conversations, backed by vector similarity search. +weight: 45 +author: kagent.dev +--- + +A {{< gloss "Session" >}}Session{{< /gloss >}} remembers its own conversation, because it holds the {{< gloss "Transcript" >}}transcript{{< /gloss >}}. Long-term memory is different: it carries what an agent learned in one conversation into later ones. kagent stores those memories as vectors and retrieves them by similarity to whatever the user just said. Each agent on a {{< gloss "Harness" >}}Harness{{< /gloss >}} keeps its own memories, scoped to the user who created them. + +Memory is configured on the Harness rather than on an AgentTemplate, so it applies to every agent that the Harness runs. + +## How an agent uses memory + +Enabling memory adds three tools to every agent on the Harness, and appends a short instruction to its system prompt telling it that the tools exist. + +| Tool | What it does | +| ---- | ------------ | +| `save_memory` | Stores a specific fact, preference, or finding for later. | +| `load_memory` | Searches stored memories by query when the agent needs context it does not have. | +| `prefetch_memory` | Retrieves memories relevant to the current message before the agent answers. | + +Retrieval works by similarity rather than by keyword. kagent encodes the query as a vector, compares it against stored memories by cosine similarity, and puts the closest matches into the agent's context. A search returns at most five memories, and only those that score above 0.3. The agent also saves memories from a finished session on its own, so a user does not need to ask it to remember anything. + +Memories are scoped to the agent and to the user who created them, and carry the time that kagent wrote them. One agent cannot read another agent's memories, even on the same Harness. + +## Before you begin + +Agent memory requires your own PostgreSQL database with the [pgvector](https://github.com/pgvector/pgvector) extension installed. The default database that `kagent install` deploys runs the stock `postgres` image, which omits pgvector, so that database cannot store memories. + +If your installation already uses your own PostgreSQL database, skip step 1 and install pgvector in that database. The vector migration adds only the memory tables, so existing Sessions are kept. + +> [!WARNING] +> On the default database, step 1 moves kagent to a new, empty database. Existing Sessions do not carry over. + +1. Prepare the database for kagent and create its two connection Secrets. Follow steps 1 to 3 of [Use an external PostgreSQL instance]({{< link path="operations/operational-considerations#use-an-external-postgresql-instance" >}}) to create the `kagent_owner` and `kagent_readwrite` roles and the `my-postgres-owner` and `my-postgres-readwrite` Secrets. + +2. Install the pgvector extension in the schema that kagent reads it from. The `config.database.vectorSchema` value names this schema and defaults to `extensions`. Migrations and runtime queries both use the extension, so grant `USAGE` on the schema to both kagent roles. + ```sql + CREATE SCHEMA IF NOT EXISTS extensions; + CREATE EXTENSION IF NOT EXISTS vector SCHEMA extensions; + GRANT USAGE ON SCHEMA extensions TO kagent_owner, kagent_readwrite; + ``` + + The kagent migrations create the memory tables, and they never create the extension. Be sure to install it yourself before you enable memory. + +3. Save the following Helm values to point kagent at your database and run the migration that creates the memory tables. + ```yaml + cat > kagent-memory-values.yaml <}}/{{< reuse "kagent-docs/snippets/helm-kagent.md" >}} \ + --version {{< reuse "kagent-docs/versions/kagent.md" >}} \ + --namespace kagent --timeout 10m --reuse-values \ + -f kagent-memory-values.yaml + ``` + +5. Wait for the controller to roll out with the new configuration. + ```bash + kubectl rollout status deployment/kagent-controller -n kagent --timeout=300s + ``` + +## Choose an embedding model + +Any provider that the `kagent` harness supports can serve the embedding model, so the choice typically depends on whichever provider already holds your credentials. The embedding ModelConfig is resolved in the same way that a chat ModelConfig is, which means that the same [provider limitations]({{< link path="setup/model-providers/about-model-providers" >}}) apply to it. + +kagent stores every memory as a **768-dimensional** vector. Models that produce wider vectors are truncated and re-normalized to that width, so no dimension setting is needed and models of different widths can coexist. + +An Amazon Bedrock embedding model, for example, looks like the following: + +```yaml +spec: + provider: Bedrock + model: amazon.titan-embed-text-v2:0 + bedrock: + region: us-east-1 +``` + +## Enable memory + +Add memory to a Harness that already exists. The examples in these steps use `my-first-harness` in the `kagent` namespace, which you create in [Your first agent]({{< link path="get-started/your-first-agent" >}}). + +1. Create a `ModelConfig` for the embedding model in the same namespace as your Harness. The model is an embedding model rather than a chat model. + ```yaml + kubectl apply -f - <}} To review other available Harness fields, see [Agent harness]({{< link path="agents/agent-harness" >}}). + | Field | Description | + | ----- | ----------- | + | `memory.modelConfigRef.name` | The ModelConfig supplying the embedding model. Required when `memory` is set. The namespace defaults to the Harness's namespace. To use a ModelConfig in another namespace, set `memory.modelConfigRef.namespace`. That ModelConfig must admit the Harness's namespace, as described in [Share a ModelConfig across namespaces]({{< link path="setup/model-providers/about-model-providers#share-a-modelconfig-across-namespaces" >}}). | + | `memory.ttlDays` | How many days a stored memory stays valid. Minimum 1. When omitted, the server applies a default of 15 days. | + +3. Create a new Session against an {{< gloss "Agent" >}}Agent{{< /gloss >}} that uses this Harness. Editing the Harness builds a new {{< gloss "Revision" >}}revision{{< /gloss >}}, and an existing Session keeps running the revision it was created from. An agent that was already running does not gain memory until you start a new Session. + ```bash + kagent agent session create --agent my-first-agent + ``` + + The command returns output only after the Session reaches the `READY` state. Example output: + ```console + +--------------------------------------+----------------+-------+----------------------+ + | ID | AGENT | STATE | CREATED | + +--------------------------------------+----------------+-------+----------------------+ + | 0198c3d7-4f2a-7b61-9c3e-5d8f7a2b4e10 | my-first-agent | READY | 2026-08-31T15:02:10Z | + +--------------------------------------+----------------+-------+----------------------+ + ``` + +> [!NOTE] +> Only the `kagent` harness supports memory. The `memory` block exists only under `spec.kagent`, so a Harness that selects `codex`, `claude`, or `byo` does not take memory settings. + +## Verify that memory works + +Memory is working when a fact from one conversation reaches a later one. A Session holds the transcript of its own conversation, so the check needs a second Session that never saw the first. + +1. Create a Session and save its ID. + ```bash + export FIRST_SESSION_ID=$(kagent agent session create --agent my-first-agent -o json | jq -r '.session.id') + ``` + +2. Tell the agent a fact that is worth remembering. + ```bash + kagent agent invoke --session $FIRST_SESSION_ID \ + --task "Remember that I deploy to the staging cluster on Fridays." + ``` + +3. Create a second Session against the same Agent, and save its ID. The new Session starts with an empty transcript. + ```bash + export SECOND_SESSION_ID=$(kagent agent session create --agent my-first-agent -o json | jq -r '.session.id') + ``` + +4. Ask the second Session about the fact. An answer that includes the fact can only have come from memory, because this Session never saw the earlier conversation. + ```bash + kagent agent invoke --session $SECOND_SESSION_ID --task "When do I deploy to staging?" + ``` + + Example output: + ```console + You deploy to the staging cluster on Fridays. + ``` + +### What kagent stored + +Each memory is one row in the `memory` table, which the vector migration created in your database. A memory that the agent saves at the end of a session is summarized first, so one conversation typically produces several short rows rather than one long one. + +1. Connect to the database with the connection string that you stored in the Secret. Any PostgreSQL client works. + ```bash + psql 'postgres://:@:5432/?sslmode=require' + ``` + +2. Read what an agent saved. + ```sql + SELECT agent_name, user_id, content, created_at, expires_at + FROM memory ORDER BY created_at DESC LIMIT 10; + ``` + + Review the following table to understand the `memory` table output. + | Column | What it holds | + | ------ | ------------- | + | `content` | The text that the agent saved. Retrieval returns it to a later conversation. | + | `agent_name` | The agent that owns the memory, written as `/`, such as `kagent/my-first-agent`. The name identifies one {{< gloss "Agent" >}}Agent{{< /gloss >}}, so the same AgentTemplate paired into two Agents owns two separate sets of memories. A memory that an earlier release saved keeps its original key, such as `kagent__NS__my_first_agent`, and kagent does not migrate it. The agent no longer retrieves that memory. To list or delete it, pass the original key. | + | `user_id` | The user that the memory belongs to. | + | `embedding` | The 768-dimensional vector that similarity search compares a query against. | + | `created_at` and `expires_at` | When kagent wrote the memory, and `ttlDays` after that. | + | `access_count` | How many times retrieval has returned this memory. | + +## Manage memories + +The SQL query only reads the table. To list or clear memories, call the `MemoryService` that the kagent controller serves over gRPC. + +No CLI command manages memories, so these examples call the service with [grpcurl](https://github.com/fullstorydev/grpcurl). Each call acts on the memories of the user that the call authenticates as, and does not take a user ID. For how the controller resolves that user, see [Controller authentication modes]({{< link path="about/substrate-runtime/identity#controller-authentication-modes" >}}). In the default `insecure` mode, a call without an identity runs as `admin@kagent.dev`. In that case, the call returns an empty list for memories that belong to any other user, and a Delete call succeeds without deleting anything. To act on another user's memories in `insecure` mode, add `-H 'X-User-Id: '` to each grpcurl call, with the value from the `user_id` column. + +Each call also takes the `agent_name` exactly as the memory table stores it. An incorrect name fails silently, so copy the value from the table instead of typing it by hand. + +1. Port-forward the controller's gRPC port, and confirm that your kagent installation sets `config.server.grpcReflection=true`. + ```bash + kubectl port-forward -n kagent svc/kagent-controller 8083:8083 + ``` + +2. List the memories that an agent stores. + ```bash + grpcurl -plaintext -d '{ + "agent_name": "kagent/my-first-agent" + }' localhost:8083 kagent.runtime.v1.MemoryService/List + ``` + + Each entry returns the `id`, `content`, `access_count`, `created_at`, and `expires_at` fields. Results are ranked by how often retrieval has returned them, so the most-used memories appear first. + +3. Delete the memories that an agent stores. + ```bash + grpcurl -plaintext -d '{ + "agent_name": "kagent/my-first-agent" + }' localhost:8083 kagent.runtime.v1.MemoryService/Delete + ``` + +The memory service also exposes `Search`, `AddSession`, and `AddSessionBatch`. Each method takes a 768-dimensional vector rather than text, because kagent does not embed on the caller's behalf. Call them from a program that already has an embedding model, rather than by hand. + +## Memory lifetime + +A memory expires `ttlDays` after it is written, which defaults to 15 days. Expiry is per memory rather than per session, so an old preference ages out while a recent one survives. + +Changing `ttlDays` on the Harness applies to memories written by Sessions created after the change, because the value is built into the revision. + +## Known limitations + +- **Memories are deleted for an agent and user together.** `Delete` clears everything for that pair, and no call removes a single memory. +- **Memories are not shared between agents.** Each agent has its own store, so one agent cannot read what another learned, even on the same Harness and for the same user. +- **The memory implementation is not pluggable.** kagent builds on the Google Agent Development Kit (ADK) memory implementation, and it cannot be swapped for another memory system. To use an alternative, run it as a Model Context Protocol (MCP) server, [bind it as a tool]({{< link path="skills-and-mcp/about-tools" >}}), and instruct the agent to use that instead of the built-in tools. + +## Next steps + +{{< cards >}} + {{< card link=`{{< link path="agents/declarative-agents" >}}` title="Declarative agents" subtitle="Configure the rest of the kagent harness that memory is enabled on." >}} + {{< card link=`{{< link path="setup/model-providers/about-model-providers" >}}` title="About model providers" subtitle="Understand how a ModelConfig reaches a running agent." >}} +{{< /cards >}} diff --git a/docs-site/content/kagent/main/agents/bring-your-own-agent.md b/docs-site/content/kagent/main/agents/bring-your-own-agent.md new file mode 100644 index 00000000..e55f2603 --- /dev/null +++ b/docs-site/content/kagent/main/agents/bring-your-own-agent.md @@ -0,0 +1,285 @@ +--- +title: Bring your own agent +description: Run a container image of your own as a kagent agent by exposing the A2A service that the byo harness expects. +weight: 20 +author: kagent.dev +--- + +The `byo` harness runs a container image that you build, and treats what is inside it as opaque. {{< reuse "kagent-docs/snippets/name-product.md" >}} still builds a {{< gloss "Revision" >}}revision{{< /gloss >}} from the {{< gloss "Harness" >}}Harness{{< /gloss >}} and the {{< gloss "AgentTemplate" >}}AgentTemplate{{< /gloss >}}, schedules that revision onto {{< gloss "Agent Substrate" >}}Agent Substrate{{< /gloss >}}, and routes conversations to it. What the image does with a message is yours to decide. Choose this harness when you already have an agent that is built with an agent framework, and you want to run that agent in kagent. + +## Responsibilities on each side of the A2A boundary + +{{< gloss "A2A" >}}A2A{{< /gloss >}} (Agent2Agent) is the contract between kagent and a BYO agent. + +Your image owns one side of the A2A contract by: + +- Exposing an A2A service that accepts a message and returns a reply. +- Exposing a readiness endpoint that reports when the service can take traffic. +- Deciding what the agent actually does, including model calls, tool calls, and conversation state. + +kagent runs a BYO agent exactly as it runs one on a managed harness, and owns the other side of the A2A contract by: + +- Building an immutable revision from the {{< gloss "Harness" >}}Harness{{< /gloss >}} and the AgentTemplate, and creating a {{< gloss "Session" >}}Session{{< /gloss >}} from it. +- Running the {{< gloss "Actor" >}}Actor{{< /gloss >}} that hosts your image inside a sandbox. The {{< gloss "WorkerPool" >}}WorkerPool{{< /gloss >}} that the Harness references sets the class: `gvisor`, which every installation provides, or `microvm`, which needs its own Agent Substrate provisioning. For more information, see [Sandbox classes]({{< link path="about/substrate-runtime/sandboxing#sandbox-classes" >}}). +- [Suspending and resuming]({{< link path="about/substrate-runtime/suspend-and-resume#suspension-between-turns" >}}) the Actor between turns. A suspend keeps only the durable volume mounted at `/data`, so an image must write any state that it needs across turns to that directory. +- Routing every conversation through the A2A gateway, so callers address the {{< gloss "Agent" >}}Agent{{< /gloss >}} rather than the Actor behind it. +- Publishing the agent configuration and agent card in a bootstrap bundle for the revision. At startup, the kagent helpers fetch the bundle and write its contents to `/config/config.json` and `/config/agent-card.json`. + +Because Agent Substrate snapshots the whole Actor, an image that keeps conversation state in memory keeps it across a suspend. Your image does not need to persist anything to survive the gap between turns. + +## The A2A contract + +Both the Harness and the image must meet the following requirements, and kagent enforces them at different moments. Nothing checks the A2A service until a caller invokes the agent, so a misconfigured image reaches a healthy-looking state first. For examples of how to satisfy each requirement, see the steps in [Configure a BYO Harness](#configure-a-byo-harness) and [Build the image](#build-the-image). + +| Requirement | Checked | Detail | +| ----------- | ------- | ------ | +| `spec.workload.command` is set on the Harness | When you apply the Harness | You must set the entrypoint override on a `byo` Harness. A manifest that omits it is rejected with `BYO harnesses must specify workload.command`. | +| `spec.workload.image` is pinned by digest | When you apply the Harness | A `byo` Harness must name an image, and every Harness must pin it by `sha256` digest. A manifest that omits the image is rejected with `BYO harnesses must specify workload.image`, and a tag alone fails the field's pattern. | +| The container serves gRPC on port 80 | When a caller invokes the agent | kagent builds the agent card with a single interface, `http://127.0.0.1:80` bound to gRPC, and that address is not configurable. The service is `lf.a2a.v1.A2AService`. | +| The container answers `GET /readyz` on port 8081 | When Agent Substrate starts the Actor | Agent Substrate probes this path to decide when the Actor is ready, with a 30-second timeout. The probe is on a different port from the A2A service on purpose, so serve it independently. | +| The A2A service speaks gRPC, not JSON-RPC | When a caller invokes the agent | The agent card fixes the protocol binding to gRPC. An image that serves A2A over only JSON-RPC is never reached, regardless of the port it listens on. | + +> [!WARNING] +> **An image that serves A2A on another port still reports `READY`.** Agent Substrate probes readiness on port 8081. That probe succeeds whether or not the A2A service on port 80 works, so the mismatch stays hidden until the first invoke. The invoke then fails in the gateway with `Connect: tunnel failed`. That message reports only that the gateway could not open a connection to the agent. It does not name a port. To bind the right port, read the `runtime.port` field of the configuration that kagent delivers. An image that ignores the configuration must bind port 80 itself. + +## Compare BYO and declarative agents + +kagent resolves the AgentTemplate in the same way for a BYO agent as for a [declarative agent]({{< link path="agents/declarative-agents" >}}), then publishes the result as a JSON document in the revision's bootstrap bundle. Both harnesses resolve a ModelConfig through the same code, so the same model providers work and the same ones are rejected. For the provider matrix, see [Model provider support]({{< link path="agents/agent-harness#model-provider-support" >}}). + +The kagent harness reads that document and assembles the agent from it. A BYO image decides whether to read it: + +- **An opaque agent ignores the document.** The AgentTemplate exists to give the agent an identity and a description, and the image decides everything else, including which model to call and which tools to offer. Every field on an opaque AgentTemplate is optional, `modelConfig` included. +- **A configured agent reads the document.** The image applies the AgentTemplate's system prompt, ModelConfig, tool bindings, skills, and plugins itself. Editing the AgentTemplate changes what the agent does without a rebuild. + +Two AgentTemplate features continue to apply regardless of this choice. First, a `byo` Harness rejects structured output, with `Harness runtime "byo" does not support structured output`. Second, the agent card that kagent advertises for a `byo` Agent leaves out the AgentTemplate's skills and declares no extension, so a client cannot discover human-in-the-loop support the way it can for a declarative agent. + +Both kinds of BYO agent receive the same configuration. kagent stores the agent configuration and the agent card in a bootstrap bundle for each revision, and limits the two documents together to 256 KiB. The Actor's environment contains only a small descriptor of that bundle. At startup, the kagent helpers fetch the bundle from the kagent API, verify it against the descriptor, save it to `/data/kagent/bootstrap/bundle.pb`, and write the two documents to files. A restarted process or a resumed Session reads the saved bundle, and does not fetch it again. + +Build the image with the helpers from the same kagent release as the controller, because the helpers of each release read the configuration in the way that the release delivers it. + +The two files exist only after the helpers run. An image without the helpers reads the descriptor and fetches the bundle itself. + +| Item | Contents | +| ---- | -------- | +| `/config/config.json`, which the helpers write | The agent configuration, including the resolved system prompt, the model and its settings, every tool, skill, and plugin that the AgentTemplate binds, and a `runtime` object that sets the process settings. | +| `/config/agent-card.json`, which the helpers write | The agent card that kagent advertises for this agent, rendered as JSON. | +| `KAGENT_BOOTSTRAP_JSON` | The bootstrap descriptor, with the control-plane URL, the revision, the format version, and the SHA-256 digest of the bundle. | +| Anything in `spec.env` | The literal values that the Harness sets. | +| Certificate-authority variables and `SSL_CERT_DIR` | The egress gateway's trust bundle, which every Actor receives. For the list and the limit on how many variables an Actor takes, see [Environment variables on an Actor]({{< link path="agents/agent-harness#environment-variables-on-an-actor" >}}). | + +The `runtime` object is the root of the configuration document, and subagents inherit it. The object includes the settings that a process needs before it serves a request. + +```json +{ + "runtime": { + "control_plane_url": "http://kagent-controller.kagent.svc:8083", + "gateway_url": "http://kagent-controller.kagent.svc:8083", + "name": "my-byo-agent", + "namespace": "kagent", + "log_level": "info", + "port": "80", + "max_content_length": 0 + } +} +``` + +| Field | Contents | +| ----- | -------- | +| `control_plane_url` | The kagent API that the agent reports task state to. Required by the Go and Python helpers. | +| `gateway_url` | The A2A gateway that the agent reaches other agents and MCP servers through. | +| `name` and `namespace` | The Agent's name and the AgentTemplate's namespace, which kagent's helpers use to scope sessions. | +| `port` | The port to serve A2A on. The controller sets `80` for every harness, and the agent card advertises that same port. | +| `log_level` | `debug`, `info`, `warn`, or `error`, from the `config.runtimeDefaults.logging.level` Helm value. | +| `max_content_length` | The largest A2A request body to accept, in bytes. Zero means unlimited, and an absent field keeps the helper's own default. | + +## Configure a BYO Harness + +A `byo` Harness takes the same fields as any other Harness. Its own block, `spec.byo`, is an empty object, because the image defines the behavior that the other harnesses take settings for. + +```yaml +kubectl apply -f - </my-agent@sha256: + command: ["/my-agent"] + substrate: + workerPoolRef: + name: kagent-default + snapshotPolicy: + location: s3://ate-snapshots/kagent/ +EOF +``` + +{{< reuse "kagent-docs/snippets/review-table.md" >}} For the fields that every Harness shares, see [Configure a Harness]({{< link path="agents/agent-harness#configure-a-harness" >}}). + +| Field | Required | Description | +| ----- | -------- | ----------- | +| `byo` | Yes | Selects this harness. The object is always empty, and naming a second harness alongside it is rejected. | +| `workload.image` | Yes | Your image, pinned by `sha256` digest. A tag alone is rejected, because a revision must be reproducible. | +| `workload.command` | Yes | The entrypoint to run, up to 32 entries. Required for `byo` and optional for every other harness. | +| `env` | No | Any setting that your image reads. Each entry takes a literal `value` only. Put credentials in a ModelConfig or a RemoteMCPServer instead. | + +An opaque agent's AgentTemplate sets only a description for the agent card. An Agent then pairs it with the Harness. + +```yaml +kubectl apply -f - <}}). + +## Build the image + +The A2A contract is a gRPC service and a readiness endpoint, so any language with a gRPC server can satisfy it. kagent ships helpers for Go and Python. To build and run a minimal image end to end before writing your own, see [Run your own agent image]({{< link path="examples/a2a-byo" >}}). + +> [!TIP] +> The Go helper produces smaller images, faster startup, and lower memory use than the Python one. Where both suit the agent that you are building, prefer Go. + +{{< tabs >}} +{{% tab name="Go" %}} +`github.com/kagent-dev/kagent/go/adk/pkg/app` serves the A2A gRPC service, the readiness endpoint on 8081, and the agent card, given any type that implements `a2asrv.AgentExecutor`. The helper is framework-agnostic, so the executor is the only part that you write. `config.MaterializeFromBootstrap` fetches the bootstrap bundle and writes its documents to `/config`, and `config.LoadAgentConfig` reads the configuration back. + +```go +import ( + "cmp" + "context" + "fmt" + "iter" + "os" + + a2atype "github.com/a2aproject/a2a-go/v2/a2a" + "github.com/a2aproject/a2a-go/v2/a2asrv" + "github.com/kagent-dev/kagent/go/adk/pkg/app" + "github.com/kagent-dev/kagent/go/adk/pkg/config" + "github.com/kagent-dev/kagent/go/pkg/logging" +) + +if err := config.MaterializeFromBootstrap(context.Background(), "/config"); err != nil { + return err +} +compiled, err := config.LoadAgentConfig("/config/config.json") +if err != nil { + return err +} +runtime := compiled.ProcessConfig() + +application, err := app.New(app.AppConfig{ + AgentCard: a2atype.AgentCard{ + Name: "my-agent", + Version: "v1", + Capabilities: a2atype.AgentCapabilities{Streaming: true}, + }, + ControlPlaneURL: runtime.ControlPlaneURL, + MaxContentLength: runtime.MaxContentLength, + Port: cmp.Or(runtime.Port, "80"), + AppName: cmp.Or(runtime.Name, "my-agent"), + Logger: logger, +}, myExecutor{}) +if err != nil { + return err +} +return application.Run() +``` + +`app.New` takes an application configuration and an executor, and returns an application that serves the A2A gRPC service and the agent card on the port that the configuration names. These settings in `app.AppConfig` decide whether kagent can reach the result: + +- `Port` is the port the A2A listener binds. `app.AppConfig` defaults it to `8080`, which kagent never dials. Pass `cmp.Or(runtime.Port, "80")` instead to take the port from the configuration and fall back to `80` when the configuration omits it. +- `ControlPlaneURL`, or a `ControllerClient` of your own, is required because the helper reports task state to the kagent API. +- `AppName` scopes sessions. The `runtime.Name` value is the Agent's name. + +For a complete executor, see [`go/core/test/byoa2a/main.go`](https://github.com/kagent-dev/kagent/blob/main/go/core/test/byoa2a/main.go) in the kagent repository. +{{% /tab %}} +{{% tab name="Python" %}} +The `kagent-adk` package provides helpers for agents that are built with the Google Agent Development Kit (ADK). `KAgentApp` wraps an ADK agent rather than an arbitrary framework, serves A2A over gRPC, and answers the readiness endpoint on 8081. `materialize_from_bootstrap` fetches the bootstrap bundle and writes its documents to `/config`, and `KAgentConfig.from_file` reads the `runtime` object back. The listener defaults to `[::]:80`, which matches what kagent dials. + +```python +from kagent.adk import KAgentApp +from kagent.core import KAgentConfig, materialize_from_bootstrap + +materialize_from_bootstrap("/config") +config = KAgentConfig.from_file() + +app = KAgentApp( + root_agent_factory=build_agent, + agent_card=card, + kagent_api_url=config.api_url, + app_name=config.app_name, +).build() +``` + +`from_file` raises `ValueError` when the document omits `control_plane_url`, `gateway_url`, `name`, or `namespace`. Because `from_file` only reads the file, a standalone process that runs outside Agent Substrate skips `materialize_from_bootstrap` and supplies its own `/config/config.json`. + +The page uses two configuration objects. `KAgentConfig` holds only the `runtime` object, which supplies the API URL and the app name. `kagent.adk.types.parse_config` parses the whole document into a `Config`, and `KAgentApp` reads the listener address from that `Config`. To bind the address that the configuration names, pass a parsed `Config` as the `config` argument. `parse_config` takes JSON text or a dict rather than a path, so read the file first. Call `materialize_from_bootstrap` before either reader, because only `materialize_from_bootstrap` writes the file. + +```python +from pathlib import Path + +from kagent.adk.types import parse_config + +app = KAgentApp( + root_agent_factory=build_agent, + agent_card=card, + kagent_api_url=config.api_url, + app_name=config.app_name, + config=parse_config(Path("/config/config.json").read_text()), +).build() +``` + +The `a2a_grpc_address` argument overrides both the default and the configured address. +{{% /tab %}} +{{% tab name="Any other language" %}} +Implement the contract directly: + +1. Serve `lf.a2a.v1.A2AService` over gRPC on port 80. Generate the stubs from the [A2A protocol](https://a2a-protocol.org) definitions. +2. Serve `GET /readyz` over HTTP on port 8081, returning `200`. +3. To honor the AgentTemplate, fetch the bootstrap bundle at startup. An agent that skips this step runs as an opaque agent. + 1. Read the descriptor from the `KAGENT_BOOTSTRAP_JSON` environment variable. + 2. Call `kagent.runtime.v1.BootstrapService/GetBootstrapConfig` over gRPC at the descriptor's `control_plane_url`. Send an `authorization` header with any bearer value, because Agent Substrate replaces that header with the Actor's own token. + 3. Check that the bundle's `revision` and `sha256` match the descriptor. For the digest algorithm, see [`go/api/bootstrap/bundle.go`](https://github.com/kagent-dev/kagent/blob/main/go/api/bootstrap/bundle.go) in the kagent repository. + 4. Read the agent configuration from the bundle's `config_json` field and the agent card from `agent_card_json`. + +The kagent helpers also save the bundle to `/data/kagent/bootstrap/bundle.pb`, and read that file on a restart without calling the service again. +{{% /tab %}} +{{< /tabs >}} + +## Known limitations + +- **The A2A interface is fixed.** kagent advertises `http://127.0.0.1:80` over gRPC, and `spec.byo` does not take a field that changes the address, the port, or the protocol. An image that serves A2A over only HTTP JSON-RPC cannot run on this harness. +- **The LangGraph, CrewAI, and OpenAI adapters do not satisfy the contract.** The `kagent-langgraph`, `kagent-crewai`, and `kagent-openai` packages each build a FastAPI application with A2A JSON-RPC routes, without a gRPC server. Running LangGraph, CrewAI, or the OpenAI Agents SDK under `byo` means serving A2A over gRPC yourself, as the **Any other language** tab in [Build the image](#build-the-image) describes. +- **Skills load only when the image reads them.** kagent mounts each skill and plugin that an AgentTemplate attaches, and describes them in the agent configuration. The Go helper reads that description. The Python helper does not yet read it, so a Python image runs without its skills. +- **Long-term memory is unavailable.** Memory is configured under `spec.kagent.memory`, and only the `kagent` harness wires it up. A `byo` Harness does not take an equivalent setting. For what a BYO image would need to replace, see [Agent memory]({{< link path="agents/agent-memory" >}}). +- **A broken port mapping looks like a healthy agent.** For the failure and what avoids it, see [The A2A contract](#the-a2a-contract). + +## Next steps + +{{< cards >}} + {{< card link=`{{< link path="examples/a2a-byo" >}}` title="Run your own agent image" subtitle="Build the minimal BYO agent, run it on a byo Harness, and invoke it." >}} + {{< card link=`{{< link path="agents/agent-harness" >}}` title="Agent harness" subtitle="Compare the byo harness against the ones that kagent executes itself." >}} + {{< card link=`{{< link path="examples/a2a-agents" >}}` title="Call an agent over A2A" subtitle="Send messages to an Agent with the same protocol that a BYO image serves." >}} + {{< card link=`{{< link path="about/substrate-runtime/suspend-and-resume" >}}` title="Suspend and resume" subtitle="Understand what Agent Substrate snapshots while your image is idle." >}} +{{< /cards >}} diff --git a/docs-site/content/kagent/main/agents/context-management.md b/docs-site/content/kagent/main/agents/context-management.md new file mode 100644 index 00000000..81674a60 --- /dev/null +++ b/docs-site/content/kagent/main/agents/context-management.md @@ -0,0 +1,110 @@ +--- +title: Context management +description: Summarize older session events so an agent's prompt stays bounded as a conversation grows, with two strategies configured on the Harness. +weight: 40 +author: kagent.dev +--- + +An agent's prompt grows with the conversation: every message, tool call, and tool result accumulates, and long sessions eventually exceed the model's context window or degrade answer quality. Context compaction solves that by summarizing older session events so the prompt stays bounded while the agent keeps enough recent context to stay coherent. + +This page applies to [declarative agents]({{< link path="agents/declarative-agents" >}}), which run on the `kagent` harness. Compaction is a `spec.kagent` setting, so a Harness that selects `codex`, `claude`, or `byo` does not take compaction settings. Omitting the `compaction` block leaves the history uncompacted. + +Compaction is configured on the {{< gloss "Harness" >}}Harness{{< /gloss >}}, not on the {{< gloss "AgentTemplate" >}}AgentTemplate{{< /gloss >}}. The setting belongs to the runner that drives the root agent, so the same AgentTemplate compacts differently on two Harnesses that set different values. An AgentTemplate cannot enable, disable, or tune compaction on its own. [Agent memory]({{< link path="agents/agent-memory" >}}) follows the same Harness-versus-AgentTemplate split. + +Two strategies ship, and both are configured on the same field: + +- **Sliding window** — `compactionInterval`, optionally with `overlapSize` — summarizes each group of completed user-initiated invocations, pulling already-compacted invocations back into the next window so consecutive summaries overlap. +- **Tail retention** — `tokenThreshold` with `eventRetentionSize` — bounds the prompt once it passes a token count by summarizing the history but keeping the most recent events uncompacted. + +## Configure compaction + +The following manifest enables both strategies on a Harness resource that already selects the `kagent` harness. The `compaction` block is the only change. The rest of the Harness is unchanged. + +```yaml +apiVersion: api.kagent.dev/v1alpha3 +kind: Harness +metadata: + name: my-harness + namespace: kagent +spec: + kagent: + compaction: + # Sliding window: summarize every 5 completed user-initiated invocations, + # pulling the 2 most recent ones back into the next window so summaries overlap. + compactionInterval: 5 + overlapSize: 2 + # Tail retention: once the prompt passes 24000 tokens, summarize the history + # but keep the 10 most recent events uncompacted. + tokenThreshold: 24000 + eventRetentionSize: 10 + # Optional: name a ModelConfig to write the summaries. Add namespace to use + # a shared ModelConfig. When you omit it, the agent's own model summarizes. + summarizer: + modelConfigRef: + name: summarizer-model-config + # Optional: replace the runtime's default summarization prompt. + # Must contain {conversation_history}, which the runtime replaces with + # the rendered events. + promptTemplate: > + Summarize the following conversation so the agent can continue. + {conversation_history} + workload: + image: {{< reuse "kagent-docs/versions/runtime-image.md" >}} + substrate: + workerPoolRef: + name: kagent-default + snapshotPolicy: + location: s3://ate-snapshots/kagent/ +``` + +`Harness.spec.kagent.compaction` is the only field this page owns. The generated [API reference]({{< link path="reference/api-ref#kagentharnesscompaction" >}}) carries the field-by-field schema, with types, defaults, and validation rules. [Agent harness]({{< link path="agents/agent-harness" >}}) carries the complete Harness schema, including `workload`, `substrate`, and the harness selection. + +The table below lists the compaction fields and their requirements: + +| Field | Required | Description | +| ----- | -------- | ----------- | +| `compactionInterval` | Optional | The number of new user-initiated invocations that, once fully represented in the session, triggers a sliding-window compaction of those invocations. Minimum 1. | +| `overlapSize` | Optional | The number of already-compacted invocations pulled back into the next sliding window so consecutive summaries overlap. Requires `compactionInterval`. Minimum 0. | +| `tokenThreshold` | Optional | The prompt token count at which tail-retention compaction summarizes the history before the next model call. Requires `eventRetentionSize`. Minimum 1. | +| `eventRetentionSize` | Optional | The number of most recent events that tail retention keeps uncompacted. Requires `tokenThreshold`. Minimum 1. | +| `summarizer.modelConfigRef` | Optional | The {{< gloss "ModelConfig" >}}ModelConfig{{< /gloss >}} that writes the summaries, by `name` and optional `namespace`. The namespace defaults to the Harness namespace. A ModelConfig in another namespace must admit the Harness namespace, as described in [Share a ModelConfig across namespaces]({{< link path="setup/model-providers/about-model-providers#share-a-modelconfig-across-namespaces" >}}). When you omit the field, the agent's own model summarizes. | +| `summarizer.promptTemplate` | Optional | Replaces the runtime's default summarization prompt. Must contain `{conversation_history}`, which the runtime replaces with the rendered events. | + +## Admission rules + +These are CEL validations on the CRD, so a manifest that breaks one is rejected by the API server at admission rather than failing at runtime. They are rules, not advice: + +- At least one strategy must be configured — `compactionInterval` or `tokenThreshold`. +- `tokenThreshold` and `eventRetentionSize` must be set together. +- `overlapSize` requires `compactionInterval`. +- A `promptTemplate` must contain `{conversation_history}`, which the runtime replaces with the rendered events. + +## Summarizer model and revisions + +The summarizer model resolves in the same way as the agent's own model: its config lands in the revision, its credentials and egress join the revision, and it joins the provenance. Changing the summarizer model therefore builds a new revision for every {{< gloss "Agent" >}}Agent{{< /gloss >}} that uses the Harness. A {{< gloss "Session" >}}Session{{< /gloss >}} that is already running keeps summarizing with the model that it started with. + +Naming an AgentTemplate's own model in `summarizer.modelConfigRef` is semantically a no-op for that AgentTemplate, because the runtime already summarizes by using that model by default. The setting applies to every Agent that uses the Harness, so templates that use a different model summarize with the named one instead. Setting the field still builds a new revision. A {{< gloss "Revision" >}}revision{{< /gloss >}} ID is a digest of the inputs that kagent resolved, and those inputs include each source resource's `metadata.generation`. Editing the Harness bumps that generation regardless of whether or not the result differs. + +## Verify the configuration + +If the summarizer names a ModelConfig that does not exist, the Agent is not ready. The failure surfaces on the Agent as `ResolvedRefs=False`, with a reason such as `resolve summarizer ModelConfig "x": resolve ModelConfig "/x": not found`. Confirm that the Agent is ready before relying on compaction: + +```bash +kagent agent get +``` + +`READY` reports whether kagent built and prepared a revision. If `READY` stays `False`, inspect the Agent's conditions to find the failing reference: + +```bash +kubectl get agent -n kagent -o jsonpath='{.status.conditions}' | jq +``` + +For more information, see [Your first agent]({{< link path="get-started/your-first-agent" >}}). + +## Next steps + +{{< cards >}} + {{< card link=`{{< link path="agents/declarative-agents" >}}` title="Declarative agents" subtitle="Configure the rest of the kagent harness that compaction is enabled on." >}} + {{< card link=`{{< link path="agents/agent-memory" >}}` title="Agent memory" subtitle="Give agents on this Harness memory that outlasts a single conversation." >}} + {{< card link=`{{< link path="reference/api-ref#kagentharnesscompaction" >}}` title="API reference" subtitle="Field-by-field schema for Harness.spec.kagent.compaction." >}} +{{< /cards >}} diff --git a/docs-site/content/kagent/main/agents/declarative-agents.md b/docs-site/content/kagent/main/agents/declarative-agents.md new file mode 100644 index 00000000..43d1e5c7 --- /dev/null +++ b/docs-site/content/kagent/main/agents/declarative-agents.md @@ -0,0 +1,128 @@ +--- +title: Declarative agents +description: Build an agent from a system prompt, tools, skills, and plugins, and let the kagent harness run the agentic loop. +weight: 15 +author: kagent.dev +--- + +A declarative agent is an agent that you describe rather than build. In an {{< gloss "AgentTemplate" >}}AgentTemplate{{< /gloss >}} resource, you write a system prompt, name a model, and bind the tools, skills, and plugins that the agent can use. The `kagent` {{< gloss "Harness" >}}Harness{{< /gloss >}} then runs kagent's built-in Go engine, which reads the AgentTemplate configuration and runs the agentic loop from it. To change what the agent does, you edit the AgentTemplate. + +## Declarative agents and BYO agents + +kagent runs an agent in one of two ways depending on who owns the agentic loop. A declarative agent hands the loop to kagent. A [BYO agent]({{< link path="agents/bring-your-own-agent" >}}) keeps the loop inside a container image that you build and maintain, and kagent runs the image instead. + +| | Declarative agent | BYO agent | +| --- | --- | --- | +| Harness | `kagent` | `byo` | +| What executes the agent | kagent's built-in Go engine | The container image that you build | +| Where the behavior is defined | The AgentTemplate | The image, which might read the AgentTemplate | +| Changing the behavior | Edit the AgentTemplate | Rebuild the image, unless the image reads the configuration that kagent delivers | +| [Structured output]({{< link path="agents/structured-output" >}}), [long-term memory]({{< link path="agents/agent-memory" >}}), and UI file attachments | Supported | Unsupported | + +Both routes receive the same resolved agent configuration, run in the same sandbox, follow the same snapshot policy, and route through the same gateway. For how a BYO image reads that configuration, and what its agent card leaves out, see [Compare BYO and declarative agents]({{< link path="agents/bring-your-own-agent#compare-byo-and-declarative-agents" >}}). + +The `codex` and `claude` harnesses provide a third route, in which a coding agent owns the loop and takes its configuration from the AgentTemplate. To compare all options side by side, see [Choose a harness]({{< link path="agents/agent-harness#choose-a-harness" >}}). + +## Configure the kagent harness + +Define settings that belong to the engine rather than to the agent in the `spec.kagent` section of the Harness resource. The section takes memory settings, which persist facts across conversations, and compaction settings, which keep a single conversation's prompt within a limit as the conversation grows. + +```yaml +spec: + kagent: + memory: + modelConfigRef: + name: embedding-model-config + ttlDays: 30 + compaction: + tokenThreshold: 120000 + eventRetentionSize: 20 +``` + +Compaction offers two strategies. Tail retention summarizes the older history once the prompt passes a token count, while a sliding window summarizes each group of completed invocations as the conversation goes. Setting `compaction` requires at least one of these two strategies, and setting both is valid. For the rules that the API server enforces, see [Context management]({{< link path="agents/context-management" >}}). + +| Field | Description | +| ----- | ----------- | +| `memory.modelConfigRef.name` | The {{< gloss "ModelConfig" >}}ModelConfig{{< /gloss >}} that supplies the embedding model. Required when `memory` is set. To use a ModelConfig outside the Harness's namespace, also set `memory.modelConfigRef.namespace`. That ModelConfig must admit the Harness's namespace, as described in [Share a ModelConfig across namespaces]({{< link path="setup/model-providers/about-model-providers#share-a-modelconfig-across-namespaces" >}}). Setting `memory` gives every agent on this Harness memory that persists across conversations. For how agents store and retrieve it, see [Agent memory]({{< link path="agents/agent-memory" >}}). | +| `memory.ttlDays` | How many days a stored memory entry stays valid. Minimum 1. When omitted, the server applies a default of 15 days. | +| `compaction.tokenThreshold` | Tail retention. The prompt token count that triggers a summary. Once the prompt passes this count, kagent summarizes the older history before the next model call. Requires `compaction.eventRetentionSize`. | +| `compaction.eventRetentionSize` | Tail retention. The number of most recent events that kagent leaves out of that summary, so the latest exchanges reach the model in full. Requires `compaction.tokenThreshold`. | +| `compaction.compactionInterval` | Sliding window. The number of new user-initiated invocations that triggers a summary of those invocations. Set `5` to summarize each group of five. | +| `compaction.overlapSize` | Sliding window. The number of already-summarized invocations that kagent includes in the next summary, so that consecutive summaries overlap and carry context forward. Requires `compaction.compactionInterval`. | +| `compaction.summarizer` | Both strategies. The ModelConfig and prompt template that write the summaries. When omitted, kagent summarizes with the agent's own model and the engine's default prompt. | + +For the fields that every Harness shares, such as the workload image and the Agent Substrate placement, see [Configure a Harness]({{< link path="agents/agent-harness#configure-a-harness" >}}). + +## Apply a complete declarative agent + +The following example applies a declarative agent end to end, from the ModelConfig to the Agent that runs it. The ModelConfig reads its API key from the `kagent-openai` Secret that the [OpenAI]({{< link path="setup/model-providers/openai#create-the-modelconfig" >}}) guide creates. `spec.kagent` is an empty object because memory and compaction are both optional. + +```yaml +kubectl apply -f - <}} + substrate: + workerPoolRef: + name: kagent-default + snapshotPolicy: + location: s3://ate-snapshots/kagent/ +--- +apiVersion: api.kagent.dev/v1alpha3 +kind: AgentTemplate +metadata: + name: kagent-template + namespace: kagent +spec: + modelConfig: + name: kagent-model-config + systemPrompt: You are a concise, helpful assistant. +--- +apiVersion: api.kagent.dev/v1alpha3 +kind: Agent +metadata: + name: my-agent + namespace: kagent +spec: + template: + ref: + name: kagent-template + harnessRef: + name: kagent-harness +EOF +``` + +To confirm that the Agent is ready, run the following command. In the output, `READY` reports whether kagent built and prepared a {{< gloss "Revision" >}}revision{{< /gloss >}} for the pair. + +```bash +kagent agent get my-agent +``` + +A `READY` value of `False` immediately after you apply the Agent is expected, because kagent builds a snapshot of the agent before it reports the Agent ready. If `READY` stays `False`, read the conditions on the Agent to find the stage that failed, asdescribed in [Your first agent]({{< link path="get-started/your-first-agent#create-a-harness-an-agenttemplate-and-an-agent" >}}). To start a conversation with the agent, see [Create a Session]({{< link path="get-started/your-first-agent#create-a-session" >}}). + +## Next steps + +{{< cards >}} + {{< card link=`{{< link path="agents/system-prompts" >}}` title="System prompts" subtitle="Write the instruction that shapes what a declarative agent does." >}} + {{< card link=`{{< link path="skills-and-mcp/about-tools" >}}` title="About tools" subtitle="Bind the tools that the agent can call." >}} + {{< card link=`{{< link path="agents/agent-harness" >}}` title="Agent harness" subtitle="Compare the kagent harness against the other three." >}} + {{< card link=`{{< link path="agents/bring-your-own-agent" >}}` title="Bring your own agent" subtitle="Run an agent that you built yourself, on your own container image." >}} +{{< /cards >}} diff --git a/docs-site/content/kagent/main/agents/egress-policies.md b/docs-site/content/kagent/main/agents/egress-policies.md new file mode 100644 index 00000000..9efa1144 --- /dev/null +++ b/docs-site/content/kagent/main/agents/egress-policies.md @@ -0,0 +1,282 @@ +--- +title: Egress policies +description: Grant an Agent or a standalone sandbox access to HTTP and HTTPS destinations beyond its model and tools, and inject a credential into requests to those destinations. +weight: 55 +author: kagent.dev +--- + +An {{< gloss "Agent" >}}Agent{{< /gloss >}} reaches only the destinations that its AgentTemplate needs, such as its model provider, its Model Context Protocol (MCP) servers, and its skill sources. {{< reuse "kagent-docs/snippets/name-product.md" >}} infers these destinations and compiles them into the agent's [egress policy]({{< link path="about/substrate-runtime/networking-and-egress#policy-generation" >}}). The egress gateway denies every other destination. + +To grant an Agent or SandboxTemplate access to destinations beyond the inferred set, create an EgressPolicy resource. The Agent or SandboxTemplate selects one or more policies by name in its `spec.egressPolicyRefs` field. An EgressPolicy only adds access, and cannot remove a destination that kagent inferred from the model or the tools. + +A policy can also have the gateway add a credential to each request, either a value from a Kubernetes Secret or a token that identifies the calling agent. The credential never enters the agent's sandbox. Typical uses include a package registry that a skill installs from, or an internal API that a tool script calls. + +## Before you begin + +Create the resource that you want to grant access to. The examples on this page use the `kagent` namespace. +* **Agent**: [Create your first agent]({{< link path="get-started/your-first-agent" >}}), so that you have the `my-first-agent` Agent. Use commands in the "Agent" tabs in the following steps. +* **Standalone sandbox**: Create the `scratch` SandboxTemplate, as described in [Create a SandboxTemplate]({{< link path="about/substrate-runtime/standalone-sandboxes#create-a-sandboxtemplate" >}}). Use commands in the "Standalone sandbox" tabs in the following steps. + +## Grant access to a destination + +Create an EgressPolicy that lists the destinations that you want to grant access to, and then select the policy on the Agent or SandboxTemplate. + +1. Create the EgressPolicy in the namespace of the Agent or SandboxTemplate. A resource can select only the policies in its own namespace. This example grants access to the npm registry and the Node.js download site. + ```yaml + kubectl apply -f - <}} Each destination grants one origin, and the grant covers every path on that origin. For the complete schema, see the [API reference]({{< link path="reference/api-ref#egresspolicy" >}}). + + | Field | Description | + | ----- | ----------- | + | `destinations` | The destinations to grant, from 1 to 64 entries. | + | `destinations[].origin` | A scheme, a DNS hostname, and an optional port, such as `https://registry.npmjs.org` or `https://search.example.com:8443`. For the accepted forms, see [Origin format](#origin-format). | + | `destinations[].authentication` | Optional. A credential for the gateway to add to each request. For more information, see [Inject a credential](#inject-a-credential). | + +2. Select the policy in the `spec.egressPolicyRefs` field. + {{< tabs >}} + {{% tab name="Agent" %}} + The AgentTemplate and the Harness cannot select a policy, so two Agents that share an AgentTemplate can have different access. The grant applies to the whole {{< gloss "Actor" >}}Actor{{< /gloss >}}, including the subagents that run in it. + ```bash + kubectl patch agent my-first-agent -n kagent --type merge \ + -p '{"spec":{"egressPolicyRefs":[{"name":"node-dependencies"}]}}' + ``` + {{% /tab %}} + {{% tab name="Standalone sandbox" %}} + Every Sandbox that is created from the template receives the grant. + ```bash + kubectl patch sandboxtemplate scratch -n kagent --type merge \ + -p '{"spec":{"egressPolicyRefs":[{"name":"node-dependencies"}]}}' + ``` + {{% /tab %}} + {{< /tabs >}} + +3. Check that kagent admitted the grant. When kagent admits every selected policy, the `EgressPoliciesApplied` condition reports `True` with the reason `Applied`. + {{< tabs >}} + {{% tab name="Agent" %}} + ```bash + kubectl get agent my-first-agent -n kagent \ + -o jsonpath='{range .status.conditions[?(@.type=="EgressPoliciesApplied")]}{.status} {.reason} {.message}{end}' + ``` + {{% /tab %}} + {{% tab name="Standalone sandbox" %}} + ```bash + kubectl get sandboxtemplate scratch -n kagent \ + -o jsonpath='{range .status.conditions[?(@.type=="EgressPoliciesApplied")]}{.status} {.reason} {.message}{end}' + ``` + {{% /tab %}} + {{< /tabs >}} + + Example output: + ```console + True Applied All requested egress grants are admitted + ``` + +The grant reaches existing workloads as well as new ones. kagent updates the policy of every running and suspended Actor without restarting it. For an Agent, this update covers every {{< gloss "Session" >}}Session{{< /gloss >}}. For a SandboxTemplate, the update covers every Sandbox that was created from the template. The update is asynchronous, so a request that is sent immediately after the change might still use the previous policy. + +## Origin format + +An origin names exactly one scheme, host, and port. The scheme must be `http` or `https`. When the port is omitted, kagent uses `443` for `https` and `80` for `http`. The API server rejects an origin that contains a path, a query, a fragment, user information, or an IP address. + +A wildcard can replace the leftmost label of the hostname. For example, `https://*.example.com` matches `api.example.com`, but it matches neither `example.com` nor `a.b.example.com`. The API server rejects `*` and `https://*.com`. + +A redirect to another origin needs its own grant. For example, a package registry often serves downloads from a separate host. An install fails at that host until the policy grants it as well. + +The API server rejects an `egressPolicyRefs` list that selects more than 16 policies and a policy that lists more than 64 destinations. The compiled policy limit also counts the destinations that kagent inferred, so an Agent with many tools reaches it sooner. + +| Limit | Value | +| ----- | ----- | +| Policies that one Agent or SandboxTemplate selects | 16 | +| Destinations in one policy | 64 | +| Rules in the compiled policy of one Actor | 256 | +| Credential headers for one origin | 16 | + +## Inject a credential + +The gateway can add a credential to each request for an origin. The agent sends the header with a placeholder value, and the gateway replaces this value as the request passes through. The credential is not stored in the agent's revision and never reaches the sandbox. The following steps inject a value from a Kubernetes Secret. To inject a token that identifies the calling agent, see [Inject the Actor's token](#inject-the-actors-token). + +1. Create the Secret in the namespace of the EgressPolicy. To rotate the credential later, update the Secret. The gateway reads the value from the Secret, so the agent's revision does not change. + ```bash + kubectl create secret generic ticket-auth -n kagent \ + --from-literal=token= + ``` + +2. Create an EgressPolicy with an `authentication` block for the destination. + ```yaml + kubectl apply -f - <}} The gateway sets the header to the prefix followed by the Secret value. + + | Field | Description | + | ----- | ----------- | + | `origin` | The destination. An origin with a credential must be exact, so the API server rejects a wildcard origin. On a cleartext `http` origin, the credential is readable on the path between the gateway and the destination. Use an `http` origin with a credential only for an in-cluster service whose traffic a service mesh encrypts. | + | `authentication.header` | The request header that carries the credential, such as `Authorization` or `X-API-Key`. The API server rejects the headers that control routing and framing, such as `Host` and `Content-Length`, and the `Proxy-Authorization`, `Proxy-Authenticate`, and `Cookie` headers. | + | `authentication.prefix` | Optional. Text that the gateway puts before the Secret value. The prefix is literal. Include the trailing space, as in `"Bearer "`. When the prefix is omitted or empty, the header contains only the Secret value. | + | `authentication.secretKeyRef` | The Secret and key that store the credential. Both `name` and `key` are required, and the reference cannot be optional. A destination sets exactly one of `secretKeyRef` and `actorJwt`. For the token alternative, see [Inject the Actor's token](#inject-the-actors-token). | + +3. Select the policy. A merge patch replaces the whole `egressPolicyRefs` list, so the command also keeps `node-dependencies`. Include every policy that the resource needs. + {{< tabs >}} + {{% tab name="Agent" %}} + ```bash + kubectl patch agent my-first-agent -n kagent --type merge \ + -p '{"spec":{"egressPolicyRefs":[{"name":"node-dependencies"},{"name":"ticket-api"}]}}' + ``` + {{% /tab %}} + {{% tab name="Standalone sandbox" %}} + ```bash + kubectl patch sandboxtemplate scratch -n kagent --type merge \ + -p '{"spec":{"egressPolicyRefs":[{"name":"node-dependencies"},{"name":"ticket-api"}]}}' + ``` + {{% /tab %}} + {{< /tabs >}} + +4. In the script or SDK that sends the request, send the header with a placeholder value. The gateway replaces a header that the request already sends, and it does not add a header that is missing. Selecting a policy does not configure a tool or set an environment variable. The client must also trust the egress gateway's certificate authority. kagent provides this certificate authority to every Actor. For the variables that carry the trust bundle, see [Environment variables on an Actor]({{< link path="agents/agent-harness#environment-variables-on-an-actor" >}}). + ```bash + curl -H 'Authorization: Bearer kagent-credential-injected' \ + https://tickets.example.com/tickets + ``` + +### Inject the Actor's token + +A Secret-backed credential is the same for every agent that selects the policy, so the destination cannot tell the agents apart. When an internal API must authorize or audit each agent, inject a token that identifies the calling {{< gloss "Actor" >}}Actor{{< /gloss >}} instead. This method covers the requests that an agent's own code sends, such as a `curl` command or a script that a skill runs. For model and MCP server calls, set `actorJwt` on the ModelConfig or the RemoteMCPServer, as described in [Actor tokens for other services]({{< link path="about/substrate-runtime/identity#actor-tokens-for-other-services" >}}). + +The gateway mints, injects, and refreshes the token, so the policy needs no Secret, and you have no credential to rotate. + +1. Create an EgressPolicy whose `authentication` block sets `actorJwt`. The gateway replaces the header with the prefix followed by a newly issued Actor token. + ```yaml + kubectl apply -f - <}} For the token's claims and how the destination validates it, see [Actor tokens for other services]({{< link path="about/substrate-runtime/identity#actor-tokens-for-other-services" >}}). + + | Field | Description | + | ----- | ----------- | + | `authentication.actorJwt.audiences` | The services that can accept the token, between 1 and 16 values. The destination checks that its own audience is in the list. `kagent-runtime` is reserved and rejected. | + | `authentication.actorJwt.expirationSeconds` | Optional. The token lifetime, between 300 and 3600 seconds. Defaults to 900. | + +2. Select the policy. A merge patch replaces the whole `egressPolicyRefs` list, so include every policy that the resource needs. + {{< tabs >}} + {{% tab name="Agent" %}} + ```bash + kubectl patch agent my-first-agent -n kagent --type merge \ + -p '{"spec":{"egressPolicyRefs":[{"name":"node-dependencies"},{"name":"ticket-api"}]}}' + ``` + {{% /tab %}} + {{% tab name="Standalone sandbox" %}} + ```bash + kubectl patch sandboxtemplate scratch -n kagent --type merge \ + -p '{"spec":{"egressPolicyRefs":[{"name":"node-dependencies"},{"name":"ticket-api"}]}}' + ``` + {{% /tab %}} + {{< /tabs >}} + +3. In the client that sends the request, send the header with a placeholder value. + ```bash + curl -H 'Authorization: Bearer kagent-credential-injected' \ + https://tickets.example.com/tickets + ``` + +## Troubleshoot a grant + +The `EgressPoliciesApplied` condition on the Agent or SandboxTemplate reports whether kagent admitted the grants. This condition is independent of `Ready`. An Agent with a missing or invalid policy still reports `Ready`, and it still serves Sessions with its inferred destinations. + +The condition message names the affected policies, and the reason identifies the problem. + +| Status | Reason | Meaning | +| ------ | ------ | ------- | +| `True` | `Applied` | kagent admitted every selected policy. | +| `False` | `EgressPolicyNotFound` | A selected policy does not exist, or is being deleted. | +| `False` | `InvalidEgressPolicy` | A policy, a Secret reference, or the `egressPolicyRefs` list is invalid. For example, the Secret or its key does not exist. | +| `False` | `EgressGrantConflict` | A new grant conflicts with an existing one. kagent holds the new grants, and keeps the previously admitted grants that are still selected and compatible. | + +A problem with one policy does not block the others. A policy that is missing or invalid does not grant any access, and the remaining policies still apply. An `egressPolicyRefs` list that is itself invalid revokes every grant for that resource. For example, one name that is not a valid Kubernetes resource name, such as `Node-Deps`, revokes the grants of every other policy in the list. + +A grant conflict happens in the following cases: + +- Two policies set a different Secret or prefix for the same origin and header. +- A grant sets a credential for an origin that already has a credential from the model or a tool. The credential from the model or the tool always takes precedence. +- A grant sets a credential for an origin that uses token passthrough. Token passthrough and a Secret credential cannot share an origin. +- The compiled policy would exceed 256 rules. + +In a standalone Sandbox, a destination that no grant allows does not resolve. A client therefore reports a name-resolution error, not an HTTP error. For example, `curl` fails with `Could not resolve host` and exit code `6`. For how to find a denied request in the gateway's log, see [Diagnose a denied request]({{< link path="about/substrate-runtime/networking-and-egress#diagnose-a-denied-request" >}}). + +## Access control + +An EgressPolicy is namespace-scoped, and kagent does not check who may select it. A user who can create or edit an Agent or a SandboxTemplate in a namespace can select every EgressPolicy in that namespace and use its credentials. Treat the namespace as the boundary for sharing policies and the Secrets that they reference. To give each team a namespace of its own, see [Run agents in multiple namespaces]({{< link path="operations/multiple-namespaces" >}}). For more information about how Kubernetes role-based access control (RBAC) governs who can write these resources, see [The Kubernetes plane]({{< link path="about/substrate-runtime/identity#the-kubernetes-plane" >}}). + +To control who can grant access, give the permission to create and edit EgressPolicy resources only to administrators. The kagent controller also writes EgressPolicy resources, with its own service account, for the **Policies** page of the kagent UI and the `EgressPolicyService` API. By default, kagent does not authenticate that path, so any caller that reaches the kagent API can create, edit, or delete a policy regardless of their Kubernetes permissions. For more information, see [Resources that the API writes]({{< link path="about/substrate-runtime/identity#resources-that-the-api-writes" >}}). + +The gateway keeps a credential out of the sandbox, but it cannot stop an allowed destination from returning the credential in a response. Grant a credential only to a destination that you trust with it. + +## Clean up + +1. Remove the policies from the resource. + {{< tabs >}} + {{% tab name="Agent" %}} + ```bash + kubectl patch agent my-first-agent -n kagent --type json \ + -p '[{"op":"remove","path":"/spec/egressPolicyRefs"}]' + ``` + {{% /tab %}} + {{% tab name="Standalone sandbox" %}} + ```bash + kubectl patch sandboxtemplate scratch -n kagent --type json \ + -p '[{"op":"remove","path":"/spec/egressPolicyRefs"}]' + ``` + {{% /tab %}} + {{< /tabs >}} + +2. Delete the EgressPolicy resources and the Secret. + ```bash + kubectl delete egresspolicy node-dependencies ticket-api -n kagent + kubectl delete secret ticket-auth -n kagent + ``` + +## Next steps + +{{< cards >}} + {{< card link=`{{< link path="about/substrate-runtime/networking-and-egress" >}}` title="Egress control" subtitle="Learn how the egress gateway authorizes each connection." >}} + {{< card link=`{{< link path="about/substrate-runtime/standalone-sandboxes" >}}` title="Standalone sandboxes" subtitle="Create sandboxes that agents and scripts can run commands in." >}} +{{< /cards >}} diff --git a/docs-site/content/kagent/main/agents/human-in-the-loop.md b/docs-site/content/kagent/main/agents/human-in-the-loop.md new file mode 100644 index 00000000..914c7c8f --- /dev/null +++ b/docs-site/content/kagent/main/agents/human-in-the-loop.md @@ -0,0 +1,275 @@ +--- +title: Human in the loop +description: Pause an agent before it acts, return the request to a person, and resume the turn once the person answers. +weight: 50 +author: kagent.dev +--- + +Some agents only answer questions, and those can run unattended. Others call tools that change a cluster, spend money, or read sensitive information. An agent that takes actions like those should get a person's approval first. The human in the loop (HITL) mechanism lets an agent stop mid-turn, return a question or a pending tool call to a person, and continue once that person answers. + +A working HITL setup needs support from both the agent and the client. To make the agent pause, set `requireApproval` on a tool binding in an {{< gloss "AgentTemplate" >}}AgentTemplate{{< /gloss >}}. The client must support the Agent2Agent (A2A) human-in-the-loop extension, so that the client can present the request to a person and return the decision to the agent. Before you configure an agent to pause, confirm that the client that you plan to use supports the extension. For the clients that support the extension, see [Client support](#client-support). + +## Pause lifecycle + +A client opens every turn by telling the agent which optional A2A extensions the client can handle. A client that requests the HITL extension on that message can answer a pause later in the turn. A client that does not request the extension still receives a reply when the turn finishes. When the agent stops for a person, that client does not receive a payload to answer. + +The following diagram traces one turn in which the agent stops. + +```mermaid +flowchart TB + caller["Client sends a message and asks
to activate the HITL extension"] + working["Agent works on the turn"] + decision{"Does the agent need
a person?"} + pause["Task state becomes
INPUT_REQUIRED"] + request["Status message contains a
tool_approval_request
or ask_user_request"] + answer["Client sends a response message
on the same task"] + done["Agent finishes the turn"] + + caller --> working + working --> decision + decision -->|no| done + decision -->|yes| pause + pause --> request + request --> answer + answer --> working +``` + +The agent works until it either finishes the turn or needs a person. When it needs a person, the task moves to `INPUT_REQUIRED` and its status message contains either a `tool_approval_request` or an `ask_user_request`. The client answers by sending a response message on the same task, and the agent resumes the turn where it left off. + +## Pause kinds + +An agent pauses either to get permission before it acts or to ask a question. A tool approval and a question raise different requests, but both pause and resume in the same way. A client that handles one handles the other with a different payload. + +| Request | Raised when | The client answers with | +| ------- | ----------- | ----------------------- | +| `tool_approval_request` | The agent wants to call a tool from a binding that sets `requireApproval`. | `tool_approval_response` | +| `ask_user_request` | The agent calls the built-in `ask_user` tool because it needs information that only a person has. | `ask_user_response` | + +Every pause also adds a plain text part to its status message that states why the agent stopped. A client that never activated the extension receives that text and nothing else. The client can show a person why the agent stopped. The client does not have an `id` to answer with, so the task stays at `INPUT_REQUIRED`. Treat the text as a diagnostic, and send the turn again from a client that requests the extension. The `hint` field of an activated payload contains the same string. + +The `kagent` harness composes this text from the pending calls. The wording depends on what the agent stopped for. + +| Pause | Text | +| ----- | ---- | +| `ask_user` | The questions the agent asked, joined with `; `. | +| A tool pause whose tools supplied hints | The hints, followed by the tool names in parentheses, such as `Deleting this file requires approval (delete_file, restart_pod)`. | +| A tool pause with no hints | `Approval is required for tool(s): delete_file` | +| A pause that does not include a tool call | `Human input is required before the agent can continue.` | + +The `codex` and `claude` harnesses compose the text differently. Both pass through the hint that Codex or Claude Code supplied with the request, so the wording comes from the coding agent. + +### Disable questions + +When an agent runs unattended, nobody answers its question, so the task waits at `INPUT_REQUIRED`. To stop agents on the `kagent` harness from asking questions, set the `config.runtimeDefaults.builtinTools.disableAskUser` Helm value to `true`. The value removes the `ask_user` tool from every agent and subagent in the installation, and an AgentTemplate cannot override it. The value defaults to `false`. + +```yaml +config: + runtimeDefaults: + builtinTools: + disableAskUser: true +``` + +The value removes only the question tool. A binding that sets `requireApproval` still pauses the agent for approval, and a remote A2A agent that the agent calls can still pause the turn with its own question. The `codex` and `claude` harnesses do not read the value, because the coding agent supplies its own tools. A `byo` image decides whether to read it. + +## Require approval for a tool + +If you set `requireApproval` on an `mcp` tool binding in the AgentTemplate, the agent stops before each call to a tool that the binding exposes. + +### Harness support + +The Harness that runs an agent decides which pauses the agent can raise, and how precisely a binding can express approval. + +| Harness | Tool approval | `ask_user` question | Splitting one server across two bindings | +| ------- | ------------- | ------------------- | ---------------------------------------- | +| `kagent` | Supported | Supported | Supported | +| `codex` | Supported | Supported | Rejected, with `RemoteMCPServer "" is bound more than once`. | +| `claude` | Supported | Unsupported, because the upstream Claude Code tool that backed it was removed. A `claude` agent pauses for tool approval only. | Rejected, with `RemoteMCPServer "" is bound more than once`. A binding whose tool selection kagent cannot verify against the server's discovered tools exposes the whole server and reports a warning. | +| `byo` | Decided by your image | Decided by your image | Not applicable | + +kagent passes each binding's `requireApproval` setting to a `byo` image, and the image decides whether to honor it. The agent card for a `byo` Agent never declares the HITL extension. The card omits it even when the image implements the extension, so a client cannot learn from the card that the agent pauses. For what else that card omits, see [Bring your own agent]({{< link path="agents/bring-your-own-agent#compare-byo-and-declarative-agents" >}}). + +### Configure a binding + +In the following AgentTemplate, the binding sets `requireApproval`, so the agent stops before it calls either Kubernetes tool. + +```yaml +apiVersion: api.kagent.dev/v1alpha3 +kind: AgentTemplate +metadata: + name: cluster-operator + namespace: kagent +spec: + tools: + - mcp: + server: + kind: RemoteMCPServer + name: kagent-tool-server + tools: + - k8s_delete_resource + - k8s_patch_resource + requireApproval: true +``` + +{{< reuse "kagent-docs/snippets/review-table.md" >}} For the rest of the binding's fields, see [About tools]({{< link path="skills-and-mcp/about-tools" >}}). + +| Field | Description | +| ----- | ----------- | +| `mcp.tools` | The names of the tools to bind. To bind every tool that the server offers, omit the list or leave it empty. | +| `mcp.requireApproval` | Pauses before each invocation of a tool that this binding exposes. The pause covers the tools in `mcp.tools`, or every tool on the server when `mcp.tools` is omitted or empty. To run the bound tools without approval, omit the field. | + +Approval belongs to the binding rather than to the tool name, so one server can supply tools that require approval alongside tools that do not. Bind the tools that require approval in a binding that sets `requireApproval`, and bind the tools that do not in a second binding that omits it. The `codex` and `claude` harnesses reject a second binding to the same server. An agent on those harnesses applies one approval setting to every tool that the server offers. + +> [!NOTE] +> kagent 0.x named the tools that needed approval in a `requireApproval` list on the Agent resource, which matched tool names across every server. + +A tool runs without a pause unless a binding that sets `requireApproval` exposes it. These tools include every built-in tool, such as file access, shell, and web search, and every Model Context Protocol (MCP) tool on a binding that omits the field. The Actor's sandbox confines those calls. For more information, see [Sandboxing]({{< link path="about/substrate-runtime/sandboxing" >}}). + +## Implement a client that supports human approval + +HITL is an [A2A](https://a2a-protocol.org) message extension, identified by a versioned Uniform Resource Identifier (URI). The following sections follow a client through one pause, from the header that it sends to the response that it returns. + +### Request the extension + +A client requests the extension by setting the extension URI as the `A2A-Extensions` header on the call that sends a message. + +```http +A2A-Extensions: https://kagent.dev/extensions/hitl/v1 +``` + +kagent activates the extension only for calls that request it. kagent then echoes the activated URI back and attaches a structured pause payload only to an activated call. A declarative agent's card declares the extension, so a client can discover HITL support before it sends anything. + +A gRPC call specifies the {{< gloss "Agent" >}}Agent{{< /gloss >}} in the standard A2A `tenant` field, as `/`, and requests the extension through a header. Every A2A call also sends the `A2A-Version: 1.0` header. Port-forward the controller's gRPC port first, as in [Install kagent]({{< link path="setup/installation" >}}). + +```bash +grpcurl -plaintext \ + -H 'A2A-Version: 1.0' \ + -H 'A2A-Extensions: https://kagent.dev/extensions/hitl/v1' \ + -d '{ + "tenant": "kagent/cluster-operator", + "message": { + "messageId": "msg-1", + "role": "ROLE_USER", + "parts": [{"text": "Delete the obsolete pod in the production namespace."}] + } + }' localhost:8083 lf.a2a.v1.A2AService/SendStreamingMessage +``` + +A message that omits both a `contextId` and a task ID starts a new {{< gloss "Session" >}}Session{{< /gloss >}}. To continue an existing one, set the message's `contextId` to the Session ID. Over HTTP, the URL path `/agents/{namespace}/{name}` selects the Agent instead, and no tenant is needed. For both transports, see [Call an agent over A2A]({{< link path="examples/a2a-agents" >}}). + +> [!WARNING] +> A send that omits the header produces a pause that no one can answer. The turn still stops, and its status message states the question as prose, without `metadata` or a correlation `id`. The client therefore does not have anything to render or an `id` to answer with. Re-reading that task with the header does not recover the payload, because kagent never attached one. To prevent an unanswerable pause, send the header on every call. Sending it on a read is harmless, and missing it on a send cannot be undone. + +### Read the pause + +When the agent pauses, the task moves to `INPUT_REQUIRED`. The payload arrives in the status message's `metadata`, keyed by the extension URI. The message's `extensions` array lists the same URI. Each payload includes a `type` field that identifies its shape. kagent stores an attached payload with the task. A later read returns it whether or not that read requests the extension. + +Each type travels in one direction. An agent raises a request, and a client answers with the matching response. + +| Type | Direction | +| ---- | --------- | +| `tool_approval_request` | Agent to client | +| `ask_user_request` | Agent to client | +| `tool_approval_response` | Client to agent | +| `ask_user_response` | Client to agent | + +### Approve or reject a tool + +A `tool_approval_request` lists the pending calls, each with an `id`, the tool `name`, and the `args` that the agent chose. Present the pending calls to a person, then send one `tool_approval_response` that decides every call in the request. + +```json +{ + "type": "tool_approval_response", + "approvals": [ + { "id": "", "approved": true }, + { "id": "", "approved": false, "rejection_reason": "Deleting that namespace is out of scope." } + ] +} +``` + +A rejection reason is optional. The agent receives the reason and can adjust its next action. + +### Answer a question + +An `ask_user_request` contains an `id` and a list of `questions`. The response echoes the same `id` and answers the questions in order. + +```json +{ + "type": "ask_user_response", + "id": "", + "answers": [ + { "answer": ["us-east-1"] } + ] +} +``` + +## Resume a paused task + +A paused task waits for the client to send a message that contains the response payload, on the same task and context. kagent rejects a resume attempt on a task that is not waiting. + +The response message must list the extension URI in its `extensions` array as well as in its `metadata`. The `kagent` harness ignores a `metadata` payload that the array does not declare, and delivers the message as ordinary text. The task resumes and the agent replies, so the mistake is silent. The `codex` and `claude` harnesses reject the message with `tool approval response must declare https://kagent.dev/extensions/hitl/v1`. + +The following message answers the `ask_user_request` in [Answer a question](#answer-a-question). The `taskId` and `contextId` values come from the paused task, and the `id` in the payload comes from the request. + +```bash +grpcurl -plaintext \ + -H 'A2A-Version: 1.0' \ + -H 'A2A-Extensions: https://kagent.dev/extensions/hitl/v1' \ + -d '{ + "tenant": "kagent/cluster-operator", + "message": { + "messageId": "msg-2", + "role": "ROLE_USER", + "taskId": "", + "contextId": "", + "extensions": ["https://kagent.dev/extensions/hitl/v1"], + "metadata": { + "https://kagent.dev/extensions/hitl/v1": { + "type": "ask_user_response", + "id": "", + "answers": [{ "answer": ["us-east-1"] }] + } + }, + "parts": [{"text": "us-east-1"}] + } + }' localhost:8083 lf.a2a.v1.A2AService/SendMessage +``` + +The agent continues the turn with the answer. `SendMessage` returns after the turn finishes, with the task in `TASK_STATE_COMPLETED`. + +While the task waits, kagent suspends the {{< gloss "Actor" >}}Actor{{< /gloss >}} to a durable {{< gloss "Snapshot" >}}snapshot{{< /gloss >}} in the same way as at the end of a turn. The snapshot keeps the durable volume and discards the running process. Each harness records the pending request in that volume and continues the turn when the answer arrives. + +The {{< gloss "Worker" >}}Worker{{< /gloss >}} is released in the meantime, so a task that waits at `INPUT_REQUIRED` does not consume pool capacity. The wait also survives the loss of the node. For more information, see [Suspend and resume]({{< link path="about/substrate-runtime/suspend-and-resume" >}}). + +The `claude` harness resumes its deferred tool call. The `codex` harness runs the approved MCP call itself and continues with the result. On these two harnesses, only one pending request survives the suspension. The `codex` harness cancels any other request along with the interrupted turn. The `claude` harness denies a protected call that arrives in a parallel batch, but a single remaining call can still pause for approval. If an approved call fails after it starts, the harness cannot resume the task. To continue the work, start a new Session. + +The {{< gloss "Transcript" >}}transcript{{< /gloss >}} only grows, so the question and the answer both stay in the task history. A later reader can see what was asked and what a person decided. + +## Task states + +The following A2A task states mean that the task has stopped and is waiting. + +| State | Meaning | +| ----- | ------- | +| `INPUT_REQUIRED` | The agent is waiting for a person. A tool approval or a question produces this state. | +| `AUTH_REQUIRED` | The agent is waiting for credentials. kagent's own harnesses never set this state. kagent accepts a resume from `AUTH_REQUIRED`, so a `byo` image that sets this state works. | + +## Agents bound as tools + +An agent that a parent binds as a tool can raise a pause of its own. The request then includes a `nested` block that identifies the subagent, along with its task and context. A client can therefore tell the person which agent is asking. + +## Client support + +The AgentTemplate decides that a turn pauses, and the client that raised the turn decides whether a person can answer it. To add support to a client of your own, see [Implement a client that supports human approval](#implement-a-client-that-supports-human-approval). + +| Client | HITL | +| ------ | ---- | +| Your own A2A client | Supported. Request the extension URI and handle every payload type. | +| An {{< gloss "MCP" >}}MCP{{< /gloss >}} client that supports tasks | Supported. The `invoke_agent` tool returns a task, and input requests surface as MCP elicitations. | +| The kagent CLI | Not supported. `kagent agent invoke` does not request the extension, so a turn that pauses stays at `INPUT_REQUIRED` and the CLI cannot answer it. The CLI reports `Input required to continue this Session.` and stops there. | + +## Next steps + +{{< cards >}} + {{< card link=`{{< link path="skills-and-mcp/about-tools" >}}` title="About tools" subtitle="Bind the tools that an approval request would cover." >}} + {{< card link=`{{< link path="agents/system-prompts" >}}` title="System prompts" subtitle="Tell an agent when to ask rather than act." >}} +{{< /cards >}} diff --git a/docs-site/content/kagent/main/agents/scheduled-runs.md b/docs-site/content/kagent/main/agents/scheduled-runs.md new file mode 100644 index 00000000..23c17934 --- /dev/null +++ b/docs-site/content/kagent/main/agents/scheduled-runs.md @@ -0,0 +1,237 @@ +--- +title: Scheduled runs +description: Run an agent on a cron schedule that sends a fixed prompt to a new Session for each execution. +weight: 65 +author: kagent.dev +--- + +A schedule sends a fixed prompt to an {{< gloss "Agent" >}}Agent{{< /gloss >}} on a cron schedule. Each execution of the schedule starts a new {{< gloss "Session" >}}Session{{< /gloss >}}. You can open that Session afterward and send more messages to the agent. You can also start an execution immediately, outside the schedule. + +kagent stores schedules in its database, not as Kubernetes resources. You manage schedules in the kagent UI or through the `kagent.api.v1.ScheduledRunService` gRPC API. The `kagent` CLI does not manage schedules. + +## Before you begin + +1. [Install kagent]({{< link path="setup/installation" >}}). To call the API with grpcurl, set `config.server.grpcReflection=true`. grpcurl uses server reflection to find the `ScheduledRunService` methods. +2. Create an Agent that answers a prompt, such as the one in [Your first agent]({{< link path="get-started/your-first-agent" >}}). + +## Create a schedule in the UI + +The kagent UI builds the cron expression from a form, so the UI is the simplest way to create a schedule. + +1. [Open the kagent UI]({{< link path="ui/launch-ui" >}}). +2. In the navigation, click **Schedules**, and then click **New Schedule**. If the agent has no schedules yet, you can also click **Create a schedule** on the **Schedules** card of the **Agent Details** page. + - In the **Agent** list, select the agent to run. + - Optional: In **Schedule Name**, enter a name of up to 200 characters. + - In **Time zone**, enter or select an Internet Assigned Numbers Authority (IANA) time zone name, such as `America/New_York`. The field starts as `UTC`. + - In **Repeat**, select how often the agent runs. Each option shows its own fields. + * **Every few minutes** asks for an interval of 1, 5, 10, 15, 20, or 30 minutes. + * **Hourly** asks for the minute past the hour. + * **Daily** asks for the time. + * **Weekly** asks for the days and the time. + * **Monthly** asks for the day of the month and the time. + * **Custom (advanced)** asks for a cron expression with five fields. + - In **Prompt**, enter the message that the agent receives on each execution. + - Optional: In **Execution timeout (seconds)**, change the limit for each execution. The default is 900 seconds. + - To create the schedule in a paused state, turn off **Enable Schedule**. +3. Click **Create schedule**. The UI opens the page for the new schedule. + +The **Schedules** page lists the schedule with its status and its next execution time, in your browser's local time zone. + +## Manage a schedule in the UI + +The page for one schedule contains its controls and execution history. To open the page, click the schedule's name on the **Schedules** page or on the **Schedules** card of the agent's **Agent Details** page. + +* **Run** starts an execution immediately. A manual execution does not change the next scheduled execution. **Run** also works on a paused schedule. +* **Pause** stops future scheduled executions. **Resume** starts them again. +* **Edit** changes the schedule's configuration. +* **Delete schedule**, in the **Danger zone** section, stops future executions. Executions that kagent already accepted continue to the end. The execution history and the Sessions remain. + +The **Execution history** table shows the trigger, state, and failure reason of each execution. To open the Session that an execution started, click **Open conversation**. + +## Manage schedules with the API + +The `ScheduledRunService` API exposes the same operations as the UI, so a script or a continuous integration (CI) job can manage schedules. The examples in this section use [grpcurl](https://github.com/fullstorydev/grpcurl) to call the API, and they target an Agent named `my-first-agent` in the `kagent` namespace. Requests use snake_case field names, and responses use camelCase field names, such as `time_zone` and `timeZone`. + +1. Port-forward the controller's gRPC port. The Service name assumes that your Helm release is named `kagent`. + ```bash + kubectl port-forward -n kagent svc/kagent-controller 8083:8083 + ``` + +2. Create a schedule. The required `request_id` field, from 1 to 128 characters, makes the call safe to retry. A second call with the same `request_id` and the same inputs returns the existing schedule. A second call with the same `request_id` and different inputs returns `AlreadyExists`. + ```bash + grpcurl -plaintext -d '{ + "agent": {"namespace": "kagent", "name": "my-first-agent"}, + "request_id": "weekday-check-1", + "config": { + "name": "Weekday cluster check", + "schedule": "0 9 * * 1-5", + "time_zone": "America/New_York", + "prompt": "Summarize any pods that are not Running in the kagent namespace.", + "execution_timeout": "900s" + } + }' localhost:8083 kagent.api.v1.ScheduledRunService/CreateScheduledRun + ``` + + Example output: + ```json + { + "scheduledRun": { + "id": "1df2a30a-9030-4674-9841-620ab6bc22ca", + "creator": "admin@kagent.dev", + "config": { + "name": "Weekday cluster check", + "schedule": "0 9 * * 1-5", + "timeZone": "America/New_York", + "prompt": "Summarize any pods that are not Running in the kagent namespace.", + "executionTimeout": "900s" + }, + "etag": "a711c6b3-4721-4fa9-b6a7-2b85e87ea84d", + "nextExecutionTime": "2026-11-04T14:00:00Z", + ... + } + } + ``` + +3. Save the `id` and the `etag` values from the `CreateScheduledRun` response as environment variables. The trigger, pause, and delete calls take the `id`. The pause call also takes the `etag`. + ```bash + export SCHEDULE_ID= + export ETAG= + ``` + +4. Start an execution immediately. A repeated call with the same `request_id` returns the same execution. + ```bash + grpcurl -plaintext -d @ localhost:8083 kagent.api.v1.ScheduledRunService/TriggerScheduledRun < + ``` + +6. Check the execution. + ```bash + grpcurl -plaintext -d @ localhost:8083 kagent.api.v1.ScheduledRunService/GetScheduledRunExecution <}}) rules that apply to Sessions. Only the creator can see, change, trigger, or delete the schedule. A call from any other user returns `NotFound`, not a permission error. The Sessions that a schedule creates belong to the same creator. A [session share]({{< link path="agents/share-a-session" >}}) never grants access to a schedule. + +The creator's identity does not select credentials for the model or for tool calls. Each execution uses the credentials that are configured on the Agent. + +To create schedules from unattended automation, call the API as a Kubernetes ServiceAccount. For how the controller turns a ServiceAccount into a creator, see [Service account callers]({{< link path="about/substrate-runtime/identity#service-account-callers" >}}). + +> [!WARNING] +> **Creator ownership protects a schedule only when every call reaches the controller through an authenticating boundary.** `config.server.auth.mode` defaults to `insecure`. In this mode, the caller supplies its own identity in the `X-User-Id` header, and a call without that header runs as `admin@kagent.dev`. Any caller can therefore claim to be a schedule's creator and see, change, trigger, or delete the schedule. In `trusted-proxy` mode, the controller does not verify the token, so a caller that reaches port `8083` directly can also claim any creator. For more information, see [Creator ownership]({{< link path="about/substrate-runtime/identity#creator-ownership" >}}). For the modes themselves, see [Controller authentication modes]({{< link path="about/substrate-runtime/identity#controller-authentication-modes" >}}). + +## Execution behavior + +Each scheduled time or manual trigger creates one execution record. The execution record stores only the prompt that the schedule sent. The Session stores the transcript. A reply that you send later in that Session does not change the execution. + +An execution moves through these states. + +| State | Meaning | +| --- | --- | +| `PENDING` | kagent accepted the execution and is preparing its Session. | +| `RUNNING` | kagent sent the prompt to the agent. | +| `SUCCEEDED` | The agent's task completed. | +| `FAILED` | The agent's task ended in a state other than completed, or the Session was deleted before the task finished. The `failure_reason` field states the cause. | +| `TIMED_OUT` | The execution timeout elapsed before the agent's task ended. The `failure_reason` field is `Execution deadline elapsed`. kagent cancels a running task. A task that waits for [human approval]({{< link path="agents/human-in-the-loop" >}}) is not cancelled, and stays in the Session. A task that completes after the deadline also ends as `TIMED_OUT`. | + +The following rules control when an execution runs and what its record stores. + +* **Missed times are skipped.** When kagent cannot accept an execution within 30 seconds of its scheduled time, kagent skips that time and waits for the next one. A controller outage therefore does not produce a burst of executions when the controller returns. +* **Configuration changes apply to future executions.** An execution uses the prompt and the timeout that were current when kagent accepted it. +* **A prompt is sent only once.** If the agent's runtime does not accept the prompt, kagent does not resend it, because a resend could duplicate the agent's work. +* **An execution needs the Agent.** If the Agent is deleted after you create the schedule, or has no ready revision, each execution waits in `PENDING` until the timeout, and then ends as `TIMED_OUT`. +* **History outlives its Sessions.** An execution keeps its `session_id` after the Session is deleted. The ID is a historical reference and does not mean that the Session still exists. + +## Next steps + +{{< cards >}} + {{< card link=`{{< link path="about/substrate-runtime/identity" >}}` title="Controller identity" subtitle="How the controller resolves the creator that owns a schedule." >}} + {{< card link=`{{< link path="agents/human-in-the-loop" >}}` title="Human in the loop" subtitle="Approval requests during an agent's task." >}} +{{< /cards >}} diff --git a/docs-site/content/kagent/main/agents/share-a-session.md b/docs-site/content/kagent/main/agents/share-a-session.md new file mode 100644 index 00000000..d4e71992 --- /dev/null +++ b/docs-site/content/kagent/main/agents/share-a-session.md @@ -0,0 +1,44 @@ +--- +title: Session sharing +description: Give another account access to one conversation with a share token, and bound its lifetime with a ttl the controller can cap. +weight: 60 +author: kagent.dev +--- + +A share lets the owner of a {{< gloss "Session" >}}Session{{< /gloss >}} give another account access to that one conversation. Creating a share produces a token; a caller presenting that token reaches the shared Session without becoming its creator. kagent exposes three RPCs for this: `CreateSessionShare`, `ListSessionShares`, and `RevokeSessionShare`. There is no `kagent` CLI verb for sharing. Sharing is a gRPC and UI procedure, not something you script from the command line. + +## A share is a capability, not a grant to a user + +Every share is read-only. The holder can A2A `get`, `list`, and `subscribe` on the shared Session. The holder cannot send a message, cancel a task, or create a Session. + +The share does not act with the owner's access. When a caller presents a share token, the controller checks the caller's own identity against policy and uses the share only to locate the record's owner in storage. Checking the owner instead would lend the owner's grants to visitors. A holder needs their own read permission on the Session for a share to be useful: the share supplies the lookup, the holder's own credentials supply the authority. The share does not name a person, and it does not create a second principal. + +> [!WARNING] +> **`insecure` mode is not a security boundary in a default install.** `config.server.auth.mode` defaults to `insecure`, where a caller supplies its own identity in `X-User-Id` and the authorizer it installs permits every check. A caller can claim any identity and reach the Session without any share token at all. The boundary holds only under `trusted-proxy`, with a credential-validating proxy in front of the controller. For the modes themselves, see [Controller authentication modes]({{< link path="about/substrate-runtime/identity#controller-authentication-modes" >}}). + +## Bound the lifetime with `ttl` + +When you create a share you can set `CreateSessionShareRequest.ttl` to a positive lifetime, counted from creation. The returned `SessionShare.expiresAt` records when the token stops granting access; once that time passes, the token lookup returns not found, and the owner still sees the expired share in `ListSessionShares` and can revoke it. + +An operator caps share lifetimes with `config.sessions.shareMaxTtl`: + +```yaml +config: + sessions: + shareMaxTtl: 86400s +``` + +A positive cap rejects a `ttl` that exceeds it and supplies the lifetime for a request that omits `ttl`. When no cap is set, an omitted `ttl` leaves `expiresAt` unset, and the share then grants access until it is revoked or the Session is deleted. The default `0s` leaves a share unbounded unless the client sets `ttl`, and a negative value is rejected at controller startup. Durations in the `config` block are protobuf seconds, so `86400s` rather than `24h`. For the idle-Session limit that separately governs the conversation itself, see [Expire idle Sessions]({{< link path="operations/operational-considerations#expire-idle-sessions" >}}). + +## Spending and revoking a share + +A caller spends the share by presenting the token in the `X-Share-Token` header. The controller resolves the token to the Session's owner and scopes the call to that Session. + +The owner revokes a share with `RevokeSessionShare`, keyed by the `SessionShare.id`. Revoking deletes the share: the record does not remain, and it no longer appears in `ListSessionShares`. Deleting the Session removes its shares with it. + +## Next steps + +{{< cards >}} + {{< card link=`{{< link path="about/substrate-runtime/identity" >}}` title="Controller identity" subtitle="How authentication modes and shares scope who can reach a Session." >}} + {{< card link=`{{< link path="operations/operational-considerations" >}}` title="Operational considerations" subtitle="Idle Session expiry and the share lifetime cap." >}} +{{< /cards >}} diff --git a/docs-site/content/kagent/main/agents/structured-output.md b/docs-site/content/kagent/main/agents/structured-output.md new file mode 100644 index 00000000..57a075fc --- /dev/null +++ b/docs-site/content/kagent/main/agents/structured-output.md @@ -0,0 +1,381 @@ +--- +title: Structured output +description: Constrain an AgentTemplate's final answer to a JSON Schema, set inline or in a ConfigMap, and troubleshoot schemas and answers that fail validation. +weight: 35 +author: kagent.dev +--- + +An {{< gloss "AgentTemplate" >}}AgentTemplate{{< /gloss >}} can require that its final answer is JSON matching a schema. {{< reuse "kagent-docs/snippets/name-product.md" >}} checks the schema when it builds a {{< gloss "Revision" >}}revision{{< /gloss >}} for an {{< gloss "Agent" >}}Agent{{< /gloss >}}. The revision records the schema and its digest. kagent does not build a revision from a schema that fails the checks, so an Agent that has never been ready cannot start a {{< gloss "Session" >}}Session{{< /gloss >}}. At runtime, the agent validates its complete answer against the recorded schema before it publishes the answer. + +The schema goes in one of two AgentTemplate fields, `spec.outputSchema` or `spec.outputSchemaFrom`. The fields are mutually exclusive. An AgentTemplate that sets both is rejected when you apply it, with the message `outputSchema and outputSchemaFrom are mutually exclusive`. If you omit both fields, the agent's final answer is not constrained. + +| Field | Description | +| ----- | ----------- | +| `outputSchema` | The schema, written inline in the AgentTemplate. | +| `outputSchemaFrom.name` | The ConfigMap that stores the schema as JSON, in the AgentTemplate's namespace. | +| `outputSchemaFrom.key` | The key within that ConfigMap. | +| `outputSchemaStrict` | Whether the model provider constrains generation to the schema. Defaults to `false`. For the providers that honor it, see [Strict mode](#strict-mode). | + +## Before you begin + +1. [Create your first agent]({{< link path="get-started/your-first-agent" >}}), so that you have the `my-first-harness` Harness and the `default-model-config` {{< gloss "ModelConfig" >}}ModelConfig{{< /gloss >}} that the AgentTemplates in this guide use. That guide also has you install the kagent CLI and `jq`. + + Structured output needs a Harness resource that selects the `kagent` harness with the `golang-adk` image, as `my-first-harness` does. For a Harness of your own, use the `kagent` block and the same `workload.image` as `my-first-harness`. On the `codex`, `claude`, or `byo` harness, an Agent whose template sets a schema is rejected with `Harness runtime "" does not support structured output`. For the differences between harnesses, see [Choose a harness]({{< link path="agents/agent-harness#choose-a-harness" >}}). + +2. Check your kagent CLI version. The steps on this page need the {{< reuse "kagent-docs/versions/kagent.md" >}} CLI. A CLI from another release can fail with `unknown command`. + ```bash + kagent version + ``` + + When `kagent_version` in the output differs from {{< reuse "kagent-docs/versions/kagent.md" >}}, replace the CLI. + + {{< reuse "kagent-docs/snippets/install-cli.md" >}} + +## Set the schema inline + +Set `spec.outputSchema` to keep the schema in the AgentTemplate, so that the schema and the rest of the agent's configuration change together. + +1. Apply an AgentTemplate with a schema, and an Agent that pairs it with the `my-first-harness` Harness. + ```yaml + kubectl apply -f - <`, which cannot be recursive +- `$schema` and `$id` +- `pattern`, in RE2 syntax, on a schema whose `type` includes `string` +- `format`, on a schema whose `type` includes `string`, with one of the values `date`, `date-time`, `duration`, `email`, `hostname`, `ipv4`, `ipv6`, `time`, or `uuid` + +Anything else is rejected, including `oneOf` and `allOf`, conditional schemas such as `if` and `then`, tuple arrays, references outside `$defs`, and validation keywords such as `minimum` and `minLength`. Claude structured outputs do not support word boundaries, so kagent also rejects a `pattern` that uses `\b` or `\B`. + +JSON Schema treats `format` as an annotation. kagent forwards `format` to the model provider, but the runtime does not check it on the answer. + +A Gemini model answers by calling the `set_model_response` tool of the Go Agent Development Kit (ADK), and that tool checks each property against one type. The check applies to a model whose name, after its last `/`, starts with `gemini-`, on any provider. For such a model, kagent rejects a property that has no `type`, such as one that sets only `enum` or `const`. kagent also rejects a property that uses `anyOf`, `type: null`, or a type list with more than one non-null type. kagent also rejects an array property that does not set `items`. To allow `null` in such a property, use a type list such as `["string", "null"]`. The check skips models on the `GeminiVertexAI` provider. + +kagent also enforces the following limits when it builds the revision. A schema that exceeds a limit is rejected, before any agent runs with it. + +| Limit | Maximum | +| ----- | ------- | +| Size of the schema as kagent reads it | 64 KiB | +| Nesting depth | 32 levels | +| Schema nodes | 1,000 | + +kagent counts depth and nodes through `properties`, `items`, and `anyOf`, and counts a `$defs` entry again each time that it is referenced. The size limit applies before kagent normalizes the schema, so whitespace in a ConfigMap value counts toward it. + +## Strict mode + +OpenAI models follow `format` and `pattern` during generation only in strict mode. To constrain generation to the schema, set `outputSchemaStrict` to `true` in the AgentTemplate. + +Strict mode accepts fewer schemas. kagent checks the following rules for every provider, so a schema stays valid when you change the provider of the ModelConfig. + +- The root does not use `anyOf`. +- Every node sets `type`, `anyOf`, `enum`, or `$ref`. +- A node that sets `$ref` does not set any other keyword. To add `null`, wrap the reference in `anyOf`, such as `anyOf: [{$ref: "#/$defs/item"}, {type: "null"}]`. +- Every object sets `additionalProperties` to `false` and lists all of its properties in `required`. A map with free-form keys cannot meet this rule. + +To make a property optional in strict mode, list it in `required` and add `null` to its type list. + +```yaml +spec: + outputSchemaStrict: true + outputSchema: + type: object + properties: + ticket: + type: string + pattern: "^OPS-[0-9]+$" + due: + type: [string, "null"] + format: date + required: + - ticket + - due + additionalProperties: false +``` + +The following fields turn on strict mode, make `due` optional, and constrain both values. + +| Field | Description | +| ----- | ----------- | +| `outputSchemaStrict` | Sends the schema in strict mode to the OpenAI, Azure OpenAI, and Azure AI Foundry providers. Azure AI Foundry receives it only with the `OpenAI` API format. Other providers ignore the field. | +| `outputSchema.properties.ticket.pattern` | Requires a ticket ID such as `OPS-123`. | +| `outputSchema.properties.due.type` | Allows `null`, which makes `due` optional. | +| `outputSchema.properties.due.format` | Asks for a date such as `2026-11-03`. | + +An AgentTemplate that sets `outputSchemaStrict` to `true` without a schema is rejected when you apply it, with the message `outputSchemaStrict requires outputSchema or outputSchemaFrom`. The runtime validates every answer against the schema in both modes. + +## Read the answer + +A successful answer is published as one {{< gloss "A2A" >}}A2A{{< /gloss >}} (Agent2Agent) `DataPart` with the media type `application/json`. The part's metadata includes the SHA-256 digest of the schema that the answer was validated against, under the key `kagent.dev/a2a/output-schema-sha256`. kagent computes the digest after it normalizes the schema, so two schemas that differ only in formatting or key order have the same digest. The Agent's card lists `application/json` as its default output mode, instead of `text`. + +To see the part and its metadata, print the task as JSON. + +```bash +kagent agent invoke --session $SESSION_ID --task "What is 3 plus 5?" -o json \ + | jq '.task.artifacts[-1].parts' +``` + +Example output: +```json +[ + { + "data": { + "answer": 8, + "explanation": "Three plus five equals eight." + }, + "metadata": { + "kagent.dev/a2a/output-schema-sha256": "59a6341782be5eacb0762a62133c7b45ec785770392216e712f109f11afb6a1b" + }, + "mediaType": "application/json" + } +] +``` + +The runtime holds back the agent's partial output while the agent writes its answer, and publishes only the complete answer. Progress updates, tool calls, approval requests, and input-required messages keep their usual form, and the agent can call tools before it answers. A streaming client that needs only the answer reads the last artifact with content before the task reaches `TASK_STATE_COMPLETED`. That artifact's part includes the `kagent.dev/a2a/output-schema-sha256` metadata key. + +## Agents as tools + +The schema applies only to the root agent, which is the agent of the AgentTemplate that the Agent names. An AgentTemplate [bound to it as a subagent]({{< link path="skills-and-mcp/about-tools#subagents-as-tools" >}}) does not inherit the schema. If the bound AgentTemplate has a schema of its own, that schema applies only to Agents that name it directly. + +If the subagent uses a `Shared` binding, the root agent can hand the conversation to the bound agent, which then answers in its place, in that turn and in the turns that follow. Those tasks fail with `output_validation_failed: root agent produced no result artifact`, because the answers do not come from the root agent. The bound agent's answers still reach the caller as text. Give a schema only to an AgentTemplate whose own agent writes the final answer. + +## Answers that fail validation + +Structured output does not have a fallback to text. If the model refuses, stops early, or returns an answer that is not JSON or does not match the schema, the task fails. kagent does not publish the invalid answer or include it in the failure message, because it can contain sensitive data. When the agent uses a Gemini model on the `Gemini` provider, however, the agent is told to give its answer as the arguments of a `set_model_response` tool call. kagent publishes that call like any other tool call, before it validates the answer, so an answer given that way reaches the caller even when it fails validation. + +For example, an answer that does not match the schema fails the `kagent agent invoke` command with output like the following. The `output_validation_failed:` message states the cause. + +```console +processor failed: output_validation_failed: root agent output does not conform to its schema +Error: Session task 01a0ec27-157f-7034-bb89-09855f485b2a ended in TASK_STATE_FAILED +``` + +In the CLI output, each of the following messages follows the `processor failed:` prefix. + +| Message | Cause | +| ------- | ----- | +| `output_validation_failed: root agent output does not conform to its schema` | The answer is JSON, but it does not match the schema. | +| `output_validation_failed: root agent output is not valid JSON` | The answer is not JSON. | +| `output_validation_failed: root agent did not complete structured output` | The model stopped for a reason other than finishing normally, such as reaching its token limit. | +| `output_validation_failed: root agent produced no structured value` | The model did not return any answer text. | +| `output_validation_failed: root agent produced no result artifact` | The task ended without a structured answer from the root agent, for example because the root agent handed the conversation to an [agent bound as a tool](#agents-as-tools). | + +## Troubleshooting + +A schema problem appears on the Agent's status conditions. `ResolvedRefs` reports a ConfigMap or key that kagent cannot find. `Compatible` reports a schema that kagent found but cannot accept, or a Harness whose runtime is not `kagent`. + +```bash +kubectl get agent structured-answer-shared -n kagent -o json \ + | jq '[.status.conditions[] | select(.type == "ResolvedRefs" or .type == "Compatible")]' +``` + +For example, if the `shared-schemas` ConfigMap no longer has the `arithmetic-answer` key, the output is similar to the following. When `ResolvedRefs` fails, `Compatible` reports `Blocked` instead of a result of its own. + +```json +[ + { + "lastTransitionTime": "2026-09-29T07:52:08Z", + "message": "resolve output schema ConfigMap \"shared-schemas\": key \"arithmetic-answer\" not found", + "observedGeneration": 1, + "reason": "ReferenceResolutionFailed", + "status": "False", + "type": "ResolvedRefs" + }, + { + "lastTransitionTime": "2026-09-29T07:52:08Z", + "message": "blocked by ResolvedRefs", + "observedGeneration": 1, + "reason": "Blocked", + "status": "False", + "type": "Compatible" + } +] +``` + +The `message` field of the failing condition identifies the cause. + +| Message | Condition and reason | Cause | +| ------- | -------------------- | ----- | +| `resolve output schema ConfigMap "x": not found` | `ResolvedRefs`, `ReferenceResolutionFailed` | The ConfigMap does not exist in the AgentTemplate's namespace. | +| `resolve output schema ConfigMap "x": key "y" not found` | `ResolvedRefs`, `ReferenceResolutionFailed` | The ConfigMap does not have that key under `data`. | +| `invalid output schema: schema is empty` | `Compatible`, `UnsupportedConfiguration` | The key's value is empty. | +| `invalid output schema: schema exceeds 65536 bytes` | `Compatible`, `UnsupportedConfiguration` | The schema is larger than 64 KiB. | +| `invalid output schema: decode JSON: ...` | `Compatible`, `UnsupportedConfiguration` | The value is not valid JSON. | +| `invalid output schema: decode trailing JSON: ...` | `Compatible`, `UnsupportedConfiguration` | Text that is not JSON follows the JSON value. | +| `invalid output schema: schema must contain one JSON value` | `Compatible`, `UnsupportedConfiguration` | The value contains more than one JSON value, such as two objects in a row. | +| `invalid output schema: schema does not match the portable output profile: ...` | `Compatible`, `UnsupportedConfiguration` | The schema uses a keyword or form outside the [portable output profile](#supported-schemas), or its root is not an object. | +| `invalid output schema: resolve schema: ...` | `Compatible`, `UnsupportedConfiguration` | kagent cannot resolve the schema, for example because a `$ref` names a `$defs` entry that does not exist. | +| `invalid output schema: pattern ...` | `Compatible`, `UnsupportedConfiguration` | A `pattern` is not valid RE2, or uses a word boundary. | +| `invalid output schema: patterns compile to more than 100000 regexp instructions ...` | `Compatible`, `UnsupportedConfiguration` | The schema's patterns together compile to more than 100,000 regexp instructions. | +| `output schema for Gemini: ...` | `Compatible`, `UnsupportedConfiguration` | A property breaks the [Gemini rule](#supported-schemas) for a single type, or an array property does not set `items`. | +| `outputSchemaStrict: ...` | `Compatible`, `UnsupportedConfiguration` | The schema breaks a [strict mode](#strict-mode) rule. | +| `output schema is incompatible with Go ADK: ...` | `Compatible`, `UnsupportedConfiguration` | The schema has a recursive reference, or exceeds the depth or node limit. | +| `Harness runtime "x" does not support structured output` | `Compatible`, `UnsupportedConfiguration` | The Harness does not use the `kagent` harness. | + +If the Agent has never been ready, `kagent agent session create` fails with `Agent does not have a ready prepared revision`. + +> [!WARNING] +> A Session keeps the revision that it was created from, and validates its answers against that revision's schema. After you change a schema, create a new Session to use it. A changed schema that fails the checks does not stop new Sessions from starting. They start from the Agent's latest ready revision, which `status.latestSuccessfulRevision` reports, so they use the previous schema. + +A change to a ConfigMap does not change the AgentTemplate's `metadata.generation`, so check each condition's `lastTransitionTime` to tell whether kagent has seen your fix. + +## Clean up + +1. Delete the Sessions that you created in this guide. + ```bash + kagent agent session list -o json \ + | jq -r '.sessions[] | select(.agent.name == "structured-answer" or .agent.name == "structured-answer-shared") | .id' \ + | xargs -n1 kagent agent session delete + ``` + +2. Delete the Agents, the AgentTemplates, and the ConfigMap. + ```bash + kubectl delete agent structured-answer structured-answer-shared -n kagent + kubectl delete agenttemplate structured-answer structured-answer-shared -n kagent + kubectl delete configmap shared-schemas -n kagent + ``` + +## Next steps + +{{< cards >}} + {{< card link=`{{< link path="agents/system-prompts" >}}` title="System prompts" subtitle="Tell the agent what to put in the fields that the schema requires." >}} + {{< card link=`{{< link path="agents/agent-harness" >}}` title="Agent harness" subtitle="Compare the four harnesses and what each one supports." >}} +{{< /cards >}} diff --git a/docs-site/content/kagent/main/agents/system-prompts.md b/docs-site/content/kagent/main/agents/system-prompts.md new file mode 100644 index 00000000..7c720f06 --- /dev/null +++ b/docs-site/content/kagent/main/agents/system-prompts.md @@ -0,0 +1,167 @@ +--- +title: System prompts +description: Set an agent's system prompt inline or from a ConfigMap, and template it with values that kagent resolves when it builds a revision. +weight: 30 +author: kagent.dev +--- + +An {{< gloss "AgentTemplate" >}}AgentTemplate{{< /gloss >}}'s system prompt defines the agent's role and how it should behave. {{< reuse "kagent-docs/snippets/name-product.md" >}} resolves the prompt when it builds a {{< gloss "Revision" >}}revision{{< /gloss >}}, so the text that an agent runs with is fixed for the life of a {{< gloss "Session" >}}Session{{< /gloss >}}. Editing the prompt affects sessions created after the edit, but not ones that are already running. + +## Write an effective prompt + +A prompt that works tends to carry four things, in roughly this order. + +- **Role.** What the agent is, in a sentence, such as `You are a Kubernetes assistant.` +- **Scope.** What the agent should and should not take on, which matters more as you give it more tools. +- **Instructions.** How the agent should behave in the cases that you care about: when to ask for clarification, what to do when a tool fails, and when to refuse. +- **Response format.** What a good answer looks like. For example, ask for Markdown, or for a summary before detail. + +Two things are worth stating explicitly, because models otherwise guess: what the agent should do when it does not know an answer, and whether it should act or ask first when an action is consequential. For approval gates that the runtime enforces rather than the prompt, see [Human in the loop]({{< link path="agents/human-in-the-loop" >}}). + +## Set the prompt inline + +Set `spec.systemPrompt` to keep the prompt in the AgentTemplate itself, so that the prompt and the rest of the agent's configuration change together. Inline suits a prompt that only one AgentTemplate uses. + +```yaml +spec: + systemPrompt: |- + You are a Kubernetes assistant. You help users understand what is running in their cluster. + + # Instructions + - Ask for clarification before running a tool when a request is ambiguous. + - Answer from tool output rather than from memory of how clusters usually look. + - Say so plainly when a question cannot be answered with the tools you have. +``` + +Write the value as a YAML block scalar, such as the `|-` in the example, so that the blank lines, headings, and lists that structure the prompt reach the model as written. A folded scalar collapses them into a single paragraph. + +## Store the prompt in a ConfigMap + +Use `systemPromptFrom` to keep the prompt outside the AgentTemplate. Several AgentTemplates can then share one prompt, and a prompt can change without editing the agent. + +1. Create a ConfigMap holding the prompt. + ```yaml + kubectl apply -f - < [!NOTE] +> A prompt can come only from a ConfigMap. Earlier versions of kagent also accepted a Secret, through a `systemMessageFrom.type` field that v1alpha3 does not have. A system prompt is not a credential, so keep secrets out of it and pass them to the runtime as [Harness environment variables]({{< link path="agents/agent-harness" >}}) instead. + +## Template the prompt + +Set `spec.promptTemplate` to run the prompt through [Go templates](https://pkg.go.dev/text/template) before it reaches the model. Templating applies to both inline prompts or prompts set in a ConfigMap. + +```yaml +spec: + description: Answers questions about a Kubernetes cluster. + systemPrompt: |- + You are {{ .AgentTemplateName }}. {{ .Description }} + + You have these tools available: {{ .ToolNames }} + + {{ include "shared-prompts/response-format" }} + promptTemplate: + dataSources: + - name: shared-prompts +``` + +The following values are available to a template. + +| Value | Description | +| ----- | ----------- | +| `.AgentTemplateName` | The AgentTemplate's `metadata.name`. | +| `.AgentTemplateNamespace` | The AgentTemplate's `metadata.namespace`. | +| `.Description` | The AgentTemplate's `spec.description`. | +| `.ToolNames` | The tool names selected from every {{< gloss "MCP" >}}MCP{{< /gloss >}} server the AgentTemplate binds. Agents bound as tools are not included. Prefer this value over listing tools by hand, because it cannot drift from the bindings that the AgentTemplate declares. | + +Additionally, the `include` function pulls in one key from a ConfigMap that `dataSources` lists. The argument is formatted `"/"`, where the source is the ConfigMap's name. Every key in every listed ConfigMap becomes available, so two sources that share a key name collide. kagent rejects the collision when it builds the revision. + +| Field | Description | +| ----- | ----------- | +| `dataSources[].name` | A ConfigMap in the AgentTemplate's namespace, up to 20 entries. | +| `dataSources[].alias` | An alternative identifier to use in `include` paths. The ConfigMap is still looked up by `name`. | + +An alias only changes what you type. In this example configuration, `include "house-style/tone"` fails and `include "style/tone"` succeeds. + +```yaml +spec: + systemPrompt: |- + {{ include "shared-prompts/response-format" }} + + {{ include "style/tone" }} + promptTemplate: + dataSources: + - name: shared-prompts + - name: house-style + alias: style +``` + +> [!NOTE] +> A prompt template can reach only the supported values and the ConfigMaps that `dataSources` names. Templates cannot read arbitrary Kubernetes objects. + +## Troubleshooting + +Prompt problems surface on the `ResolvedRefs` condition, because they are reference failures rather than runtime errors. kagent does not build a new revision. An {{< gloss "Agent" >}}Agent{{< /gloss >}} that has never been ready cannot start a Session, and one that was ready before keeps starting new Sessions from its last ready revision. + +An AgentTemplate does not report status of its own, so read the conditions from the Agent that names it. + +```bash +kubectl get agent my-first-agent -n kagent -o json \ + | jq '[.status.conditions[] | select(.type == "ResolvedRefs")]' +``` + +In the output, a failure sets the reason to `ReferenceResolutionFailed` and the resolve error is listed in the message. Example output: + +```json +[ + { + "type": "ResolvedRefs", + "status": "False", + "observedGeneration": 2, + "lastTransitionTime": "2026-08-31T15:02:10Z", + "reason": "ReferenceResolutionFailed", + "message": "resolve systemPromptFrom: ConfigMap \"shared-prompts\" does not contain key \"kubernetes-assistant\"" + } + ] +} +``` + +| Message | Cause | +| ------- | ----- | +| `resolve systemPromptFrom: ConfigMap "x" not found` | The ConfigMap does not exist in the AgentTemplate's namespace. | +| `resolve systemPromptFrom: ConfigMap "x" does not contain key "y"` | The ConfigMap exists but has no such key. | +| `resolve prompt source "x": ConfigMap not found` | A `dataSources` entry names a ConfigMap that does not exist. | +| `duplicate prompt template identifier "x/y"` | Two data sources expose the same `source/key` path. Give one of them an `alias`. | +| `prompt template "x/y" not found, available: [...]` | An `include` path does not match any available key. The error lists every path that is available. | +| `parse system message template: ...` | The template is not valid Go template syntax. | + +## Next steps + +{{< cards >}} + {{< card link=`{{< link path="skills-and-mcp/about-tools" >}}` title="About tools" subtitle="Bind the tools that a prompt can refer to." >}} + {{< card link=`{{< link path="agents/agent-memory" >}}` title="Agent memory" subtitle="Let an agent carry what it learned into later conversations." >}} +{{< /cards >}} diff --git a/docs-site/content/kagent/main/examples/_index.md b/docs-site/content/kagent/main/examples/_index.md new file mode 100644 index 00000000..65bb6f3c --- /dev/null +++ b/docs-site/content/kagent/main/examples/_index.md @@ -0,0 +1,6 @@ +--- +title: Examples +description: Work through end-to-end examples that put kagent's runtime, tools, and Agent Substrate features together. +weight: 80 +author: kagent.dev +--- diff --git a/docs-site/content/kagent/main/examples/a2a-agents.md b/docs-site/content/kagent/main/examples/a2a-agents.md new file mode 100644 index 00000000..10f10318 --- /dev/null +++ b/docs-site/content/kagent/main/examples/a2a-agents.md @@ -0,0 +1,436 @@ +--- +title: Call an agent over A2A +description: Use the A2A service that the kagent controller serves to read an Agent's card, send it a message, and stream a reply. +weight: 30 +author: kagent.dev +--- + +Every {{< gloss "Agent" >}}Agent{{< /gloss >}} is reachable over the {{< gloss "A2A" >}}A2A{{< /gloss >}} (Agent2Agent) protocol through the kagent controller. This example calls that service directly to read an Agent's card, send it a message, stream a reply, and answer the agent when it pauses for a person. + +In {{< reuse "kagent-docs/snippets/name-product.md" >}}, agent traffic runs over A2A. The CLI and the [Model Context Protocol (MCP) server]({{< link path="examples/agents-via-mcp" >}}) reach an Agent through the same service. + +This example uses [grpcurl](https://github.com/fullstorydev/grpcurl) to show the requests and replies directly. Real callers use an A2A client library rather than assembling requests by hand. + +## About the kagent A2A service + +The controller serves `lf.a2a.v1.A2AService` on port `8083`, alongside its REST API and its MCP endpoint. The kagent CLI reaches the same port and the same service. + +Every A2A call must send the `A2A-Version: 1.0` header, as gRPC metadata or as an HTTP header. Without it, the controller rejects the call with `this version is not supported: requested "0.3", served "1.0"`. The A2A client libraries set the header automatically, but a hand-assembled request such as a `grpcurl` call must set it. + +A gRPC caller identifies the Agent in the standard A2A `tenant` field, as `/`, and the controller routes the call to that Agent. The same controller serves A2A over HTTP JSON-RPC, where the URL path selects the Agent instead: + +| Transport | How a caller selects the Agent | +| --------- | ----------------------------- | +| gRPC | The `tenant` field on each request, as `/` | +| HTTP JSON-RPC | `POST /agents/{namespace}/{name}`, with the card at `GET /agents/{namespace}/{name}/.well-known/agent-card.json` | + +An HTTP request that also sets the `tenant` field must set a value that matches its URL. The controller does not serve a deployment-wide agent card, because it serves many Agents. + +Both forms replace the `/api/a2a///` URL paths that kagent 0.x served over HTTP. + +A caller addresses the Agent rather than one Session. The message's `contextId` selects the Session. Omitting both a `contextId` and a task ID starts a new {{< gloss "Session" >}}Session{{< /gloss >}}, and repeating a `contextId` continues that Session. + +If the Agent, the Session, or the task does not exist, an A2A call fails with `TaskNotFound` (JSON-RPC code `-32001`, or gRPC `NOT_FOUND`). Over HTTP, a request for the card of an Agent that does not exist returns `404`. To learn when kagent deletes a Session, see [Expire idle Sessions]({{< link path="operations/operational-considerations#expire-idle-sessions" >}}). + +{{< reuse "kagent-docs/snippets/examples/a2a-port-warning.md" >}} + +### A2A methods + +The following table covers the methods that this example uses. The `TaskPushNotificationConfig` calls manage a task's callbacks, and this example reaches them through the kagent CLI. For the further methods that `lf.a2a.v1.A2AService` defines, see the [A2A protocol specification](https://a2a-protocol.org/latest/specification/#31-core-operations). + +| Method | What it does | +| ------ | ------------ | +| `GetExtendedAgentCard` | Returns the agent card describing the Agent. | +| `SendMessage` | Sends a message and returns the task after the agent either finishes the turn or pauses for a person. | +| `SendStreamingMessage` | Sends a message and streams events as the agent works. | +| `GetTask` | Reads a task that a previous call created. | +| `CancelTask` | Stops a task that is still running. | +| `CreateTaskPushNotificationConfig` | Adds or replaces a callback for a task that is still active. | +| `ListTaskPushNotificationConfigs` | Lists the active callbacks for one task. | +| `DeleteTaskPushNotificationConfig` | Removes a task callback. | + +## Before you begin + +{{< reuse "kagent-docs/snippets/grpcurl-prerequisites.md" >}} + +## Read the agent card + +An A2A client typically starts by reading the agent card, which tells it what the agent is and which protocol features the agent supports. + +1. Fetch the card for your Agent. + ```bash + grpcurl -plaintext \ + -H 'A2A-Version: 1.0' \ + -d '{"tenant":"'"$AGENT"'"}' \ + localhost:8083 lf.a2a.v1.A2AService/GetExtendedAgentCard + ``` + + Example output: + ```json + { + "name": "my_first_agent", + "description": "My first kagent agent", + "supportedInterfaces": [ + { + "url": "http://kagent-controller.kagent.svc:8083/agents/kagent/my-first-agent", + "protocolBinding": "JSONRPC", + "protocolVersion": "1.0" + }, + { + "url": "http://kagent-controller.kagent.svc:8083", + "protocolBinding": "GRPC", + "protocolVersion": "1.0", + "tenant": "kagent/my-first-agent" + } + ], + "version": "v1", + "capabilities": { + "streaming": true, + "pushNotifications": true, + "extensions": [ + { + "uri": "https://kagent.dev/extensions/hitl/v1", + "description": "Human in the loop for tool approval, ask user, and nested subagents" + }, + { + "uri": "https://kagent.dev/extensions/usage/v1", + "description": "Task-lifetime token usage" + } + ], + "extendedAgentCard": true + }, + "defaultInputModes": ["text"], + "defaultOutputModes": ["text"] + } + ``` + +2. Read the card for what a caller acts on. + * Both `name` and `description` come from the {{< gloss "AgentTemplate" >}}AgentTemplate{{< /gloss >}}. In `name`, kagent replaces hyphens with underscores. The `description` value is `spec.description` verbatim, so a caller sees the description that you wrote. + * Each `supportedInterfaces` URL is the controller's in-cluster address rather than the Actor's, because a caller reaches the agent through the controller. The card advertises JSON-RPC first, then gRPC. The JSON-RPC URL identifies the Agent in its path. The gRPC entry sets the `tenant` value instead. + * The `capabilities.pushNotifications` value advertises task callbacks. Each callback belongs to one task, not to the whole Session. + * The `capabilities.extensions` list identifies the optional extensions that the agent supports. A client opts into the human-in-the-loop extension per call to answer pauses. Through the usage extension, the agent reports the tokens that a task uses. For the payload, see [Read token usage](#read-token-usage). + +> [!NOTE] +> The card does not include `skills`. kagent 0.x let you declare agent card skills in an `a2aConfig` block. However, v1alpha3 does not have that field, so kagent generates the card only from the AgentTemplate's name and description. An AgentTemplate's `spec.skills` field is a different feature: those are [Agent Skills]({{< link path="skills-and-mcp/skills" >}}) that the agent can use, not advertisements to a caller. + +## Send a message + +`SendMessage` blocks until the agent finishes the turn, or pauses to ask a person, and then returns the whole task. A message needs its own ID, a role, and at least one part. + +A message part can also contain a file. Put the file's bytes in the part's `raw` field, which JSON encodes as base64, and set `filename` and `mediaType` on the same part. To send a link to a file, set the part's `url` field in place of `raw`. When the ModelConfig sets `apiFormat: responses`, or the provider is Gemini, the model fetches the file from the link. Every other model receives the link as text. + +> [!NOTE] +> Unlike the UI, which checks each file as you stage it, this path accepts any media type. When the ModelConfig sets `apiFormat: responses`, a PDF, `.docx`, `.xlsx`, or `.pptx` file reaches the model as a file. Any other file whose media type is not text-like reaches the model as the note `[Uploaded file "report.pdf" (application/pdf) could not be read as text.]` instead of contents. The controller's gRPC server rejects any message over 16 MiB, framing included. For how files reach the model, and the file types and sizes that the UI accepts, see [Attach files to a message]({{< link path="ui/launch-ui#attach-files-to-a-message" >}}). + +1. Send a message to the Agent. Omitting a `contextId` starts a new Session. + ```bash + grpcurl -plaintext \ + -H 'A2A-Version: 1.0' \ + -d '{"tenant":"'"$AGENT"'","message":{"messageId":"'"$(uuidgen)"'","role":"ROLE_USER","parts":[{"text":"What is 7 times 6? Answer with just the number."}]}}' \ + localhost:8083 lf.a2a.v1.A2AService/SendMessage + ``` + + The reply text arrives in `artifacts`, not in `status`. Example output, with the message history omitted: + ```json + { + "task": { + "id": "01a06cfb-a9ae-7ddb-be98-baaf17414998", + "contextId": "01a068e3-aeb6-7abc-8d6f-5ba9becd3143", + "status": { + "state": "TASK_STATE_COMPLETED", + "timestamp": "2026-09-04T15:13:52.990137169Z" + }, + "artifacts": [ + { + "artifactId": "01a06cfb-bf2c-70ea-8e65-e3ef15133a96", + "parts": [{ "text": "42" }] + } + ], + "history": [ ] + } + } + ``` + The task returns two identifiers, `id` and `contextId`, and a caller uses them differently. + * The `id` identifies one turn, and every message returns a new one. + * The `contextId` is the Session ID. A second message that repeats this `contextId` continues the same Session. + + Task metadata uses the `kagent.dev/a2a/` namespace. An artifact includes `kagent.dev/a2a/timeline-position`, an RFC 3339 timestamp that orders task history. The task itself includes `kagent.dev/a2a/task-created-at`. A runtime that reports per-event model usage publishes it as `kagent.dev/a2a/usage` on a task event rather than on the artifact. For task-lifetime usage, see [Read token usage](#read-token-usage). + +2. Save the task's `id` and the Session's `contextId`, so that later steps can read the task and continue the same Session. + ```bash + export TASK_ID= + export SESSION_ID= + ``` + +## Stream a reply + +`SendStreamingMessage` takes the same request and returns a sequence of events instead of one result. A caller can then show a reply as the agent produces it. + +1. Send a message on the streaming method. + ```bash + grpcurl -plaintext \ + -H 'A2A-Version: 1.0' \ + -d '{"tenant":"'"$AGENT"'","message":{"messageId":"'"$(uuidgen)"'","contextId":"'"$SESSION_ID"'","role":"ROLE_USER","parts":[{"text":"Count from 1 to 3."}]}}' \ + localhost:8083 lf.a2a.v1.A2AService/SendStreamingMessage + ``` + +2. Read the event sequence. The stream opens with the task at `TASK_STATE_SUBMITTED`, moves to `TASK_STATE_WORKING`, and then emits an artifact update for each chunk of the reply. Every chunk shares one `artifactId`, so a client appends the chunks into a single artifact. Example output, abbreviated to the text of each event: + ```console + "state": "TASK_STATE_SUBMITTED" + "state": "TASK_STATE_WORKING" + "artifactId": "01a06cfd-1c3c-7e65-8650-2e86f5d7f5eb", "text": "1" + "artifactId": "01a06cfd-1c3c-7e65-8650-2e86f5d7f5eb", "text": "," + "artifactId": "01a06cfd-1c3c-7e65-8650-2e86f5d7f5eb", "text": " " + "artifactId": "01a06cfd-1c3c-7e65-8650-2e86f5d7f5eb", "text": "2" + ``` + +## Read a task later + +A task outlives the call that created it, so a caller that lost its connection can still read the result. + +1. Read the task by ID. + ```bash + grpcurl -plaintext \ + -H 'A2A-Version: 1.0' \ + -d '{"tenant":"'"$AGENT"'","id":"'"$TASK_ID"'"}' \ + localhost:8083 lf.a2a.v1.A2AService/GetTask + ``` + +2. Read the task's fields. `GetTask` returns the task itself, rather than wrapping it in a `task` field the way `SendMessage` does. The `status`, `artifacts`, and `history` values are the ones that the original call returned. + ```json + { + "id": "01a06cfb-a9ae-7ddb-be98-baaf17414998", + "contextId": "01a068e3-aeb6-7abc-8d6f-5ba9becd3143", + "status": { + "state": "TASK_STATE_COMPLETED", + "timestamp": "2026-09-04T15:13:52.990137169Z" + }, + "artifacts": [ + { + "artifactId": "01a06cfb-bf2c-70ea-8e65-e3ef15133a96", + "parts": [{ "text": "42" }] + } + ], + "history": [ ] + } + ``` + +A task that is still running reports `TASK_STATE_WORKING` and does not have artifacts yet. To stop a task that is still running, call `CancelTask` with the same `id`. + +## Read token usage {#read-token-usage} + +The `kagent` harness reports the model tokens that a task uses through the optional usage extension, `https://kagent.dev/extensions/usage/v1`. The agent card of an Agent on this harness advertises the extension. A `byo` image that is built with the Go or Python helper also reports usage, but the agent card does not advertise the extension. The `codex` and `claude` harnesses do not report usage. + +1. Read a task that completed, paused for input, failed, or was canceled. Use the `GetTask` call from [Read a task later](#read-a-task-later). + +2. In the task's `metadata` field, find the entry that is keyed by the usage extension URI. + ```json + "metadata": { + "https://kagent.dev/extensions/usage/v1": { + "inputTokens": 1000, + "outputTokens": 150, + "reasoningTokens": 40, + "cachedInputTokens": 800, + "totalTokens": 1190, + "models": [ + { + "model": "gpt-4o-2024-11-20", + "inputTokens": 1000, + "outputTokens": 150, + "reasoningTokens": 40, + "cachedInputTokens": 800, + "totalTokens": 1190 + } + ] + } + } + ``` + + Each count covers every model call in the task, not only the calls of the latest execution. + + | Field | Meaning | + | ----- | ------- | + | `inputTokens` | Prompt tokens sent to the model. | + | `outputTokens` | Generated tokens, excluding reasoning tokens. | + | `reasoningTokens` | Reasoning tokens, when the provider reports them. | + | `cachedInputTokens` | Input tokens served from the provider's cache. This value is a subset of `inputTokens`. | + | `totalTokens` | The provider-reported total, or `inputTokens + outputTokens + reasoningTokens` when a call does not return a total. | + | `models` | The same counts per model, ordered by model name. The top-level totals also count calls that do not name a model, but `models` does not list those calls. | + + When a task pauses and resumes, use the latest payload instead of adding it to an earlier one. For a canceled task, the payload includes the model calls that completed before the cancellation. The runtime does not count a call that the cancellation cuts off before the provider sends its final response. + + If no model call returns token counts, the task metadata does not include the usage entry. + +## Register task callbacks {#task-callbacks} + +A task callback lets a webhook learn when a task needs input or reaches a final state. The callback receives a status update for future `INPUT_REQUIRED`, `AUTH_REQUIRED`, `COMPLETED`, `FAILED`, `CANCELED`, and `REJECTED` states. After the callback arrives, the receiver calls authenticated `GetTask` to read the current state and artifacts. + +kagent keeps one pending delivery for each callback, and a newer state replaces a delivery that has not yet succeeded. When a task resumes, kagent discards the pending delivery for the state that the task left, so a receiver does not see every state. + +The kagent CLI wraps the push notification configuration methods, so this section uses the CLI. The CLI defaults to port 8083, which the prerequisites already forward. + +1. To register a callback before a new task is dispatched, add `--push-url` to the kagent CLI invocation. The command returns immediately with the task ID and callback ID. + ```bash + kagent agent invoke \ + --session $SESSION_ID \ + --task "Run the long task." \ + --push-url https://receiver.example/callback \ + -o json + ``` + +2. Save the task ID from the command output so that later steps can address the task. + ```bash + export PUSH_TASK_ID= + ``` + +3. To add a second callback to a task that is still active, create one explicitly. The server rejects new callbacks after the task reaches a terminal state. + ```bash + kagent agent session push create \ + $SESSION_ID $PUSH_TASK_ID \ + --id audit-webhook \ + --url https://receiver.example/callback + ``` + +4. List the callbacks for the task. + ```bash + kagent agent session push list $SESSION_ID $PUSH_TASK_ID + ``` + +5. Delete a callback that should no longer receive task updates. + ```bash + kagent agent session push delete $SESSION_ID $PUSH_TASK_ID audit-webhook + ``` + +### Secure callback deliveries + +A callback URL is a public endpoint that receives task state from kagent. Before you register a callback in production, decide how the receiver establishes that a delivery came from kagent. + +> [!IMPORTANT] +> kagent signs deliveries only when the installation sets a signing key, and `controller.push.signing.secretName` is empty by default. Until you set it, kagent posts each callback without an `Authorization` header, and the controller returns `404` at `/.well-known/jwks.json`. A receiver cannot distinguish an unsigned kagent delivery from any other caller, so use the default configuration only on a trusted network. + +kagent does not generate its own signing key, so you are responsible for creating the key, keeping it, and rotating it. The following steps turn signing on. + +1. Generate an Ed25519 private key in unencrypted PKCS#8 PEM form. + ```bash + openssl genpkey -algorithm ed25519 -out private-key.pem + ``` + + > [!NOTE] + > The `openssl` binary that ships with macOS is LibreSSL, which rejects this command with `Algorithm ed25519 not found`. Install OpenSSL 3, or generate the key on another machine. + +2. Store the key in a Secret. kagent reads only the `private-key.pem` entry, so the entry must have that name. + ```bash + kubectl create secret generic kagent-push-signing -n kagent \ + --from-file=private-key.pem=private-key.pem + ``` + +3. Set `controller.push.signing.secretName` to the Secret name, and set `config.push.issuer` to a URL that reaches the controller. Use the Helm release name and version that you installed kagent with. + ```bash + helm upgrade kagent oci://ghcr.io/kagent-dev/kagent/helm/kagent \ + --namespace kagent --reuse-values \ + --set controller.push.signing.secretName=kagent-push-signing \ + --set config.push.issuer=https://kagent.example + ``` + + The issuer that a receiver trusts must match the host that it fetches the JSON Web Key Set (JWKS) from. When you omit `config.push.issuer`, Helm derives it from `ui.externalUrl` and then from the in-cluster A2A gateway URL. Either fallback can point to a host that does not serve the JWKS. + +4. Confirm that the controller publishes a verification key. Before you set the Secret, this path returns `404`. + ```bash + curl -s http://localhost:8083/.well-known/jwks.json + ``` + + Example output, which includes only the public key: + ```json + {"keys":[{"kty":"OKP","crv":"Ed25519","alg":"EdDSA","use":"sig","kid":"B1BP8zPNOTEGYyBUVgUAeTyA5hPlnrNc0OB416NY-CM","x":"xIc8dv_I-NrYSvWKg4aKm_X1CjHmiUBSMf9cDXEKesM"}]} + ``` + +Receivers verify a signed delivery against the JWKS. The controller serves the path on the same port as its gRPC, A2A, and MCP endpoints, so a receiver outside of the cluster needs a stable HTTPS URL that routes to the controller. The UI does not proxy the path, so publishing only the UI leaves a receiver without a key to verify against. + +Receivers must not send a token or credential when they register a callback. kagent signs each delivery attempt with a fresh five-minute JSON Web Token (JWT) whose audience is the exact callback URL, including the query string. Before you acknowledge the request with a 2xx response, verify the JWT signature, issuer, audience, expiry, and `taskId` claim. The `taskId` claim repeats the `statusUpdate.taskId` value in the body, so a receiver that compares them rejects a token replayed against a different task. + +By default, callback URLs must use HTTPS. Callback URLs must not resolve to private, loopback, link-local, unspecified, or multicast addresses. To reach a test receiver inside the cluster, set `config.push.allowHttp=true` or `config.push.allowPrivateNetworks=true`. kagent does not follow redirects, and failed attempts are retried from the database-backed queue. + +## When an agent needs a person + +An agent can stop mid-task to ask a question or to request approval for a tool call. The task then reports `TASK_STATE_INPUT_REQUIRED` and waits until a caller answers it. By default, every agent on the `kagent` harness can raise the question kind, because the runtime gives each one a built-in `ask_user` tool. A system prompt that tells an agent to ask before it answers is therefore enough to see a pause. + +Answering a pause needs the human-in-the-loop extension, which a caller requests per call. The extension is a versioned URI. A request activates the extension through the `A2A-Extensions` header. A response that answers a pause must also list the URI in the message's own `extensions` list. + +1. Send a message that requests the extension. + ```bash + grpcurl -plaintext \ + -H 'A2A-Version: 1.0' \ + -H "A2A-Extensions: https://kagent.dev/extensions/hitl/v1" \ + -d '{"tenant":"'"$AGENT"'","message":{"messageId":"'"$(uuidgen)"'","role":"ROLE_USER","extensions":["https://kagent.dev/extensions/hitl/v1"],"parts":[{"text":"Should I increase the replica count?"}]}}' \ + localhost:8083 lf.a2a.v1.A2AService/SendMessage + ``` + +2. Read the pause. The task stops at `TASK_STATE_INPUT_REQUIRED`, and the status message's `metadata` contains the request, keyed by the extension URI. Example output: + ```json + { + "task": { + "id": "01a0828d-6bc4-700a-b27e-9115b3174827", + "status": { + "state": "TASK_STATE_INPUT_REQUIRED", + "message": { + "parts": [{ "text": "Which environment do you mean for increasing the replica count?" }], + "metadata": { + "https://kagent.dev/extensions/hitl/v1": { + "type": "ask_user_request", + "id": "adk-823f48ab-4a0b-4652-a23f-e9db9724d35f", + "questions": [ + { + "question": "Which environment do you mean for increasing the replica count?", + "choices": ["development", "staging", "production"], + "multiple": false + } + ] + } + }, + "extensions": ["https://kagent.dev/extensions/hitl/v1"] + } + } + } + } + ``` + +3. Save the task ID, the Session's `contextId`, and the request `id` to environment variables. The request `id` is within the extension metadata and starts with `adk-`. + ```bash + export PAUSED_TASK_ID= + export PAUSED_SESSION_ID= + export REQUEST_ID= + ``` + +4. Answer on the same task. The response goes in the same metadata key, references the request `id` that it answers, and sets `taskId` so that it answers the paused task rather than starting a new turn. + ```bash + grpcurl -plaintext \ + -H 'A2A-Version: 1.0' \ + -H "A2A-Extensions: https://kagent.dev/extensions/hitl/v1" \ + -d '{"tenant":"'"$AGENT"'","message":{"messageId":"'"$(uuidgen)"'","taskId":"'"$PAUSED_TASK_ID"'","contextId":"'"$PAUSED_SESSION_ID"'","role":"ROLE_USER","extensions":["https://kagent.dev/extensions/hitl/v1"],"parts":[{"text":"staging"}],"metadata":{"https://kagent.dev/extensions/hitl/v1":{"type":"ask_user_response","id":"'"$REQUEST_ID"'","answers":[{"answer":["staging"]}]}}}}' \ + localhost:8083 lf.a2a.v1.A2AService/SendMessage + ``` + + The agent resumes where it paused and finishes the turn. Example output: + ```console + "state": "TASK_STATE_COMPLETED" + "text": "For the staging environment, I recommend increasing the replica count to ensure better load distribution and fault tolerance during testing." + ``` + +> [!IMPORTANT] +> A caller that does not request the extension is still interrupted. The task stops at `TASK_STATE_INPUT_REQUIRED`, and the question arrives as ordinary text on the status message. However, the metadata does not include a request `id`. Without that `id`, the caller cannot answer the pause, so the task waits until the task is canceled. Request the extension on any call to an agent that can pause. + +A tool approval works in the same way, but the agent sends `tool_approval_request`, and the caller answers with `tool_approval_response`. For the pause kinds, the approval model, and what a nested agent's pause looks like, see [Human in the loop]({{< link path="agents/human-in-the-loop" >}}). + +## Clean up + +* This example does not create Kubernetes resources, so you have nothing to delete. +* You can stop the 8083 port-forward for the kagent-controller service with `Ctrl+C`. +* The tasks that your messages created stay on the Session, and deleting the Session removes them. + +## Next steps + +{{< cards >}} + {{< card link=`{{< link path="examples/agents-via-mcp" >}}` title="Use agents from an MCP client" subtitle="Reach the same agents through MCP instead, from Claude Code or Cursor." >}} + {{< card link=`{{< link path="agents/human-in-the-loop" >}}` title="Human in the loop" subtitle="Handle an agent that pauses to ask a question or request approval." >}} + {{< card link=`{{< link path="observability/tracing" >}}` title="Tracing" subtitle="Follow one A2A call from the controller through to the Actor that served it." >}} +{{< /cards >}} diff --git a/docs-site/content/kagent/main/examples/a2a-byo.md b/docs-site/content/kagent/main/examples/a2a-byo.md new file mode 100644 index 00000000..d95069d1 --- /dev/null +++ b/docs-site/content/kagent/main/examples/a2a-byo.md @@ -0,0 +1,282 @@ +--- +title: Run your own agent image +description: Build a minimal BYO agent image, run it on a byo Harness, and invoke it the same way as any other kagent agent. +weight: 70 +author: kagent.dev +--- + +The `byo` runtime runs a container image that you build, so long as the image implements kagent's {{< gloss "A2A" >}}A2A{{< /gloss >}} (Agent2Agent) contract. This example takes the shortest path through that contract: build the minimal BYO agent that kagent tests itself against, run it on a {{< gloss "Harness" >}}Harness{{< /gloss >}}, and invoke it. + +The agent that you build here does not call a model or bind tools. It answers every message with a fixed string, which makes it a poor agent and a clear demonstration: everything that happens between `kagent agent invoke` and that reply is kagent's half of the contract. To review the contract and the configured agents that read their AgentTemplate instead of ignoring it, see [Bring your own agent]({{< link path="agents/bring-your-own-agent" >}}). + +## Before you begin + +1. [Install kagent]({{< link path="setup/installation#verify-the-installation" >}}), including the port-forward to the controller's gRPC API. + +2. [Create your first agent]({{< link path="get-started/your-first-agent" >}}) so that you have a Harness, an AgentTemplate, and an Agent to model these on, and a snapshot location to reuse. + +3. Install the following tools. + * [Docker](https://docs.docker.com/get-started/get-docker/) + * [git](https://git-scm.com/downloads) + +4. Export the container registry that your cluster can pull from. For example, a local kind cluster created with `make create-kind-cluster` runs one on `localhost:5001`. + ```bash + export DOCKER_REGISTRY=localhost:5001 + ``` + + > [!TIP] + > On a local kind cluster, `atelet` pulls each agent image itself, so it must reach the registry at the address that the kind network uses. In the `substrate-atelet` release, point `localhost:5001` image references at the `kind-registry` container. + > ```bash + > helm upgrade substrate-atelet \ + > {{< reuse "kagent-docs/snippets/helm-substrate-path.md" >}}/substrate-atelet \ + > --version {{< reuse "kagent-docs/versions/agent-substrate-chart.md" >}} \ + > --namespace ate-system --reuse-values \ + > --set 'atelet.extraArgs[0]=--localhost-registry-replacement=kind-registry:5000' \ + > --wait + > ``` + +5. Know which {{< gloss "WorkerPool" >}}WorkerPool{{< /gloss >}} and snapshot location your installation uses. The Harness that you created carries both. + ```bash + kubectl get harness my-first-harness -n kagent \ + -o custom-columns=WORKERPOOL:.spec.substrate.workerPoolRef.name,SNAPSHOT:.spec.substrate.snapshotPolicy.location + ``` + + Example output: + ```console + WORKERPOOL SNAPSHOT + kagent-default s3://ate-snapshots/kagent/ + ``` + +## Build the agent image + +kagent's own end-to-end suite runs an opaque BYO agent from `go/core/test/byoa2a/main.go`, and the repository has a make target that builds it. Building that image rather than writing one from scratch means starting from a version that is proven against the current contract. + +1. Clone the kagent repository and navigate to it. + ```bash + git clone https://github.com/kagent-dev/kagent.git + cd kagent + ``` + +2. Build the image and push it to your registry. The target builds `go/core/test/byoa2a/main.go` with the repository's Go Dockerfile, which produces a single binary at `/app`. + ```bash + make build-byo-a2a DOCKER_REGISTRY=$DOCKER_REGISTRY VERSION=byo-example + ``` + +3. Resolve the digest, and save the pinned reference. A Harness rejects an image that names only a tag, because a {{< gloss "Revision" >}}revision{{< /gloss >}} must be reproducible. + ```bash + export BYO_IMAGE=$DOCKER_REGISTRY/kagent-dev/kagent/byo-a2a@$(docker buildx imagetools inspect \ + $DOCKER_REGISTRY/kagent-dev/kagent/byo-a2a:byo-example \ + | awk '$1 == "Digest:" { print $2; exit }') + echo $BYO_IMAGE + ``` + + Example output: + ```console + localhost:5001/kagent-dev/kagent/byo-a2a@sha256:ea596db3dac8da570980143210efeb2b47bcfb0a3afc5aa0f325a6063c5cf009 + ``` + +## Create the Harness and the AgentTemplate + +A `byo` Harness carries two fields that the other runtimes do not need: an empty `byo` block to select the runtime, and `workload.command` to override the image entrypoint. The AgentTemplate stays almost empty, because this agent ignores everything that an AgentTemplate would configure. + +1. Create the Harness. The repository's Go Dockerfile puts the binary at `/app`, so `command` names that path. + ```yaml + kubectl apply -f - < [!NOTE] + > This Harness does not set a port, because `byoa2a/main.go` reads `runtime.port` from the compiled configuration and falls back to `80`. An image that binds another port still reports `READY`. The mismatch surfaces only on the first invoke. For the requirements that a BYO image must meet, and the error that the invoke returns, see [Bring your own agent]({{< link path="agents/bring-your-own-agent#the-a2a-contract" >}}). + +2. Create the AgentTemplate and an Agent that pairs it with the Harness. `description` is the only field that this template needs. + ```yaml + kubectl apply -f - <}}Session{{< /gloss >}} is the conversation, and the CLI reaches it through the controller's A2A service. Nothing in these commands names the runtime. + +1. Create a Session against the Agent. + ```bash + kagent agent session create --agent byo-example-agent + ``` + + The command returns only after the Session reaches `READY`. Example output: + ```console + +--------------------------------------+-------------------+-------+----------------------+ + | ID | AGENT | STATE | CREATED | + +--------------------------------------+-------------------+-------+----------------------+ + | 01a08301-8cd0-72c8-818f-26c7490ce37d | byo-example-agent | READY | 2026-09-14T14:22:07Z | + +--------------------------------------+-------------------+-------+----------------------+ + ``` + +2. Save the Session ID. + ```bash + export SESSION_ID=$(kagent agent session create --agent byo-example-agent -o json | jq -r '.session.id') + ``` + +3. Send it a message. + ```bash + kagent agent invoke --session $SESSION_ID --task "hello" + ``` + + Example output: + ```console + BYO agent response + ``` + + That string is hardcoded, so the reply itself proves nothing. Its path proves the contract: kagent compiled a revision, Agent Substrate started a sandboxed {{< gloss "Actor" >}}Actor{{< /gloss >}} from your image, the controller routed the message to it, and your executor answered. + +4. Send another message to the same Session. The reply does not change, but the message reaches the same Actor. Agent Substrate suspended that Actor after the first turn and resumed it for this message. For more information about the Actor lifecycle, see [Suspend and resume]({{< link path="about/substrate-runtime/suspend-and-resume#suspension-between-turns" >}}). + ```bash + kagent agent invoke --session $SESSION_ID --task "hello again" + ``` + +## Change what the agent does + +The whole agent is one type with two methods. `Execute` receives a request and yields A2A events until the turn ends, and `Cancel` handles a caller stopping a task that is still running. + +```go +type executor struct{} + +func (executor) Execute(_ context.Context, request *a2asrv.ExecutorContext) iter.Seq2[a2atype.Event, error] { + return func(yield func(a2atype.Event, error) bool) { + if !yield(a2atype.NewSubmittedTask(request, request.Message), nil) { + return + } + message := a2atype.NewMessage(a2atype.MessageRoleAgent, a2atype.NewTextPart("BYO agent response")) + message.ContextID, message.TaskID = request.ContextID, request.TaskID + yield(a2atype.NewStatusUpdateEvent(request, a2atype.TaskStateCompleted, message), nil) + } +} + +func (executor) Cancel(context.Context, *a2asrv.ExecutorContext) iter.Seq2[a2atype.Event, error] { + return func(func(a2atype.Event, error) bool) {} +} +``` + +Two events make a complete turn. `NewSubmittedTask` acknowledges the message and opens the task, and a `TaskStateCompleted` status update carrying an agent message ends it. Between them, a real agent yields whatever its work produces. + +`main` sets the rest up. First, `config.MaterializeFromBootstrap` fetches the revision's bootstrap bundle from the kagent API and writes its documents to `/config`. Then `config.LoadAgentConfig` reads the configuration back, and `config.ProcessConfig` returns its `runtime` object. `app.New` serves the A2A gRPC service and the readiness endpoint. + +```go +compiled, err := config.LoadAgentConfig("/config/config.json") +runtime := config.ProcessConfig(compiled) + +application, err := app.New(app.AppConfig{ + ControlPlaneURL: runtime.ControlPlaneUrl, + AgentCard: a2atype.AgentCard{Name: "opaque-byo", Version: "v1", Capabilities: a2atype.AgentCapabilities{Streaming: true}}, + Port: fmt.Sprint(cmp.Or(runtime.GetPort(), 80)), + AppName: cmp.Or(runtime.Name, "opaque-byo"), + Logger: logger, +}, executor{}) +``` + +`ControlPlaneURL` is required because the helper reports task state to the kagent API. The `cmp.Or` fallbacks cover a configuration document that omits the port or the name. All of this stays as it is in an agent of your own. + +This agent ignores the rest of the configuration. To honor the AgentTemplate instead, read the other fields of the document that `LoadAgentConfig` returns. To review what the document contains, see [Compare BYO and declarative agents]({{< link path="agents/bring-your-own-agent#compare-byo-and-declarative-agents" >}}). + +## Build and run your own agent + +An agent of your own takes the same path as the example image, with two differences: the build names your package, and the Harness moves to the image that it produces. + +1. Replace the body of `Execute` with the work that your agent does, and leave the rest of the file as-is. + +2. Build and push the image. The `build-byo-a2a` target names its package inline, so a package of your own means calling Docker directly. The Dockerfile takes the package as a build argument, relative to the `go` directory. + ```bash + docker build --build-arg BUILD_PACKAGE=core/test/myagent/main.go \ + -t $DOCKER_REGISTRY/kagent-dev/kagent/my-agent:v1 -f go/Dockerfile ./go + docker push $DOCKER_REGISTRY/kagent-dev/kagent/my-agent:v1 + ``` + + That Dockerfile copies `api`, `core`, `adk`, `harness`, and `pkg` from the kagent module, so it suits an agent written inside a checkout. An agent in a module of your own needs a Dockerfile of your own. kagent constrains what the image serves rather than how you build it. + +3. Resolve the digest of the new image, as in [Build the agent image](#build-the-agent-image). A Harness rejects an image that names only a tag. + ```bash + export BYO_IMAGE=$DOCKER_REGISTRY/kagent-dev/kagent/my-agent@$(docker buildx imagetools inspect \ + $DOCKER_REGISTRY/kagent-dev/kagent/my-agent:v1 \ + | awk '$1 == "Digest:" { print $2; exit }') + ``` + +4. Point the Harness at the new image. kagent compiles a revision for the updated pair. + ```bash + kubectl patch harness byo-example -n kagent --type=merge \ + -p "{\"spec\":{\"workload\":{\"image\":\"$BYO_IMAGE\"}}}" + ``` + +5. Create a Session on the updated Harness, and invoke it as in [Invoke the agent](#invoke-the-agent). A Session pins the revision that it was created from, so the one from earlier keeps running the example image. + ```bash + kagent agent session create --agent byo-example-agent + ``` + +## Clean up + +1. Delete the Session. Repeat for any Session that you created from an image of your own. + ```bash + kagent agent session delete $SESSION_ID + ``` + +2. Delete the AgentTemplate and the Harness. + ```bash + kubectl delete agent byo-example-agent -n kagent + kubectl delete agenttemplate byo-example-template -n kagent + kubectl delete harness byo-example -n kagent + ``` + +## Next steps + +{{< cards >}} + {{< card link=`{{< link path="agents/bring-your-own-agent" >}}` title="Bring your own agent" subtitle="Read the full A2A contract and the limits of the byo runtime." >}} + {{< card link=`{{< link path="examples/a2a-agents" >}}` title="Call an agent over A2A" subtitle="Talk to this Agent with grpcurl instead of the CLI." >}} + {{< card link=`{{< link path="examples/agent-substrate" >}}` title="Agent Substrate" subtitle="Watch the Actor behind your image suspend and checkpoint." >}} +{{< /cards >}} diff --git a/docs-site/content/kagent/main/examples/agent-delegation.md b/docs-site/content/kagent/main/examples/agent-delegation.md new file mode 100644 index 00000000..06a17d4e --- /dev/null +++ b/docs-site/content/kagent/main/examples/agent-delegation.md @@ -0,0 +1,202 @@ +--- +title: Delegate work to another agent +description: Bind one AgentTemplate to another as a subagent tool, and watch a front-line agent hand a conversation to a specialist. +weight: 50 +author: kagent.dev +--- + +An {{< gloss "AgentTemplate" >}}AgentTemplate{{< /gloss >}} can bind another AgentTemplate as a tool, so that a general agent passes work to a specialist instead of answering itself. This example builds a front-line incident agent with two specialists behind it, and shows what a caller sees when a hand-off happens. + +For the binding fields and the rules that constrain a tree, see [About tools]({{< link path="skills-and-mcp/about-tools" >}}). For the Harness that all of these templates run on, see [Agent harness]({{< link path="agents/agent-harness" >}}). + +## About subagent bindings + +A binding names a second AgentTemplate in the same namespace and describes when to use it. + +> [!IMPORTANT] +> **A subagent binding hands over the conversation rather than returning an answer.** kagent compiles the binding into a subagent of the parent and runs the whole tree in one Actor, so the parent's model can transfer the turn to it. Once that happens, the bound agent answers, and it keeps answering the turns that follow in the same conversation. The parent does not receive the bound agent's output and cannot summarize it or combine it with a second agent's. Plan a tree around routing a conversation to the right specialist, rather than around a coordinator that collects results. + +Every field is required. For the complete schema, see the [API reference]({{< link path="reference/api-ref#subagenttoolbinding" >}}). + +| Field | Description | +| ----- | ----------- | +| `tools[].subAgent.name` | The name that the parent's model sees for this binding. It replaces the bound template's own name. | +| `tools[].subAgent.description` | What the binding is for. The parent's model reads this to decide when to hand work over, so it does the same job that a tool description does. | +| `tools[].subAgent.templateRef.name` | The AgentTemplate to bind, in the same namespace. | + +> [!NOTE] +> Dedicated subagents, which would give a bound agent its own {{< gloss "Actor" >}}Actor{{< /gloss >}}, sandbox, and Session, are not part of the served API. The `subAgent.agentRef` field that would select one is deferred until a dedicated subagent can create and invoke its own Session. + +Every template in the tree is an ordinary AgentTemplate, and only the coordinator needs an {{< gloss "Agent" >}}Agent{{< /gloss >}}. A bound template compiles under the coordinator's Agent and its Harness, so the specialists need neither an Agent nor a Harness reference of their own. + +## Before you begin + +1. [Install kagent]({{< link path="setup/installation" >}}). + +2. [Create your first agent]({{< link path="get-started/your-first-agent" >}}), so that you have a Harness to pair with. This example uses a Harness named `my-first-harness` in the `kagent` namespace. + +## Create specialist agents + +Apply two ordinary AgentTemplates. Nothing marks them as bound, because a template does not know that something binds it. + +1. Create the specialists. Each system prompt makes its replies recognizable so that you can tell which agent answered. + ```bash + kubectl apply -f - < [!NOTE] + > Deleting a bound specialist while the coordinator still binds it breaks the coordinator rather than the specialist. Its `ResolvedRefs` condition reports `resolve AgentTemplate "incident-researcher": not found`, and `Compatible` and `Ready` both report `blocked by ResolvedRefs`. Recreating the specialist, or removing the binding, clears it. + +## Next steps + +{{< cards >}} + {{< card link=`{{< link path="skills-and-mcp/about-tools" >}}` title="About tools" subtitle="Read the binding fields and the rules that constrain a subagent tree." >}} + {{< card link=`{{< link path="agents/agent-harness" >}}` title="Agent harness" subtitle="Understand the Harness that every template in the tree runs on." >}} + {{< card link=`{{< link path="examples/a2a-agents" >}}` title="Call an agent over A2A" subtitle="Reach the same agents directly, and see what a hand-off looks like on the wire." >}} +{{< /cards >}} diff --git a/docs-site/content/kagent/main/examples/agent-substrate.md b/docs-site/content/kagent/main/examples/agent-substrate.md new file mode 100644 index 00000000..a474b176 --- /dev/null +++ b/docs-site/content/kagent/main/examples/agent-substrate.md @@ -0,0 +1,212 @@ +--- +title: Agent Substrate +description: Watch an agent's Actor suspend between turns, pin its state with a checkpoint, and fork that checkpoint into a second Session. +weight: 10 +author: kagent.dev +--- + +[Agent Substrate]({{< link path="about/architecture/agent-substrate" >}}) runs every agent as an Actor: a sandboxed unit of compute that holds a {{< gloss "Worker" >}}Worker{{< /gloss >}} only while a turn is in progress. This example follows one agent through suspend, resume, and checkpoint creation. + +The Actor that these steps follow is also the isolation boundary. Every Actor runs in its own {{< gloss "gVisor" >}}gVisor{{< /gloss >}} sandbox rather than sharing one with its neighbors. This isolation allows a model to safely run tools and execute commands. For what the sandbox blocks, see [Sandboxing]({{< link path="about/substrate-runtime/sandboxing" >}}). + +## Before you begin + +Checkpoints have no kagent CLI commands yet, so this example calls `CheckpointService` with grpcurl. + +{{< reuse "kagent-docs/snippets/grpcurl-prerequisites.md" >}} + +5. Install `kubectl-ate`, the Agent Substrate command line tool. + ```bash + curl -fsSL -o kubectl-ate \ + "https://github.com/kagent-dev/substrate/releases/download/v{{< reuse "kagent-docs/versions/agent-substrate.md" >}}/kubectl-ate-$(uname -s | tr '[:upper:]' '[:lower:]')-$(uname -m | sed 's/x86_64/amd64/; s/aarch64/arm64/')" + chmod +x kubectl-ate + sudo mv kubectl-ate /usr/local/bin/ + kubectl ate --help + ``` + +6. Create a {{< gloss "Session" >}}Session{{< /gloss >}} and send it a message. Then save the Session's ID and the ID of the task that the message produced. A checkpoint pins a completed turn, so the conversation needs at least one. + ```bash + export SESSION_ID=$(kagent agent session create --agent my-first-agent -o json | jq -r '.session.id') + export TASK_ID=$(kagent agent invoke --session $SESSION_ID --task "What is 2+2?" -o json | jq -r '.task.id') + echo "$SESSION_ID $TASK_ID" + ``` + +## Watch the Actor suspend between turns + +1. List the Actors in your namespace's {{< gloss "Atespace" >}}atespace{{< /gloss >}}. A Session's Actor name is formatted `session-`. + ```bash + kubectl ate get actors --atespace kagent + ``` + + Between turns, the Actor reports `ACTOR_STATE_SUSPENDED` and does not hold a Worker, so the `ATEOM POD` column reads `` and the `ATEOM IP` column is blank. The `VERSION` column is the Actor record's revision counter, which increases each time the record is updated. Example output: + ```console + ATESPACE NAME TEMPLATE STATE ATEOM POD ATEOM IP VERSION AGE + kagent session-0198c3d7-4f2a-7b61-9c3e-5d8f7a2b4e10 kagent/my-first-agent-5f2b3c1a9e8d ACTOR_STATE_SUSPENDED 4 11m + ``` + +2. Send the agent another message. Nothing in the command acknowledges that the Actor was suspended, because resuming is automatic. + ```bash + kagent agent invoke --session $SESSION_ID --task "Summarize this conversation so far." + ``` + +3. From a second terminal, list the Actors again while the turn is still running. + ```bash + kubectl ate get actors --atespace kagent + ``` + + The same Actor now reports `ACTOR_STATE_RUNNING`, names the Worker pod that it resumed onto, and carries a higher `VERSION`. Example output: + ```console + ATESPACE NAME TEMPLATE STATE ATEOM POD ATEOM IP VERSION AGE + kagent session-0198c3d7-4f2a-7b61-9c3e-5d8f7a2b4e10 kagent/my-first-agent-5f2b3c1a9e8d ACTOR_STATE_RUNNING kagent/kagent-default-7c9f8b6d54-x2n4p 10.244.1.37 6 12m + ``` + + If the listing already reads `ACTOR_STATE_SUSPENDED`, the turn finished before the command ran. Repeat steps 2 and 3 to catch the Actor mid-turn. A turn is short, and the two transitions on either side of one, `ACTOR_STATE_RESUMING` and `ACTOR_STATE_SUSPENDING`, pass quickly enough that a single listing rarely catches them. + +4. After the turn finishes, list the Actors again. + ```bash + kubectl ate get actors --atespace kagent + ``` + + The Actor is back to `ACTOR_STATE_SUSPENDED` and again does not hold a Worker, at a higher `VERSION` than the listing in step 1. The `NAME` and `AGE` columns confirm that this is the same Actor throughout, rather than a new one per turn. Example output: + ```console + ATESPACE NAME TEMPLATE STATE ATEOM POD ATEOM IP VERSION AGE + kagent session-0198c3d7-4f2a-7b61-9c3e-5d8f7a2b4e10 kagent/my-first-agent-5f2b3c1a9e8d ACTOR_STATE_SUSPENDED 8 13m + ``` + +5. Check the Session while its Actor is suspended. + ```bash + kagent agent session list + ``` + + The Session reports `READY`, even though the Actor that runs it does not hold a Worker. Example output: + ```console + +--------------------------------------+----------------+-------+----------------------+ + | ID | AGENT | STATE | CREATED | + +--------------------------------------+----------------+-------+----------------------+ + | 0198c3d7-4f2a-7b61-9c3e-5d8f7a2b4e10 | my-first-agent | READY | 2026-08-31T15:02:10Z | + +--------------------------------------+----------------+-------+----------------------+ + ``` + +The Session stays `READY` throughout all steps. A suspended agent remains listed and readable because suspension is a property of the Actor underneath the conversation, not of the conversation itself. For the full cycle, see [Suspend and resume]({{< link path="about/substrate-runtime/suspend-and-resume" >}}). + +> [!NOTE] +> Two objects report state on this page, and each interface names its states differently. `kubectl ate get actors` reports the Actor's state in full, such as `ACTOR_STATE_SUSPENDED`, because the command prints the Agent Substrate enum name. The kagent CLI trims the prefix from the Session's state and prints `READY`, and the same value reaches you as `RUNTIME_STATE_READY` in a `grpcurl` response. Checkpoints have no CLI command yet, so the next section calls the API directly and reads the checkpoint's state in full, as `CHECKPOINT_STATE_READY`. + +## Pin the conversation with a checkpoint + +Each suspend writes a {{< gloss "Snapshot" >}}snapshot{{< /gloss >}}, and Agent Substrate is free to collect that snapshot once a newer one supersedes it. A {{< gloss "Checkpoint" >}}checkpoint{{< /gloss >}} pins a snapshot so that you can come back to it. + +1. Create a checkpoint. A checkpoint keeps the conversation through the turn that you name. `expectedHeadTaskId` names the finished turn to pin. When the conversation has advanced past that task, the call fails rather than pinning the wrong turn. `requestId` is a required idempotency key of 1 to 128 characters. Repeating a `requestId` returns the checkpoint that the earlier call created rather than pinning a second one. + ```bash + grpcurl -plaintext \ + -d '{"sessionId":"'"$SESSION_ID"'","expectedHeadTaskId":"'"$TASK_ID"'","requestId":"'"$(uuidgen)"'"}' \ + localhost:8083 kagent.api.v1.CheckpointService/CreateCheckpoint + ``` + + Example output: + ```json + { + "checkpoint": { + "id": "0198c3e2-8a41-7d05-b6c2-1f4e9a7b3c58", + "sessionId": "0198c3d7-4f2a-7b61-9c3e-5d8f7a2b4e10", + "headTaskId": "0198c3d9-b7e3-7a24-8f10-6c2d5e8a1b47", + "historySequence": "4", + "state": "CHECKPOINT_STATE_READY", + "createdAt": "2026-08-31T15:12:44Z", + "name": "my-first-agent turn 4" + } + } + ``` + + > [!NOTE] + > A checkpoint captures a turn boundary. The call succeeds only once every turn on the Session has reached a final state, meaning `TASK_STATE_COMPLETED`, `TASK_STATE_FAILED`, `TASK_STATE_CANCELED`, or `TASK_STATE_REJECTED`. + > + > * When a turn on the Session is still running, the call fails with `checkpoint requires all tasks to be terminal`. Wait for the turn to finish, then send the request again. + > * When a newer turn has already started, the call fails with `conversation advanced beyond the expected task`. Name the current task ID and send the request again. + > * When Agent Substrate is still writing the snapshot, the call fails with `snapshot for the expected task is not ready`. Send the same request again. + +2. Save the checkpoint's `id` so that you can delete it during cleanup. + ```bash + export CHECKPOINT_ID= + ``` + +3. List the checkpoints on the Session at any time. Omit `limit` for the default page of 50, up to a maximum of 100. + ```bash + grpcurl -plaintext \ + -d '{"sessionId":"'"$SESSION_ID"'","page":{"limit":50}}' \ + localhost:8083 kagent.api.v1.CheckpointService/ListCheckpoints + ``` + +Underneath, the checkpoint attaches a {{< gloss "Tag" >}}Tag{{< /gloss >}} named `checkpoint-` to the snapshot, and Agent Substrate does not collect a snapshot while a tag names it. You can see the tag by running `kubectl ate get tags --atespace kagent`. + +## Fork the checkpoint into a second Session + +A fork starts a new Session from a checkpoint. The new Session replays the conversation through the pinned turn and then continues on its own. You can therefore take a conversation in a second direction without disturbing the first. + +1. Fork the checkpoint. `requestId` is an idempotency key of 1 to 128 characters. Repeating a `requestId` returns the Session that the earlier call created rather than forking a second one. + ```bash + grpcurl -plaintext \ + -d '{"checkpointId":"'"$CHECKPOINT_ID"'","requestId":"'"$(uuidgen)"'"}' \ + localhost:8083 kagent.api.v1.CheckpointService/ForkSession + ``` + + The fork takes the checkpoint's name, and runs the revision that the source Session was created on. Example output: + ```json + { + "session": { + "id": "0198c3f1-2d7a-7c48-9e03-4b1f6a8d2c95", + "creator": "admin@kagent.dev", + "agent": { + "namespace": "kagent", + "name": "my-first-agent" + }, + "state": "RUNTIME_STATE_READY", + "createdAt": "2026-08-31T15:19:02Z", + "name": "my-first-agent turn 4", + "contextId": "0198c3f1-2d7a-7c48-9e03-4b1f6a8d2c95" + } + } + ``` + + > [!NOTE] + > The fork receives its own conversation ID and its own task IDs, so `contextId` is the fork's own ID rather than the source Session's. The replayed turns keep the timestamps they had on the source Session. Agent Substrate measures the fork's idle lifetime from its creation time instead, so a fork of an old conversation is not collected as idle. + +2. Save the forked Session's ID. + ```bash + export FORK_SESSION_ID= + ``` + +3. Ask the fork about the replayed conversation. The agent answers from the turns that the checkpoint pinned, which is what makes the fork a continuation rather than a new conversation. + ```bash + kagent agent invoke --session $FORK_SESSION_ID --task "What did I ask you first?" + ``` + +4. Confirm that the two Sessions are now independent. Send the source Session a message, and note that the fork's transcript does not change. + ```bash + kagent agent invoke --session $SESSION_ID --task "Remember the number 7." + kagent agent invoke --session $FORK_SESSION_ID --task "What number did I ask you to remember?" + ``` + + The fork does not know the number, because the two conversations diverged at the checkpoint. + +## Clean up + +1. Delete the forked Session. + ```bash + kagent agent session delete $FORK_SESSION_ID + ``` + +2. Delete the checkpoint. Deleting removes the Tag and releases the pin, and Agent Substrate can collect the snapshot whenever no tag names it. + ```bash + grpcurl -plaintext \ + -d '{"checkpointId":"'"$CHECKPOINT_ID"'"}' \ + localhost:8083 kagent.api.v1.CheckpointService/DeleteCheckpoint + ``` + +## Next steps + +{{< cards >}} + {{< card link=`{{< link path="about/substrate-runtime/suspend-and-resume" >}}` title="Suspend and resume" subtitle="Understand the snapshot cycle that checkpoints pin." >}} + {{< card link=`{{< link path="about/substrate-runtime/sandboxing" >}}` title="Sandboxing" subtitle="See what the sandbox around each Actor isolates." >}} + {{< card link=`{{< link path="about/substrate-runtime/identity" >}}` title="Identity" subtitle="See who owns a Session and the checkpoints taken on it." >}} +{{< /cards >}} diff --git a/docs-site/content/kagent/main/examples/agentgateway.md b/docs-site/content/kagent/main/examples/agentgateway.md new file mode 100644 index 00000000..6e796624 --- /dev/null +++ b/docs-site/content/kagent/main/examples/agentgateway.md @@ -0,0 +1,400 @@ +--- +title: Block PII in an agent's model requests +description: Add a prompt guard to an agentgateway model so that a prompt carrying personally identifiable information never reaches the provider. +weight: 80 +author: kagent.dev +--- + +An agent sends every turn of a conversation to a model provider, and each of those requests carries whatever the person typed, personally identifiable information (PII) included. When agentgateway routes that traffic, the gateway sees each request before the provider does, so you can inspect and stop a prompt at the gateway. This example adds a prompt guard to the model that an agent calls, then watches the gateway reject a prompt that carries an email address. + +This guide builds on the routing configured in [Agentgateway model routing]({{< link path="setup/model-providers/byo-agentgateway#set-up-agentgateway-model-routing" >}}), including extending the `AgentgatewayModel` and the {{< gloss "ModelConfig" >}}ModelConfig{{< /gloss >}} that those steps create. + +## About prompt guards on a model + +A prompt guard inspects the body of an OpenAI-compatible request as the request passes through the gateway, and either rejects the request or masks the matched text. Because the guard runs at the gateway, no change to an {{< gloss "AgentTemplate" >}}AgentTemplate{{< /gloss >}} is needed, and every agent that shares the model inherits the guard. + +A guard can live in two places, and the choice follows the routing that you already use: + +- **On the `AgentgatewayModel`**, under `spec.policies.promptGuard`. The guard applies to that one model, and no other resource is involved. This example uses this form, because [agentgateway model routing]({{< link path="setup/model-providers/byo-agentgateway#set-up-agentgateway-model-routing" >}}) attaches models straight to a `Gateway` listener. +- **On an `AgentgatewayPolicy`** that targets an `HTTPRoute`. An `AgentgatewayPolicy` cannot name an `AgentgatewayModel` in its `targetRefs`, so this form requires the model to attach to an HTTPRoute rather than to the listener. For more information about this variant, see [Attach the guard to a route instead](#attach-the-guard-to-a-route-instead). + +The following `AgentgatewayModel` carries the guard that the rest of this example applies. + +```yaml +apiVersion: agentgateway.dev/v1alpha1 +kind: AgentgatewayModel +metadata: + name: gpt-4o-mini + namespace: agentgateway-system +spec: + parentRefs: + - group: gateway.networking.k8s.io + kind: Gateway + name: agentgateway-proxy + sectionName: http + provider: OpenAI + policies: + auth: + secretRef: + name: openai-key + promptGuard: + request: + - regex: + builtins: + - Email + - Ssn + - CreditCard + action: Reject + response: + message: The prompt contained personally identifiable information. +``` + +Only `policies.promptGuard` is new in this example. The routing fields and `policies.auth` come from the prerequisite setup, and they appear here because a reapply must carry them. + +| Field | Description | +| ----- | ----------- | +| `parentRefs` and `provider` | Route the model to a `Gateway` listener, and name the provider that agentgateway calls. | +| `policies.auth` | The Secret holding the credentials that agentgateway uses to reach the provider. | +| `policies.promptGuard.request[]` | The guards to apply to requests that the agent sends. Each entry sets exactly one guard kind: `regex` matches patterns in the gateway itself, while `webhook`, `openAIModeration`, `bedrockGuardrails`, and `googleModelArmor` hand the content to an external service instead. A separate `response` list guards what the provider sends back. | +| `regex.builtins` | Built-in patterns for common kinds of PII. The five values are `Email`, `Ssn`, `CreditCard`, `PhoneNumber`, and `CaSin`. To add your own patterns, use `regex.matches`, which holds a list of regular expressions and is additive with `builtins`. | +| `regex.action` | What to do with a match, one of `Mask`, `Reject`, and `Audit`. Omit to default to `Mask`, which is also the safer choice for an agent. `Audit` records the action that the guard would have taken and lets the content through, so use it to trial a guard before you enforce it. A `Reject` guard ends the conversation permanently, for the reason described in [Mask instead of reject](#mask-instead-of-reject). | +| `response.message` | The message that the gateway returns to the caller on a rejection. Omit to default to `The request was rejected due to inappropriate content`. A sibling `response.statusCode` field sets the status code, and defaults to `403`. | + +> [!IMPORTANT] +> **A request guard inspects the system prompt as well as the messages.** The default scope is `SystemPrompt` and `Messages`, so an AgentTemplate whose `systemPrompt` contains an example email address fails every turn rather than only the turns where a person types one. Check the system prompts of the agents that share a model before you turn on `Reject`. + +## Before you begin + +1. [Install kagent]({{< link path="setup/installation" >}}), and [create your first agent]({{< link path="get-started/your-first-agent" >}}) so that you have a {{< gloss "Harness" >}}Harness{{< /gloss >}} to pair with. This example uses a Harness named `my-first-harness` in the `kagent` namespace. + +2. Set up [agentgateway model routing]({{< link path="setup/model-providers/byo-agentgateway#set-up-agentgateway-model-routing" >}}), which installs agentgateway with `--set agentgatewayModels.enabled=true` and creates a `Gateway` named `agentgateway-proxy` and an `AgentgatewayModel` named `gpt-4o-mini`. + +3. Create the [ModelConfig]({{< link path="setup/model-providers/byo-agentgateway#create-the-modelconfig" >}}) that points at the gateway. This example uses a ModelConfig named `agentgateway-model-config` in the `kagent` namespace, and the deployment that it reaches does not enforce API key authentication. + +## Add the prompt guard to the model + +Adding `promptGuard` to a model that already routes traffic changes nothing about the routing, so you reapply the same resource with the guard attached. Keep `policies.auth` in the resource that you reapply, because an apply replaces `policies` wholesale. A model that loses its `auth` is still accepted and programmed, and every request through it returns the provider's own `401`. + +1. Reapply the `AgentgatewayModel` with a request guard that rejects three kinds of PII. + ```bash + kubectl apply -f - <}} + {{% tab name="Cloud Provider LoadBalancer" %}} + ```bash + export AGENTGATEWAY_ADDRESS=$(kubectl get svc -n agentgateway-system agentgateway-proxy -o jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}") + echo $AGENTGATEWAY_ADDRESS + ``` + {{% /tab %}} + {{% tab name="Port-forward for local testing" %}} + ```bash + kubectl port-forward -n agentgateway-system service/agentgateway-proxy 8080:80 + export AGENTGATEWAY_ADDRESS=localhost:8080 + ``` + {{% /tab %}} + {{< /tabs >}} + +2. Send a prompt that contains an email address. + ```bash + curl -i http://$AGENTGATEWAY_ADDRESS/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"Summarize the ticket from alex@example.com"}]}' + ``` + + The gateway returns the status code and message from the guard, and the provider never receives the request. Example output: + ```console + HTTP/1.1 403 Forbidden + + The prompt contained personally identifiable information. + ``` + +3. To confirm that ordinary traffic still reaches the provider, send a prompt that does not contain PII. + ```bash + curl -i http://$AGENTGATEWAY_ADDRESS/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"Summarize the most recent ticket"}]}' + ``` + + The response is an ordinary completion. Example output, truncated: + ```console + HTTP/1.1 200 OK + content-type: application/json + + {"model":"gpt-4o-mini-2024-07-18","usage":{"prompt_tokens":14,"completion_tokens":33,...},"choices":[{"message":{"content":"I'm sorry, but I don't have access to specific ... + ``` + +4. Create an agent that uses the guarded model, and a {{< gloss "Session" >}}Session{{< /gloss >}} to talk to it. + ```bash + kubectl apply -f - < [!NOTE] +> **The guard's `response.message` does not reach the person talking to the agent.** A caller who reads the gateway's response directly sees the message, as the curl steps show, but kagent surfaces only the status code and the failed task. Treat `response.message` as something for an operator reading gateway logs rather than as an explanation for the end user, and put the explanation your users need in the agent's system prompt instead. + +> [!CAUTION] +> **A rejected turn strands the conversation permanently.** An agent is not a single-shot chat client: the {{< gloss "Transcript" >}}transcript{{< /gloss >}} is append-only, and a failed turn does not remove the message that failed. The Session resends the whole conversation on every turn, so the prompt that tripped the guard is sent again, and rejected again, for the rest of that Session's life. +> +> The following sequence reproduces it. A fresh Session answers `Summarize the most recent ticket` normally. Send `Summarize the ticket from alex@example.com` next, and that turn fails with the `403`. Then send `What is 2+2?`, which does not contain PII of its own: that turn fails with a `403` as well, and so does every turn after it. +> +> Creating a new Session is the only way to recover a conversation that a `Reject` guard has stopped. For agents, that cost is the strongest argument for [masking instead](#mask-instead-of-reject). + +## Mask instead of reject + +A `Reject` guard ends the conversation for good, as the [previous section](#watch-the-gateway-reject-a-prompt) demonstrates. A `Mask` guard instead lets the turn through with the matched text replaced. Masking is therefore the better default for an agent, and `Reject` belongs to a policy that forbids PII outright and accepts a dead conversation as the price. + +Masking also repairs a conversation that a `Reject` guard already stopped. Changing the action re-masks the offending message on the next turn rather than rejecting it, so the stranded Session answers again without being recreated. + +1. Change the action on the request guard to `Mask`. + ```bash + kubectl patch agentgatewaymodel gpt-4o-mini -n agentgateway-system --type merge -p \ + '{"spec":{"policies":{"promptGuard":{"request":[{"regex":{"builtins":["Email","Ssn","CreditCard"],"action":"Mask"}}]}}}}' + ``` + +2. Send the same prompt again, to the same Session that the `Reject` guard stranded. + ```bash + kagent agent invoke --session $SESSION_ID \ + --task "Summarize this ticket: alex@example.com reports that checkout returns 503 errors during peak hours." + ``` + + The turn succeeds this time. The agent answers, or asks a follow-up question, and the address never reaches the provider. The patch replaces the whole `request` list, so it also drops the `response.message` that the rejection used. A `Mask` guard does not return a message, because it rejects nothing. + +> [!WARNING] +> **A guard on the response does not inspect streamed content unless you enable it.** Prompt guards default to skipping streaming responses to preserve throughput. Set `policies.promptGuard.streaming: Enabled` to guard them, and note that `Mask` is never applied to a streamed response even then: a guard can reject streamed content, and matched text in a stream that is not rejected passes through unmodified. + +## Guard what an agent's tools send + +An agent differs from a chat client in that much of its traffic originates from tools rather than from a person. A tool that reads a ticket, a database row, or a Kubernetes resource can feed PII back to the model, and the default scope does not inspect that content. + +Add `scope` to the guard to cover tool traffic. `ToolOutput` covers the results that a tool feeds back to the model, and `ToolInput` covers the arguments that the model produces for a tool call. + +```yaml + policies: + promptGuard: + request: + - scope: + - SystemPrompt + - Messages + - ToolOutput + regex: + builtins: + - Email + - Ssn + - CreditCard + action: Mask +``` + +> [!NOTE] +> Listing `scope` replaces the default rather than adding to it, so name `SystemPrompt` and `Messages` alongside the tool scopes to keep guarding the conversation. In an API that sends tool arguments as opaque JSON, a `Mask` action on `ToolInput` can rewrite those arguments into invalid JSON, so prefer `ToolOutput` unless you have a reason to inspect the arguments. + +## Attach the guard to a route instead + +An `AgentgatewayPolicy` holds the same `promptGuard` configuration, and one policy can govern every model behind a route rather than one model at a time. Because `targetRefs` accepts an `HTTPRoute` and not an `AgentgatewayModel`, this form needs the model to attach to a route instead of to the listener. + +1. Create an `HTTPRoute` whose rule sends traffic to every model, and point the model at the route rather than at the `Gateway`. + ```bash + kubectl apply -f - < [!NOTE] +> The listener must allow the `HTTPRoute` kind for this variant. The Gateway that [agentgateway model routing]({{< link path="setup/model-providers/byo-agentgateway#set-up-agentgateway-model-routing" >}}) creates already allows both `HTTPRoute` and `AgentgatewayModel`, so no change to the Gateway is needed. + +## Clean up + +1. Delete the Session, the Agent, and the AgentTemplate. + ```bash + kagent agent session delete $SESSION_ID + kubectl delete agent support-triage -n kagent + kubectl delete agenttemplate support-triage -n kagent + ``` + +2. Remove the guard from the model, leaving the routing and the provider credentials in place. The patch names `promptGuard` rather than `policies`, because `policies` also holds the `auth` that lets the gateway reach the provider. A merge patch also succeeds when the model does not name a guard, which is the case if you followed [Attach the guard to a route instead](#attach-the-guard-to-a-route-instead). + ```bash + kubectl patch agentgatewaymodel gpt-4o-mini -n agentgateway-system --type merge -p '{"spec":{"policies":{"promptGuard":null}}}' + ``` + +3. If you followed [Attach the guard to a route instead](#attach-the-guard-to-a-route-instead), delete the policy and the route, and repoint the model at the Gateway listener. + ```bash + kubectl delete agentgatewaypolicy block-pii -n agentgateway-system + kubectl delete httproute model-traffic -n agentgateway-system + kubectl apply -f - <}} + {{< card link=`{{< link path="setup/model-providers/byo-agentgateway" >}}` title="agentgateway model routing" subtitle="Review the Gateway, AgentgatewayModel, and ModelConfig that carry an agent's model traffic." >}} + {{< card link=`{{< link path="observability/audit-prompts" >}}` title="Audit prompts" subtitle="Review where each runtime records the prompts and replies that your agents exchange with a model, and how to capture them for an audit." >}} + {{< card link="https://agentgateway.dev/docs/kubernetes/latest/documentation/llm/guardrails/regex/#block-requests-with-pii" title="agentgateway regex guardrails" subtitle="Read the full set of regex guards, masking rules, and built-in patterns in the agentgateway documentation." >}} +{{< /cards >}} diff --git a/docs-site/content/kagent/main/examples/agents-via-mcp.md b/docs-site/content/kagent/main/examples/agents-via-mcp.md new file mode 100644 index 00000000..9e5c7435 --- /dev/null +++ b/docs-site/content/kagent/main/examples/agents-via-mcp.md @@ -0,0 +1,585 @@ +--- +title: Use agents from an MCP client +description: Connect Claude Code, Cursor, or another MCP client to kagent's MCP server, then discover your agents and invoke them as tools. +weight: 20 +author: kagent.dev +--- + +The kagent controller runs a {{< gloss "Model Context Protocol" >}}Model Context Protocol{{< /gloss >}} (MCP) server that exposes your agents and their {{< gloss "Session" >}}Sessions{{< /gloss >}} as tools. Any MCP client can then discover the agents in your cluster and delegate work to them. This mechanism allows one agent to orchestrate another as a subagent. + +This example runs in the opposite direction to [Your first MCP tool]({{< link path="get-started/your-first-mcp-tool" >}}). There, kagent is the MCP client and an external server provides the tools. Here, kagent is the MCP server and your agents are the tools. + +An MCP client such as Claude Code or Cursor builds each call. To see each call, or to drive the endpoint without a client, send raw `curl` requests. + +## About the kagent MCP server + +The MCP server runs on the controller's HTTP port, so a default installation already serves it at `/mcp` on port `8083`. + +- **Transport**: Clients connect over Streamable HTTP. Standalone Server-Sent Events (SSE) and stdio are unsupported. +- **State**: The handler is stateless. Each request stands alone, and a client does not need to establish an MCP session first. +- **Extensions**: The server advertises the `io.modelcontextprotocol/tasks` extension, which changes how invocations behave. For more information, see [Invoke without waiting](#invoke-without-waiting). + +{{< reuse "kagent-docs/snippets/examples/mcp-endpoint-warning.md" >}} + +## Before you begin + +1. [Install kagent]({{< link path="setup/installation" >}}), and confirm that your installation sets `config.server.grpcReflection=true`. Reflection lets a gRPC client discover the controller's methods without a local copy of kagent's protocol buffer definitions. + +2. [Create your first agent]({{< link path="get-started/your-first-agent" >}}), so that the cluster has an {{< gloss "Agent" >}}Agent{{< /gloss >}} to invoke. + +3. Install [grpcurl](https://github.com/fullstorydev/grpcurl) to delete the checkpoint in [Clean up](#clean-up). + +## Connect a client + +A client reaches the MCP server through a port-forward to the controller's HTTP port. After the port-forward runs, register the endpoint in your client, or confirm the endpoint with a `curl` request. + +1. Port-forward the controller's HTTP port, and leave the command running. + ```bash + kubectl port-forward -n kagent svc/kagent-controller 8083:8083 + ``` + +2. Add `http://localhost:8083/mcp` to your client. + {{< tabs >}} + {{% tab name="Claude Code" %}} + To limit the entry to the current project, add `--scope project`. + ```bash + claude mcp add --transport http kagent http://localhost:8083/mcp + ``` + + Continue with the steps in [Use agents from Claude Code or Cursor](#use-agents-from-claude-code-or-cursor). + {{% /tab %}} + {{% tab name="Cursor" %}} + Add the server to your Cursor MCP settings. + ```json + { + "mcpServers": { + "kagent": { + "url": "http://localhost:8083/mcp" + } + } + } + ``` + + Continue with the steps in [Use agents from Claude Code or Cursor](#use-agents-from-claude-code-or-cursor). + {{% /tab %}} + {{% tab name="curl" %}} + A `curl` client does not need to register the server. Before you continue, confirm the endpoint. Streamable HTTP replies are framed as events, so each response arrives on a `data:` line. + ```bash + curl -s -X POST http://localhost:8083/mcp \ + -H 'Content-Type: application/json' \ + -H 'Accept: application/json, text/event-stream' \ + -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{},"clientInfo":{"name":"curl","version":"0"}}}' + ``` + Example output: + ```console + event: message + data: {"jsonrpc":"2.0","id":1,"result":{"capabilities":{"extensions":{"io.modelcontextprotocol/tasks":{}},"tools":{"listChanged":true}},"protocolVersion":"2025-06-18","serverInfo":{"name":"kagent","version":"v1.0.0"}}} + ``` + + Continue with the steps in [Use agents from curl](#use-agents-from-curl). + {{% /tab %}} + {{< /tabs >}} + +## Use agents from Claude Code or Cursor + +Claude Code and Cursor read the tool schemas and build each call, so you work in plain language. You discover the agents in your cluster, start a Session with one, answer the agent when it stops to ask you something, and checkpoint its Session. + +### List and invoke an agent + +The `list_agents` tool returns the Agents that the caller can see. The `invoke_agent` tool sends one of them a message and starts a Session. + +1. Ask for your agents. The client calls `list_agents` and reports the namespace and name of each Agent. A listed Agent is not necessarily ready, and the caller might lack permission to invoke it. + ```console + > List my kagent agents. + ``` + +2. Ask the agent a question by naming the Agent. The client calls `invoke_agent` with the Agent's namespace and name. The call starts a new Session, and the reply includes the Session ID. + ```console + > Ask the kagent agent my-first-agent in the kagent namespace: what is 2+2? + Answer with just the number. + ``` + + If your client declares the `io.modelcontextprotocol/tasks` extension, the reply is a task ID instead, and the client polls the task in the background. A client declares its own capabilities on each request, so nothing on the server decides whether a call blocks. + +3. Ask a follow-up that depends on the previous answer, such as `Multiply that by 10.`, in the same Session. The client calls `invoke_agent` again with the `session_id` from the first reply. The word "that" points back to the earlier answer. The model can follow the reference because the transcript belongs to the Session rather than to the client. + + Because a Session _is_ one conversation, a second message to the same `session_id` continues where the first left off. The reply's `context_id` is the Session's own ID. To hold two independent conversations with one Agent, invoke the Agent by name twice, which starts two Sessions. + +4. To continue a Session that already exists, ask the client to list your Sessions. + ```console + > List my kagent sessions. + ``` + + The `list_sessions` tool returns only the Sessions that the calling identity created, and cannot widen that scope. The kagent command line interface (CLI) and this endpoint both default to `admin@kagent.dev`, so they see each other's Sessions. If you created the Session as another user with `kagent --user-id `, send a matching `X-User-Id` header from the client. + +### Answer an agent's question + +When an agent pauses to ask something, kagent returns the question as an MCP elicitation, and your client presents its own prompt with the agent's question and any choices that the agent offered. When you answer the prompt, the agent resumes the turn where it left off. When you refuse it, the agent is told that the person declined, and the agent can adapt to the refusal. + +Only a client that declares the tasks extension can answer an agent. A blocking `invoke_agent` call cannot show the question, so an agent that pauses leaves that call waiting. + +The pause is the same one that any other client sees, reached through MCP instead of Agent2Agent (A2A). For the pause kinds, the approval model, and what the agent receives, see [Human in the loop]({{< link path="agents/human-in-the-loop" >}}). + +### Checkpoint a Session + +Before an agent tries something risky, you can checkpoint its Session. The {{< gloss "Checkpoint" >}}checkpoint{{< /gloss >}} tools give the client the same checkpoint operations that are performed from the command line in the [Agent Substrate example]({{< link path="examples/agent-substrate" >}}). + +1. Ask the client to checkpoint the Session. + ```console + > Checkpoint kagent session . + ``` + + The client reports the checkpoint's own ID, the turn that it pinned, and a state of `CHECKPOINT_STATE_READY`. Because the client keeps that result in context, you can refer to the checkpoint without repeating its ID. To inspect or delete the checkpoint later, keep its ID. + +You can now safely [clean up these resources](#clean-up). + +## Use agents from curl + +With `curl` you build each request yourself, so every field is visible, including the tool name, its arguments, and the `_meta` field that decides whether a call blocks or returns a task to poll. You list the agents in your cluster, start a Session with one, answer the agent when it pauses, and checkpoint the Session. Every request is a `tools/call` to `/mcp` unless it names a `tasks/` method, and no request needs an `initialize` handshake first. + +Do not put `io.modelcontextprotocol/protocolVersion` in a request's `_meta`. If a request includes it, the server requires a matching `Mcp-Protocol-Version` header and rejects the call without one. None of these requests need either field. + +### List and invoke an agent by using curl + +Each request in this section is a blocking `tools/call`. The reply returns after the agent finishes the turn. + +1. List the Agents that you can see. To limit the list to one namespace, add a `namespace` argument. + ```bash + curl -s -X POST http://localhost:8083/mcp \ + -H 'Content-Type: application/json' \ + -H 'Accept: application/json, text/event-stream' \ + -d '{ + "jsonrpc": "2.0", + "id": 1, + "method": "tools/call", + "params": { + "name": "list_agents", + "arguments": {} + } + }' + ``` + + The reply contains an `agents` array of namespace and name references, plus a `next_page_token` when more Agents remain. A listed Agent is not necessarily ready, and the caller might lack permission to invoke it. Example output: + ```json + { + "agents": [ + { "namespace": "kagent", "name": "my-first-agent" } + ] + } + ``` + +2. Start a Session by sending the Agent a message, and wait for the reply. The `agent` argument takes a reference from the `list_agents` result. A blocking call does not need `_meta`, because by default the tool does not require a client capability. + ```bash + curl -s -X POST http://localhost:8083/mcp \ + -H 'Content-Type: application/json' \ + -H 'Accept: application/json, text/event-stream' \ + -d '{ + "jsonrpc": "2.0", + "id": 2, + "method": "tools/call", + "params": { + "name": "invoke_agent", + "arguments": { + "agent": { "namespace": "kagent", "name": "my-first-agent" }, + "message": "What is 2+2? Answer with just the number." + } + } + }' + ``` + + The reply text comes back as the tool's content, with the new Session's ID and the task identifiers alongside it. Example output: + ```json + { + "session_id": "01a068e3-aeb6-7abc-8d6f-5ba9becd3143", + "task_id": "01a06d0d-5fcf-7b07-aae3-1f470a8ee157", + "context_id": "01a068e3-aeb6-7abc-8d6f-5ba9becd3143", + "state": "TASK_STATE_COMPLETED", + "text": "4" + } + ``` + +3. Save the Session ID and the task ID from the reply. The checkpoint steps use the task ID. + ```bash + export SESSION_ID= + export HEAD_TASK_ID= + ``` + +4. Send `Multiply that by 10.` to the same Session. This time, `session_id` replaces `agent`, because the call takes exactly one of the two. The word "that" points back to the earlier answer. The model can follow the reference because the transcript belongs to the Session. In the reply, `session_id` and `context_id` are unchanged and `task_id` is new, so the second turn continued the same Session. Save the new task ID. + ```bash + curl -s -X POST http://localhost:8083/mcp \ + -H 'Content-Type: application/json' \ + -H 'Accept: application/json, text/event-stream' \ + -d '{ + "jsonrpc": "2.0", + "id": 2, + "method": "tools/call", + "params": { + "name": "invoke_agent", + "arguments": { + "session_id": "'"$SESSION_ID"'", + "message": "Multiply that by 10." + } + } + }' + ``` + Example output: + ```json + { + "session_id": "01a068e3-aeb6-7abc-8d6f-5ba9becd3143", + "task_id": "01a06d0d-6864-79f1-a4cb-8547f77638ba", + "context_id": "01a068e3-aeb6-7abc-8d6f-5ba9becd3143", + "state": "TASK_STATE_COMPLETED", + "text": "40" + } + ``` + ```bash + export HEAD_TASK_ID= + ``` + +### Invoke without waiting + +Declaring the `io.modelcontextprotocol/tasks` extension changes the same tool's result. Rather than blocking, `invoke_agent` returns immediately with a task to poll, which keeps a long agent run from holding a request open. Because the handler is stateless, every request, including each poll, repeats the declaration in its `_meta`. The `tasks/cancel` method stops a run that is still working, and the `tasks/update` method answers an agent that is waiting on a person. + +1. Invoke the agent with the extension declared in `params._meta`. If you omit the declaration, the tool blocks, as in [List and invoke an agent by using curl](#list-and-invoke-an-agent-by-using-curl). + ```bash + curl -s -X POST http://localhost:8083/mcp \ + -H 'Content-Type: application/json' \ + -H 'Accept: application/json, text/event-stream' \ + -d '{ + "jsonrpc": "2.0", + "id": 3, + "method": "tools/call", + "params": { + "name": "invoke_agent", + "arguments": { + "session_id": "'"$SESSION_ID"'", + "message": "Count to three." + }, + "_meta": { + "io.modelcontextprotocol/clientCapabilities": { + "extensions": { "io.modelcontextprotocol/tasks": {} } + } + } + } + }' + ``` + + The task has an opaque ID of the form `v1.` that identifies the Session and the A2A task. Pass the ID back verbatim. The checkpoint tools do not accept this ID. Example output: + ```json + { + "taskId": "v1.eyJuYW1lc3BhY2UiOiJrYWdlbnQiLCJpbnN0YW5jZUlkIjoi...", + "status": "working", + "createdAt": "2026-09-04T15:35:25.722159606Z", + "lastUpdatedAt": "2026-09-04T15:35:25.722159606Z", + "ttlMs": null, + "pollIntervalMs": 1000, + "resultType": "task" + } + ``` + +2. Save the task ID. The value ends in base64, so quote it. + ```bash + export TASK_ID='' + ``` + +3. Poll the task with `tasks/get`. Include the same `_meta` declaration on every poll. The server rejects a poll that omits it with `-32021 tasks capability required but not declared by client`. + ```bash + curl -s -X POST http://localhost:8083/mcp \ + -H 'Content-Type: application/json' \ + -H 'Accept: application/json, text/event-stream' \ + -d '{ + "jsonrpc": "2.0", + "id": 4, + "method": "tasks/get", + "params": { + "taskId": "'"$TASK_ID"'", + "_meta": { + "io.modelcontextprotocol/clientCapabilities": { + "extensions": { "io.modelcontextprotocol/tasks": {} } + } + } + } + }' + ``` + +4. If the `status` reports `working`, send the `tasks/get` request again after the interval that `pollIntervalMs` suggests. On a `working` task, `statusMessage` contains the raw task record, because the agent has not produced text yet, so do not present it as progress text. A status of `input_required` means the agent is waiting on a person. To answer it, see [Answer an agent's question by using curl](#answer-an-agents-question-by-using-curl). A status of `cancelled` means that a `tasks/cancel` call stopped the run, so stop polling. + +5. When the `status` reads `completed`, stop polling. Read `status`, not `resultType`, to decide that a run is over, because a paused task reports `resultType` as `complete` while its `status` is still `input_required`. The `result` field contains the same `content` and `structuredContent` that a blocking call returns. A run that fails also reports `completed`, so read `structuredContent.state` to tell success from failure. Example output: + ```json + { + "status": "completed", + "statusMessage": "1, 2, 3", + "resultType": "complete", + "result": { + "content": [{ "type": "text", "text": "1, 2, 3" }], + "structuredContent": { + "session_id": "01a068e3-aeb6-7abc-8d6f-5ba9becd3143", + "task_id": "01a06d0d-6f2a-7c41-9b5e-31d0c2a7e8f4", + "context_id": "01a068e3-aeb6-7abc-8d6f-5ba9becd3143", + "state": "TASK_STATE_COMPLETED", + "text": "1, 2, 3" + } + } + } + ``` + +6. Save the `structuredContent.task_id` value. The checkpoint steps use this task ID, not the `v1.` task ID. + ```bash + export HEAD_TASK_ID= + ``` + + +### Answer an agent's question by using curl + +When an agent pauses to ask something, the task's status becomes `input_required` and `tasks/get` returns an `inputRequests` object describing what the agent needs. kagent builds the request as an MCP elicitation. To answer it, send a `tasks/update` call. + +1. Poll the paused task and read `inputRequests`. `inputRequests` is keyed by request ID, and each entry is an `elicitation/create` call whose `requestedSchema` is the schema that kagent built for the pause. Note the key, because the `tasks/update` call requires it. Example output: + ```json + { + "status": "input_required", + "statusMessage": "Which database should we use?", + "inputRequests": { + "01a06d12-4832-7e1f-873d-c25bc6b6b70b": { + "method": "elicitation/create", + "params": { + "mode": "form", + "message": "Which database should we use?", + "requestedSchema": { + "type": "object", + "properties": { + "response": { + "type": "string", + "description": "Which database should we use?", + "enum": ["PostgreSQL", "MySQL"] + } + }, + "required": ["response"], + "additionalProperties": false + } + } + } + } + } + ``` + + The schema's shape depends on what the agent asked for. + - **A question** becomes one string field per question, named `response` when the agent asks one question and `response_1`, `response_2`, and so on when it asks more than one. A question with a fixed set of choices restricts the field to those values with `enum`, and one that accepts more than one answer takes an array. + - **A tool approval** becomes one boolean field per tool, named `approve_1`, `approve_2`, and so on. A response must decide every tool in the request. + +2. Save the request ID that keys the entry. + ```bash + export INPUT_REQUEST_ID= + ``` + +3. Send the answer with `tasks/update`, keying `inputResponses` by the same request ID. An `accept` result sends the answers to the agent. A `decline` or `cancel` result tells the agent that the person refused. If the response key does not match the `inputRequests` key, the server rejects the response with an `Invalid params` error (`-32602`), and the task stays `input_required`. Read the key from `tasks/get`, and do not reuse a task ID or a checkpoint ID. + ```bash + curl -s -X POST http://localhost:8083/mcp \ + -H 'Content-Type: application/json' \ + -H 'Accept: application/json, text/event-stream' \ + -d '{ + "jsonrpc": "2.0", + "id": 5, + "method": "tasks/update", + "params": { + "taskId": "'"$TASK_ID"'", + "inputResponses": { + "'"$INPUT_REQUEST_ID"'": { + "action": "accept", + "content": { "response": "PostgreSQL" } + } + }, + "_meta": { + "io.modelcontextprotocol/clientCapabilities": { + "extensions": { "io.modelcontextprotocol/tasks": {} } + } + } + } + }' + ``` + + The reply confirms only that the update was accepted, and does not include agent output. Example output: + ```json + { "resultType": "complete" } + ``` + +4. Poll the task again to collect the resumed turn. The agent picks up where it paused, so the status returns to `working` and then `completed` with the answer that your response produced. Example output: + ```json + { + "status": "completed", + "statusMessage": "You chose PostgreSQL as the database to use.", + "resultType": "complete", + "result": { + "content": [{ "type": "text", "text": "You chose PostgreSQL as the database to use." }] + } + } + ``` + +For the pause kinds, the approval model, and what the agent receives, see [Human in the loop]({{< link path="agents/human-in-the-loop" >}}). + +### Checkpoint a Session by using curl + +The checkpoint tools pin a Session's state. These tools do not need the tasks extension, so their requests omit `_meta`. + +1. Create a checkpoint of the latest finished turn. `expected_head_task_id` identifies the turn to pin. If the Session has advanced past that turn, the call fails. Use the `$HEAD_TASK_ID` from the latest turn. The `v1.` task ID that `tasks/get` uses fails with `conversation advanced beyond the expected task`. `request_id` is an idempotency key, so a repeated call with the same value returns the existing checkpoint. + ```bash + curl -s -X POST http://localhost:8083/mcp \ + -H 'Content-Type: application/json' \ + -H 'Accept: application/json, text/event-stream' \ + -d '{ + "jsonrpc": "2.0", + "id": 6, + "method": "tools/call", + "params": { + "name": "create_session_checkpoint", + "arguments": { + "session_id": "'"$SESSION_ID"'", + "expected_head_task_id": "'"$HEAD_TASK_ID"'", + "request_id": "my-first-checkpoint" + } + } + }' + ``` + + The result identifies the checkpoint and the turn that it pinned. Example output: + ```json + { + "checkpoint": { + "id": "01a06d19-540a-7040-befb-ec4499c96ff2", + "session_id": "01a068e3-aeb6-7abc-8d6f-5ba9becd3143", + "head_task_id": "01a06d13-2504-7245-9eaf-9c1870c51d26", + "history_sequence": 398, + "state": "CHECKPOINT_STATE_READY", + "created_at": "2026-09-04T15:46:11.594634Z" + } + } + ``` + +2. Save the checkpoint ID. + ```bash + export CHECKPOINT_ID= + ``` + +3. List the Session's checkpoints. + ```bash + curl -s -X POST http://localhost:8083/mcp \ + -H 'Content-Type: application/json' \ + -H 'Accept: application/json, text/event-stream' \ + -d '{ + "jsonrpc": "2.0", + "id": 7, + "method": "tools/call", + "params": { + "name": "list_session_checkpoints", + "arguments": { + "session_id": "'"$SESSION_ID"'" + } + } + }' + ``` + + The result is a `checkpoints` array of the same records, oldest first. Example output: + ```json + { + "checkpoints": [ + { + "id": "01a0690f-5548-7935-b7ca-70919fc9c221", + "session_id": "01a068e3-aeb6-7abc-8d6f-5ba9becd3143", + "head_task_id": "01a0690f-058d-7d29-a880-9b5d6d30b772", + "history_sequence": 91, + "state": "CHECKPOINT_STATE_READY", + "created_at": "2026-09-03T20:56:47.689204Z" + }, + { + "id": "01a06d19-540a-7040-befb-ec4499c96ff2", + "session_id": "01a068e3-aeb6-7abc-8d6f-5ba9becd3143", + "head_task_id": "01a06d13-2504-7245-9eaf-9c1870c51d26", + "history_sequence": 398, + "state": "CHECKPOINT_STATE_READY", + "created_at": "2026-09-04T15:46:11.594634Z" + } + ] + } + ``` + +4. Start a second Session from the checkpoint. `request_id` is an idempotency key, as in the checkpoint request. For what a fork replays and where it diverges, see [Suspend and resume]({{< link path="about/substrate-runtime/suspend-and-resume#checkpoints" >}}). + ```bash + curl -s -X POST http://localhost:8083/mcp \ + -H 'Content-Type: application/json' \ + -H 'Accept: application/json, text/event-stream' \ + -d '{ + "jsonrpc": "2.0", + "id": 8, + "method": "tools/call", + "params": { + "name": "fork_session", + "arguments": { + "checkpoint_id": "'"$CHECKPOINT_ID"'", + "request_id": "my-first-fork" + } + } + }' + ``` + + The result identifies the new Session, which is ready for a message. Example output: + ```json + { + "session": { + "agent": "kagent/my-first-agent", + "id": "01a06d1a-2b7c-7d40-8e13-5c9f04b1a6d2", + "state": "RUNTIME_STATE_READY" + } + } + ``` + +You can now safely [clean up these resources](#clean-up). + +## Clean up + +1. Delete the checkpoint that you created with the `delete_session_checkpoint` tool. If you used Claude Code or Cursor, set `CHECKPOINT_ID` to the ID that the client reported, or ask the client to delete the checkpoint. + ```bash + curl -s -X POST http://localhost:8083/mcp \ + -H 'Content-Type: application/json' \ + -H 'Accept: application/json, text/event-stream' \ + -d '{ + "jsonrpc": "2.0", + "id": 9, + "method": "tools/call", + "params": { + "name": "delete_session_checkpoint", + "arguments": { "checkpoint_id": "'"$CHECKPOINT_ID"'" } + } + }' + ``` + +2. Remove the server entry from your client. In Claude Code, run `claude mcp remove kagent`. In Cursor, delete the `kagent` entry from your MCP settings. + +3. Stop the port-forward with `Ctrl+C`. + +## MCP tool reference + +The server exposes the following agent and Session tools. They cover discovery, conversation, reading back what an agent did, and the {{< gloss "Checkpoint" >}}checkpoint{{< /gloss >}} operations, so a client can manage an agent's state as well as talk to it. An Agent is addressed by its namespace and name, and a Session by its own universally unique identifier (UUID). Every tool that returns a Session reports its `agent` as `/`. The server also exposes a set of [standalone sandbox]({{< link path="about/substrate-runtime/standalone-sandboxes" >}}) tools, which this example does not cover. + +| Tool | Required arguments | What it does | +| ---- | ------------------ | ------------ | +| `list_agents` | None | Lists the Agents that the caller can see. Takes `namespace` to filter, and `page_size` and `page_token`. | +| `list_sessions` | None | Lists the ready Sessions that the caller created. Takes `agent` to filter by Agent, `all_states` to include failed and suspended Sessions, and `page_size` and `page_token`. | +| `create_session` | `agent` | Starts a Session with the Agent. Takes `message` to send a first message, `name` for a display name, and `request_id` for idempotency. With `message`, the call returns once the agent accepts the turn. To read the reply, call `get_session`. | +| `invoke_agent` | `message`, and exactly one of `agent` or `session_id` | Starts a new Session with `agent`, or continues one with `session_id`, and returns the agent's reply with the Session and task IDs. Takes `message_id` for idempotency. | +| `get_session` | `session_id` | Returns the Session's state, its failure if it has one, and its turns, oldest first. Each turn summarizes the user message, the reply, and the tool calls. Takes `last_turns` to return only the most recent turns, and `max_bytes` to cap the answer, which drops the oldest turns first. To poll a running turn, pass `last_turns` set to `1`. | +| `list_tasks` | `session_id` | Pages through the Session's tasks, oldest first, each as the same summary that `get_session` returns. Takes `page_token` and `max_bytes`, and `page_size`, which accepts 1 to 100 and defaults to 50. | +| `get_task` | `session_id`, `task_id` | Returns one task's A2A messages and artifacts in timeline order. Takes `tool_call`, an index or an ID, to return only the A2A parts of one tool call, `from_item` to continue a paged answer, and `max_bytes`. | +| `rename_session` | `session_id`, `name` | Sets the Session's display name. An empty `name` clears it. | +| `delete_session` | `session_id` | Deletes the Session and its runtime state when no turn or other operation on the Session is in progress. | +| `create_session_checkpoint` | `session_id`, `expected_head_task_id` | Pins the Session at a turn boundary. The call fails if the Session has advanced past that task. Takes `request_id` for idempotency. | +| `list_session_checkpoints` | `session_id` | Lists that Session's checkpoints. Takes `page_size` and `page_token`. | +| `get_session_checkpoint` | `checkpoint_id` | Returns one checkpoint's state, and its failure if it has one. | +| `delete_session_checkpoint` | `checkpoint_id` | Deletes the checkpoint. | +| `fork_session` | `checkpoint_id` | Starts a second Session from a checkpoint, replaying the Session's transcript through the pinned turn. Takes `request_id` for idempotency. | + +## Next steps + +{{< cards >}} + {{< card link=`{{< link path="examples/agent-substrate" >}}` title="Agent Substrate" subtitle="Watch the Actor beneath an agent suspend between turns and resume on demand." >}} + {{< card link=`{{< link path="agents/human-in-the-loop" >}}` title="Human in the loop" subtitle="Understand the pauses that an agent can raise and how a client answers them." >}} + {{< card link=`{{< link path="get-started/your-first-mcp-tool" >}}` title="Your first MCP tool" subtitle="Point kagent at an MCP server so that your agent gains tools of its own." >}} +{{< /cards >}} diff --git a/docs-site/content/kagent/main/examples/documentation-agent.md b/docs-site/content/kagent/main/examples/documentation-agent.md new file mode 100644 index 00000000..c3848dc7 --- /dev/null +++ b/docs-site/content/kagent/main/examples/documentation-agent.md @@ -0,0 +1,327 @@ +--- +title: Build a documentation agent +description: Crawl a documentation site into a vector database, serve it over MCP, and give an agent semantic search over it. +weight: 60 +author: kagent.dev +--- + +An agent answers from its model unless you give it a source, such as in a vector database. This example crawls a documentation website into a vector database with [doc2vec](https://github.com/kagent-dev/doc2vec), serves that database over the {{< gloss "Model Context Protocol" >}}Model Context Protocol{{< /gloss >}} (MCP), and binds it to an agent so that the agent searches your documentation before it answers. + +Only the last two steps are kagent's. doc2vec and its MCP server are a separate project, so this guide treats them as a supplied tool server and spends its detail on the kagent side. To bind your first MCP tool without the crawl, see [Your first MCP tool]({{< link path="get-started/your-first-mcp-tool" >}}). + +## About the pieces + +This guide installs the following components. +* **doc2vec**: A command line tool that crawls a website or repository, splits it into chunks, embeds each chunk, and writes the vectors to a database. You run it once, outside the cluster. +* **The doc2vec MCP server**: A published container image that reads that database and exposes semantic search as MCP tools. It runs in your cluster as an ordinary Deployment. +* **kagent**: A `RemoteMCPServer` that points at the MCP server, and an AgentTemplate that binds one or more of its tools. + +The MCP server exposes three tools, and an AgentTemplate names the ones it wants. + +| Tool | Function | +| ---- | -------- | +| `query_documentation` | Semantic search over the crawled documentation, filtered by product and version. | +| `query_code` | Semantic search over crawled source code. | +| `get_chunks` | Returns the chunks of one document, so an agent can read a page rather than a snippet. | + +## Before you begin + +1. [Install kagent]({{< link path="setup/installation" >}}), and [create your first agent]({{< link path="get-started/your-first-agent" >}}) so that you have a Harness and know which label it admits. + +2. Install [Node.js](https://nodejs.org) 20 or later, to run doc2vec. + +3. Export the API key that doc2vec uses for embeddings. + > [!NOTE] + > The MCP server embeds each incoming query, so it needs its own Secret with a model provider key. The agent's ModelConfig does not supply this key, even when both point at the same provider and you reuse the same value. + ```bash + export OPENAI_API_KEY= + ``` + +## Crawl your documentation + +doc2vec reads a YAML file that names what to crawl and where to put the vectors. This example writes to [Qdrant](https://qdrant.tech), so that the cluster reads the vectors over the network rather than needing a database file inside the pod. + +1. Install Qdrant in your cluster. + ```bash + helm repo add qdrant https://qdrant.to/helm + helm repo update + helm upgrade --install qdrant qdrant/qdrant --namespace kagent --wait + ``` + +2. Forward the Qdrant port and leave the command running, so that doc2vec can write to it from your machine. + ```bash + kubectl port-forward -n kagent svc/qdrant 6333:6333 + ``` + +3. Write a doc2vec configuration. Each entry under `sources` becomes one crawl. + ```bash + cat > config.yaml <<'EOF' + embedding: + provider: 'openai' + dimension: 3072 + openai: + model: 'text-embedding-3-large' + + sources: + - type: website + product_name: 'kagent' + version: 'latest' + url: 'https://kagent.dev/docs/' + max_size: 1048576 + database_config: + type: 'qdrant' + params: + qdrant_url: 'http://localhost' + qdrant_port: 6333 + EOF + ``` + + | Field | Description | + | ----- | ----------- | + | `embedding.dimension` | Must match the model. `text-embedding-3-large` produces 3072 dimensions. | + | `product_name` and `version` | Stored on every chunk. The `query_documentation` tool filters on them, so an agent can search one product without seeing another. | + | `database_config.type` | `qdrant` or `sqlite`. Use `sqlite` only when the database file can sit beside the MCP server. | + +4. Run the crawl. The crawl embeds every chunk, so it costs model provider usage and takes longer for a large site. + ```bash + npx doc2vec config.yaml + ``` + +## Deploy the MCP server + +The doc2vec project publishes a prebuilt image of the MCP server, so you deploy it rather than build it from source. + +1. Store the embedding key that the server uses for incoming queries. + ```bash + kubectl create secret generic doc2vec-openai -n kagent \ + --from-literal=OPENAI_API_KEY="$OPENAI_API_KEY" + ``` + +2. Deploy the server and a Service for it. The `TRANSPORT_TYPE` must be `http`, because kagent speaks streamable HTTP to a remote MCP server. + + ```bash + kubectl apply -f - <<'EOF' + apiVersion: apps/v1 + kind: Deployment + metadata: + name: doc2vec-mcp + namespace: kagent + spec: + replicas: 1 + selector: + matchLabels: + app: doc2vec-mcp + template: + metadata: + labels: + app: doc2vec-mcp + spec: + containers: + - name: mcp + # Pin the image to 2.11.0. Tag v2.15.1 accepts a query + # and then fails every search with 'Error querying + # documentation: fetch failed', from an 'InvalidArgumentError: + # invalid onError method' inside the image. + image: ghcr.io/kagent-dev/doc2vec/mcp:2.11.0 + ports: + - containerPort: 3001 + env: + - name: TRANSPORT_TYPE + value: "http" + - name: PORT + value: "3001" + - name: VECTOR_DB_TYPE + value: "qdrant" + - name: QDRANT_URL + value: "http://qdrant.kagent:6333" + - name: OPENAI_API_KEY + valueFrom: + secretKeyRef: + name: doc2vec-openai + key: OPENAI_API_KEY + --- + apiVersion: v1 + kind: Service + metadata: + name: doc2vec-mcp + namespace: kagent + spec: + selector: + app: doc2vec-mcp + ports: + - port: 3001 + targetPort: 3001 + EOF + ``` + +3. Confirm that the server is running on the HTTP transport. + ```bash + kubectl logs -n kagent -l app=doc2vec-mcp --tail=5 + ``` + + Example output: + ```console + Starting MCP server with HTTP transport... + MCP server is running on port 3001 with HTTP transport + Connect to: http://localhost:3001/mcp + ``` + +## Register the server with kagent + +A `RemoteMCPServer` gives kagent the address of the running MCP server. kagent connects to that address, reads the server's tool catalog, and records the tool names in the resource's status. An AgentTemplate then binds names from that list. + +1. Create a `RemoteMCPServer` pointing at the Service. The path is `/mcp`. + ```bash + kubectl apply -f - <<'EOF' + apiVersion: api.kagent.dev/v1alpha3 + kind: RemoteMCPServer + metadata: + name: doc2vec + namespace: kagent + spec: + description: Semantic search over crawled documentation. + protocol: STREAMABLE_HTTP + url: http://doc2vec-mcp.kagent:3001/mcp + timeout: 30s + EOF + ``` + +2. Confirm that kagent reached the server and read its catalog. `ACCEPTED` reports the result of that discovery. + ```bash + kubectl get remotemcpserver doc2vec -n kagent + ``` + + Example output: + ```console + NAME PROTOCOL URL ACCEPTED + doc2vec STREAMABLE_HTTP http://doc2vec-mcp.kagent:3001/mcp True + ``` + +3. Read the tool names from the cluster, rather than assuming them. + ```bash + kubectl get remotemcpserver doc2vec -n kagent \ + -o jsonpath='{.status.discoveredTools[*].name}{"\n"}' + ``` + + Example output: + ```console + get_chunks query_code query_documentation + ``` + +## Create the agent + +The AgentTemplate binds the tools and sets the system prompt that makes the agent search before it answers. This example binds `query_documentation` and `get_chunks`, and leaves out `query_code`, because the crawl covered a website rather than a repository. The prompt also names `productName` and `version` on every query, because `query_documentation` needs both filters together. Passing `productName` alone returns `Error querying documentation: Not Found`, an error that reads like a broken database rather than a missing filter. + +1. Apply an AgentTemplate that binds the search tools, and an Agent that pairs it with your Harness. + ```bash + kubectl apply -f - < [!IMPORTANT] + > Do not set `apiGroup` on `tools[].mcp.server`. The API rejects it with `spec.tools[0].mcp.server: Invalid value: apiGroup must be omitted`. Give the reference a `kind` and a `name` only. + + > [!WARNING] + > kagent resolves the server but never checks the tool names against what the server serves. A misspelled name compiles into a ready revision and fails silently at run time, so copy the names from `status.discoveredTools`. + +2. Create a Session and save its ID. + ```bash + export SESSION_ID=$(kagent agent session create --agent docs-agent -o json | jq -r '.session.id') + echo $SESSION_ID + ``` + +## Ask a question + +Ask the agent something that the crawled documentation covers, and then something that it does not, so that you can tell retrieval from recall. + +1. Ask a question that the crawl covers. + ```bash + kagent agent invoke --session $SESSION_ID \ + --task "What does a Harness do? Answer in two sentences." + ``` + + The agent calls `query_documentation`, receives the matching chunks, and answers from them. Example output: + ```console + A Harness is a Kubernetes custom resource that defines how an agent is allowed to run by + specifying the runtime engine, workload container image and environment, and substrate + policy for scheduling and storage. It selects exactly one runtime out of kagent, codex, + claude, or byo, and determines which AgentTemplates can run on it via a selector. + ``` + +2. Ask a question that the crawl does not cover, to confirm that the agent refuses rather than falling back on the model. + ```bash + kagent agent invoke --session $SESSION_ID \ + --task "How do I configure Istio ambient mode mTLS? Two sentences." + ``` + + Example output: + ```console + The provided documentation does not cover how to configure Istio ambient mode mTLS. + Would you like me to try another query or help with something else? + ``` + +## Clean up + +1. Delete the Session, the Agent, and the AgentTemplate. + ```bash + kagent agent session delete $SESSION_ID + kubectl delete agent docs-agent -n kagent + kubectl delete agenttemplate docs-agent -n kagent + ``` + +2. Delete the tool server and its registration. + ```bash + kubectl delete remotemcpserver doc2vec -n kagent + kubectl delete deployment doc2vec-mcp -n kagent + kubectl delete service doc2vec-mcp -n kagent + kubectl delete secret doc2vec-openai -n kagent + ``` + +3. Uninstall Qdrant, and stop the port-forward with `Ctrl+C`. The chart's StatefulSet volume claim outlives the release, so delete the volume as well. + ```bash + helm uninstall qdrant -n kagent + kubectl delete pvc qdrant-storage-qdrant-0 -n kagent + ``` + +## Next steps + +{{< cards >}} + {{< card link=`{{< link path="get-started/your-first-mcp-tool" >}}` title="Your first MCP tool" subtitle="Bind a tool from the MCP server that ships with kagent." >}} + {{< card link=`{{< link path="skills-and-mcp/about-tools" >}}` title="About tools" subtitle="Read the binding fields and how kagent resolves a tool server." >}} + {{< card link=`{{< link path="examples/skills" >}}` title="Add a skill to an agent" subtitle="Give the same agent packaged instructions alongside its tools." >}} +{{< /cards >}} diff --git a/docs-site/content/kagent/main/examples/skills.md b/docs-site/content/kagent/main/examples/skills.md new file mode 100644 index 00000000..a89257a5 --- /dev/null +++ b/docs-site/content/kagent/main/examples/skills.md @@ -0,0 +1,430 @@ +--- +title: Add a skill to an agent +description: Package instructions and a script as a skill, publish it as an OCI image, and attach it to an AgentTemplate. +weight: 40 +author: kagent.dev +--- + +A {{< gloss "Skill" >}}skill{{< /gloss >}} packages know-how that an agent picks up at run time. A skill consists of a `SKILL.md` file of instructions, together with the scripts and reference files that those instructions depend on. This example builds a skill that turns raw commit subjects into release notes, publishes it as an Open Container Initiative (OCI) image, and attaches it to an {{< gloss "AgentTemplate" >}}AgentTemplate{{< /gloss >}}. + +For the fields that attach a skill and the rules that govern their names, see [Skills]({{< link path="skills-and-mcp/skills" >}}). For the format of a plugin, see [Plugins]({{< link path="skills-and-mcp/plugins" >}}). + +## About skills at run time + +kagent does not fetch a skill when you apply an AgentTemplate. The compiled revision records the source of each skill, and the skill arrives when the agent starts. For an `oci` source, Agent Substrate pulls the image from the Worker and mounts it as read-only into the {{< gloss "Actor" >}}Actor{{< /gloss >}} before the runtime starts. For a `git` or `bucket` source, the Actor fetches the files itself. The runtime then copies each enabled skill into the directory that its Harness reads, which is `/skills/` on a `kagent` Harness. + +Because kagent fetches a skill this late, review the following considerations. + +* **A wrong source still compiles.** kagent validates skill names before it accepts an AgentTemplate, but it never checks that the artifact exists or that it contains a `SKILL.md` file. A bad digest produces a revision that reports `Ready`, and the agent then fails to start. +* **A private registry needs Worker credentials.** Agent Substrate pulls an `oci` source with the registry credentials that the Worker uses for workload images. Kubernetes `imagePullSecrets` on a pod do not configure that pull. +* **Each Actor mounts up to 29 distinct skill images.** Images shared by several skills or plugins count once. A revision that needs more fails when kagent builds its ActorTemplate. +* **Scripts run in the runtime image.** A skill's scripts get whatever the Harness image provides. The kagent runtime image is Alpine Linux with `bash`, `git`, and the standard Alpine utilities, and it does **not** include Python. + +### Skill tools and built-in tools + +On a `kagent` Harness, every agent has kagent's built-in tools, even without a skill. A skill runs its scripts with the built-in `bash` tool. Attaching a skill also adds the skill tools, which read skills. The `claude` and `codex` Harness runtimes also take skills. They expose the skills through their own coding agent's tools. + +| Tool | Availability | What it does | +| ---- | ------------ | ------------ | +| `list_skills` | With a skill | Lists the attached skills with their names and descriptions. | +| `load_skill` | With a skill | Reads a skill's full `SKILL.md` instructions. | +| `load_skill_resource` | With a skill | Reads one file inside a skill directory, such as a reference document. | +| `read_file` | Built-in | Reads a file, with line numbers. | +| `write_file` | Built-in | Writes a file, and creates its parent directories. | +| `edit_file` | Built-in | Replaces an exact string in a file. The edit fails when the string occurs more than once, unless the agent asks to replace every occurrence. | +| `list_files` | Built-in | Lists the contents of a directory. | +| `grep_file` | Built-in | Searches a file, or a directory recursively, for a regular expression. | +| `bash` | Built-in | Runs a command in a new shell. A command stops after 60 seconds by default. The agent can raise that limit to 600 seconds for one call. The tool combines standard output and standard error, and truncates the result at 64 KiB. When the call returns, the tool stops every process that the command started. | + +The built-in tools accept an absolute path or a path relative to the workspace, `/data/workspace`. The root agent and its subagents share the workspace. The workspace starts empty, and its files survive when the Actor is [suspended and resumed]({{< link path="about/substrate-runtime/suspend-and-resume#what-a-snapshot-captures" >}}). A command that is still running does not survive. + +A Harness does not have a per-agent setting for the built-in tools. The `config.runtimeDefaults.builtinTools` Helm values change the built-in tools for every agent in the installation. + +```yaml +config: + runtimeDefaults: + builtinTools: + disableFileSearch: true + disableBash: true +``` + +| Value | Description | +| ----- | ----------- | +| `disableFileSearch` | Set to `true` to remove `list_files` and `grep_file` from the root agent and from every subagent. Defaults to `false`. This value replaces `skillTools.fileSearchTools` from earlier releases, with the opposite meaning. | +| `disableBash` | Set to `true` to remove the `bash` tool from the root agent and from every subagent. The agent keeps the file tools, and still loads skills. Defaults to `false`. | +| `disableAskUser` | Set to `true` to remove the `ask_user` tool. Defaults to `false`. For more information, see [Disable questions]({{< link path="agents/human-in-the-loop#disable-questions" >}}). | +| `bashVenvPath` | The path to a Python virtual environment in the Harness image. The `bash` tool adds the environment's `bin` directory to the start of `PATH`, and sets `VIRTUAL_ENV`. The kagent runtime image does not include Python. To use this value, build a Harness image that includes the environment. Defaults to empty. | + +When you attach a skill, the runtime appends the name and description of each attached skill to the model request, together with an instruction to call `load_skill` before the agent acts on a skill. The model therefore sees each skill before the agent calls any tool. + +> [!IMPORTANT] +> The `bash` tool gives the agent shell access inside its own Actor sandbox, and the sandbox is the boundary that contains the agent. The file tools are not limited to the workspace. They read and write anywhere in the sandbox, including `/skills`, so one skill's instructions can change another skill's files. Review a skill before you attach it, and treat the [egress]({{< link path="about/substrate-runtime/sandboxing" >}}) that you grant the Actor as the reach that the skill has. + +## Before you begin + +1. [Install kagent]({{< link path="setup/installation" >}}). + +2. [Create your first agent]({{< link path="get-started/your-first-agent" >}}), so that you have a Harness and an AgentTemplate to attach a skill to. + +3. Install [Docker](https://docs.docker.com/get-started/get-docker/) to build the skill image. + +4. Choose a container registry that your cluster can reach over HTTPS, and set it as an environment variable. Replace the example value with your own repository. + ```bash + export SKILL_REPO=ghcr.io//release-notes + ``` + + > [!WARNING] + > Agent Substrate pulls a skill image on the Worker. API version v1alpha3 does not have an option to skip certificate verification. kagent 0.x accepted an `insecureSkipVerify` flag for a local registry, but that field does not exist in 1.x. A `localhost` registry that only the host can reach fails at agent startup. + +## Build the skill + +A skill is a directory whose root holds a `SKILL.md` file. Everything else in the directory is available to the agent through the skill tools. + +1. Create the skill directory. + ```bash + mkdir -p release-notes/scripts + cd release-notes + ``` + +2. Write `SKILL.md`. The YAML front matter must include a `name` and a `description`. The body contains the instructions that the agent follows. + ```bash + cat > SKILL.md <<'EOF' + --- + name: release-notes + description: Group a list of conventional commit subjects into release notes with Added, Fixed, and Changed sections. Use this skill whenever the user supplies raw commit subjects and wants them turned into release notes. + --- + # Release notes + + Turn raw commit subjects into release notes grouped by change type. + + ## Instructions + + 1. Ask the user for the commit subjects if they have not supplied them. One subject per line. + 2. Write the subjects to `commits.txt` in your working directory with the `write_file` tool. + 3. Run `bash /skills/release-notes/scripts/group.sh commits.txt` with the `bash` tool. + 4. Return the script's output unchanged. Do not re-order or re-word the entries. + + ## Notes + + - The script reads conventional commit prefixes: `feat:` becomes Added, `fix:` becomes Fixed, and everything else becomes Changed. + - A section with no entries is omitted. + EOF + ``` + + The model reads the `description` to decide whether to load the skill, in the same way as the model decides whether to call a tool. In the `description`, state plainly when the skill applies. The agent reads the instructions in the body only after it calls `load_skill`. + +3. Add the script that the instructions call. The script runs in the agent's runtime image, so it uses `bash` rather than Python. + ```bash + cat > scripts/group.sh <<'EOF' + #!/usr/bin/env bash + # Group conventional commit subjects into release note sections. + set -euo pipefail + + input="${1:?usage: group.sh }" + added="^feat(\([^)]*\))?!?:" + fixed="^fix(\([^)]*\))?!?:" + + section() { + local heading="$1" body="$2" + [ -n "$body" ] || return 0 + printf '### %s\n%s\n\n' "$heading" "$body" + } + + strip() { + sed -E 's/^[a-z]+(\([^)]*\))?!?: *//; s/^/- /' + } + + section Added "$(grep -E "$added" "$input" | strip || true)" + section Fixed "$(grep -E "$fixed" "$input" | strip || true)" + section Changed "$(grep -Ev "$added|$fixed" "$input" | strip || true)" + EOF + chmod +x scripts/group.sh + ``` + +4. Before you publish the skill, confirm that the script works. The `bash` tool returns a failed command's error to the model, not to you. A broken script therefore produces an unreliable answer, not a failed resource. + ```bash + printf 'feat: add checkpoint API\nfix: correct revision digest\ndocs: update install guide\n' > /tmp/commits.txt + bash scripts/group.sh /tmp/commits.txt + ``` + + Example output: + ```console + ### Added + - add checkpoint API + + ### Fixed + - correct revision digest + + ### Changed + - update install guide + ``` + +## Publish the skill as an OCI image + +Agent Substrate mounts an `oci` source as a container image. The image must contain the skill directory and nothing else. Build it from `scratch`, which produces an image whose root **is** the skill root. + +1. Create the Dockerfile. + ```bash + cat > Dockerfile <<'EOF' + FROM scratch + COPY . / + EOF + ``` + +2. Build and push the image. Build for the architecture that your worker nodes run, because kagent pulls the `linux/amd64` or `linux/arm64` manifest that matches the node. + ```bash + docker buildx build --push --platform linux/amd64 -t "$SKILL_REPO:1.0.0" . + ``` + +3. Read the image digest and save the pinned reference. An `oci` source must be pinned to a digest, and a tag alone is rejected. + ```bash + export SKILL_OCI="$SKILL_REPO@$(docker buildx imagetools inspect --format '{{.Manifest.Digest}}' "$SKILL_REPO:1.0.0")" + echo "$SKILL_OCI" + ``` + + Example output: + ```console + ghcr.io/example-org/release-notes@sha256:3091b917d23de93c40e38a574aea1e5615989ca4d7b38f79431c87e04adfa58a + ``` + +## Attach the skill to an AgentTemplate + +To give an agent the skill, reference the skill image from the AgentTemplate that the agent runs, and start a Session on the revision that kagent compiles. + +1. Add a `skills` entry to an AgentTemplate, and pair it with your Harness through an Agent. Keep the model configuration that your existing template uses, and change only the name and the skill. + ```bash + kubectl apply -f - <@sha256:`. | + +2. Confirm that the Agent compiled. The revision is ready when `desiredRevision` and `latestSuccessfulRevision` report the same value. + ```bash + kubectl get agent release-writer -n kagent \ + -o jsonpath='{.status.desiredRevision}{"\t"}{.status.latestSuccessfulRevision}{"\n"}' + ``` + + > [!NOTE] + > A ready revision means that kagent accepted the reference, not that the image exists. A wrong digest surfaces only when the agent starts. + +3. Create a {{< gloss "Session" >}}Session{{< /gloss >}}, and save its ID to an environment variable. A Session pins the revision that it was created on, so a Session that already exists does not pick up the skill. + ```bash + export SESSION_ID=$(kagent agent session create --agent release-writer -o json | jq -r '.session.id') + echo $SESSION_ID + ``` + +## Ask the agent to use the skill + +Send requests to the Session that you created, and check that the agent runs the skill's script. + +1. Send the agent a request that matches the skill's description. + ```bash + kagent agent invoke --session $SESSION_ID \ + --task "Turn these commit subjects into release notes. feat: add checkpoint API. fix: correct revision digest. docs: update install guide." + ``` + +2. Read the reply. The agent calls `load_skill` to read the instructions, `write_file` to stage the commit subjects, and `bash` to run the script, then returns the script's output. + + Example output: + ```console + ### Added + - add checkpoint API + + ### Fixed + - correct revision digest + + ### Changed + - update install guide + ``` + +3. To confirm the attachment from the agent's own side, ask the agent which skills it has. + ```bash + kagent agent invoke --session $SESSION_ID --task "What skills do you have?" + ``` + +## Publish a new version of the skill + +A source is immutable. To change a skill, publish new content under a new digest. + +1. Edit the skill. + +2. Build and push the skill under a new tag and read the new digest. + ```bash + docker buildx build --push --platform linux/amd64 -t "$SKILL_REPO:1.1.0" . + export SKILL_OCI="$SKILL_REPO@$(docker buildx imagetools inspect --format '{{.Manifest.Digest}}' "$SKILL_REPO:1.1.0")" + ``` + +3. Update `skills[].source.oci` on the AgentTemplate with the new digest. kagent compiles a new revision. + +4. Create a new Session. Sessions that already exist keep the skill content of their pinned revision. + +## Bundle the skill in a plugin + +While a standalone source contains one skill, a {{< gloss "Plugin" >}}plugin{{< /gloss >}} contains multiple skills. An AgentTemplate attaches the plugin once and lists the skills that it wants. To ship a set of skills together, or to have the same artifact contribute [Model Context Protocol (MCP) servers]({{< link path="skills-and-mcp/plugins#mcp-servers" >}}) as well, use a plugin. + +1. Restructure the directory so that each skill sits under `skills/`, and add the manifest that makes it a plugin. + ```bash + cd .. + mkdir -p release-tools/skills + mv release-notes release-tools/skills/release-notes + cd release-tools + cat > plugin.json <<'EOF' + { + "$schema": "https://agent-plugins.org/schemas/1.0.0/plugin.schema.json", + "name": "release-tools" + } + EOF + ``` + + > [!NOTE] + > kagent compares the `$schema` value literally and rejects anything else, so copy it exactly. Of the remaining manifest fields, kagent reads only `name`. + +2. Build and push the plugin, and read its digest. A plugin image is built in the same way as a single skill from `scratch`, so that the plugin root is the image root. + ```bash + export PLUGIN_REPO=ghcr.io//release-tools + mv skills/release-notes/Dockerfile . + docker buildx build --push --platform linux/amd64 -t "$PLUGIN_REPO:1.0.0" . + export PLUGIN_OCI="$PLUGIN_REPO@$(docker buildx imagetools inspect --format '{{.Manifest.Digest}}' "$PLUGIN_REPO:1.0.0")" + ``` + +3. Attach the plugin with `plugins` instead of `skills`, and list the skills to enable. + ```bash + kubectl apply -f - < [!IMPORTANT] + > A plugin enables only the skills that you list. If you omit `plugins[].skills`, or leave it empty, the agent does not get any of the skills, and kagent accepts the empty list without an error. For why an explicit list is the safer default, see [Plugins]({{< link path="skills-and-mcp/plugins#skill-allowlist" >}}). + +4. Create a Session on the new revision, and confirm that the agent still has the skill. The `$SESSION_ID` Session is pinned to the revision that included the standalone skill. + ```bash + export PLUGIN_SESSION_ID=$(kagent agent session create --agent release-writer -o json | jq -r '.session.id') + kagent agent invoke --session $PLUGIN_SESSION_ID --task "What skills do you have?" + ``` + + The agent reports `release-notes` exactly as before. A skill behaves in the same way whether it arrives on its own or inside a plugin, because kagent copies both into `/skills` before the agent starts. + +## Troubleshoot a skill that does not load + +A skill that kagent cannot fetch stops the agent from starting at all, rather than producing an agent without that skill. The agent logs the failure and exits, and the Agent never becomes ready. + +1. Check the Agent's `Ready` condition. A skill that cannot be fetched leaves the Agent waiting, because the Actor that builds the golden snapshot also fetches the skill. + ```bash + kubectl get agent release-writer -n kagent \ + -o jsonpath='{range .status.conditions[?(@.type=="Ready")]}{.status} {.reason} {.message}{end}' + ``` + + Example output: + ```console + False ActorTemplatePending waiting for the ActorTemplate golden snapshot + ``` + + > [!NOTE] + > This condition does not name the skill, and reports the same reason for any Actor that has not yet produced a snapshot. An agent that is merely still starting looks identical to one whose skill cannot be fetched. + +2. Find the WorkerPool that the Harness runs on, and read its Workers' logs. The Actor writes the failure to the Worker logs, not to the Agent. + + ```bash + export WORKER_POOL=$(kubectl get harness my-first-harness -n kagent \ + -o jsonpath='{.spec.substrate.workerPoolRef.name}') + kubectl logs -n kagent -l ate.dev/worker-pool=$WORKER_POOL --tail=200 \ + | grep -i "materialize" + ``` + + Example output: + ```console + {"error":"materialize agent plugins: materialize skill \"release-notes\": pull ghcr.io/example-org/release-notes@sha256:3091b91...: Get \"https://ghcr.io/v2/\": EOF","labels":{"ate.atespace":"ate-golden","ate.template.name":"release-writer-my-first-harness-23c20dcb296d"},"level":"ERROR","msg":"failed to materialize Agent Plugins"} + ``` + + A pool runs the Workers for every agent on it. To find the entries for one agent, filter by the `ate.template.name` label. Its value is the AgentTemplate name, the Harness name, and the revision's short form, joined by hyphens. + +3. Match the message to its cause. + + | Message | Cause | + | ------- | ----- | + | `pull : ... 401 Unauthorized` | The registry needs credentials that the cluster does not have. | + | `pull : ... x509` or a Transport Layer Security (TLS) error | The registry does not serve HTTPS with a certificate that the runtime trusts. | + | `materialize skill "": exit status 128`, after `Could not resolve host` | The source is `git`, which the Actor that builds the golden snapshot cannot download. Use an `oci` source, as described in [Every source is immutable]({{< link path="skills-and-mcp/skills#every-source-is-immutable" >}}). | + | `SKILL.md is required` | The artifact was fetched, but the root that `source.path` selects does not contain a `SKILL.md` file. | + | `symlink "..." escapes artifact root` | A symlink in the artifact points outside it. | + | `artifact contains more than 10000 filesystem entries`, or `artifact exceeds 104857600 bytes` | The artifact exceeds one of the plugin limits. | + +> [!TIP] +> Build the skill image with `--platform` set to the architecture of your worker nodes. kagent asks the registry for the `linux/amd64` or `linux/arm64` manifest that matches the node it runs on, so an image published for one architecture alone fails on the other. + +## Clean up + +1. Delete the Sessions that you created. Deleting the Agent does not remove them. If you did not complete the plugin section, skip the `$PLUGIN_SESSION_ID` command. + ```bash + kagent agent session delete $SESSION_ID + kagent agent session delete $PLUGIN_SESSION_ID + ``` + +2. Delete the Agent and the AgentTemplate. + ```bash + kubectl delete agent release-writer -n kagent + kubectl delete agenttemplate release-writer -n kagent + ``` + +3. Remove the skill directory from your machine. The directory is `release-notes`, or `release-tools` if you completed the plugin section. + ```bash + cd .. + rm -rf release-notes release-tools + ``` + +4. Delete the images that you pushed, `$SKILL_REPO` and `$PLUGIN_REPO`, by using your registry's own tooling. + +## Next steps + +{{< cards >}} + {{< card link=`{{< link path="skills-and-mcp/plugins" >}}` title="Plugins" subtitle="Bundle skills and MCP servers into one plugin that an AgentTemplate attaches at once." >}} + {{< card link=`{{< link path="skills-and-mcp/skills" >}}` title="Skills" subtitle="Read the full set of skill fields, source kinds, and naming rules." >}} + {{< card link=`{{< link path="get-started/your-first-mcp-tool" >}}` title="Your first MCP tool" subtitle="Give the same agent a tool from an MCP server alongside its skills." >}} +{{< /cards >}} diff --git a/docs-site/content/kagent/main/get-started/_index.md b/docs-site/content/kagent/main/get-started/_index.md new file mode 100644 index 00000000..91326661 --- /dev/null +++ b/docs-site/content/kagent/main/get-started/_index.md @@ -0,0 +1,6 @@ +--- +title: Get started +description: Install kagent and create your first agent. +weight: 10 +author: kagent.dev +--- \ No newline at end of file diff --git a/docs-site/content/kagent/main/get-started/your-first-agent.md b/docs-site/content/kagent/main/get-started/your-first-agent.md new file mode 100644 index 00000000..f0b19a7c --- /dev/null +++ b/docs-site/content/kagent/main/get-started/your-first-agent.md @@ -0,0 +1,181 @@ +--- +title: Your first agent +description: Create and communicate with your first agent by using the kagent project. +weight: 10 +author: kagent.dev +--- + +This guide walks you through creating an agent, from applying a Harness and an AgentTemplate to holding a conversation with the Agent that pairs them. You apply the Harness, the AgentTemplate, and the Agent as Kubernetes resources. Then, you create and talk to a Session with the kagent CLI. For definitions of each of these components, review the [core concepts]({{< link path="about/core-concepts" >}}). For an overview of how each component fits together in {{< reuse "kagent-docs/snippets/name-product.md" >}}, review the [architecture]({{< link path="about/architecture/kagent" >}}). For the complete schema of every field that this guide sets, see the [API reference]({{< link path="reference/api-ref" >}}). + +## Before you begin + +1. [Install kagent with a WorkerPool provisioned]({{< link path="setup/installation" >}}). +2. {{< reuse "kagent-docs/snippets/install-cli.md" >}} + +3. Install [`jq`](https://jqlang.org/download/), to read the Session ID out of the CLI's JSON output. + +> [!NOTE] +> The CLI reaches the kagent controller at `localhost:8083`. When nothing serves that port, the CLI runs `kubectl port-forward` against the `kagent-controller` service, and closes the forward when the command exits. Keep `kubectl` on your path, and keep your kubeconfig pointed at the cluster that runs kagent. To reach a controller at another address, pass the `--api-url` flag. The CLI then needs a kubeconfig only for `kagent install`, `kagent uninstall`, `kagent dashboard`, and `kagent bug-report`. + +## Create a Harness, an AgentTemplate, and an Agent + +Three resources define a runnable agent: a Harness holds the runtime, an AgentTemplate holds the behavior, and an Agent pairs one of each. + +1. Apply a `Harness` that uses kagent's native runtime. Its `substrate` section names the [WorkerPool]({{< link path="about/architecture/agent-substrate#workers-and-workerpools" >}}) that this Harness's Actors run on, and the object storage location for their [snapshots]({{< link path="about/architecture/agent-substrate#suspend-snapshot-and-resume" >}}). + ```yaml + apiVersion: api.kagent.dev/v1alpha3 + kind: Harness + metadata: + name: my-first-harness + namespace: kagent + spec: + kagent: {} + workload: + # kagent's native runtime image, pinned by digest + image: {{< reuse "kagent-docs/versions/runtime-image.md" >}} + substrate: + workerPoolRef: + name: kagent-default + snapshotPolicy: + # The bucket that the Agent Substrate chart creates in its bundled object store. + # If your Substrate installation uses your own object storage, use that location instead. + location: s3://ate-snapshots/kagent/ + ``` + +2. Apply an `AgentTemplate`. The `modelConfig` field references the `default-model-config` {{< gloss "ModelConfig" >}}ModelConfig{{< /gloss >}} that was automatically created for the model provider API key that you provided during kagent installation. + ```yaml + apiVersion: api.kagent.dev/v1alpha3 + kind: AgentTemplate + metadata: + name: my-first-template + namespace: kagent + spec: + description: My first kagent agent + modelConfig: + # Default config created by the kagent install guide + name: default-model-config + systemPrompt: You are a concise, helpful assistant. + ``` + +3. Apply an `Agent` that pairs a Harness and an AgentTemplate. + ```yaml + apiVersion: api.kagent.dev/v1alpha3 + kind: Agent + metadata: + name: my-first-agent + namespace: kagent + spec: + template: + ref: + name: my-first-template + harnessRef: + name: my-first-harness + ``` + > [!NOTE] + > `spec.template` takes either a reference, as shown here, or a complete AgentTemplate spec under `inline`. An inline template is a complete value rather than an override, and kagent does not create a Kubernetes object to back it. `spec.harnessRef` takes a reference only, so a Harness must exist before an Agent can name it. + +4. Confirm that the Agent is ready. `READY` reports whether kagent compiled and prepared a runtime {{< gloss "Revision" >}}revision{{< /gloss >}} for the pair. + ```bash + kagent agent get my-first-agent + ``` + + Example output: + ```console + +----------------+-------+----------------------+ + | NAME | READY | CREATED | + +----------------+-------+----------------------+ + | my-first-agent | True | 2026-08-31T15:01:44Z | + +----------------+-------+----------------------+ + ``` + + A `READY` value of `UNKNOWN` means that the kagent controller has not yet reconciled the Agent. Wait a few seconds, then check again. A `READY` value of `False` right after you apply the Agent is also expected, because kagent builds a snapshot of the agent's runtime before it reports the Agent ready. This step can take a minute. If `READY` stays `False`, inspect the individual conditions to find which stage failed. + ```bash + kubectl get agent my-first-agent -n kagent -o jsonpath='{.status.conditions}' | jq + ``` + + An Agent reports four conditions, ending in `Ready`. The `Accepted` condition covers the shape of the spec, and `ResolvedRefs` covers the AgentTemplate, Harness, ModelConfig, and tool references. `Compatible` covers whether the resolved configuration suits the Harness runtime, and `Ready` covers the compiled revision itself. `status.warnings` lists non-blocking compatibility decisions that the compiler made. + +## Create a Session + +A Session is one running conversation with an Agent. Creating it starts an Actor on the WorkerPool from the Agent's latest successful revision. + +1. Create a Session against the Agent, and save its ID to an environment variable. + ```bash + export SESSION_ID=$(kagent agent session create --agent my-first-agent -o json | jq -r '.session.id') + echo $SESSION_ID + ``` + + The command returns only after the Session reaches the `READY` state. Run it without `-o json` to see the table instead: + ```console + +--------------------------------------+----------------+-------+----------------------+ + | ID | AGENT | STATE | CREATED | + +--------------------------------------+----------------+-------+----------------------+ + | 0198c3d7-4f2a-7b61-9c3e-5d8f7a2b4e10 | my-first-agent | READY | 2026-08-31T15:02:10Z | + +--------------------------------------+----------------+-------+----------------------+ + ``` + + An error reporting that the Agent has no ready prepared revision means that the Agent is not `Ready` yet. Return to step 4 of the previous section to check the conditions. + +## Talk to your agent + +1. Send a message to the Session. The CLI holds the conversation over the {{< gloss "A2A" >}}A2A{{< /gloss >}} (Agent2Agent) protocol. + ```bash + kagent agent invoke --session $SESSION_ID --task "What is 2+2?" + ``` + + The agent's reply prints as text. + ```console + 4 + ``` + +2. Send a follow-up message to the same Session. A Session holds the {{< gloss "Transcript" >}}transcript{{< /gloss >}} of its conversation, so the agent answers with the earlier turns in context. + ```bash + kagent agent invoke --session $SESSION_ID --task "What did I just ask you?" + ``` + + ```console + You asked what 2+2 is. + ``` + +> [!NOTE] +> A Session gives its Worker back at the end of every turn. The Session itself stays `READY`, because suspension applies to the Actor running underneath it rather than to the conversation, and the next `kagent agent invoke` resumes that Actor automatically. To understand what happens to the Actor in between, see [Suspend and resume]({{< link path="about/substrate-runtime/suspend-and-resume" >}}). + +The `invoke` command takes a few more options that are useful beyond a first conversation. + +| Option | Description | +| ------ | ----------- | +| `--file` | Read the task from a file, or from standard input with `-`, instead of passing it inline with `--task`. | +| `--stream` | Print the reply as the agent produces it, rather than waiting for the complete answer. | + +> [!TIP] +> Run `kagent` with no arguments to open an interactive workspace in your terminal, where you can browse your Sessions and chat with them without passing an ID to each command. + +## Next steps + +{{< cards >}} + {{< card link=`{{< link path="get-started/your-first-mcp-tool" >}}` title="Your first MCP tool" subtitle="Bind a Model Context Protocol tool so that your agent can act on live cluster data." >}} + {{< card link=`{{< link path="about/architecture/agent-substrate" >}}` title="Agent Substrate architecture" subtitle="Understand what happens to your Session's Actor when it sits idle." >}} + {{< card link=`{{< link path="agents/agent-harness" >}}` title="Agent harness" subtitle="Choose from the full set of Harness runtime options." >}} + {{< card link=`{{< link path="skills-and-mcp/skills" >}}` title="Skills" subtitle="Give your agent capabilities beyond its system prompt." >}} +{{< /cards >}} + +## Optional: Clean up + +> [!IMPORTANT] +> Other guides build on the Harness, AgentTemplate, and Agent that you created here, including [Your first MCP tool]({{< link path="get-started/your-first-mcp-tool" >}}) and [Agent Substrate]({{< link path="examples/agent-substrate" >}}). Unless you are finished with the kagent guides, leave the resources in place. + +To remove the resources, follow these steps. + +1. Delete every Session that was created against the Agent. Later guides create their own sessions against the same Agent, so delete them all rather than only the one that you saved. Deleting the Agent does not delete the Sessions that were created against it, so delete the sessions first. + ```bash + kagent agent session list -o json \ + | jq -r '.sessions[] | select(.agent.name == "my-first-agent") | .id' \ + | xargs -n1 kagent agent session delete + ``` + +2. Delete the Agent, the AgentTemplate, and the Harness. + ```bash + kubectl delete agent my-first-agent -n kagent + kubectl delete agenttemplate my-first-template -n kagent + kubectl delete harness my-first-harness -n kagent + ``` diff --git a/docs-site/content/kagent/main/get-started/your-first-mcp-tool.md b/docs-site/content/kagent/main/get-started/your-first-mcp-tool.md new file mode 100644 index 00000000..34a6b7cc --- /dev/null +++ b/docs-site/content/kagent/main/get-started/your-first-mcp-tool.md @@ -0,0 +1,188 @@ +--- +title: Your first MCP tool +description: Give an agent a Model Context Protocol tool by binding an MCP server to its AgentTemplate. +weight: 20 +author: kagent.dev +--- + +A system prompt tells an agent how to behave. Tools tell it what it can do. This guide binds a {{< gloss "Model Context Protocol" >}}Model Context Protocol{{< /gloss >}} (MCP) tool to the agent that you built in [Your first agent]({{< link path="get-started/your-first-agent" >}}), so that the agent can read live data out of your cluster instead of answering from the model alone. For the full {{< gloss "Tool binding" >}}tool binding{{< /gloss >}} schema, including binding one AgentTemplate as another agent's subagent, see [About tools]({{< link path="skills-and-mcp/about-tools" >}}). + +## Before you begin + +1. Complete [Your first agent]({{< link path="get-started/your-first-agent" >}}). This guide edits the `my-first-template` AgentTemplate that the agent guide creates, so keep that AgentTemplate, the `my-first-harness` Harness, and the `my-first-agent` Agent in place. + +2. Confirm that you have the kagent CLI and [`jq`](https://jqlang.org/download/) installed. + +## Bind the tool to your AgentTemplate + +kagent ships an MCP server of its own, and installs a `RemoteMCPServer` that points at it, so the built-in server is the shortest path to a working tool. An {{< gloss "AgentTemplate" >}}AgentTemplate{{< /gloss >}} takes tools through an `mcp` binding, which names one server and, optionally, the tools to take from it. This guide names the tools specifically, so that the agent gets only the two tools it needs rather than the server's whole catalog. + +kagent records what it discovered on the server's status, so the tool names come from the cluster. This guide binds `k8s_get_resources` and `k8s_get_pod_logs`. For the full catalog that the built-in server serves, see the [tools ecosystem reference]({{< link path="reference/tools-ecosystem" >}}). + +1. List the {{< gloss "RemoteMCPServer" >}}RemoteMCPServers{{< /gloss >}} in the `kagent` namespace. + ```bash + kubectl get remotemcpserver -n kagent + ``` + + Example output: The `ACCEPTED` column reports whether kagent reached the server and read its catalog. No tool can be bound from a server that is not `True`. + ```console + NAME PROTOCOL URL ACCEPTED + kagent-tool-server STREAMABLE_HTTP http://kagent-tools.kagent.svc.cluster.local:8084/mcp True + ``` + + To see the tools that the server offers, read the discovered set from its status. + ```bash + kubectl get remotemcpserver kagent-tool-server -n kagent \ + -o jsonpath='{range .status.discoveredTools[*]}{.name}{"\t"}{.description}{"\n"}{end}' + ``` + + > [!NOTE] + > The built-in server is installed only when the `kagent-tools.enabled` Helm value is `true`, which is the default. If the command does not return any resources, either re-install with that value enabled, or use your own server as described in [Bind your own MCP server](#bind-your-own-mcp-server). + +2. Re-apply the `my-first-template` AgentTemplate with a `spec.tools` list and a system prompt that tells the model what the tools are for. + ```yaml + kubectl apply -f - <}}). | + | `mcp.tools` | Optional. The names of the tools to bind, up to 50. An omitted or empty list exposes every tool on the server. An AgentTemplate takes at most 50 bindings in total. | + +3. Confirm that kagent compiled a new {{< gloss "Revision" >}}revision{{< /gloss >}} for the Agent that uses the edited template. Every edit produces a new desired revision, and the Agent is current when the latest successful revision matches it. + ```bash + kubectl get agent my-first-agent -n kagent \ + -o jsonpath='{.status.desiredRevision}{"\t"}{.status.latestSuccessfulRevision}{"\n"}' + ``` + + Example output: + ```console + 7c1f9a2b4e8d3f60a5b7c9e1d2f4a6b8c0d2e4f68a9b1c3d5e7f9a1b3c5d7e9f 7c1f9a2b4e8d3f60a5b7c9e1d2f4a6b8c0d2e4f68a9b1c3d5e7f9a1b3c5d7e9f + ``` + + When the two values differ, kagent is still compiling, or compilation failed. A binding that names a RemoteMCPServer that does not exist in the namespace fails at the `ResolvedRefs` condition with the reason `ReferenceResolutionFailed`. + + kagent resolves the server, but it does not check the tool names against the tools that the server actually serves. A misspelled tool name compiles into a ready revision, and the only symptom is an agent that never calls the tool that you expected. If a bound tool appears to be missing, check the spelling against the server's catalog. + +## Create a Session that has the tool + +A {{< gloss "Session" >}}Session{{< /gloss >}} runs the revision that it was created from, and keeps running that revision for its whole life. The session from the agent guide still runs the revision without tools, so create a second session to pick up the binding. + +1. Create a second Session against the same Agent, and save its ID to an environment variable. The Agent has a newer revision now, so this session picks up the tools. + ```bash + export TOOL_SESSION_ID=$(kagent agent session create --agent my-first-agent -o json | jq -r '.session.id') + echo $TOOL_SESSION_ID + ``` + + Run the command without `-o json` to see the table instead: + ```console + +--------------------------------------+----------------+-------+----------------------+ + | ID | AGENT | STATE | CREATED | + +--------------------------------------+----------------+-------+----------------------+ + | 0198c4e2-8b3f-7d45-a1c6-9e2f4b8d6a03 | my-first-agent | READY | 2026-08-31T16:20:38Z | + +--------------------------------------+----------------+-------+----------------------+ + ``` + +2. Ask the agent something that it can answer only by calling a tool. + ```bash + kagent agent invoke --session $TOOL_SESSION_ID --task "Which pods are running in the kagent namespace?" + ``` + + The agent calls `k8s_get_resources` and answers from the result rather than from the model's own knowledge. + +3. Ask a follow-up question that uses the second tool. The Session holds the {{< gloss "Transcript" >}}transcript{{< /gloss >}} of the conversation, so the agent can act on the pods that it just listed. + ```bash + kagent agent invoke --session $TOOL_SESSION_ID --task "Show me the last few log lines from the kagent controller pod." + ``` + +## Bind your own MCP server + +A `RemoteMCPServer` points at any MCP server that the cluster can reach, whether it runs in the cluster or outside it. Create one, then bind it in the same way that you bound the built-in server. + +1. Apply a `RemoteMCPServer` for your own server. + ```yaml + kubectl apply -f - < [!IMPORTANT] +> Leave the Harness, AgentTemplate, Agent, and Sessions in place. Other guides build on them, and [Your first agent]({{< link path="get-started/your-first-agent#optional-clean-up" >}}) covers removing them when you are finished with the kagent guides. Leave `kagent-tool-server` in place as well, because the kagent installation owns it. + +If you created a RemoteMCPServer of your own in [Bind your own MCP server](#bind-your-own-mcp-server), delete it. + +```bash +kubectl delete remotemcpserver my-mcp-server -n kagent +``` + +## Next steps + +{{< cards >}} + {{< card link=`{{< link path="skills-and-mcp/about-tools" >}}` title="About tools" subtitle="Read the full tool binding schema, including binding one AgentTemplate as another agent's subagent." >}} + {{< card link=`{{< link path="skills-and-mcp/skills" >}}` title="Skills" subtitle="Give your agent capabilities that no MCP server provides." >}} + {{< card link=`{{< link path="examples/agent-substrate" >}}` title="Agent Substrate" subtitle="Watch your agent's Actor suspend between turns, then checkpoint the conversation." >}} +{{< /cards >}} diff --git a/docs-site/content/kagent/main/not-in-version.md b/docs-site/content/kagent/main/not-in-version.md new file mode 100644 index 00000000..6a9b3694 --- /dev/null +++ b/docs-site/content/kagent/main/not-in-version.md @@ -0,0 +1,8 @@ +--- +title: Topic not available in this version +description: This topic isn't available in this version of the documentation. +build: + list: never +--- + +This topic doesn't exist in this version of the kagent docs yet. Use the version switcher to go back to a version where it's available, or browse the [kagent docs](https://kagent.dev/docs/) landing page. diff --git a/docs-site/content/kagent/main/observability/_index.md b/docs-site/content/kagent/main/observability/_index.md new file mode 100644 index 00000000..54137b72 --- /dev/null +++ b/docs-site/content/kagent/main/observability/_index.md @@ -0,0 +1,6 @@ +--- +title: Observability +description: Set up an OpenTelemetry stack and review the traces, metrics, and logs that kagent and Agent Substrate report. +weight: 70 +author: kagent.dev +--- \ No newline at end of file diff --git a/docs-site/content/kagent/main/observability/audit-prompts.md b/docs-site/content/kagent/main/observability/audit-prompts.md new file mode 100644 index 00000000..c5b385ee --- /dev/null +++ b/docs-site/content/kagent/main/observability/audit-prompts.md @@ -0,0 +1,157 @@ +--- +title: Audit prompts +description: Capture the prompts and replies that your agents exchange with a model in your traces, then find them in your tracing backend for security and compliance review. +weight: 50 +author: kagent.dev +--- + +Audit the prompts (inputs) and replies (outputs) that pass between your agents and their models. Security and compliance teams use these records to review how people use your {{< reuse "kagent-docs/snippets/name-product.md" >}} environment. For example, you can confirm that no request sends personally identifiable information (PII) to a model. You can also reconstruct the instructions that an agent received in an earlier conversation. + +## About prompt auditing + +{{< reuse "kagent-docs/snippets/name-product.md" >}} records prompts and replies in traces. When you turn on content capture, the runtime records the agent's request and the model's reply on its spans. You then find the records in your tracing backend. + +The `kagent` runtime does not write prompts or replies to log records, so a logging backend on its own cannot hold a prompt audit trail. Capture is off by default, because the content can include sensitive user and model data. + +### Content capture by harness + +Each harness records the content on its own instrumentation, so where the content lands differs by harness. The settings in the last column are the fields in the `config.runtimeDefaults.telemetry` Helm block. For the available harnesses, see [Choose a harness]({{< link path="agents/agent-harness#choose-a-harness" >}}). + +| Harness | Where the content goes | Settings | +| ------- | ---------------------- | -------- | +| `kagent` | The `generate_content` span of each model call, in three attributes. See [What a record holds](#what-a-record-holds). | `captureContent` | +| `codex` | The `invoke_agent` span for prompts and replies, and the `execute_tool` span for tool arguments and successful results. Codex also records prompts in its own telemetry. | `captureContent` | +| `claude` | The `invoke_agent` span for prompts and replies, and the `execute_tool` span for tool arguments and successful results. Claude Code also records prompts and tool details on its own spans, and assistant replies in its own log records. With `captureRawApiBodies`, the log records also include the complete provider request and response bodies. These bodies are a fuller record than the spans. | `captureContent`, `captureRawApiBodies` | +| `byo` | Wherever the image's own instrumentation records it. The controller sets `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` on this harness, but an image that does not read the variable records nothing. | `captureContent` | + +For each setting, see the agent harness [telemetry content settings]({{< link path="agents/agent-harness#telemetry-content-settings" >}}). + +### What a record holds + +On the `kagent` runtime, each `generate_content` span carries the following three attributes, as JSON. + +| Attribute | What it holds | +| --------- | ------------- | +| `gen_ai.system_instructions` | The `systemPrompt` field of your AgentTemplate, followed by the instructions that the runtime appends. | +| `gen_ai.input.messages` | The chat history sent to the model, as a list of messages with a `role` and typed `parts`. | +| `gen_ai.output.messages` | The model's reply, with its `finish_reason`. On a turn that calls a tool, the reply holds the tool call and its arguments instead of text. | + +The record holds more than the prompts that your team wrote. + +The system instruction holds the instructions that the runtime appends as well as the prompt your team wrote, and the message history holds the person's messages, the agent's earlier turns, and the model's tool calls. + +{{< reuse "kagent-docs/snippets/observability/audit-tool-attributes.md" >}} + +On the `codex` and `claude` runtimes, the `invoke_agent` span holds only the current turn's prompt and the reply to that prompt. The `chat` spans hold no content. The `execute_tool` span holds tool arguments and successful results in the `gen_ai.tool.call.arguments` and `gen_ai.tool.call.result` attributes. + +On the `kagent` runtime, the span of each model call holds the full conversation history. A long conversation therefore repeats its earlier messages in every span. Account for that volume when you set a retention period. + +The `kagent` runtime does not shorten these attributes. The `claude` and `codex` runtimes bound each captured prompt, reply, tool argument, and tool result to the `config.runtimeDefaults.telemetry.maxCaptureBytes` budget, which defaults to 16384 bytes. A shortened capture sets the `kagent.capture.input_truncated` or `kagent.capture.output_truncated` attribute. Your tracing backend might also limit the size of an attribute, so check its limits before you rely on it for long conversations. + +### Delivery + +Agent Substrate suspends an Actor as soon as a response completes. The controller therefore sets the `kagent` and `codex` runtimes to flush their spans before each response completes, so that the records of a turn reach your backend before the Actor suspends. On the `claude` runtime, kagent flushes the spans that it writes in the same way, but not the spans of Claude Code itself. Claude Code's records of a conversation's last turn can therefore be lost. For more information, see [Traces from a suspended Actor]({{< link path="observability/tracing#traces-from-a-suspended-actor" >}}). + +> [!IMPORTANT] +> Traces are a best-effort record. An exporter drops spans without an error when the collector is unreachable or its queue is full, and nothing in the trace shows that a record is missing. Treat captured spans as a review aid, not as a complete or tamper-proof compliance log. + +## Before you begin + +1. [Install kagent]({{< link path="setup/installation" >}}). +2. [Create your first agent]({{< link path="get-started/your-first-agent" >}}), so that you have an {{< gloss "Agent" >}}Agent{{< /gloss >}} to send requests to. That guide also installs the kagent CLI. The steps on this page need the {{< reuse "kagent-docs/versions/kagent.md" >}} CLI, because the CLIs of other releases, newer ones included, do not have the Session commands that these steps use. To check your version, run `kagent version`. +3. Install [`jq`](https://jqlang.org/download/), to read the Session ID and revision out of the CLI's JSON output. +4. Set up a tracing backend, and turn on tracing. {{< reuse "kagent-docs/snippets/observability/tracing-backend.md" >}} + +## Turn on content capture + +Turn on content capture in the kagent Helm release, then create a Session that picks up the new setting. + +1. Save the current revision of your Agent. A later step uses it to tell when kagent rebuilds the Agent with the new setting. The command first waits for any rebuild that is still in progress, such as one from an earlier Helm upgrade, so that it saves a finished revision. + ```bash + for i in $(seq 1 60); do + REVISIONS=$(kubectl get agent my-first-agent -n kagent \ + -o jsonpath='{.status.desiredRevision} {.status.latestSuccessfulRevision}') + [ "${REVISIONS% *}" = "${REVISIONS#* }" ] && break + sleep 5 + done + export OLD_REVISION=${REVISIONS#* } + echo "Current revision: $OLD_REVISION" + ``` + +2. Upgrade the kagent Helm release. The `--reuse-values` flag keeps the tracing settings that you already set. + ```bash + helm upgrade kagent \ + {{< reuse "kagent-docs/snippets/helm-path.md" >}}/{{< reuse "kagent-docs/snippets/helm-kagent.md" >}} \ + --version {{< reuse "kagent-docs/versions/kagent.md" >}} \ + --namespace kagent \ + --reuse-values \ + --set config.runtimeDefaults.telemetry.captureContent=true + ``` + + For an agent on the `claude` runtime, also set `config.runtimeDefaults.telemetry.logs.endpoint` to the address of your collector, and send the logs to a backend that stores them. {{< reuse "kagent-docs/snippets/observability/claude-logs-backend.md" >}} Without log export, the replies of a `claude` agent are not recorded. + +3. Wait for the controller to roll out. + ```bash + kubectl rollout status deployment/kagent-controller -n kagent --timeout=300s + ``` + +4. Wait for kagent to rebuild the Agent. The controller rebuilds each Agent after the controller restarts. A Session that you create before the rebuild finishes starts from the previous revision, without the new setting. The following command prints `Recompiled` when the new revision is ready. + ```bash + for i in $(seq 1 60); do + [ "$(kubectl get agent my-first-agent -n kagent \ + -o jsonpath='{.status.latestSuccessfulRevision}')" != "$OLD_REVISION" ] \ + && echo "Recompiled" && break + sleep 5 + done + ``` + If the command finishes without printing `Recompiled`, the upgrade did not change the settings that kagent compiles into the Agent. Either the setting was already in place, or the chart did not recognize the key. Helm accepts a key that a chart does not define without an error, so check that you upgraded to version {{< reuse "kagent-docs/versions/kagent.md" >}} of the chart, which uses the key on this page. + +5. Create a new Session. A Session keeps the runtime configuration that it was created with, so only a new Session captures content. + ```bash + kagent agent session create --agent my-first-agent + ``` + +6. Confirm that the Session runs the Agent's current revision. If the command prints `Outdated`, the Session was created from an earlier revision, and does not capture content. Create another Session, and run the command again. + ```bash + for i in $(seq 1 60); do + REVISIONS=$(kubectl get agent my-first-agent -n kagent \ + -o jsonpath='{.status.desiredRevision} {.status.latestSuccessfulRevision}') + [ "${REVISIONS% *}" = "${REVISIONS#* }" ] && break + sleep 5 + done + SESSION_REVISION=$(kagent agent session get $SESSION_ID -o json | jq -r '.session.preparedRevision') + [ "$SESSION_REVISION" = "${REVISIONS#* }" ] && echo "Current" || echo "Outdated" + ``` + +## Verify the setup + +Send a request that contains a distinctive phrase, then find that phrase in the captured request. + +{{< reuse "kagent-docs/snippets/observability/audit-verify-steps.md" >}} + +> [!CAUTION] +> Anyone who can read your tracing or logging backend can now read the prompts and replies of every agent. Restrict access to the backend, and set a retention period that meets your compliance requirements. + +## Turn off content capture + +Turn off content capture, then create a new Session so that the change takes effect. + +1. Turn off content capture in the kagent Helm release. Tracing stays on. + ```bash + helm upgrade kagent \ + {{< reuse "kagent-docs/snippets/helm-path.md" >}}/{{< reuse "kagent-docs/snippets/helm-kagent.md" >}} \ + --version {{< reuse "kagent-docs/versions/kagent.md" >}} \ + --namespace kagent --reuse-values \ + --set config.runtimeDefaults.telemetry.captureContent=false + ``` + +2. Create a new Session, because an existing Actor keeps the configuration that it started with. The spans of a Session that still captures content keep carrying it until you delete the Session. + +3. Delete the captured content from your backend when your retention policy requires it. Turning off capture does not remove the spans that your backend already stores. + +## Next steps + +{{< cards >}} + {{< card link=`{{< link path="observability/tracing" >}}` title="Tracing" subtitle="Read the spans and attributes of an agent request." >}} + {{< card link=`{{< link path="agents/agent-harness#telemetry-content-settings" >}}` title="Telemetry content settings" subtitle="Review how each content setting applies to each harness." >}} +{{< /cards >}} diff --git a/docs-site/content/kagent/main/observability/lightweight-otel-stack.md b/docs-site/content/kagent/main/observability/lightweight-otel-stack.md new file mode 100644 index 00000000..9c3984d9 --- /dev/null +++ b/docs-site/content/kagent/main/observability/lightweight-otel-stack.md @@ -0,0 +1,410 @@ +--- +title: Lightweight OTel stack +description: Install an OpenTelemetry collector with Jaeger and Prometheus, then send traces and metrics from kagent and Agent Substrate to it. +weight: 25 +author: kagent.dev +--- + +Deploy a small observability stack based on OpenTelemetry (OTel) that collects traces and metrics from both kagent and Agent Substrate. The stack includes the following components: + +- **Traces**: Distributed tracing with [Jaeger](https://github.com/jaegertracing/jaeger). +- **Metrics**: Time-series metrics collection with [Prometheus](https://github.com/prometheus/prometheus). +- **Collection**: Telemetry collection and routing with the [OpenTelemetry Collector](https://github.com/open-telemetry/opentelemetry-collector). +- **Logs**: No logging backend. The collector prints the logs that it receives to its own output. + +This stack follows the setup that Agent Substrate uses for its own local testing, and it needs less memory than the [OTel stack]({{< link path="observability/otel-stack" >}}) with Grafana. + +## About the stack + +The stack has no logging backend. The collector prints the logs that it receives to its own output, which is enough to confirm that logs arrive. To store and query logs, such as the Actor state changes that Agent Substrate records, use the [OTel stack]({{< link path="observability/otel-stack" >}}), which adds Loki. +

+ +```mermaid +flowchart LR + subgraph sources["Your cluster"] + kagent["kagent controller
and agent runtimes"] + substrate["Agent Substrate"] + end + collector["OTel collector"] + subgraph backends["Backends"] + jaeger["Jaeger"] + prom["Prometheus"] + end + kagent -->|traces, logs| collector + substrate -->|traces, logs, metrics| collector + kagent -->|metrics scrape| prom + collector --> jaeger + collector -->|metrics scrape| prom + classDef boundary fill:#a78bfa26,stroke:#a78bfa,stroke-width:2px + classDef inner fill:#80808033,stroke:#9ca3af,stroke-width:1px + class sources,backends boundary + class kagent,substrate,collector,jaeger,prom inner +``` + +For what kagent and Agent Substrate each send, see [About the stack]({{< link path="observability/otel-stack#about-the-stack" >}}) in the OTel stack guide. + +## Before you begin + +1. [Install kagent]({{< link path="setup/installation" >}}). +2. [Create your first agent]({{< link path="get-started/your-first-agent" >}}), so that you have an {{< gloss "Agent" >}}Agent{{< /gloss >}} to send requests to. That guide also installs the kagent CLI. The steps on this page need the {{< reuse "kagent-docs/versions/kagent.md" >}} CLI, because the CLIs of other releases, newer ones included, do not have the Session commands that these steps use. To check your version, run `kagent version`. +3. Install [`jq`](https://jqlang.org/download/), to read the Session ID and revision out of the CLI's JSON output. + +## Install Jaeger + +Install [Jaeger](https://www.jaegertracing.io/) in all-in-one mode, which stores traces in memory and does not need other components. + +```bash +helm upgrade --install jaeger jaeger \ + --repo https://jaegertracing.github.io/helm-charts \ + --version {{< reuse "kagent-docs/versions/jaeger.md" >}} \ + --namespace telemetry --create-namespace \ + --values - <}} \ + --namespace telemetry \ + --values - <}} \ + --namespace telemetry \ + --values - <}} + +## Send kagent telemetry to the collector + +Turn on the kagent trace, log, and metrics exporters, and point them at the collector. Also turn on the controller's metrics endpoint, and allow Prometheus to scrape it. + +1. Save the current revision of your Agent. A later step uses it to tell when kagent recompiles the Agent with the new settings. The command first waits for any recompile that is still in progress, such as one from an earlier Helm upgrade, so that it saves a finished revision. + ```bash + for i in $(seq 1 60); do + REVISIONS=$(kubectl get agent my-first-agent -n kagent \ + -o jsonpath='{.status.desiredRevision} {.status.latestSuccessfulRevision}') + [ "${REVISIONS% *}" = "${REVISIONS#* }" ] && break + sleep 5 + done + export OLD_REVISION=${REVISIONS#* } + echo "Current revision: $OLD_REVISION" + ``` + +2. Upgrade the kagent Helm release to export traces, logs, and metrics to the collector, and to serve controller metrics. + ```bash + helm upgrade kagent \ + {{< reuse "kagent-docs/snippets/helm-path.md" >}}/{{< reuse "kagent-docs/snippets/helm-kagent.md" >}} \ + --version {{< reuse "kagent-docs/versions/kagent.md" >}} \ + --namespace kagent \ + --reuse-values \ + --values - <}} of the chart, which uses the keys on this page. + +## Send Agent Substrate telemetry to the collector + +Upgrade both Agent Substrate Helm releases. A single `otel.endpoint` setting turns on traces, logs, and metrics for every Agent Substrate component. Agent Substrate keeps 1% of its traces by default, so the command keeps every trace instead. Lower `samplingRatio` again for production. + +```bash +helm upgrade substrate-atelet \ + {{< reuse "kagent-docs/snippets/helm-substrate-path.md" >}}/substrate-atelet \ + --version {{< reuse "kagent-docs/versions/agent-substrate-chart.md" >}} \ + --namespace ate-system \ + --reuse-values \ + --set otel.endpoint=http://otel-collector.telemetry.svc.cluster.local:4317 \ + --set otel.traces.samplingRatio=1.0 \ + --wait --timeout 10m +helm upgrade substrate \ + {{< reuse "kagent-docs/snippets/helm-substrate.md" >}} \ + --version {{< reuse "kagent-docs/versions/agent-substrate-chart.md" >}} \ + --namespace ate-system \ + --reuse-values \ + --set otel.endpoint=http://otel-collector.telemetry.svc.cluster.local:4317 \ + --set otel.traces.samplingRatio=1.0 \ + --wait --timeout 10m +``` + +## Send a request + +Create a Session that picks up the new telemetry settings, and send it a request to produce traces and metrics. + +1. Create a new Session. A Session keeps the runtime configuration that it was created with, so only a new Session exports traces. + ```bash + kagent agent session create --agent my-first-agent + ``` + +2. Confirm that the Session runs the Agent's current revision. The command waits until kagent finishes compiling the Agent, then compares that revision with the one that the Session started from. If the command prints `Outdated`, the Session was created from an earlier revision, and exports without the new settings. Create another Session, and run the command again. + ```bash + for i in $(seq 1 60); do + REVISIONS=$(kubectl get agent my-first-agent -n kagent \ + -o jsonpath='{.status.desiredRevision} {.status.latestSuccessfulRevision}') + [ "${REVISIONS% *}" = "${REVISIONS#* }" ] && break + sleep 5 + done + SESSION_REVISION=$(kagent agent session get $SESSION_ID -o json | jq -r '.session.preparedRevision') + [ "$SESSION_REVISION" = "${REVISIONS#* }" ] && echo "Current" || echo "Outdated" + ``` + +3. Send a request to produce telemetry. + ```bash + export SESSION_ID=$(kagent agent session create --agent my-first-agent -o json | jq -r '.session.id') + kagent agent invoke --session $SESSION_ID --task "What is 2+2?" + ``` + +## Review the telemetry + +Check each signal in turn: traces in Jaeger, metrics in Prometheus, and logs in the collector's own output. + +1. Review the traces in Jaeger. + 1. Forward the Jaeger query port, and leave the command running. + ```bash + kubectl port-forward -n telemetry svc/jaeger 16686:16686 + ``` + 2. In your browser, open Jaeger at [http://localhost:16686](http://localhost:16686). + 3. From the **Service** list, select `my-first-agent`, and click **Find Traces**. To see Agent Substrate's own work, select one of its services instead: `ateapi`, `atelet`, `atecontroller`, or `ateom-gvisor`. For what each service reports, see [Agent Substrate traces]({{< link path="observability/tracing#agent-substrate-traces" >}}). + +2. Review the metrics in Prometheus. + 1. Forward the Prometheus port, and leave the command running. + ```bash + kubectl port-forward -n telemetry svc/prometheus-server 9090:80 + ``` + 2. In your browser, open Prometheus at [http://localhost:9090/targets](http://localhost:9090/targets), and confirm that the `otel-collector` and `kagent-controller` targets are `UP`. + 3. Open the **Query** page, and run a query such as `ate_workerpool_ready_workers`. For more metrics, see [Metrics]({{< link path="observability/metrics" >}}). + +3. Check that the collector receives logs. Each line reports a batch of log records from one source. + ```bash + kubectl logs -n telemetry deploy/otel-collector | grep '"otelcol.signal": "logs"' + ``` + +## Clean up + +1. Turn off telemetry export in kagent and Agent Substrate. + ```bash + helm upgrade kagent \ + {{< reuse "kagent-docs/snippets/helm-path.md" >}}/{{< reuse "kagent-docs/snippets/helm-kagent.md" >}} \ + --version {{< reuse "kagent-docs/versions/kagent.md" >}} \ + --namespace kagent --reuse-values \ + --set config.telemetry=null \ + --set config.runtimeDefaults.telemetry=null \ + --set controller.metrics.enabled=false + helm upgrade substrate-atelet \ + {{< reuse "kagent-docs/snippets/helm-substrate-path.md" >}}/substrate-atelet \ + --version {{< reuse "kagent-docs/versions/agent-substrate-chart.md" >}} \ + --namespace ate-system --reuse-values \ + --set otel.endpoint="" \ + --set otel.traces.samplingRatio=0.01 + helm upgrade substrate \ + {{< reuse "kagent-docs/snippets/helm-substrate.md" >}} \ + --version {{< reuse "kagent-docs/versions/agent-substrate-chart.md" >}} \ + --namespace ate-system --reuse-values \ + --set otel.endpoint="" \ + --set otel.traces.samplingRatio=0.01 + ``` + +2. Remove the role binding, the collector, the backends, and the `telemetry` namespace. + ```bash + kubectl delete clusterrolebinding kagent-metrics-reader-prometheus + helm uninstall otel-collector prometheus jaeger -n telemetry + kubectl delete namespace telemetry + ``` + +## Next steps + +{{< cards >}} + {{< card link=`{{< link path="observability/tracing" >}}` title="Tracing" subtitle="Read the spans and attributes of an agent request." >}} + {{< card link=`{{< link path="observability/metrics" >}}` title="Metrics" subtitle="Review the metrics that kagent and Agent Substrate report." >}} + {{< card link=`{{< link path="observability/otel-stack" >}}` title="OTel stack" subtitle="Add Loki and Grafana to store logs and view every signal in one place." >}} +{{< /cards >}} diff --git a/docs-site/content/kagent/main/observability/metrics.md b/docs-site/content/kagent/main/observability/metrics.md new file mode 100644 index 00000000..f9368c07 --- /dev/null +++ b/docs-site/content/kagent/main/observability/metrics.md @@ -0,0 +1,145 @@ +--- +title: Metrics +description: Scrape the kagent controller's Prometheus metrics, export Agent Substrate metrics over OpenTelemetry, and review what each metric measures. +weight: 40 +author: kagent.dev +--- + +{{< reuse "kagent-docs/snippets/name-product.md" >}} and Agent Substrate report metrics in two different ways. The kagent controller serves a Prometheus `/metrics` endpoint that Prometheus scrapes. Agent runtimes and Agent Substrate push metrics over the OpenTelemetry Protocol (OTLP) to a collector, which serves them for Prometheus to scrape. Both collection paths are off by default. The [OTel stack]({{< link path="observability/otel-stack" >}}) and [Lightweight OTel stack]({{< link path="observability/lightweight-otel-stack" >}}) guides turn on both collection paths and set up Prometheus to collect them. + +## kagent controller metrics + +### Enable the metrics endpoint + +The controller serves its metrics over HTTPS, and accepts a scrape only from a ServiceAccount that holds the `-metrics-reader` ClusterRole. Enabling the endpoint creates a dedicated metrics Service and that ClusterRole. + +```yaml +controller: + metrics: + enabled: true +config: + metrics: + bindAddress: ":8443" + secure: true +``` + +These two sections divide the listener from its Kubernetes packaging. `config.metrics` sets the address that the controller binds and whether it serves over HTTPS. `controller.metrics` creates the Service, the ClusterRoles, and the optional ServiceMonitor. Setting `controller.metrics.enabled` to `false` overrides `config.metrics.bindAddress` with `0`, which disables the listener as well. Change the port through `config.metrics.bindAddress`, because the chart derives the Service `targetPort` and the pod `containerPort` from it at template time. + +Grant your Prometheus instance access in one of the following ways. + +- **With the Prometheus Operator**: Set `controller.metrics.serviceMonitor.enabled` to `true`. The chart creates a ServiceMonitor for the metrics Service. Also set `controller.metrics.serviceMonitor.prometheusServiceAccount.name` and `.namespace` to the ServiceAccount that Prometheus runs as, and the chart binds the metrics reader ClusterRole to it. The chart renders the ServiceMonitor only when the cluster serves the `monitoring.coreos.com/v1` API. For an example, see [OTel stack]({{< link path="observability/otel-stack#send-kagent-telemetry-to-the-collector" >}}). +- **Without the Prometheus Operator**: Bind the `-metrics-reader` ClusterRole to the Prometheus ServiceAccount yourself, and add a scrape job for the `-controller-metrics` Service on port `8443`. The job must use HTTPS, send the ServiceAccount token, and skip verification of the controller's self-signed certificate. For an example, see [Lightweight OTel stack]({{< link path="observability/lightweight-otel-stack#install-prometheus" >}}). + +### Metric reference + +Besides the standard Go runtime and process metrics, the controller reports the following metrics. + +| Metric | Type | Measures | +| ------ | ---- | -------- | +| `kagent_grpc_server_requests_total` | Counter | Requests to the controller's gRPC API, by `method`, `code`, and `rpc_type`. The `method` label names the full gRPC method, such as `/kagent.api.v1.SessionService/CreateSession`. | +| `kagent_grpc_server_request_duration_seconds` | Histogram | Latency of requests to the controller's gRPC API, with the same labels. | +| `controller_runtime_reconcile_total`, `controller_runtime_reconcile_errors_total`, `controller_runtime_reconcile_time_seconds` | Counter, counter, histogram | Reconciliations of kagent resources, by controller, and how long each one takes. A rising error count means that the controller cannot bring a resource to its desired state. | +| `workqueue_depth`, `workqueue_queue_duration_seconds`, `workqueue_retries_total` | Gauge, histogram, counter | Work waiting for a reconciler, how long it waits, and how often it is retried. | +| `leader_election_master_status` | Gauge | `1` on the controller replica that holds the leader lease. With several replicas, exactly one reports `1`. | +| `rest_client_requests_total` | Counter | Requests from the controller to the Kubernetes API server, by status code. | +| `kagent_runtime_revision_gc_pending` | Gauge | Runtime revisions that the last successful discovery found eligible for cleanup. The controller reports a cached count, so a scrape does not make a database or network call. The metric is absent until a discovery succeeds, and `0` reports a discovery that matched nothing. A failed discovery holds the previous value rather than clearing it. | +| `kagent_runtime_revision_gc_duration_seconds` | Histogram | Time that one revision cleanup attempt takes, by the `kagent_gc_stage` label: `discovery` or `collection`. A `collection` attempt covers the database claim, the Agent Substrate read and deletion, and the database finalization. A failed attempt carries an `error_type` label holding a gRPC status code name, or `_OTHER` for every other failure, so the series that carry the label give both the failure rate and the reason. | + +The Agent Substrate `atecontroller` component reports the same `controller_runtime_*` and `workqueue_*` metrics for its own reconcilers. To keep the two apart in a query, filter by the scrape job, such as `job="kagent-controller-metrics"` in the OTel stack. + +## Agent runtime metrics + +When the kagent Helm release sets an endpoint in `config.runtimeDefaults.telemetry.metrics`, agent runtimes export metrics over OTLP. A runtime reports the histogram in this section only when kagent's own A2A wrapper opens the `invoke_agent` span for it. For the runtimes that open that span themselves, both stack guides derive the metric from the spans instead. The stack guides convert these OpenTelemetry metrics to Prometheus names in the collector. + +| Metric | Type | Measures | +| ------ | ---- | -------- | +| `gen_ai_invoke_agent_duration_seconds` | Histogram | Time that one `invoke_agent` execution segment takes. The `claude`, `codex`, `langgraph`, `crewai`, and `byo` runtimes record it themselves, because kagent's wrapper opens their `invoke_agent` span and measures it. The `kagent`, `adk-python`, and `openai-agents` runtimes open that span in their own instrumentation, so the wrapper stays a transport span and records no duration. Both stack guides add a `signal_to_metrics` connector that derives the metric from those spans before any sampling in the collector. Between the two paths, every runtime reports the metric, except a `byo` image that is not built on `kagent-core` and opens no `invoke_agent` span of its own. | +| `gen_ai_client_inference_duration_seconds` | Histogram | Time that one `chat` span takes on the `codex` or `claude` runtime. Codex reports no call times, so those durations are estimates. A zero-length call records no duration. | +| `gen_ai_client_inference_operation_input_tokens` and `gen_ai_client_inference_operation_output_tokens` | Histogram | Input and output tokens for one `codex` or `claude` model call. A partial call records input tokens, but not output tokens, because its output count is a lower bound. | +| `gen_ai_client_inference_usage_input_tokens_total`, `gen_ai_client_inference_usage_output_tokens_total`, `gen_ai_client_inference_usage_cache_read_input_tokens_total`, `gen_ai_client_inference_usage_cache_write_input_tokens_total`, and `gen_ai_client_inference_usage_reasoning_output_tokens_total` | Counter | Token totals from `codex` and `claude` model calls. At the end of a Claude turn, the runtime also adds the tokens that Claude's own total reports but no `chat` span records, such as most of a subagent's output. | +| `gen_ai_execute_tool_duration_seconds` | Histogram | Time that one `execute_tool` span takes on the `codex` or `claude` runtime. A declined tool call has no span, so it records no duration. | +| `gen_ai_invoke_agent_inference_calls` and `gen_ai_invoke_agent_tool_calls` | Histogram | Counts of the agent's own model calls and tool calls in one `codex` or `claude` execution segment. The counts exclude subagent calls. | + +The metric carries labels for the Agent name, model, and error type when those values exist. When the collector copies resource attributes to labels, the metric also carries labels such as `service_name`, `service_namespace`, `service_version`, `kagent_runtime`, and `gen_ai_main_agent_id`. + +## Agent Substrate metrics + +### Export the metrics + +Agent Substrate exports metrics over OTLP when both of its Helm releases have an OTLP endpoint. The same setting also turns on Agent Substrate traces and logs. + +```bash +helm upgrade substrate-atelet \ + {{< reuse "kagent-docs/snippets/helm-substrate-path.md" >}}/substrate-atelet \ + --version {{< reuse "kagent-docs/versions/agent-substrate-chart.md" >}} \ + --namespace ate-system \ + --reuse-values \ + --set otel.endpoint=http://otel-collector.telemetry.svc.cluster.local:4317 +helm upgrade substrate \ + {{< reuse "kagent-docs/snippets/helm-substrate.md" >}} \ + --version {{< reuse "kagent-docs/versions/agent-substrate-chart.md" >}} \ + --namespace ate-system \ + --reuse-values \ + --set otel.endpoint=http://otel-collector.telemetry.svc.cluster.local:4317 +``` + +To send metrics somewhere other than traces and logs, set `otel.metrics.endpoint` instead. To stop exporting metrics while keeping the other signals, leave `otel.endpoint` empty, and set `otel.traces.endpoint` and `otel.logs.endpoint` instead. Setting `otel.metrics.enabled` to `false` is not enough in version {{< reuse "kagent-docs/versions/agent-substrate.md" >}}, because only the router's agentgateway reads it, and the other Agent Substrate components keep exporting. Agent Substrate exports a batch of metrics every 60 seconds, so a new installation shows its first values about a minute after you set the endpoint. + +### Metric reference + +The collector's Prometheus exporter converts each OpenTelemetry metric name to a Prometheus name. The exporter replaces dots with underscores, and adds a unit suffix, such as `_seconds` or `_bytes`, and a `_total` suffix for counters. The following table lists the Prometheus names. Every metric also carries a `service_name` label for the Agent Substrate component that reported it, when the collector copies resource attributes to labels, as both stack guides configure. + +| Metric | Reported by | Type | Measures | +| ------ | ----------- | ---- | -------- | +| `ate_workerpool_desired_workers` | `atecontroller` | Gauge | Worker pods that a WorkerPool requests, from its `spec.replicas`. | +| `ate_workerpool_ready_workers` | `atecontroller` | Gauge | Worker pods that are ready in a WorkerPool. A value below `ate_workerpool_desired_workers` means that the pool cannot run as many Actors as it should. | +| `ate_workerpool_workers` | `ateapi` | Gauge | Live Workers per pool, split by the `ate_worker_state` label into `idle` and `assigned`. Few idle Workers means that new and resumed Actors are about to wait for capacity. | +| `ate_scheduler_assignment_duration_seconds` | `ateapi` | Histogram | Time to assign an Actor to a Worker, by the `ate_scheduler_outcome` label: `assigned`, `no_free_worker`, or `error`. | +| `ate_scheduler_eligible_workers` | `ateapi` | Histogram | Unassigned Workers that could take an Actor at scheduling time. | +| `ate_actor_lifecycle_operation_duration_seconds` | `ateapi` | Histogram | Time that each Actor operation takes, by the `ate_actor_operation_name` label, such as `create`, `resume`, and `suspend`. A failed operation carries an `error_type` label. | +| `ate_actor_restore_duration_seconds` | `atelet` | Histogram | Time for each phase of restoring an Actor from a snapshot on its Worker's node, by the `ate_snapshot_phase` label, such as `download`, `oci_unpack`, and `ateom_restore`. This is where most of the time to resume an Actor goes. | +| `ate_actor_checkpoint_duration_seconds` | `atelet` | Histogram | Time for each phase of writing a snapshot when an Actor suspends. | +| `atelet_snapshot_size_bytes` | `atelet` | Histogram | Uncompressed size of each snapshot. | +| `ate_imagecache_requests_total` | `atelet` | Counter | Lookups in the node-local image cache, by the `ate_imagecache_outcome` label. A cache miss pays for pulling and unpacking the image, so a low hit ratio predicts slow resumes. | +| `ate_actor_stats_cpu_time_seconds_total` | `atelet` | Counter | CPU time of the Actors that run on a node, by template. Use `rate()` to find the templates that use a node's CPU. | +| `ate_actor_stats_memory_usage_bytes` | `atelet` | Gauge | Memory of the Actors that run on a node, by template, including page cache that the node can reclaim. | +| `ate_actor_stats_memory_working_set_bytes` | `atelet` | Gauge | Working set memory of the Actors that run on a node, by template, without reclaimable page cache. Compare this value with a memory limit. | +| `ate_actor_stats_sampled_actors` | `atelet` | Gauge | Actors on a node that have a current resource measurement. Divide the other `ate_actor_stats_*` metrics by this value to get a per-Actor average for a template. | +| `ate_actor_crashes_total` | `ateapi` | Counter | Actors that moved to the terminal crashed state, by failure reason. The metric appears only after the first crash. | +| `rpc_server_call_duration_seconds`, `rpc_client_call_duration_seconds` | `ateapi`, `atelet` | Histogram | Latency, rate, and errors of the gRPC calls between Agent Substrate components. | + +## Example queries + +Run the following queries in Prometheus or Grafana. + +- Ready Workers compared with desired Workers, per WorkerPool. + ```text + ate_workerpool_ready_workers / ate_workerpool_desired_workers + ``` + +- The 95th percentile time to resume an Actor, per phase. + ```text + histogram_quantile(0.95, sum by (le, ate_snapshot_phase) (rate(ate_actor_restore_duration_seconds_bucket[5m]))) + ``` + +- The rate of requests to the kagent gRPC API, by method. + ```text + sum by (method) (rate(kagent_grpc_server_requests_total[5m])) + ``` + +- The rate of failed reconciliations in the kagent controller, by reconciler. The query uses the job name from the OTel stack. In the lightweight stack, the job is `kagent-controller`. + ```text + sum by (controller) (rate(controller_runtime_reconcile_errors_total{job="kagent-controller-metrics"}[5m])) + ``` + +- The 95th percentile agent invocation duration, per Agent. Because both stack guides configure the collector to derive the metric, the query covers every runtime that opens an `invoke_agent` span. + ```text + histogram_quantile(0.95, sum by (le, gen_ai_agent_name) (rate(gen_ai_invoke_agent_duration_seconds_bucket[5m]))) + ``` + +## Next steps + +{{< cards >}} + {{< card link=`{{< link path="observability/otel-stack" >}}` title="OTel stack" subtitle="Collect traces, logs, and metrics in Grafana." >}} + {{< card link=`{{< link path="operations/tune-agent-substrate" >}}` title="Tune Agent Substrate" subtitle="Size WorkerPools and adjust the runtime settings that these metrics measure." >}} +{{< /cards >}} diff --git a/docs-site/content/kagent/main/observability/otel-stack.md b/docs-site/content/kagent/main/observability/otel-stack.md new file mode 100644 index 00000000..255a3db6 --- /dev/null +++ b/docs-site/content/kagent/main/observability/otel-stack.md @@ -0,0 +1,590 @@ +--- +title: OTel stack +description: Install an OpenTelemetry collector with Grafana Tempo, Loki, Prometheus, and Grafana, then send traces, logs, and metrics from kagent and Agent Substrate to it. +weight: 20 +author: kagent.dev +--- + +Deploy an open source observability stack based on OpenTelemetry (OTel) that collects telemetry from both kagent and Agent Substrate. The stack includes the following components: + +- **Logs**: Log collection and storage with Grafana [Loki](https://github.com/grafana/loki). +- **Traces**: Distributed tracing with Grafana [Tempo](https://github.com/grafana/tempo). +- **Metrics**: Time-series metrics collection with [Prometheus](https://github.com/prometheus/prometheus). +- **Collection**: Telemetry collection and routing with the [OpenTelemetry Collector](https://github.com/open-telemetry/opentelemetry-collector). +- **Visualization**: Dashboards and queries across all three signals with [Grafana](https://github.com/grafana/grafana). + +For a smaller stack with Jaeger and Prometheus only, see [Lightweight OTel stack]({{< link path="observability/lightweight-otel-stack" >}}). + +## About the stack + +{{< reuse "kagent-docs/snippets/name-product.md" >}} and Agent Substrate each export their own telemetry over the OpenTelemetry Protocol (OTLP), and neither exports anything by default. The collector gives both a single address to send to, and holds the rules for where each signal goes next. The kagent controller also serves Prometheus metrics, which Prometheus scrapes directly. +

+ +```mermaid +flowchart LR + subgraph sources["Your cluster"] + kagent["kagent controller
and agent runtimes"] + substrate["Agent Substrate"] + end + collector["OTel collector"] + subgraph backends["Backends"] + tempo["Tempo"] + loki["Loki"] + prom["Prometheus"] + end + grafana["Grafana"] + kagent -->|traces, logs, runtime metrics| collector + substrate -->|traces, logs, metrics| collector + kagent -->|metrics scrape| prom + collector --> tempo + collector --> loki + collector -->|metrics scrape| prom + tempo --> grafana + loki --> grafana + prom --> grafana + classDef boundary fill:#a78bfa26,stroke:#a78bfa,stroke-width:2px + classDef inner fill:#80808033,stroke:#9ca3af,stroke-width:1px + class sources,backends boundary + class kagent,substrate,collector,tempo,loki,prom,grafana inner +``` + +The following table lists what each source sends. For the meaning of each signal, see [Tracing]({{< link path="observability/tracing" >}}) and [Metrics]({{< link path="observability/metrics" >}}). + +| Source | Traces | Logs | Metrics | +| ------ | ------ | ---- | ------- | +| kagent controller and agent runtimes | One trace per agent request, across the controller, the Agent Substrate proxy, and the agent runtime. | Log records from the agent runtimes. Prompts and replies are not among them on the `kagent` runtime, which puts them on spans instead. For how to capture them, see [Audit prompts]({{< link path="observability/audit-prompts" >}}). | Agent runtimes export runtime metrics over OTLP. Prometheus scrapes reconciliation, work queue, and gRPC API metrics from the controller's `/metrics` endpoint. | +| Agent Substrate | Separate traces for its own work, such as routing a request to a Worker and resuming an Actor. | A record of every Actor state change, and the router access log. | Actor lifecycle, scheduling, snapshot, image cache, and WorkerPool capacity metrics. | + +Agent Substrate traces do not join the trace of the agent request that caused them. To see why a request was slow to start, look up the Agent Substrate trace from the same time window. + +## Before you begin + +1. [Install kagent]({{< link path="setup/installation" >}}). +2. [Create your first agent]({{< link path="get-started/your-first-agent" >}}), so that you have an {{< gloss "Agent" >}}Agent{{< /gloss >}} to send requests to. That guide also installs the kagent CLI. The steps on this page need the {{< reuse "kagent-docs/versions/kagent.md" >}} CLI, because the CLIs of other releases, newer ones included, do not have the Session commands that these steps use. To check your version, run `kagent version`. +3. Install [`jq`](https://jqlang.org/download/), to read the Session ID and revision out of the CLI's JSON output. +4. Make sure that your cluster has about 1.5 GB of memory available in addition to kagent. On a kind cluster, the memory limit is the memory that you give Docker. + +## Install Tempo and Loki + +Install the backends that store traces and logs. Both run as a single replica, which suits evaluation. Tempo keeps its traces in memory, and Loki keeps its logs on a small persistent volume, so your cluster needs a default StorageClass, which a kind cluster has. For production, follow the Grafana guidance for [Tempo](https://grafana.com/docs/tempo/latest/set-up-for-tracing/setup-tempo/deploy/kubernetes/helm-chart/) and [Loki](https://grafana.com/docs/loki/latest/setup/install/helm/). + +1. Install Tempo, with an OTLP receiver for traces. + ```bash + helm upgrade --install tempo tempo \ + --repo https://grafana.github.io/helm-charts \ + --version {{< reuse "kagent-docs/versions/tempo.md" >}} \ + --namespace telemetry --create-namespace \ + --values - <}} \ + --namespace telemetry \ + --values - <}} \ + --namespace telemetry \ + --values - <}} + + | Setting | Description | + | ------- | ----------- | + | `serviceMonitorSelectorNilUsesHelmValues` | Set to `false` so that Prometheus scrapes every ServiceMonitor in the cluster, including the ones that the kagent chart and this guide create. By default, Prometheus scrapes only the ServiceMonitors that carry the chart's release label. | + | `additionalDataSources` | Adds Tempo and Loki to Grafana. Tempo serves its query API on port `3100` in this chart version. The chart adds Prometheus as a data source on its own. | + +2. Verify that Prometheus and Grafana are running. + ```bash + kubectl get pods -n telemetry -l 'app.kubernetes.io/name in (prometheus,grafana)' + ``` + Example output: + ```console + NAME READY STATUS RESTARTS AGE + kube-prometheus-stack-grafana-7599d6f796-lb4l5 3/3 Running 0 2m + prometheus-kube-prometheus-stack-prometheus-0 2/2 Running 0 2m + ``` + +## Install the OTel collector + +Install a collector that receives OTLP on ports `4317` for gRPC and `4318` for HTTP, and forwards each signal to its backend. The collector sends traces to Tempo and logs to Loki, and serves metrics on port `8889` for Prometheus to scrape. The configuration uses the `contrib` collector image, because the Prometheus exporter is not part of the core image. + +1. Install the collector. + ```bash + helm upgrade --install otel-collector opentelemetry-collector \ + --repo https://open-telemetry.github.io/opentelemetry-helm-charts \ + --version {{< reuse "kagent-docs/versions/otel-collector.md" >}} \ + --namespace telemetry \ + --values - <}} + +2. Create a ServiceMonitor so that Prometheus scrapes the metrics that the collector serves. + ```yaml + kubectl apply -f - <}}/{{< reuse "kagent-docs/snippets/helm-kagent.md" >}} \ + --version {{< reuse "kagent-docs/versions/kagent.md" >}} \ + --namespace kagent \ + --reuse-values \ + --values - <}} + + | Setting | Description | + | ------- | ----------- | + | `config.telemetry` | The endpoint and protocol that the controller's own signals export to. An `http://` endpoint sends plaintext. For each field, see [Tracing]({{< link path="observability/tracing#enable-tracing" >}}). | + | `config.runtimeDefaults.telemetry` | The same set for every agent runtime that the controller starts. The controller's own settings do not propagate to a runtime, so both blocks name the collector. | + | `telemetry.traces` | Exports traces. A signal exports when you set its endpoint, and stays off when you omit the endpoint, so neither block takes an `enabled` field. | + | `telemetry.logs` | Exports log records. The `kagent` runtime never puts prompts or replies in a log record. For where that content goes, see [Audit prompts]({{< link path="observability/audit-prompts" >}}). | + | `telemetry.metrics` | Exports runtime metrics, such as `gen_ai.invoke_agent.duration`. | + | `controller.metrics.enabled` | Serves the controller's Prometheus metrics on port `8443` over HTTPS. | + | `controller.metrics.serviceMonitor` | Creates a ServiceMonitor for the metrics endpoint. The `prometheusServiceAccount` setting binds the metrics reader role to the Prometheus ServiceAccount, which authorizes the scrape. | + +3. Wait for the controller to roll out. + ```bash + kubectl rollout status deployment/kagent-controller -n kagent --timeout=300s + ``` + +4. Wait for kagent to recompile the Agent. The controller rebuilds each Agent after the controller restarts. A Session that you create before the rebuild finishes starts from the previous revision, without the new settings. The following command prints `Recompiled` when the new revision is ready. + ```bash + for i in $(seq 1 60); do + [ "$(kubectl get agent my-first-agent -n kagent \ + -o jsonpath='{.status.latestSuccessfulRevision}')" != "$OLD_REVISION" ] \ + && echo "Recompiled" && break + sleep 5 + done + ``` + If the command finishes without printing `Recompiled`, the upgrade did not change the settings that kagent compiles into the Agent. Either the settings were already in place, or the chart did not recognize the `otel` keys. Helm accepts a key that a chart does not define without an error, so check that you upgraded to version {{< reuse "kagent-docs/versions/kagent.md" >}} of the chart, which uses the keys on this page. + +## Send Agent Substrate telemetry to the collector + +Point Agent Substrate at the collector. A single `otel.endpoint` setting turns on traces, logs, and metrics for every Agent Substrate component. + +1. Upgrade both Agent Substrate Helm releases. + ```bash + helm upgrade substrate-atelet \ + {{< reuse "kagent-docs/snippets/helm-substrate-path.md" >}}/substrate-atelet \ + --version {{< reuse "kagent-docs/versions/agent-substrate-chart.md" >}} \ + --namespace ate-system \ + --reuse-values \ + --set otel.endpoint=http://otel-collector.telemetry.svc.cluster.local:4317 \ + --set otel.traces.samplingRatio=1.0 \ + --wait --timeout 10m + helm upgrade substrate \ + {{< reuse "kagent-docs/snippets/helm-substrate.md" >}} \ + --version {{< reuse "kagent-docs/versions/agent-substrate-chart.md" >}} \ + --namespace ate-system \ + --reuse-values \ + --set otel.endpoint=http://otel-collector.telemetry.svc.cluster.local:4317 \ + --set otel.traces.samplingRatio=1.0 \ + --wait --timeout 10m + ``` + + Agent Substrate keeps 1% of its traces by default, so a few test requests rarely produce one. A `samplingRatio` of `1.0` keeps every trace so that you can see results right away. Lower it again for production, because the router then records a trace for every request that it forwards. + +2. Verify that the Agent Substrate pods are running. + ```bash + kubectl get pods -n ate-system + ``` + +## Send a request + +Create a Session that picks up the new telemetry settings, and send it a few requests to produce traces, logs, and metrics. + +1. Create a new Session. A Session keeps the runtime configuration that it was created with, so only a new Session exports telemetry. + ```bash + kagent agent session create --agent my-first-agent + ``` + +2. Confirm that the Session runs the Agent's current revision. The command waits until kagent finishes compiling the Agent, then compares that revision with the one that the Session started from. If the command prints `Outdated`, the Session was created from an earlier revision, and exports without the new settings. Create another Session, and run the command again. + ```bash + for i in $(seq 1 60); do + REVISIONS=$(kubectl get agent my-first-agent -n kagent \ + -o jsonpath='{.status.desiredRevision} {.status.latestSuccessfulRevision}') + [ "${REVISIONS% *}" = "${REVISIONS#* }" ] && break + sleep 5 + done + SESSION_REVISION=$(kagent agent session get $SESSION_ID -o json | jq -r '.session.preparedRevision') + [ "$SESSION_REVISION" = "${REVISIONS#* }" ] && echo "Current" || echo "Outdated" + ``` + +3. Send a few requests to produce telemetry. + ```bash + export SESSION_ID=$(kagent agent session create --agent my-first-agent -o json | jq -r '.session.id') + kagent agent invoke --session $SESSION_ID --task "What is 2+2?" + kagent agent invoke --session $SESSION_ID --task "What did I just ask you?" + ``` + +## Explore the telemetry in Grafana + +Log in to Grafana, and query each backend from the **Explore** view. + +1. Get the Grafana password for the `admin` user. + ```bash + kubectl get secret -n telemetry kube-prometheus-stack-grafana \ + -o jsonpath='{.data.admin-password}' | base64 --decode; echo + ``` + +2. Forward the Grafana port, and leave the command running. + ```bash + kubectl port-forward -n telemetry svc/kube-prometheus-stack-grafana 3000:80 + ``` + +3. In your browser, open Grafana at [http://localhost:3000](http://localhost:3000), and log in as `admin` with the password from the first step. + +4. Open **Explore**, and review each signal. + {{< tabs >}} + {{% tab name="Traces" %}} + Select the **Tempo** data source, then select the **Search** query type. From the **Service Name** list, select `my-first-agent`, the service that the Agent reports as, and run the query. Open a trace to see the controller, proxy, and agent runtime spans of one request. + + To see Agent Substrate's own work, select one of its services from the **Service Name** list instead: `ateapi`, `atelet`, `atecontroller`, or `ateom-gvisor`. For what each service reports, see [Agent Substrate traces]({{< link path="observability/tracing#agent-substrate-traces" >}}). + {{% /tab %}} + {{% tab name="Logs" %}} + Select the **Loki** data source, and run the following query to show the state changes of your agent's Actor. Agent Substrate names the Actor `session-` followed by the Session ID, so replace `` with the value of `$SESSION_ID` from the previous section. + ```text + {service_name="ateapi"} | ate_actor_name="session-" + ``` + + Each record carries the new state of the Actor, such as `resuming`, `running`, or `suspended`. For the meaning of each record, see [Actor state changes]({{< link path="observability/substrate-telemetry#actor-state-changes" >}}). + + To see the router access log, with an entry for every request that the router forwards, query `{service_name="agentgateway"}` instead. + + Loki does not hold the prompts and replies of the `kagent` runtime, because that runtime puts them on spans rather than in log records. To audit them, see [Audit prompts]({{< link path="observability/audit-prompts" >}}). + {{% /tab %}} + {{% tab name="Metrics" %}} + Select the **Prometheus** data source, and run a query. For example, the following query returns the number of ready Workers in each WorkerPool. + ```text + ate_workerpool_ready_workers + ``` + + The following query returns the rate of gRPC requests that the kagent controller serves, by method. + ```text + sum by (method) (rate(kagent_grpc_server_requests_total[5m])) + ``` + + For more metrics, see [Metrics]({{< link path="observability/metrics" >}}). + {{% /tab %}} + {{< /tabs >}} + +## Clean up + +1. Turn off telemetry export in kagent and Agent Substrate. + ```bash + helm upgrade kagent \ + {{< reuse "kagent-docs/snippets/helm-path.md" >}}/{{< reuse "kagent-docs/snippets/helm-kagent.md" >}} \ + --version {{< reuse "kagent-docs/versions/kagent.md" >}} \ + --namespace kagent --reuse-values \ + --set config.telemetry=null \ + --set config.runtimeDefaults.telemetry=null \ + --set controller.metrics.enabled=false \ + --set controller.metrics.serviceMonitor.enabled=false + helm upgrade substrate-atelet \ + {{< reuse "kagent-docs/snippets/helm-substrate-path.md" >}}/substrate-atelet \ + --version {{< reuse "kagent-docs/versions/agent-substrate-chart.md" >}} \ + --namespace ate-system --reuse-values \ + --set otel.endpoint="" \ + --set otel.traces.samplingRatio=0.01 + helm upgrade substrate \ + {{< reuse "kagent-docs/snippets/helm-substrate.md" >}} \ + --version {{< reuse "kagent-docs/versions/agent-substrate-chart.md" >}} \ + --namespace ate-system --reuse-values \ + --set otel.endpoint="" \ + --set otel.traces.samplingRatio=0.01 + ``` + +2. Remove the collector, the backends, and the `telemetry` namespace. + ```bash + kubectl delete servicemonitor otel-collector -n telemetry + helm uninstall otel-collector kube-prometheus-stack loki tempo -n telemetry + kubectl delete namespace telemetry + ``` + +## Next steps + +{{< cards >}} + {{< card link=`{{< link path="observability/tracing" >}}` title="Tracing" subtitle="Read the spans and attributes of an agent request." >}} + {{< card link=`{{< link path="observability/metrics" >}}` title="Metrics" subtitle="Review the metrics that kagent and Agent Substrate report." >}} +{{< /cards >}} diff --git a/docs-site/content/kagent/main/observability/substrate-telemetry.md b/docs-site/content/kagent/main/observability/substrate-telemetry.md new file mode 100644 index 00000000..75d678e6 --- /dev/null +++ b/docs-site/content/kagent/main/observability/substrate-telemetry.md @@ -0,0 +1,192 @@ +--- +title: Agent Substrate telemetry +description: Follow an agent's Actor across suspend and resume by using the Actor identity that Agent Substrate adds to its logs, traces, and metrics. +weight: 45 +author: kagent.dev +--- + +Every {{< reuse "kagent-docs/snippets/name-product.md" >}} agent runs as an {{< gloss "Actor" >}}Actor{{< /gloss >}} on Agent Substrate. An Actor suspends after each turn, and can resume on a different {{< gloss "Worker" >}}Worker{{< /gloss >}} pod the next time, so telemetry that is keyed to a pod loses track of an agent between turns. To keep one agent's telemetry together, Agent Substrate labels its telemetry with the identity of the Actor rather than the pod that ran it. + +## What Agent Substrate reports + +Agent Substrate reports telemetry from its own components, separately from the telemetry that kagent reports for each agent request. + +| Signal | Where it goes | Details | +| ------ | ------------- | ------- | +| Actor logs | The stdout of the Worker pod that runs the Actor | The Actor's own output, and records for each suspend and resume. See [Read an Actor's logs](#read-an-actors-logs). | +| Actor events | Over OTLP when an OTLP endpoint is set. Otherwise, the stdout of the component that records them. | An event for every Actor state change, and samples of each Actor's CPU and memory use. See [Actor state changes](#actor-state-changes) and [Actor usage samples](#actor-usage-samples). | +| Component logs | The stdout of each component | Logs from the Agent Substrate components, including an access log entry for every request that the router receives. The router also exports its access log over OTLP when an OTLP endpoint is set. | +| Traces | Over OTLP, when an OTLP endpoint is set | Scheduling, resume, and suspend work, in traces that are separate from the agent request trace. The router's `agentgateway` spans are the exception, and join the agent request trace. See [Agent Substrate traces]({{< link path="observability/tracing#agent-substrate-traces" >}}). | +| Metrics | Over OTLP, when an OTLP endpoint is set | Worker capacity, scheduling, and snapshot timing. See [Agent Substrate metrics]({{< link path="observability/metrics#agent-substrate-metrics" >}}). | + +The Agent Substrate Helm chart sets one OTLP endpoint for traces, metrics, and logs through `otel.endpoint`, and samples traces at a ratio of `0.01` by default through `otel.traces.samplingRatio`. {{< reuse "kagent-docs/snippets/observability/substrate-otel-setup.md" >}} + +The router access log and the Actor events are the Agent Substrate logs over OTLP, and the `otel.logs` settings control them together. To send them to a different backend than traces and metrics, set `otel.logs.endpoint`. To stop exporting them, set `otel.logs.enabled` to `false`. The chart has no setting for the access log alone. + +When the Agent Substrate release sets an OTLP endpoint, the component that records an Actor event exports the event and does not write it to stdout. Read the events from your logging backend. + +## Actor identity + +Agent Substrate adds the following labels to Actor logs and to its component logs about an Actor. Each label maps to a kagent object, so you can find an agent's telemetry from the Session or the Agent that you already know. + +| Label | Value for a kagent agent | +| ----- | ------------------------ | +| `ate.actor.name` | `session-`, one Actor for each {{< gloss "Session" >}}Session{{< /gloss >}}. | +| `ate.actor.uid` | A unique ID for the lifetime of the Actor. A deleted and recreated Actor gets a new one. | +| `ate.atespace` | The {{< gloss "Atespace" >}}atespace{{< /gloss >}} of the Actor, which is the namespace of the Agent, such as `kagent`. The Actors that build an Agent's first snapshot run in `ate-golden` instead. | +| `ate.template.name` | The {{< gloss "Agent" >}}Agent{{< /gloss >}} name and the short form of the {{< gloss "Revision" >}}revision{{< /gloss >}}, joined by a hyphen, such as `my-first-agent-a995d20d30a7`. Every Session of one Agent at one revision shares this value. | +| `ate.template.atespace` | Where the template lives, which matches `ate.atespace` for a kagent agent. | +| `ate.actor.container.name` | The container that wrote the line, such as `kagent`. Records for a suspend or resume omit this label, because the Actor writes them rather than a container. | + +Agent Substrate metrics carry template and WorkerPool labels rather than the Actor identity, so that the number of time series does not grow with the number of conversations. To look into one agent, use its logs. + +## Before you begin + +1. [Install kagent]({{< link path="setup/installation" >}}), including the `kubectl-ate` plugin that the installation guide describes. +2. [Create your first agent]({{< link path="get-started/your-first-agent" >}}) and send it at least one message. This setup gives the `my-first-agent` Agent an Actor with logs to read. The guide also installs the kagent CLI. +3. Install [`jq`](https://jqlang.org/download/), to read the Session ID and filter the JSON log records. + +## Read an Actor's logs + +An Actor writes to the stdout of whichever Worker pod runs it, and a suspended Actor is assigned to no Worker at all. Find the Actor by its Session ID, then read the Worker pods of its WorkerPool and filter by the Actor's name. + +1. Save the ID of the Session that you want to follow. Use an existing Session ID in place of this command when you already have one. + ```bash + export SESSION_ID=$(kagent agent session create --agent my-first-agent -o json | jq -r '.session.id') + echo $SESSION_ID + ``` + + A value of `null` means that the Session was not created. The CLI lists only the Sessions that you created, so confirm that you are using the identity that created the agent. + +2. Confirm that the Actor exists, and check its state. The Actor name is `session-` followed by the Session ID. + ```bash + kubectl ate get actors session-$SESSION_ID --atespace kagent + ``` + + Example output: + ```console + ATESPACE NAME TEMPLATE STATE WORKER POD WORKER IPS VERSION AGE + kagent session-01a0d409-d249-728e-865c-ee58b55dff04 kagent/my-first-agent-a995d20d30a7 ACTOR_STATE_SUSPENDED 5 32m + ``` + + `ACTOR_STATE_SUSPENDED` with a `WORKER POD` of `` is the resting state between turns. Because a suspended Actor names no Worker, the remaining steps read every Worker in the pool rather than one pod. + +3. Find the WorkerPool that the Harness runs on. + ```bash + export WORKER_POOL=$(kubectl get harness my-first-harness -n kagent \ + -o jsonpath='{.spec.substrate.workerPoolRef.name}') + echo $WORKER_POOL + ``` + + Example output: + ```console + kagent-default + ``` + +4. Read the Actor's lines from the Workers in the pool. A Worker writes the output of every Actor that it runs, so the filter selects a single Actor by name. Worker output mixes JSON records with plain text, so the filter also drops any line that is not a JSON object. + ```bash + kubectl logs -n kagent -l ate.dev/worker-pool=$WORKER_POOL --tail=-1 \ + | jq -cR --arg actor "session-$SESSION_ID" \ + 'fromjson? | objects | select(.labels["ate.actor.name"] == $actor)' + ``` + + Example output: + ```console + {"labels":{"ate.actor.name":"session-01a0d409-d249-728e-865c-ee58b55dff04","ate.actor.uid":"c91579a8-675c-474c-9437-4482ac628774","ate.atespace":"kagent","ate.template.atespace":"kagent","ate.template.name":"my-first-agent-a995d20d30a7"},"message":"Actor restoring","span_id":"d81bb304af04c945","time":"2026-09-24T16:02:52.008842377Z","trace_flags":"00","trace_id":"3b8987f98057610937b38639136d32b5"} + {"labels":{"ate.actor.name":"session-01a0d409-d249-728e-865c-ee58b55dff04","ate.actor.uid":"c91579a8-675c-474c-9437-4482ac628774","ate.atespace":"kagent","ate.template.atespace":"kagent","ate.template.name":"my-first-agent-a995d20d30a7"},"message":"Actor checkpointed","span_id":"4914db77fcc2f59b","time":"2026-09-24T15:30:09.671009219Z","trace_flags":"01","trace_id":"c4971ca2e614428819443f64fe70f81c"} + ``` + + The filter returns the Actor's lines from every Worker that currently runs in the pool, including the lines from before the Actor last suspended. + +`kubectl logs` reads only the Worker pods that exist now. When a Worker pod is replaced, the output that it held is gone. To keep an Actor's history across Worker replacements, collect the stdout of the Worker pods with a log agent that stores structured JSON, and query it by the same labels. The OTel stack and Lightweight OTel stack guides configure their collectors with only the OTLP receiver. Neither collector reads pod stdout. + +Agent Substrate can also stream a single Actor's output directly, without the pool-wide filter. + +```bash +kubectl ate logs actor session-$SESSION_ID --atespace kagent +``` + +The command reads the Worker that the Actor runs on, so it works only while the Actor is running. A kagent agent runs only during a turn, and the command returns an error the rest of the time. + +```console +Error: actor kagent/session-01a0d409-d249-728e-865c-ee58b55dff04 is not currently running on any worker pod +``` + +## Suspend and resume records + +The records that Agent Substrate writes for an Actor mark each step of a suspend and resume cycle. Each record carries the Actor labels and a `message` field, and no container label. To list only these records for an Actor, filter out the lines that have a container label. + +```bash +kubectl logs -n kagent -l ate.dev/worker-pool=$WORKER_POOL --tail=-1 \ + | jq -cR --arg actor "session-$SESSION_ID" \ + 'fromjson? | objects | select(.labels["ate.actor.name"] == $actor and .labels["ate.actor.container.name"] == null) | {time, message}' +``` + +Example output: +```console +{"time":"2026-09-24T15:30:08.965994177Z","message":"Actor restoring"} +{"time":"2026-09-24T15:30:09.04801126Z","message":"Actor restored"} +{"time":"2026-09-24T15:30:09.582808844Z","message":"Actor checkpointing"} +{"time":"2026-09-24T15:30:09.671009219Z","message":"Actor checkpointed"} +{"time":"2026-09-24T16:02:52.008842377Z","message":"Actor restoring"} +{"time":"2026-09-24T16:02:52.093494544Z","message":"Actor restored"} +{"time":"2026-09-24T16:02:52.481758336Z","message":"Actor checkpointing"} +{"time":"2026-09-24T16:02:52.588873294Z","message":"Actor checkpointed"} +``` + +| `message` | Written when | +| --------- | ------------ | +| `Actor restoring` | A Worker begins to restore the Actor from a snapshot. A Session's Actor restores even for its first turn, from the snapshot that kagent built for the Agent, so `Actor restoring` is the first record for every Session. | +| `Actor restored` | The restore finishes and the Actor can serve the request. | +| `Actor checkpointing` | The Actor begins to suspend, and Agent Substrate begins to write its snapshot. | +| `Actor checkpointed` | The snapshot is written. The Actor is suspended, and the Worker is free for another Actor. | + +Each turn produces one cycle of the four records. The time between `Actor restoring` and `Actor restored` is how long the agent took to wake up for the turn. + +The Actor that builds a pair's first snapshot, in the `ate-golden` atespace, writes `Actor starting` and `Actor started` instead of the restore records, because it has no snapshot to restore from. + +Each record also carries `trace_id`, `span_id`, and `trace_flags`, which join the record to the Agent Substrate trace for the same operation. A `trace_flags` value of `01` means that Agent Substrate sampled the trace and exported it. A value of `00` means that the trace was not sampled, which is the case for most requests at the default ratio of `0.01`. A sampled trace reaches your tracing backend only if your collector keeps the Agent Substrate spans. {{< reuse "kagent-docs/snippets/observability/substrate-span-filter.md" >}} + +## Actor state changes + +The Agent Substrate API server, `ateapi`, writes an `Actor state changed` event each time an Actor moves to a new state. The event includes the Actor identity labels, the operation that caused the change in `ate.actor.operation.name`, and the new state in `ate.actor.state`. The last event for an Actor tells you its state, and when the Actor entered it. + +| `ate.actor.state` | Meaning | +| ----------------- | ------- | +| `resuming` | A turn woke the Actor, and a Worker is restoring it. | +| `running` | The Actor is on a Worker and can serve the turn. | +| `suspending` | The turn ended, and the Actor is writing its snapshot. | +| `suspended` | The snapshot is written, and the Actor does not hold a Worker. | +| `deleting`, `deleted` | The Session was deleted. `deleted` is the last event that an Actor gets. | + +Where the events go depends on whether the Agent Substrate release sets an OTLP endpoint. When the release sets an endpoint, `ateapi` exports each event as an OTLP log with the same attributes. A logging backend then receives the state changes of every Actor without reading pod output. Do not sample this stream. A dropped event leaves the last known state wrong, with nothing to show that an event is missing. {{< reuse "kagent-docs/snippets/observability/actor-state-records.md" >}} + +### Read state changes from stdout + +When the Agent Substrate release does not set an OTLP endpoint, `ateapi` writes each event to stdout. `ateapi` runs more than one replica, and each replica writes only the changes that it handles. Read the logs of every replica. +```bash +for pod in $(kubectl get pods -n ate-system -o name | grep ate-api-server); do + kubectl logs -n ate-system "$pod" +done | grep '"msg":"Actor state changed"' | grep "session-${SESSION_ID}" +``` + +Example output: +```console +{"time":"2026-09-24T17:00:43.007925845Z","level":"INFO","msg":"Actor state changed","ate.atespace":"kagent","ate.actor.name":"session-01a0d45b-c729-7d67-85c6-daa053202bff","ate.actor.uid":"831c62c5-b5ac-411f-b15e-7addfef97d73","ate.template.atespace":"kagent","ate.template.name":"my-first-agent-699a5ed2f709","ate.actor.operation.name":"resume","ate.actor.state":"resuming","trace_id":"402701838ffd50932dc0aec8e43a1604","span_id":"1f8f032ff37422c7","trace_flags":"00"} +{"time":"2026-09-24T17:00:43.194765137Z","level":"INFO","msg":"Actor state changed","ate.atespace":"kagent","ate.actor.name":"session-01a0d45b-c729-7d67-85c6-daa053202bff","ate.actor.uid":"831c62c5-b5ac-411f-b15e-7addfef97d73","ate.template.atespace":"kagent","ate.template.name":"my-first-agent-699a5ed2f709","ate.actor.operation.name":"resume","ate.actor.state":"running","trace_id":"402701838ffd50932dc0aec8e43a1604","span_id":"f92aecd47d89ad68","trace_flags":"00"} +``` + +## Actor usage samples + +The Worker records an `Actor usage sampled` event each time an Actor starts running and again when it stops. Each event includes the Actor identity labels, the WorkerPool, the Actor's CPU time, and its memory use, peak memory, and working set. The `ate.stats.kind` attribute is `initial` for the first sample of a run and `final` for the last. + +## Learn more + +The Agent Substrate project documents its telemetry in full, including the complete metric registry, in [Actor observability](https://github.com/agent-substrate/substrate/blob/main/docs/observability.md). That page describes the newest Agent Substrate, which can be ahead of version {{< reuse "kagent-docs/versions/agent-substrate.md" >}} that this guide uses. Records that it describes might not exist in your version. + +You can also check out the following resources in this documentation set. + +{{< cards >}} + {{< card link=`{{< link path="observability/tracing" >}}` title="Tracing" subtitle="Read the spans of an agent request, and the separate Agent Substrate traces." >}} + {{< card link=`{{< link path="about/substrate-runtime/suspend-and-resume" >}}` title="Suspend and resume" subtitle="Understand what happens to an Actor between turns." >}} +{{< /cards >}} diff --git a/docs-site/content/kagent/main/observability/tracing.md b/docs-site/content/kagent/main/observability/tracing.md new file mode 100644 index 00000000..67e55108 --- /dev/null +++ b/docs-site/content/kagent/main/observability/tracing.md @@ -0,0 +1,283 @@ +--- +title: Tracing +description: Enable OpenTelemetry tracing for kagent, then read a trace that runs from the controller through to the Actor that executed your agent. +weight: 30 +author: kagent.dev +--- + +A trace records one agent request as a tree of timed spans, so you can see where a slow or failed request spent its time and which model and tool calls it made along the way. In {{< reuse "kagent-docs/snippets/name-product.md" >}} 1.0 a single request crosses the controller, the Agent Substrate router, and the {{< gloss "Actor" >}}Actor{{< /gloss >}} that runs the agent, and one trace ties all three together. + +## About trace coverage + +A trace follows a W3C Trace Context header that the controller passes along with each request. The following diagram shows the path of one request, from the caller to the agent runtime. +

+ +```mermaid +flowchart LR + caller["Caller"] + subgraph controllerproc["kagent controller"] + grpc["gRPC API"] + gateway["A2A gateway"] + end + subgraph substrateproc["Agent Substrate"] + router["Router"] + end + subgraph actorproc["Actor"] + runtime["Agent runtime"] + end + %% Cross-subgraph edges are declared outside every subgraph block, because + %% mermaid assigns a node to the subgraph that first references it. + caller --> grpc + grpc --> gateway + gateway -->|traceparent| router + router --> runtime + classDef boundary fill:#a78bfa26,stroke:#a78bfa,stroke-width:2px + classDef inner fill:#80808033,stroke:#9ca3af,stroke-width:1px + class controllerproc,substrateproc,actorproc boundary + class grpc,gateway,router,runtime inner +``` + +A caller reaches the gRPC API on the kagent controller, which starts the trace. The controller hands the request to its A2A gateway, which opens an {{< gloss "A2A" >}}A2A{{< /gloss >}} (Agent2Agent) connection to the Session's Actor and injects a `traceparent` header into that call. The Agent Substrate router forwards the call to the Worker that runs the Actor, and adds its own spans to the trace. The agent runtime inside the Actor reads the header and continues the same trace, so the model and tool spans it produces hang off the controller's spans rather than starting a trace of their own. + +Each hop sends its spans under a separate OpenTelemetry (OTel) service name. A tracing backend groups the spans by those names. + +- **The controller** reports as `kagent-controller` in the `kagent` service namespace. Its spans also carry the pod, node, and namespace that the controller runs on. +- **The Agent Substrate router** runs the agentgateway proxy, and reports as `agentgateway`. +- **Each agent runtime** reports as its own service, named for the {{< gloss "Agent" >}}Agent{{< /gloss >}} it was compiled from. The `my-first-agent` Agent reports as `my-first-agent`. In kagent 0.x, every agent reported under a single `kagent` service. A backend that you filter by service now lists one entry for each Agent, so the list grows as you add Agents. + +### Harness coverage + +The controller passes its tracing configuration to every harness, and adds the collector to the Actor's egress allowlist. Which spans a harness then sends depends on its own instrumentation. For the available harnesses, see [Choose a harness]({{< link path="agents/agent-harness#choose-a-harness" >}}). For how the allowlist is built, see [Egress control]({{< link path="about/substrate-runtime/networking-and-egress#policy-generation" >}}). + +| Harness | What the trace shows | +| ------- | -------------------- | +| `kagent` | The runtime's spans. The span names on this page describe this runtime. | +| `codex` and `claude` | The request span, plus the `chat` spans for model calls and the `execute_tool` spans for tool calls that kagent writes. The native Claude Code or Codex process also emits its own spans. kagent marks that process with `kagent.genai.producer=runtime`, so that a collector can remove the duplicate GenAI model and usage attributes from the native spans. | +| `byo` | Spans only if the image implements OTel and reads the standard exporter variables. An image without OTel instrumentation leaves its half of the trace missing. A Harness `spec.env` entry overrides the controller's value for a variable of the same name, but an endpoint that you set this way is not added to the egress allowlist, so the image cannot reach it. | + +### Spans + +Each runtime creates spans for the request, the agent invocation, the model calls, and the tool calls. Most span names describe the operation rather than the agent. The `invoke_agent` span is the exception, because its name includes the name of the agent that ran. To narrow a search to one agent, filter by service name rather than by span name. The following table lists the spans in nesting order, starting with the span that accepts the request. + +| Span | Runtime | When it is created | +| ---- | ------- | ------------------ | +| `lf.a2a.v1.A2AService/SendMessage` | `kagent`, `codex`, `claude` | Once per request, as the root of the runtime's half of the trace. The runtime creates it when it accepts the A2A call from the controller. The controller reports spans of the same name for its own side of the call. | +| `a2a.request` | `kagent` | Once per request. Records the A2A method and the final state of the task in the `a2a.method` and `a2a.task.state` attributes. | +| `invoke_agent ` | `kagent`, `codex`, `claude` | Once per request, named for the {{< gloss "Agent" >}}Agent{{< /gloss >}} that serves it, with each hyphen replaced by an underscore. The Agent `my-first-agent` produces `invoke_agent my_first_agent`, while its service name keeps the hyphens, so the two spellings differ. On the `codex` and `claude` runtimes, this span is the request span, in place of `a2a.request`. | +| `generate_content ` | `kagent` | Once per model call, named for the model that was called. | +| `chat ` | `codex`, `claude` | Once per model call, named for the model that was called. One span covers a call and its automatic retries. For a retried call, the span includes `kagent.inference.retry.count`. For a call that ends without final usage, the span includes `kagent.inference.usage_partial=true`, so its output tokens are a lower bound. | +| `execute_tool ` | `kagent`, `codex`, `claude` | Once per tool call, named for the tool that was called. On the `kagent` runtime, the span records the call's arguments and reply in the `gcp.vertex.agent.tool_call_args` and `gcp.vertex.agent.tool_response` attributes. On the `codex` and `claude` runtimes, content capture adds tool arguments and successful results in the `gen_ai.tool.call.arguments` and `gen_ai.tool.call.result` attributes. | +| `execute_tool (merged)` | `kagent` | Once per model turn that calls more than one tool, as the parent of that turn's `execute_tool` spans. A turn that calls a single tool does not create a merged span. | + +### Correlation attributes + +Use attributes, not span names, to identify a request in a trace. The runtime copies the request attributes to every descendant span. The runtime identity and main Agent identity are resource attributes, so use them to filter all spans from the same runtime process. + +| Attribute | Where | Value | +| --------- | ----- | ----- | +| `a2a.task.id` | Span | The A2A task ID, which identifies one turn of a conversation. | +| `gen_ai.conversation.id` | Span | The A2A context ID, which identifies the conversation and is stable across its turns. | +| `gen_ai.agent.id` | Span | The Agent, as `/`. The shorter `gen_ai.agent.name` carries the name alone. | +| `user.id` | Span | The authenticated caller, such as `admin@kagent.dev`, on the request span and the spans beneath it. Earlier releases recorded this attribute as `enduser.id`. On the `claude` runtime, the Claude Code spans set their own `user.id`, which is a hash of the Claude account. Read the kagent caller from the `invoke_agent` span. | +| `gen_ai.main_agent.id` | Resource | The main Agent that the runtime process was compiled for, as `/`. This resource attribute replaces the resource-level `gen_ai.agent.id` name. | +| `kagent.runtime` | Resource | The runtime that served the request, such as `adk-go`. | +| `kagent.genai.producer` | Resource | `runtime` on a native `codex` or `claude` process. The value shows that kagent writes the GenAI model and tool spans for that process. A collector can use the value to avoid counting the native spans as duplicate model calls. | + +The runtime also adds each scalar value in the A2A message's metadata as an `a2a.message.metadata.` attribute, so a client can tag a request and search for it later. These tags stay on the `a2a.request` span alone, unlike the correlation attributes. A search on one therefore returns that span rather than the whole subtree. + +> [!WARNING] +> When the `config.runtimeDefaults.telemetry.captureContent` Helm setting is `true`, prompts and replies reach your tracing backend. On the `kagent` runtime, the `generate_content` span of each model call then carries the conversation as the `gen_ai.input.messages` and `gen_ai.output.messages` attributes, and the Agent's system prompt as `gen_ai.system_instructions`. The setting defaults to `false`, which omits all three. +> +> On the `kagent` runtime, the setting does not govern tool content. An `execute_tool` span records the call's arguments and the tool's reply in `gcp.vertex.agent.tool_call_args` and `gcp.vertex.agent.tool_response` whether the setting is `true` or `false`. A tool that returns sensitive data sends it to your tracing backend on the default settings. On the `codex` and `claude` runtimes, content capture controls tool arguments and successful results. Turn tracing off for an agent whose tools return data that must not leave the cluster. For how to use this content as an audit record, see [Audit prompts]({{< link path="observability/audit-prompts" >}}). + +## Before you begin + +1. [Install kagent]({{< link path="setup/installation" >}}). +2. [Create your first agent]({{< link path="get-started/your-first-agent" >}}), so that you have an {{< gloss "Agent" >}}Agent{{< /gloss >}} to send a request to. That guide also installs the kagent CLI. The steps on this page need the {{< reuse "kagent-docs/versions/kagent.md" >}} CLI, because the CLIs of other releases, newer ones included, do not have the Session commands that these steps use. To check your version, run `kagent version`. +3. Set up a tracing backend. + + The [OTel stack]({{< link path="observability/otel-stack" >}}) sends traces to Tempo, and the [Lightweight OTel stack]({{< link path="observability/lightweight-otel-stack" >}}) sends traces to Jaeger. Both guides turn on tracing, so you can skip to [Review a trace](#review-a-trace). + +## Enable tracing + +Tracing is off by default. Turning it on is a Helm change, because the controller reads its tracing configuration from the environment and passes that configuration to the agent runtimes that the controller starts. The following steps send traces over OTLP to the collector that both stack guides install. To send traces to another OTLP backend, change the endpoint. + +1. Save the current revision of your Agent. A later step uses it to tell when kagent recompiles the Agent with the new settings. The command first waits for any recompile that is still in progress, such as one from an earlier Helm upgrade, so that it saves a finished revision. + ```bash + for i in $(seq 1 60); do + REVISIONS=$(kubectl get agent my-first-agent -n kagent \ + -o jsonpath='{.status.desiredRevision} {.status.latestSuccessfulRevision}') + [ "${REVISIONS% *}" = "${REVISIONS#* }" ] && break + sleep 5 + done + export OLD_REVISION=${REVISIONS#* } + echo "Current revision: $OLD_REVISION" + ``` + +2. Get your current Helm values for kagent. + ```shell + helm get values kagent -n kagent -o yaml > values.yaml + ``` + +3. Add the tracing settings to the values file. + ```yaml + config: + telemetry: + traces: + endpoint: http://otel-collector.telemetry.svc.cluster.local:4317 + protocol: grpc + timeout: 15s + runtimeDefaults: + telemetry: + traces: + endpoint: http://otel-collector.telemetry.svc.cluster.local:4317 + protocol: grpc + timeout: 15s + ``` + + {{< reuse "kagent-docs/snippets/review-table.md" >}} + + | Field | Description | + | ----- | ----------- | + | `config.telemetry.traces` | The traces of the kagent controller itself. | + | `config.runtimeDefaults.telemetry.traces` | The traces of every agent runtime that the controller compiles. The controller's own settings do not propagate to a runtime, so set both blocks to trace the whole path of a request. | + | `traces.endpoint` | Where the traces go, as an `http://` or `https://` URL. An `http://` endpoint sends plaintext. On the `http/protobuf` protocol, include the signal path, as in `http://otel-collector.telemetry.svc.cluster.local:4318/v1/traces`. | + | `traces.protocol` | `grpc` or `http/protobuf`. Defaults to `grpc`, which matches the port `4317` in the example endpoint. Point `http/protobuf` at port `4318` instead. | + | `traces.timeout` | The export timeout, as a duration in seconds with an `s` suffix. Defaults to `10s`. | + + A signal does not have a separate switch. When you set a signal's endpoint, the controller exports the signal. When you omit a signal's endpoint, the controller compiles an explicit off state for the signal. Omit the endpoint key rather than leaving it empty, because an empty endpoint fails validation. An endpoint must be an absolute `http://` or `https://` URL. The controller rejects one that includes credentials, a query, or a fragment. Check the controller's logs after you change these values. + +4. Upgrade the kagent Helm release. + ```bash + helm upgrade kagent \ + {{< reuse "kagent-docs/snippets/helm-path.md" >}}/{{< reuse "kagent-docs/snippets/helm-kagent.md" >}} \ + --version {{< reuse "kagent-docs/versions/kagent.md" >}} \ + --namespace kagent \ + --values values.yaml + ``` + +5. Wait for kagent to recompile the Agent. The controller rebuilds each Agent after the controller restarts. A Session that you create before the rebuild finishes starts from the previous revision, without the new settings. The following command prints `Recompiled` when the new revision is ready. + ```bash + for i in $(seq 1 60); do + [ "$(kubectl get agent my-first-agent -n kagent \ + -o jsonpath='{.status.latestSuccessfulRevision}')" != "$OLD_REVISION" ] \ + && echo "Recompiled" && break + sleep 5 + done + ``` + If the command finishes without printing `Recompiled`, the upgrade did not change the settings that kagent compiles into the Agent. Either the settings were already in place, or the chart did not recognize the `config` keys. Helm accepts a key that a chart does not define without an error, so check that you upgraded to version {{< reuse "kagent-docs/versions/kagent.md" >}} of the chart, which uses the keys on this page. + +6. Create a new Session, so that its Actor starts from a runtime that has the tracing configuration. + ```bash + kagent agent session create --agent my-first-agent + ``` + +7. Confirm that the Session runs the Agent's current revision. The command waits until kagent finishes compiling the Agent, then compares that revision with the one that the Session started from. If the command prints `Outdated`, the Session was created from an earlier revision, and exports without the new settings. Create another Session, and run the command again. + ```bash + for i in $(seq 1 60); do + REVISIONS=$(kubectl get agent my-first-agent -n kagent \ + -o jsonpath='{.status.desiredRevision} {.status.latestSuccessfulRevision}') + [ "${REVISIONS% *}" = "${REVISIONS#* }" ] && break + sleep 5 + done + SESSION_REVISION=$(kagent agent session get $SESSION_ID -o json | jq -r '.session.preparedRevision') + [ "$SESSION_REVISION" = "${REVISIONS#* }" ] && echo "Current" || echo "Outdated" + ``` + +## Review a trace + +Send a request to a new Session, then find its trace in the backend that you set up. + +1. Send a request to a new Session to produce a trace. + ```bash + export SESSION_ID=$(kagent agent session create --agent my-first-agent -o json | jq -r '.session.id') + kagent agent invoke --session $SESSION_ID --task "What is 2+2?" + ``` + +2. Open the trace in your tracing backend. + {{< tabs >}} + {{% tab name="Grafana with Tempo" %}} + 1. Forward the Grafana port, and leave the command running. + ```bash + kubectl port-forward -n telemetry svc/kube-prometheus-stack-grafana 3000:80 + ``` + 2. In your browser, open Grafana at [http://localhost:3000](http://localhost:3000), and log in. For the password, see [Explore the telemetry in Grafana]({{< link path="observability/otel-stack#explore-the-telemetry-in-grafana" >}}). + 3. Open **Explore**, select the **Tempo** data source, and select the **Search** query type. + 4. From the **Service Name** list, select `my-first-agent`, and run the query. Selecting `kagent-controller` instead returns the same traces from the controller's side. + 5. Click a trace to open it. + {{% /tab %}} + {{% tab name="Jaeger" %}} + 1. Forward the Jaeger query port, and leave the command running. + ```bash + kubectl port-forward -n telemetry svc/jaeger 16686:16686 + ``` + 2. In your browser, open Jaeger at [http://localhost:16686](http://localhost:16686). + 3. From the **Service** list, select `my-first-agent`. Selecting `kagent-controller` instead returns the same traces from the controller's side. + 4. Leave **Operation** on `all`, or select `invoke_agent ` to start from the agent's own work rather than from the A2A call that carries it, and click **Find Traces**. + 5. Click a trace to open it. + {{% /tab %}} + {{< /tabs >}} + +3. Review the span tree. The trace starts with the controller's spans, continues through the Agent Substrate router, and ends with the agent runtime's spans. The following example shows the spans of one request, with the service that reported each span. + ```console + lf.a2a.v1.A2AService/SendMessage kagent-controller + lf.a2a.v1.A2AService/SendMessage kagent-controller + POST /* agentgateway + POST agentgateway + lf.a2a.v1.A2AService/SendMessage my-first-agent + a2a.request my-first-agent + invoke_agent my_first_agent my-first-agent + generate_content gpt-4.1-mini my-first-agent + HTTP POST my-first-agent + ``` + +4. Use a span attribute, such as `gen_ai.conversation.id=`, to narrow a search to one conversation. Use a resource attribute, such as `gen_ai.main_agent.id=/`, to narrow a search to one runtime process. + +## Agent Substrate traces + +Agent Substrate records traces for its own work, such as scheduling an Actor onto a Worker and restoring it from a snapshot. These traces are separate from the agent request trace, and do not share its ID. The router's `agentgateway` spans are the exception, and join the request trace. The `POST /*` span covers the whole request at the router, including any wait for the Actor to resume, but does not show which step of the resume took the time. To investigate a slow start, look up the Agent Substrate traces from the same time window, or read the Actor's [suspend and resume records]({{< link path="observability/substrate-telemetry#suspend-and-resume-records" >}}), which carry the trace ID of each operation. + +| Service | Reports | +| ------- | ------- | +| `ateapi` | Actor lifecycle operations, such as create, resume, and suspend, and the scheduling of Actors onto Workers. | +| `atelet` | Work on a Worker's node, such as restoring an Actor from a snapshot. | +| `ateworker-gvisor` | Work inside the sandbox that runs the Actor. | +| `atecontroller` | Reconciliation of Agent Substrate resources, such as WorkerPools. | + +Agent Substrate exports traces only when its Helm release sets `otel.endpoint`, and it keeps 1% of its traces by default. To keep more, raise `otel.traces.samplingRatio`. + +For the steps, see [Send Agent Substrate telemetry to the collector]({{< link path="observability/otel-stack#send-agent-substrate-telemetry-to-the-collector" >}}). + +## Traces from a suspended Actor + +Agent Substrate {{< gloss "Checkpoint" >}}checkpoints{{< /gloss >}} an Actor as soon as the response body closes, which is sooner than a batching span exporter normally sends its buffer. Spans still in the buffer at that moment freeze inside the {{< gloss "Snapshot" >}}snapshot{{< /gloss >}} and reach the backend only when the session next resumes, or never at all for a conversation's last message. + +To avoid losing them, the `kagent`, `codex`, and `claude` runtimes flush their span buffer after each A2A handler returns, before the response completes. The flush is unconditional and waits up to three seconds, and no setting changes either. An agent on the `byo` harness flushes only if its own image does, so a conversation's last turn can lose its spans there. + +The flush lets a kagent trace arrive promptly rather than on the exporter's own schedule. To understand what suspension does to an Actor, see [Suspend and resume]({{< link path="about/substrate-runtime/suspend-and-resume" >}}). + +## Turn tracing off + +Turn off the trace exporter, then create a new Session so that the change takes effect. The controller sets the trace exporter to `none` in each runtime, so an agent does not fall back to the OpenTelemetry SDK's default of exporting to `localhost`. + +1. Remove the telemetry settings from the kagent Helm release. Remove the whole `telemetry` block rather than one signal inside it. The controller rejects a null value anywhere in its configuration and fails to start, so a values file that sets a single signal to `null` breaks the installation. A `--set` flag removes the key instead of nulling it. + ```bash + helm upgrade kagent \ + {{< reuse "kagent-docs/snippets/helm-path.md" >}}/{{< reuse "kagent-docs/snippets/helm-kagent.md" >}} \ + --version {{< reuse "kagent-docs/versions/kagent.md" >}} \ + --namespace kagent --reuse-values \ + --set config.telemetry=null \ + --set config.runtimeDefaults.telemetry=null + ``` + +2. Create a new Session to pick up the change, because an existing Actor keeps the configuration it started with. + +3. To remove the tracing backend, follow the cleanup steps in the [OTel stack]({{< link path="observability/otel-stack#clean-up" >}}) or [Lightweight OTel stack]({{< link path="observability/lightweight-otel-stack#clean-up" >}}) guide. + +## Next steps + +{{< cards >}} + {{< card link=`{{< link path="observability/audit-prompts" >}}` title="Audit prompts" subtitle="Export every prompt and reply as a log event for security and compliance review." >}} + {{< card link=`{{< link path="observability/metrics" >}}` title="Metrics" subtitle="Review the metrics that kagent and Agent Substrate report." >}} + {{< card link=`{{< link path="about/substrate-runtime/suspend-and-resume" >}}` title="Suspend and resume" subtitle="Learn what happens to an Actor between the turns of a conversation." >}} +{{< /cards >}} diff --git a/docs-site/content/kagent/main/operations/_index.md b/docs-site/content/kagent/main/operations/_index.md new file mode 100644 index 00000000..c752aa84 --- /dev/null +++ b/docs-site/content/kagent/main/operations/_index.md @@ -0,0 +1,6 @@ +--- +title: Operations +description: Run, tune, and maintain a kagent installation after it is up. +weight: 90 +author: kagent.dev +--- diff --git a/docs-site/content/kagent/main/operations/debug.md b/docs-site/content/kagent/main/operations/debug.md new file mode 100644 index 00000000..6a707203 --- /dev/null +++ b/docs-site/content/kagent/main/operations/debug.md @@ -0,0 +1,226 @@ +--- +title: Debug +description: Find where kagent reports a failure, read the readiness conditions, and work back from a symptom to its cause. +weight: 30 +author: kagent.dev +--- + +Most {{< reuse "kagent-docs/snippets/name-product.md" >}} failures surface in the readiness conditions on the {{< gloss "Agent" >}}Agent{{< /gloss >}}. This page explains where to read those conditions, how to tell the real failure from the ones that follow it, and how to work back from the symptoms that report nothing useful on their own. + +## Where kagent reports status + +A {{< gloss "Harness" >}}Harness{{< /gloss >}} and an {{< gloss "AgentTemplate" >}}AgentTemplate{{< /gloss >}} have **no status at all**, and their list views have no status column, so a healthy resource and a broken one look identical. + +```bash +kubectl get agenttemplate -n kagent +``` +Example output: +```console +NAME AGE +assistant 30h +``` + +The diagnostic surface is the Agent. An Agent pairs one AgentTemplate with one Harness, and its conditions cover that pairing. + +```bash +kubectl get agent -n kagent +``` +Example output: +```console +NAME READY AGE +assistant True 30h +``` + +The `READY` column reports the status of the `Ready` condition and nothing else. To find the condition's reason and message, read the Agent's conditions directly. + +```bash +kubectl get agent -n kagent \ + -o jsonpath='{range .status.conditions[*]}{.type}{"="}{.status}{" "}{.reason}{": "}{.message}{"\n"}{end}' +``` + +Example output for a healthy Agent: +```console +Accepted=True Accepted: Agent explicitly selects its template and harness +ResolvedRefs=True Resolved: All runtime references resolved +Compatible=True Compatible: Resolved configuration is compatible with the Harness +Ready=True Ready: ActorTemplate golden snapshot is ready +``` + +## Read the readiness conditions + +The four conditions form a pipeline rather than an unordered set, so their order tells you how far preparation reached before it stopped. + +Each condition covers one stage, and a stage runs only when the stage before it succeeded. + +| Condition | Stage it covers | A failure here means | +| --------- | --------------- | -------------------- | +| `Accepted` | The Agent selects a template and a Harness. | Nothing. kagent writes this condition only as `True`, because an Agent references both resources explicitly. | +| `ResolvedRefs` | Every resource that the Agent references was found, including its AgentTemplate, its Harness, its ModelConfig, and any tool servers. | A reference points at something that does not exist, or the controller cannot read it. | +| `Compatible` | The resolved configuration is valid for the Harness. | The configuration asks for something that its Harness does not support. | +| `Ready` | The ActorTemplate's golden {{< gloss "Snapshot" >}}snapshot{{< /gloss >}} exists, so Sessions can start from it. | Agent Substrate has not finished capturing the golden snapshot, or capture failed. | + +When a stage fails, kagent marks that stage `False` with the real reason and message, then marks **every later stage `False` with the reason `Blocked`**. Only the first failure describes an actual problem. + +**Read the first condition that is `False` and whose reason is not `Blocked`.** Everything after it is a consequence. + +In the following example, one missing ModelConfig is the whole problem, and the two conditions after it do not add independent information. + +```console +Accepted=True Accepted: Agent explicitly selects its template and harness +ResolvedRefs=False ReferenceResolutionFailed: resolve ModelConfig "does-not-exist": not found +Compatible=False Blocked: blocked by ResolvedRefs +Ready=False Blocked: blocked by ResolvedRefs +``` + +A failed golden snapshot is not final. When Agent Substrate fails to build the golden snapshot, kagent deletes the failed ActorTemplate and rebuilds it after about 15 seconds, 1 minute, and 4 minutes, and then about every 10 minutes indefinitely. While a rebuild waits, `Ready` is `False` with the reason `ActorTemplateRetrying` and a message such as `golden build failed (GoldenActorCrashed), retry 1 at