diff --git a/images/use-cases/observability/clickstack-cloud-v2.png b/images/use-cases/observability/clickstack-cloud-v2.png new file mode 100644 index 00000000..5ebc14b2 Binary files /dev/null and b/images/use-cases/observability/clickstack-cloud-v2.png differ diff --git a/images/use-cases/observability/clickstack-search.png b/images/use-cases/observability/clickstack-search.png new file mode 100644 index 00000000..0d2c7173 Binary files /dev/null and b/images/use-cases/observability/clickstack-search.png differ diff --git a/images/use-cases/observability/clickstack-start-exploring.png b/images/use-cases/observability/clickstack-start-exploring.png new file mode 100644 index 00000000..f5c4cc61 Binary files /dev/null and b/images/use-cases/observability/clickstack-start-exploring.png differ diff --git a/images/use-cases/observability/clickstack-start-ingestion.png b/images/use-cases/observability/clickstack-start-ingestion.png new file mode 100644 index 00000000..9da8e182 Binary files /dev/null and b/images/use-cases/observability/clickstack-start-ingestion.png differ diff --git a/products/clickstack/deployment/managed.mdx b/products/clickstack/deployment/managed.mdx index 8be23c0e..b52ddace 100644 --- a/products/clickstack/deployment/managed.mdx +++ b/products/clickstack/deployment/managed.mdx @@ -8,7 +8,6 @@ keywords: ['clickstack', 'deployment', 'setup', 'configuration', 'observability' --- import { Image } from "/snippets/components/Image.jsx"; -import BetaBadge from "/snippets/components/BetaBadge/BetaBadge.jsx"; import { TrackedLink } from "/snippets/components/GalaxyTrackedLink/GalaxyTrackedLink.jsx"; @@ -19,12 +18,6 @@ import SetupManagedIngestion from '/snippets/clickstack/_setup_managed_ingestion import NavigateClickStackUI from '/snippets/clickstack/_navigate_managed.mdx'; import ProviderSelection from '/snippets/clickstack/_select_provider.mdx'; - - -::::note[Beta feature] -This feature is in ClickHouse Cloud beta. -:::: - This **guide is for existing users of ClickHouse Cloud**. If you're new to ClickHouse Cloud, we recommend our [Getting Started](/products/clickstack/getting-started/managed) guide for Managed ClickStack. In this deployment pattern, both ClickHouse and the ClickStack UI (HyperDX) are hosted in ClickHouse Cloud, minimizing the number of components the user needs to self-host. diff --git a/products/clickstack/example-datasets/index.mdx b/products/clickstack/example-datasets/index.mdx index 31fcf6ab..db970c9e 100644 --- a/products/clickstack/example-datasets/index.mdx +++ b/products/clickstack/example-datasets/index.mdx @@ -14,3 +14,4 @@ This section provides various sample datasets and examples to help you get start | [Local Data](/products/clickstack/example-datasets/local-data) | Collect local system metrics and logs sending them to ClickStack for analysis | | [Remote Demo Data](/products/clickstack/example-datasets/remote-demo-data) | Connect to our remote demo cluster and explore an issue | | [Session Replay Demo](/products/clickstack/example-datasets/session-replay) | Instrument a demo web application for session replay and view your interactions in ClickStack | +| [Synthetic data with otelgen](/products/clickstack/example-datasets/otelgen) | Use `otelgen` to send synthetic logs, traces and metrics to a running ClickStack OpenTelemetry collector | diff --git a/products/clickstack/example-datasets/otelgen.mdx b/products/clickstack/example-datasets/otelgen.mdx new file mode 100644 index 00000000..decff787 --- /dev/null +++ b/products/clickstack/example-datasets/otelgen.mdx @@ -0,0 +1,204 @@ +--- +slug: /use-cases/observability/clickstack/getting-started/otelgen +title: 'Generate synthetic OpenTelemetry data with otelgen' +sidebarTitle: 'Synthetic data with otelgen' +description: 'Use otelgen to send synthetic logs, traces and metrics to a ClickStack OpenTelemetry collector' +doc_type: 'guide' +toc_max_heading_level: 2 +keywords: ['clickstack', 'otelgen', 'synthetic data', 'OpenTelemetry', 'test', 'logs', 'traces', 'metrics', 'observability'] +--- + +[`otelgen`](https://github.com/krzko/otelgen) is a small Go CLI that generates synthetic OTLP logs, traces and metrics. Use it to confirm that an existing ClickStack OpenTelemetry collector is accepting data and that events surface in the ClickStack UI. + +This guide assumes the collector is already running with OTLP endpoints on `4317` (gRPC) and `4318` (HTTP). + + + + + + + +### Prerequisites {#prerequisites-managed} + +This guide assumes you have completed the [Getting Started Guide for Managed ClickStack](/products/clickstack/deployment/managed) and have an OpenTelemetry collector running with the OTLP gRPC (`4317`) and HTTP (`4318`) endpoints reachable from the machine you run `otelgen` on. If you [secured the collector](/products/clickstack/ingesting-data/collector#securing-the-collector) with an `OTLP_AUTH_TOKEN`, keep that value handy. + + + +### Install otelgen {#install-otelgen-managed} + +Install with Homebrew: + +```shell +brew install krzko/tap/otelgen +``` + +Or install with Go: + +```shell +go install github.com/krzko/otelgen@latest +``` + + + +### Set environment variables {#set-env-vars-managed} + +Export the collector endpoint and, if the collector is secured, the auth token: + +```shell +export OTEL_ENDPOINT=:4317 +export OTLP_AUTH_TOKEN= +``` + +Use the host and port of your collector. For a collector running on the same machine, this is `localhost:4317`. + + +**Unsecured collector** + +The ClickStack OpenTelemetry collector is unauthenticated by default. If you haven't followed [Securing the collector](/products/clickstack/ingesting-data/collector#securing-the-collector) to set an `OTLP_AUTH_TOKEN`, skip `OTLP_AUTH_TOKEN` here and drop the `--header` flag from the commands below. + + + + +### Generate traces {#generate-traces-managed} + +Send a short burst of multi-span traces: + +```shell +otelgen --otel-exporter-otlp-endpoint ${OTEL_ENDPOINT} \ + --header "authorization=${OTLP_AUTH_TOKEN}" \ + --protocol grpc --insecure \ + --rate 2 --duration 10 \ + traces multi +``` + +`--rate` is traces per second and `--duration` is the run length in seconds. `--insecure` disables TLS on the gRPC connection, which is needed when pointing `otelgen` at the collector's plaintext OTLP port. + + + +### Generate logs {#generate-logs-managed} + +```shell +otelgen --otel-exporter-otlp-endpoint ${OTEL_ENDPOINT} \ + --header "authorization=${OTLP_AUTH_TOKEN}" \ + --protocol grpc --insecure \ + --rate 2 --duration 10 \ + logs multi +``` + + + +### Generate metrics {#generate-metrics-managed} + +The metrics subcommands don't honor `--duration`. Run the command and press `Ctrl+C` after a few seconds to stop it. + +```shell +otelgen --otel-exporter-otlp-endpoint ${OTEL_ENDPOINT} \ + --header "authorization=${OTLP_AUTH_TOKEN}" \ + --protocol grpc --insecure \ + --rate 2 \ + metrics sum +``` + +`otelgen` also supports `gauge`, `histogram`, `up-down-counter` and `exponential-histogram` subcommands under `metrics`. + + + +### Verify in ClickStack {#verify-managed} + +Open the ClickStack UI from the ClickHouse Cloud console. In the `Search` view, switch the source between `Logs` and `Traces` to confirm new events. Set the time range to `Last 15 minutes`. Open the `Chart Explorer`, select `Metrics`, and chart one of the metric names produced by `otelgen` (for example `otelgen.metrics.sum`) to verify metrics ingestion. + + + + + + + + + + +### Prerequisites {#prerequisites-oss} + +This guide assumes you have started Open Source ClickStack using the [instructions for the all-in-one image](/products/clickstack/getting-started/oss), and that the OTLP endpoints (`4317` gRPC and `4318` HTTP) are reachable. You also need the ingestion API key from the HyperDX UI under `Team Settings > API Keys`. + + + +### Install otelgen {#install-otelgen-oss} + +Install with Homebrew: + +```shell +brew install krzko/tap/otelgen +``` + +Or install with Go: + +```shell +go install github.com/krzko/otelgen@latest +``` + + + +### Set environment variables {#set-env-vars-oss} + +Export the collector endpoint and the ingestion API key: + +```shell +export OTEL_ENDPOINT=localhost:4317 +export CLICKSTACK_API_KEY= +``` + + + +### Generate traces {#generate-traces-oss} + +Send a short burst of multi-span traces: + +```shell +otelgen --otel-exporter-otlp-endpoint ${OTEL_ENDPOINT} \ + --header "authorization=${CLICKSTACK_API_KEY}" \ + --protocol grpc --insecure \ + --rate 2 --duration 10 \ + traces multi +``` + +`--rate` is traces per second and `--duration` is the run length in seconds. `--insecure` enables plaintext gRPC against the local collector. + + + +### Generate logs {#generate-logs-oss} + +```shell +otelgen --otel-exporter-otlp-endpoint ${OTEL_ENDPOINT} \ + --header "authorization=${CLICKSTACK_API_KEY}" \ + --protocol grpc --insecure \ + --rate 2 --duration 10 \ + logs multi +``` + + + +### Generate metrics {#generate-metrics-oss} + +The metrics subcommands don't honor `--duration`. Run the command and press `Ctrl+C` after a few seconds to stop it. + +```shell +otelgen --otel-exporter-otlp-endpoint ${OTEL_ENDPOINT} \ + --header "authorization=${CLICKSTACK_API_KEY}" \ + --protocol grpc --insecure \ + --rate 2 \ + metrics sum +``` + +`otelgen` also supports `gauge`, `histogram`, `up-down-counter` and `exponential-histogram` subcommands under `metrics`. + + + +### Verify in ClickStack {#verify-oss} + +Visit [http://localhost:8080](http://localhost:8080) to open the ClickStack UI. In the `Search` view, switch the source between `Logs` and `Traces` to confirm new events. Set the time range to `Last 15 minutes`. Open the `Chart Explorer`, select `Metrics`, and chart one of the metric names produced by `otelgen` (for example `otelgen.metrics.sum`) to verify metrics ingestion. + + + + + + diff --git a/products/clickstack/getting-started/managed.mdx b/products/clickstack/getting-started/managed.mdx index 2b47be6d..dffc6def 100644 --- a/products/clickstack/getting-started/managed.mdx +++ b/products/clickstack/getting-started/managed.mdx @@ -9,13 +9,10 @@ keywords: ['Managed ClickStack', 'getting started', 'ClickHouse Cloud'] import { Image } from "/snippets/components/Image.jsx"; -import BetaBadge from "/snippets/components/BetaBadge/BetaBadge.jsx"; import SetupManagedIngestion from '/snippets/clickstack/_setup_managed_ingestion.mdx'; import ProviderSelection from '/snippets/clickstack/_select_provider.mdx'; import NavigateClickStackUI from '/snippets/clickstack/_navigate_managed.mdx'; - - The easiest way to get started is by deploying **Managed ClickStack** on **ClickHouse Cloud**, which provides a fully managed, secure backend while retaining complete control over ingestion, schema, and observability workflows. This removes the need to operate ClickHouse yourself and delivers a range of benefits: - Automatic scaling of compute, independent of storage diff --git a/products/clickstack/managed-onboarding/setting-up-your-opentelemetry-collector.mdx b/products/clickstack/managed-onboarding/setting-up-your-opentelemetry-collector.mdx new file mode 100644 index 00000000..acb65019 --- /dev/null +++ b/products/clickstack/managed-onboarding/setting-up-your-opentelemetry-collector.mdx @@ -0,0 +1,162 @@ +--- +slug: /use-cases/observability/clickstack/setting-up-your-opentelemetry-collector +title: 'Setting up your OpenTelemetry Collector' +description: 'Setting up an OpenTelemetry Collector for Managed ClickStack' +doc_type: 'guide' +keywords: ['clickstack', 'opentelemetry', 'collector', 'managed', 'observability', 'gateway', 'otelgen'] +unlisted: true +--- + +import { Image } from "/snippets/components/Image.jsx"; + +This guide walks you through deploying an OpenTelemetry (OTel) Collector against an existing Managed ClickStack service, then verifying that data is flowing through it. + +The collector runs as a **gateway**: a single OTLP endpoint that your applications, SDKs, and agent collectors send to. The gateway batches events, applies any processing you've configured, and writes them to ClickHouse via the [ClickHouse exporter](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/clickhouseexporter). This pattern keeps collection logic out of your application code and lets you scale ingestion independently of the workloads producing data. For background on gateway versus agent roles, see [Collector roles](/products/clickstack/ingesting-data/collector#collector-roles). + +This guide assumes you've completed the [Getting started with Managed ClickStack](/products/clickstack/getting-started/managed) guide and have your connection credentials to hand. + + + + +## Gather your credentials {#gather-credentials} + +You'll need: + +- The HTTPS endpoint of your ClickHouse Cloud service, including protocol and port, for example `https://abc123xyz.us-central1.gcp.clickhouse.cloud:8443`. +- A ClickHouse username and password for ingestion. + +If you don't have these recorded, open your service in the [ClickHouse Cloud console](https://console.clickhouse.cloud) and select **Connect**. Record the url from the subsequent dialog. We will create a dedicated user for ingestion below. + +Service connect panel showing HTTPS endpoint and password + + + +## Create an ingestion user {#create-ingestion-user} + +We recommend creating a dedicated user for the collector rather than reusing `default`. Connect to your service via the SQL console and run: + +```sql +CREATE USER hyperdx_ingest IDENTIFIED WITH sha256_password BY 'ClickH0u3eRocks123!'; +GRANT SELECT, INSERT, CREATE DATABASE, CREATE TABLE, CREATE VIEW ON otel.* TO hyperdx_ingest; +``` + +**Replace the password with a strong value**. + +The collector creates the schema for logs, traces, and metrics inside the `otel` database on first use. For more guidance on production user setup, see [Going to production](/products/clickstack/managing/production#create-a-database-ingestion-user-managed). + + + +## Deploy the collector {#deploy-the-collector} + +Deploy the collector somewhere that is accessible to the applications and infrastructure sending OpenTelemetry data. In the example below, we run the collector locally and generate artificial telemetry from the same machine for simplicity. + + +In production, you would typically deploy the collector in a Kubernetes cluster, or on a virtual machine that can be reached by your OpenTelemetry SDKs, agents, and other collectors. This allows telemetry from across your environment to be centrally collected and forwarded to ClickStack. + + +Pick a shared secret to authenticate clients sending data to the collector, then export it alongside your connection details and chosen password for the `hyperdx_ingest` user: + +```shell +export CLICKHOUSE_ENDPOINT= +export CLICKHOUSE_USER=hyperdx_ingest +export CLICKHOUSE_PASSWORD=ClickH0u3eRocks123! +export OTLP_AUTH_TOKEN="a-strong-shared-secret" +``` + +Run the ClickStack OTel collector: + +```shell +docker run -d \ + -e OTLP_AUTH_TOKEN=${OTLP_AUTH_TOKEN} \ + -e CLICKHOUSE_ENDPOINT=${CLICKHOUSE_ENDPOINT} \ + -e CLICKHOUSE_USER=${CLICKHOUSE_USER} \ + -e CLICKHOUSE_PASSWORD=${CLICKHOUSE_PASSWORD} \ + -e HYPERDX_OTEL_EXPORTER_CLICKHOUSE_DATABASE=otel \ + -p 4317:4317 \ + -p 4318:4318 \ + clickhouse/clickstack-otel-collector:latest +``` + +The collector now exposes OTLP gRPC on `4317` and OTLP HTTP on `4318`. Applications, SDKs, and agent collectors should send to these ports with `authorization: $OTLP_AUTH_TOKEN` in the request headers. + + +**Production deployments** + +For production, we recommend enabling TLS on the OTLP endpoint. See [Securing the collector](/products/clickstack/ingesting-data/collector#securing-the-collector). + + + + +## Verify the endpoint {#verify-the-endpoint} + +Generate some synthetic traffic against the collector to confirm the full pipeline works. We use [`otelgen`](https://github.com/krzko/otelgen), a small CLI that emits OTLP logs, traces, and metrics. + +Install `otelgen` with Homebrew: + +```shell +brew install krzko/tap/otelgen +``` + +Or with Go: + +```shell +go install github.com/krzko/otelgen@latest +``` + +Send a short burst of logs to the collector: + +```shell + otelgen \ + --otel-exporter-otlp-endpoint localhost:4317 \ + --insecure \ + --protocol grpc \ + --header "authorization=${OTLP_AUTH_TOKEN}" \ + --rate 5 \ + --duration 60 \ + logs multi +``` + +For the equivalent trace and metrics commands, and a walkthrough of the other `otelgen` subcommands, see [Synthetic data with otelgen](/products/clickstack/example-datasets/otelgen). + + + +## Confirm in the ClickStack UI {#confirm-in-ui} + +Open your service in the [ClickHouse Cloud console](https://console.clickhouse.cloud) and select **ClickStack** from the left menu. + +Launch ClickStack + +You should be automatically directed to the **Getting Started** page. If not, select **Getting Started** from the left-hand menu, then click **Start Ingestion**. + +ClickStack Start Ingestion + +The next step can be skipped, as you've already configured your collector. Click **Next** to continue. + +ClickStack should automatically detect your tables and telemetry data, allowing you to proceed. Select **Start Exploring** to begin exploring your trace data. + +ClickStack Start Exploring + +Switch the source to `Logs` and set the time range to **Last 15 minutes**. The synthetic logs from `otelgen` should appear within a few seconds. + +ClickStack Search view with logs appearing + +If nothing shows up: + +- Confirm the `OTLP_AUTH_TOKEN` value passed to `otelgen` matches the one set on the collector. +- Tail the collector logs with `docker logs -f ` and look for export errors. +- Verify the `CLICKHOUSE_ENDPOINT` includes both the protocol and port (`https://...:8443`). + + + +## Further reading {#further-reading} + +This guide covers a single collector instance in its simplest form. The [OpenTelemetry collector reference](/products/clickstack/ingesting-data/collector) covers what to do next: + +- [Securing the collector](/products/clickstack/ingesting-data/collector#securing-the-collector) with TLS on the OTLP endpoint and least-privilege ingestion users. +- [Processing, filtering, and enriching](/products/clickstack/ingesting-data/collector#processing-filtering-transforming-enriching) events at the gateway. +- [Extending the collector configuration](/products/clickstack/ingesting-data/collector#extending-collector-config) with custom receivers, processors, and pipelines. +- [Estimating resources](/products/clickstack/ingesting-data/collector#estimating-resources) for gateway and agent deployments at your expected throughput. +- [Going to production](/products/clickstack/managing/production) for recommendations when going to production. + + + diff --git a/products/clickstack/navigation.json b/products/clickstack/navigation.json index eb640abf..09472a02 100644 --- a/products/clickstack/navigation.json +++ b/products/clickstack/navigation.json @@ -23,7 +23,8 @@ "products/clickstack/example-datasets/kubernetes", "products/clickstack/example-datasets/local-data", "products/clickstack/example-datasets/remote-demo-data", - "products/clickstack/example-datasets/session-replay" + "products/clickstack/example-datasets/session-replay", + "products/clickstack/example-datasets/otelgen" ] }, "products/clickstack/architecture",