Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -23,4 +23,6 @@ helm upgrade \
--values values.yaml \
suse-observability \
suse-observability/suse-observability
```
```

For more details, see [How much to add](https://l.stackstate.com/stackpack-suse-observability-how-much-to-add).
Original file line number Diff line number Diff line change
Expand Up @@ -19,4 +19,6 @@ helm upgrade \
--values values.yaml \
suse-observability \
suse-observability/suse-observability
```
```

For more details, see [How much to add](https://l.stackstate.com/stackpack-suse-observability-how-much-to-add).
Original file line number Diff line number Diff line change
@@ -1,2 +1,4 @@
ClickHouse has INSERT statements that are throttled due to high number of active data parts for partition in a MergeTree,
please decrease INSERT frequency https://clickhouse.com/docs/en/development/architecture/#merge-tree
please decrease INSERT frequency https://clickhouse.com/docs/en/development/architecture/#merge-tree

For more details, see [Traces are missing](https://l.stackstate.com/stackpack-suse-observability-traces-missing).
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,6 @@ ClickHouse has INSERT statements that are rejected due to high number of active
please decrease INSERT frequency:
- [MergeTreeArchitecture](https://clickhouse.com/docs/en/development/architecture/#merge-tree)
- [system.part_log](https://clickhouse.com/docs/en/operations/system-tables/part_log)
- [system.merge_tree_settings](https://clickhouse.com/docs/en/operations/system-tables/merge_tree_settings)
- [system.merge_tree_settings](https://clickhouse.com/docs/en/operations/system-tables/merge_tree_settings)

For more details, see [Traces are missing](https://l.stackstate.com/stackpack-suse-observability-traces-missing).
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,6 @@ When the latency of the transformation process is high we might have components
not present or arriving late.

### Investigate
There are a few potential causes of the high latency. To identify the root cause, please contact SUSE Observability [support](https://www.stackstate.com/company/contact-us/).
There are a few potential causes of the high latency. To identify the root cause, please contact SUSE Observability [support](https://www.stackstate.com/company/contact-us/).

For more details, see [Process and connection data is delayed](https://l.stackstate.com/stackpack-suse-observability-process-connection-delayed).
Original file line number Diff line number Diff line change
Expand Up @@ -48,4 +48,6 @@ helm upgrade \
--values values.yaml \
suse-observability \
suse-observability/suse-observability
```
```

For more details, see [Health states are delayed](https://l.stackstate.com/stackpack-suse-observability-health-states-delayed).
Original file line number Diff line number Diff line change
Expand Up @@ -49,3 +49,5 @@ helm upgrade \
suse-observability \
suse-observability/suse-observability
```

For more details, see [Events are delayed](https://l.stackstate.com/stackpack-suse-observability-events-delayed).
Original file line number Diff line number Diff line change
Expand Up @@ -14,4 +14,6 @@ Potential Reasons:
- **Batch Size and Interval**: Adjust batch size and sending interval settings in your exporter configuration to manage data flow better. Smaller batches and shorter intervals might help with overloaded endpoints.
- **Queue Management**: Monitor metrics like `stackstate_otelcol_exporter_queue_size` and `stackstate_otelcol_exporter_queue_capacity`. If the queue is often full, consider increasing its capacity or adjusting batch sizes/intervals.

You may find more details by checking [logs](/#/components/{{ componentUrnForUrl }}#logs).
You may find more details by checking [logs](/#/components/{{ componentUrnForUrl }}#logs).

For more details, see [Metrics are delayed or missing](https://l.stackstate.com/stackpack-suse-observability-metrics-delayed).
Original file line number Diff line number Diff line change
Expand Up @@ -15,4 +15,6 @@ Potential Reasons:
- Other Potential Causes:
- **Bugs in the processor**: While less likely, consider checking for known issues or bugs in the specific processor you're using.

You may find more details by checking [logs](/#/components/{{ componentUrnForUrl }}#logs).
You may find more details by checking [logs](/#/components/{{ componentUrnForUrl }}#logs).

For more details, see [Metrics are delayed or missing](https://l.stackstate.com/stackpack-suse-observability-metrics-delayed).
Original file line number Diff line number Diff line change
Expand Up @@ -16,4 +16,6 @@ Common Reasons:
- **Connectivity problems**: Ensure the collector can establish and maintain a connection to the receiving endpoint (e.g., backend). Connectivity issues might prevent metrics delivery.
- **Firewall restrictions**: Check for any firewall rules that might block communication between the collector and the receiving endpoint.

You may find more details by checking [logs](/#/components/{{ componentUrnForUrl }}#logs).
You may find more details by checking [logs](/#/components/{{ componentUrnForUrl }}#logs).

For more details, see [Metrics are delayed or missing](https://l.stackstate.com/stackpack-suse-observability-metrics-delayed).
Original file line number Diff line number Diff line change
Expand Up @@ -14,4 +14,6 @@ Potential Reasons:
- **Batch Size and Interval**: Adjust batch size and sending interval settings in your exporter configuration to manage data flow better. Smaller batches and shorter intervals might help with overloaded backends.
- **Queue Management**: Monitor metrics like `stackstate_otelcol_exporter_queue_size` and `stackstate_otelcol_exporter_queue_capacity`. If the queue is often full, consider increasing its capacity or adjusting batch sizes/intervals.

You may find more details by checking [logs](/#/components/{{ componentUrnForUrl }}#logs).
You may find more details by checking [logs](/#/components/{{ componentUrnForUrl }}#logs).

For more details, see [Traces are missing](https://l.stackstate.com/stackpack-suse-observability-traces-missing).
Original file line number Diff line number Diff line change
Expand Up @@ -15,4 +15,6 @@ Potential Reasons:
- Other Potential Causes:
- **Bugs in the processor**: While less likely, consider checking for known issues or bugs in the specific processor you're using.

You may find more details by checking [logs](/#/components/{{ componentUrnForUrl }}#logs).
You may find more details by checking [logs](/#/components/{{ componentUrnForUrl }}#logs).

For more details, see [Traces are missing](https://l.stackstate.com/stackpack-suse-observability-traces-missing).
Original file line number Diff line number Diff line change
Expand Up @@ -15,4 +15,6 @@ Common Reasons:
- **Connectivity problems**: Ensure the collector can establish and maintain a connection to the receiving endpoint (e.g., backend). Connectivity issues might prevent span delivery.
- **Firewall restrictions**: Check for any firewall rules that might block communication between the collector and the receiving endpoint.

You may find more details by checking [logs](/#/components/{{ componentUrnForUrl }}#logs).
You may find more details by checking [logs](/#/components/{{ componentUrnForUrl }}#logs).

For more details, see [Traces are missing](https://l.stackstate.com/stackpack-suse-observability-traces-missing).
Original file line number Diff line number Diff line change
@@ -1,28 +1,34 @@
Element budget **nearly saturated for `{{ labels.element_type }}`**, has high unique element budget saturation level. When the budget is saturated data starts being dropped which translates in missing topology or metrics.
The unique element budget for `{{ labels.element_type }}` is **nearly saturated**. When the budget is saturated, the receiver drops data for that element type, which shows up as missing topology or connections.

### Increase the budget of elements allowed
1. We can control the budget by setting extra environment variables for the receiver pod.
Here are the default settings:
Each budget is a value per agent multiplied by the number of agents, so it grows when you add agents.

### Check for limited agents
The sizing profile limits how many nodes you can observe. Agents above that limit are marked as limited. The receiver rejects their process, container and connection data, and they do not add to the budgets. Raising a budget does not help limited agents.

1. Run `sts agent list`. The last line shows how many agents are active, limited and stale.
2. If any agents are limited, move to the next sizing profile. A profile change can change volume sizes and the Kafka partition count, so it takes more than a `helm upgrade`.

### Raise the budget per agent
If no agents are limited, the agents send more elements than the default budget per agent allows, for example because they run many short-lived pods or open many connections.

1. Find the variable for the saturated element type:

| element type | environment variable |
|--------------|----------------------|
| component | `CONFIG_FORCE_stackstate_receiver_processAgent_maxComponentsPerHost` |
| relation | `CONFIG_FORCE_stackstate_receiver_processAgent_maxRelationsPerHost` |
| connection | `CONFIG_FORCE_stackstate_receiver_processAgent_maxConnectionObservationsPerHost` |

2. The current value per agent is `stackstate_stackstate_receiver_unique_element_passed_max` for this element type divided by `stackstate_stackstate_receiver_unique_element_passed_count{element_type="agent"}`. Raise it by 50% in your `values.yaml`, for example:
```
stackstate:
components:
receiver:
extraEnv:
open:
CONFIG_FORCE_stackstate_receiver_processAgent_maxAgents: "2000"
CONFIG_FORCE_stackstate_receiver_processAgent_maxComponents: "5000"
CONFIG_FORCE_stackstate_receiver_processAgent_maxConnections: "150000"
CONFIG_FORCE_stackstate_receiver_processAgent_maxRelations: "15000"
CONFIG_FORCE_stackstate_receiver_processAgent_maxComponentsPerHost: "450"
```

2. Identify which budget got saturated and increment the right environment variable to allow more elements to be passed:

| element type | environment variable |
|--------------|--------------------------------------------------------------|
| agent | CONFIG_FORCE_stackstate_receiver_processAgent_maxAgents |
| component | CONFIG_FORCE_stackstate_receiver_processAgent_maxComponents |
| connection | CONFIG_FORCE_stackstate_receiver_processAgent_maxConnections |
| relation | CONFIG_FORCE_stackstate_receiver_processAgent_maxRelations |

Keep in mind that allowing more elements to be processed will have an impact on downstream services such as the Correlator and the Topology Sync.
On HA profiles, set the variable under `stackstate.components.receiver.split.processAgent.extraEnv.open`.

3. Redeploy via a `helm upgrade` including any other values files or extra settings you applied previously
```
Expand All @@ -32,4 +38,8 @@ helm upgrade \
--values values.yaml \
suse-observability \
suse-observability/suse-observability
```
```

A higher budget increases the load on Correlate and Topology Sync, so watch their latency afterwards.

For more details, see [Receiver element budget saturated](https://l.stackstate.com/stackpack-suse-observability-receiver-element-budget).
Original file line number Diff line number Diff line change
@@ -1,24 +1,33 @@
Element budget **nearly saturated for `{{ labels.element_type }}`**, has high element hourly budget saturation level. When the budget is saturated data starts being dropped which translates in missing topology or metrics.
The hourly budget for new `{{ labels.element_type }}` elements is **nearly saturated**. When the budget is saturated, the receiver drops new elements of that type for the rest of the hour, which shows up as missing topology.

### Increase the budget of elements allowed
1. We can control the budget by setting extra environment variables for the receiver pod.
Here are the default settings:
Each budget is a value per agent multiplied by the number of agents, so it grows when you add agents.

### Check for limited agents
The sizing profile limits how many nodes you can observe. Agents above that limit are marked as limited. The receiver rejects their process, container and connection data, and they do not add to the budgets. Raising a budget does not help limited agents.

1. Run `sts agent list`. The last line shows how many agents are active, limited and stale.
2. If any agents are limited, move to the next sizing profile. A profile change can change volume sizes and the Kafka partition count, so it takes more than a `helm upgrade`.

### Raise the budget per agent
If no agents are limited, the agents create more new elements per hour than the default budget per agent allows, for example because they run many short-lived pods.

1. Find the variable for the saturated element type:

| element type | environment variable |
|--------------|----------------------|
| component | `CONFIG_FORCE_stackstate_receiver_processAgent_maxNewComponentsHourlyPerHost` |
| relation | `CONFIG_FORCE_stackstate_receiver_processAgent_maxNewRelationsHourlyPerHost` |

2. Raise the value by 50% in your `values.yaml`, for example:
```
stackstate:
components:
receiver:
extraEnv:
open:
CONFIG_FORCE_stackstate_receiver_processAgent_maxNewComponentsHourly: "2000"
CONFIG_FORCE_stackstate_receiver_processAgent_maxNewRelationsHourly: "5000"
CONFIG_FORCE_stackstate_receiver_processAgent_maxNewComponentsHourlyPerHost: "150"
```

2. Identify which budget got saturated and increment the right environment variable to allow more elements to be passed:

| element type | environment variable |
|--------------|--------------------------------------------------------------|
| component | CONFIG_FORCE_stackstate_receiver_processAgent_maxNewComponentsHourly |
| relation | CONFIG_FORCE_stackstate_receiver_processAgent_maxNewRelationsHourly |

Keep in mind that allowing more elements to be processed will have an impact on downstream services such as the Correlator and the Topology Sync.
On HA profiles, set the variable under `stackstate.components.receiver.split.processAgent.extraEnv.open`.

3. Redeploy via a `helm upgrade` including any other values files or extra settings you applied previously
```
Expand All @@ -28,4 +37,8 @@ helm upgrade \
--values values.yaml \
suse-observability \
suse-observability/suse-observability
```
```

A higher budget increases the load on Correlate and Topology Sync, so watch their latency afterwards.

For more details, see [Receiver element budget saturated](https://l.stackstate.com/stackpack-suse-observability-receiver-element-budget).
Original file line number Diff line number Diff line change
Expand Up @@ -8,4 +8,6 @@ This either means that there is a huge and sudden increase in load but much more
```
kubectl delete pod {{ labels.pod_name }} -n {{ labels.namespace }}
```
4. After restart check the same charts again for the next 10 minutes to make sure issue is fixed.
4. After restart check the same charts again for the next 10 minutes to make sure issue is fixed.

For more details, see [Receiver rejects data or requests time out](https://l.stackstate.com/stackpack-suse-observability-receiver-rejects-data).
Original file line number Diff line number Diff line change
Expand Up @@ -34,4 +34,6 @@ helm upgrade \
--values values.yaml \
suse-observability \
suse-observability/suse-observability
```
```

For more details, see [Receiver rejects data or requests time out](https://l.stackstate.com/stackpack-suse-observability-receiver-rejects-data).
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ SUSE Observability logs are rejected when one or more log sources (for example p

### Increase Disk Space for ElasticSearch
1. Verify the current ElasticSearch disk space and inspect its configuration to check the current size.
* Verify that the used storage class can be resized. For that please follow our [resizing documentation] https://docs.stackstate.com/self-hosted-setup/data-management/data_retention#resizing-storage
* Verify that the used storage class can be resized. See [Resizing storage](https://l.stackstate.com/stackpack-suse-observability-resizing-storage).
2. Update the values.yaml adding more
```
elasticsearch:
Expand All @@ -26,7 +26,7 @@ suse-observability \
suse-observability/suse-observability
```

For more detailed information visit our [data-retention](https://docs.stackstate.com/self-hosted-setup/data-management/data_retention) documentation.
For more detailed information, see [Data retention](https://l.stackstate.com/stackpack-suse-observability-data-retention).

### Decrease retention for logs
1. By default, SUSE Observability sets a 7-day retention for logs
Expand All @@ -47,4 +47,6 @@ helm upgrade \
--values values.yaml \
suse-observability \
suse-observability/suse-observability
```
```

For more details, see [Logs are missing](https://l.stackstate.com/stackpack-suse-observability-logs-missing).
Original file line number Diff line number Diff line change
Expand Up @@ -7,4 +7,6 @@ kubectl delete pod {{ labels.pod_name }} -n {{ labels.namespace }}
```
`Restarting the pod can help by either unblocking the issue or making it more evident in the logs`

2. Inspect the [Logs](/#/components/{{ componentUrnForUrl }}#logs) of the pod in order to find the root cause of the blockage.
2. Inspect the [Logs](/#/components/{{ componentUrnForUrl }}#logs) of the pod in order to find the root cause of the blockage.

For more details, see [Topology is delayed or components are missing](https://l.stackstate.com/stackpack-suse-observability-topology-delayed).
Original file line number Diff line number Diff line change
Expand Up @@ -53,4 +53,6 @@ suse-observability/suse-observability

### Reduce the frequency of the topology updates.
Update the settings of the agent collection of data for integration: `{{ labels.integration_type }}`,
Reducing the frequency of updates will remove pressure on the Topology Synchronization but as well means that the topology data will be less up-to-date as we don't refresh it that often.
Reducing the frequency of updates will remove pressure on the Topology Synchronization but as well means that the topology data will be less up-to-date as we don't refresh it that often.

For more details, see [Topology is delayed or components are missing](https://l.stackstate.com/stackpack-suse-observability-topology-delayed).
Original file line number Diff line number Diff line change
Expand Up @@ -53,4 +53,6 @@ suse-observability/suse-observability

### Reduce the frequency of the topology updates.
Update the settings of the agent collection of data for integration: `{{ labels.integration_type }}`,
Reducing the frequency of updates will remove pressure on the Topology Synchronization but as well means that the topology data will be less up-to-date as we don't refresh it that often.
Reducing the frequency of updates will remove pressure on the Topology Synchronization but as well means that the topology data will be less up-to-date as we don't refresh it that often.

For more details, see [Topology is delayed or components are missing](https://l.stackstate.com/stackpack-suse-observability-topology-delayed).
Original file line number Diff line number Diff line change
Expand Up @@ -31,4 +31,6 @@ helm upgrade \
--values values.yaml \
suse-observability \
suse-observability/suse-observability
```
```

For more details, see [Metrics are delayed or missing](https://l.stackstate.com/stackpack-suse-observability-metrics-delayed).
Original file line number Diff line number Diff line change
Expand Up @@ -26,4 +26,6 @@ suse-observability \
suse-observability/suse-observability
```

2b. Consider to limit the ingestion rate.
2b. Consider to limit the ingestion rate.

For more details, see [Metrics are delayed or missing](https://l.stackstate.com/stackpack-suse-observability-metrics-delayed).
Original file line number Diff line number Diff line change
Expand Up @@ -7,3 +7,5 @@ Check:
- `stackstate_vmagent_remotewrite_rate_limit_reached_total` if a remote-write rate limit is configured

Fix the downstream cause rather than growing the queue; a larger queue only delays the same loss.

For more details, see [Metrics are delayed or missing](https://l.stackstate.com/stackpack-suse-observability-metrics-delayed).
2 changes: 1 addition & 1 deletion stackpacks/suse-observability/stackpack.yaml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
name: suse-observability
version: "0.0.10"
version: "0.0.11"
schemaVersion: "2.0"
displayName: "SUSE Observability"
categories: [ "SUSE Observability" ]
Expand Down
Loading