Skip to content

docs: batching mode for Prometheus Remote Write protocol #2392

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
v0y4g3r wants to merge 3 commits into
base: main
Choose a base branch
from
+94 −6
Open

docs: batching mode for Prometheus Remote Write protocol #2392

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
07114fa
docs/prom-batching:
v0y4g3r Apr 7, 2026
4ac1cb0
docs: apply suggestion
v0y4g3r Apr 8, 2026
b0ed3d2
docs/prom-batching:
v0y4g3r Apr 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 13 additions & 3 deletions docs/user-guide/deployments-administration/configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -188,6 +188,11 @@ enable = true
[prom_store]
enable = true
with_metric_engine = true
pending_rows_flush_interval = "0s"
max_batch_rows = 100000
max_concurrent_flushes = 256
worker_channel_capacity = 65536
max_inflight_requests = 3000
```

The following table describes the options in detail:
Expand All @@ -212,9 +217,14 @@ The following table describes the options in detail:
| | enable | Boolean | Whether to enable InfluxDB protocol in HTTP API, true by default |
| opentsdb | | | OpenTSDB Protocol options |
| | enable | Boolean | Whether to enable OpenTSDB protocol in HTTP API, true by default |
| prom_store | | | Prometheus remote storage options |
| | enable | Boolean | Whether to enable Prometheus Remote Write and read in HTTP API, true by default |
| | with_metric_engine | Boolean | Whether to use the metric engine on Prometheus Remote Write, true by default |
| prom_store | | | Prometheus remote storage options |
| | enable | Boolean | Whether to enable Prometheus Remote Write and read in HTTP API, true by default |
| | with_metric_engine | Boolean | Whether to use the metric engine on Prometheus Remote Write, true by default |
| | pending_rows_flush_interval | String | Interval between batch flushes for Prometheus Remote Write. Set to a non-zero duration (e.g. `"500ms"`) to enable [batching mode](/user-guide/ingest-data/for-observability/prometheus.md#batching-mode). `"0s"` by default (disabled) |
| | max_batch_rows | Integer | Maximum number of rows per batch before a flush is triggered, 100000 by default |
| | max_concurrent_flushes | Integer | Maximum number of concurrent flush operations, 256 by default |
| | worker_channel_capacity | Integer | Capacity of the internal worker channel for receiving rows, 65536 by default |
| | max_inflight_requests | Integer | Maximum number of in-flight write requests waiting for batch completion, 3000 by default |
| postgres | | | PostgresSQL server options |
| | enable | Boolean | Whether to enable PostgresSQL protocol, true by default |
| | addr | String | Server address, "127.0.0.1:4003" by default |
Expand Down
34 changes: 34 additions & 0 deletions docs/user-guide/ingest-data/for-observability/prometheus.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -307,6 +307,40 @@ You can refer to the [pipeline's documentation](/user-guide/logs/use-custom-pipe

## Performance tuning

### Batching mode

When the metric engine is enabled, GreptimeDB supports a batching mode for Prometheus Remote Write
that reduces per-request overhead and improves ingestion throughput.
In batching mode, incoming rows are accumulated and flushed to the metric engine in larger batches,
which can yield up to **2x throughput improvement** in clustered deployments with multiple regions,
but requires careful tuning of batch parameters to match the ingestion workload and prevent increased
latency.

Batching mode is **disabled by default**.
To enable it, set `pending_rows_flush_interval` to a non-zero duration in the `[prom_store]` section of the configuration file:

```toml
[prom_store]
enable = true
with_metric_engine = true
pending_rows_flush_interval = "500ms"
```

The following table describes the batching-related options:

| Key | Type | Default | Description |
| ---------------------------- | ------- | -------- | --------------------------------------------------------------------------- |
| pending_rows_flush_interval | String | `"0s"` | Interval between batch flushes. `"0s"` disables batching. |
| max_batch_rows | Integer | `100000` | Maximum number of rows per batch before a flush is triggered. |
| max_concurrent_flushes | Integer | `256` | Maximum number of flush operations that can run concurrently. |
| max_inflight_requests | Integer | `3000` | Maximum number of in-flight write requests waiting for batch completion. |

:::tip
Batching mode only takes effect when both `with_metric_engine` is `true` and `pending_rows_flush_interval` is set to a non-zero duration.
:::

### Customized physical table

By default, the metric engine will automatically create a physical table named `greptime_physical_table` if it does not already exist. For performance optimization, you may choose to create a physical table with customized configurations.

### Enable skipping index
Expand Down
17 changes: 14 additions & 3 deletions ...gin-content-docs/current/user-guide/deployments-administration/configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -180,6 +180,12 @@ enable = true

[prom_store]
enable = true
with_metric_engine = true
pending_rows_flush_interval = "0s"
max_batch_rows = 100000
max_concurrent_flushes = 256
worker_channel_capacity = 65536
max_inflight_requests = 3000
```

下表描述了每个选项的详细信息：
Expand All @@ -204,9 +210,14 @@ enable = true
| | enable | 布尔值 | 是否在 HTTP API 中启用 InfluxDB 协议，默认为 true |
| opentsdb | | | OpenTSDB 协议选项 |
| | enable | 布尔值 | 是否启用 OpenTSDB 协议，默认为 true |
| prom_store | | | Prometheus 远程存储选项 |
| | enable | 布尔值 | 是否在 HTTP API 中启用 Prometheus 远程读写，默认为 true |
| | with_metric_engine | 布尔值 | 是否在 Prometheus 远程写入中使用 Metric Engine，默认为 true |
| prom_store | | | Prometheus 远程存储选项 |
| | enable | 布尔值 | 是否在 HTTP API 中启用 Prometheus 远程读写，默认为 true |
| | with_metric_engine | 布尔值 | 是否在 Prometheus 远程写入中使用 Metric Engine，默认为 true |
| | pending_rows_flush_interval | 字符串 | Prometheus Remote Write 批量刷写的时间间隔。设为非零值（如 `"500ms"`）以启用[批量写入模式](/user-guide/ingest-data/for-observability/prometheus.md#批量写入模式)，默认为 `"0s"`（禁用） |
| | max_batch_rows | 整数 | 触发刷写的最大批量行数，默认为 100000 |
| | max_concurrent_flushes | 整数 | 同时执行的最大刷写操作数量，默认为 256 |
| | worker_channel_capacity | 整数 | 内部接收行数据的 worker 通道容量，默认为 65536 |
| | max_inflight_requests | 整数 | 等待批量完成的最大请求数，默认为 3000 |
| postgres | | | PostgresSQL 服务器选项 |
| | enable | 布尔值 | 是否启用 PostgresSQL 协议，默认为 true |
| | addr | 字符串 | 服务器地址，默认为 "127.0.0.1:4003" |
Expand Down
33 changes: 33 additions & 0 deletions ...gin-content-docs/current/user-guide/ingest-data/for-observability/prometheus.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -293,6 +293,39 @@ mysql> select * from `go_memstats_mcache_inuse_bytes`;

## 性能优化

### 批量写入模式

当 metric engine 启用时，GreptimeDB 支持 Prometheus Remote Write 的批量写入模式，
通过减少单次请求的开销来提高写入吞吐量。
在批量写入模式下，传入的行数据会被累积并以更大的批次刷写到 metric engine 中，
在多 region 的集群部署中可以带来最高 **2 倍的吞吐量提升**，但是需要仔细调整攒批相关的参数以避免写入延迟增加。

批量写入模式**默认关闭**。
要启用它，请在配置文件的 `[prom_store]` 部分将 `pending_rows_flush_interval` 设置为非零时间间隔：

```toml
[prom_store]
enable = true
with_metric_engine = true
pending_rows_flush_interval = "500ms"
```

下表描述了批量写入相关的配置选项：

| 键 | 类型 | 默认值 | 描述 |
| ---------------------------- | ------ | --------- | -------------------------------------------------------------------- |
| pending_rows_flush_interval | 字符串 | `"0s"` | 批量刷写的时间间隔。`"0s"` 表示禁用批量写入模式。 |
| max_batch_rows | 整数 | `100000` | 触发刷写的最大批量行数。 |
| max_concurrent_flushes | 整数 | `256` | 同时执行的最大刷写操作数量。 |
| worker_channel_capacity | 整数 | `65536` | 内部接收行数据的 worker 通道容量。 |
| max_inflight_requests | 整数 | `3000` | 等待批量完成的最大请求数。 |

:::tip
批量写入模式仅在 `with_metric_engine` 为 `true` 且 `pending_rows_flush_interval` 设置为非零时间间隔时生效。
:::

### 自定义物理表

默认情况下，metric engine 会自动创建一个名为 `greptime_physical_table` 的物理表。
为了优化性能，你可以选择创建一个具有自定义配置的物理表。

Expand Down
Loading