Repeated GET requests to /v1/objects/services cause unbounded RSS growth in the icinga2 process due to glibc malloc arena fragmentation. Each request produces a large transient heap allocation (observed ~17x the wire transfer size), which is correctly freed by icinga2 at the application level, but never returned to the OS by glibc. Over repeated cycles the heap expands monotonically via brk(), leading to OOM kills on large installations.
This is not a memory leak — it is a direct consequence of the full-response-buffering architecture described in #10407.
To Reproduce
Prerequisite: ~15,000 service objects.
Execute repeated GET requests against /v1/objects/services:
curl -k -s -u "root:PASSWORD" -H 'Accept: application/json' -H 'X-HTTP-Method-Override: GET' -X POST "https://icinga2-master:5665/v1/objects/services"
Observe memory of the icinga2 service increasing with each call and never returning to baseline.
Expected behavior
Process RSS returns to approximately pre-request baseline after each response is transmitted and allocations are freed.
Enviroment Details
- Version:
icinga2 r2.15.1-1
- OS: RHEL 8.10
- Topology: Single-node master
- Objects at test time: 15,096 Services, 12 Hosts
Additional context
OOM kill confirmed via dmesg:
Out of memory: Killed process 3242359 (icinga2) total-vm:5841220kB, anon-rss:3453864kB, file-rss:0kB, UID:518
Confirmed mitigation for r2.15.1:
[Service] Environment="MALLOC_ARENA_MAX=1" Environment="MALLOC_TRIM_THRESHOLD_=0"
With these settings RSS reaches a stable high-water mark and oscillates around it rather than growing monotonically — confirmed across 500 consecutive requests.
The permanent fix is PR #10554 (targeting 2.16.0). We would request these allocator settings be backported and documented as an official mitigation for r2.15.1, given the OOM risk on production installations before 2.16.0 is widely available.
Repeated GET requests to /v1/objects/services cause unbounded RSS growth in the icinga2 process due to glibc malloc arena fragmentation. Each request produces a large transient heap allocation (observed ~17x the wire transfer size), which is correctly freed by icinga2 at the application level, but never returned to the OS by glibc. Over repeated cycles the heap expands monotonically via brk(), leading to OOM kills on large installations.
This is not a memory leak — it is a direct consequence of the full-response-buffering architecture described in #10407.
To Reproduce
Prerequisite: ~15,000 service objects.
Execute repeated GET requests against /v1/objects/services:
curl -k -s -u "root:PASSWORD" -H 'Accept: application/json' -H 'X-HTTP-Method-Override: GET' -X POST "https://icinga2-master:5665/v1/objects/services"Observe memory of the icinga2 service increasing with each call and never returning to baseline.
Expected behavior
Process RSS returns to approximately pre-request baseline after each response is transmitted and allocations are freed.
Enviroment Details
icinga2 r2.15.1-1Additional context
OOM kill confirmed via
dmesg:Out of memory: Killed process 3242359 (icinga2) total-vm:5841220kB, anon-rss:3453864kB, file-rss:0kB, UID:518
Confirmed mitigation for r2.15.1:
[Service] Environment="MALLOC_ARENA_MAX=1" Environment="MALLOC_TRIM_THRESHOLD_=0"With these settings RSS reaches a stable high-water mark and oscillates around it rather than growing monotonically — confirmed across 500 consecutive requests.
The permanent fix is PR #10554 (targeting 2.16.0). We would request these allocator settings be backported and documented as an official mitigation for r2.15.1, given the OOM risk on production installations before 2.16.0 is widely available.