feat: introduce deletion timestamp metric for daemonset, statefulset, deployment, service and pdb by IgorIgnatevBolt · Pull Request #2678 · kubernetes/kube-state-metrics

IgorIgnatevBolt · 2025-06-02T11:42:36Z

What this PR does / why we need it:

Some resources can be blocked by deletion from finalizers. To catch this and expose it to metrics, we can use the deletion timestamp metadata field.
Introduce a deletion_timestamp metric for the next resources:

deployment kube_deployment_deletion_timestamp
statefulset kube_statefulset_deletion_timestamp
daemonset kube_daemonset_deletion_timestamp
service kube_service_deletion_timestamp
poddisruptionbudget kube_poddisruptionbudget_deletion_timestamp

Also formatting tables in docs

How does this change affect the cardinality of KSM: (increases, decreases or does not change cardinality)

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

IgorIgnatevBolt · 2025-06-03T16:20:08Z

All commits were squashed into one.

CatherineF-dev · 2025-06-20T00:23:06Z

Hi, could you share more insights on use cases after these metrics are added?

Is it used for monitoring Kubernetes resources that are stuck in a terminating state?

IgorIgnatevBolt · 2025-06-20T05:18:04Z

@CatherineF-dev Hi, yes, if the resource deletion process is stuck for some reason or blocked by the finalizer, deletiontimestamp metric can help to detect such a case and raise an alert for investigation.

richabanker · 2025-06-26T16:50:21Z

/assign

CatherineF-dev · 2025-07-11T12:32:19Z

@IgorIgnatevBolt How will we know which resource should be deleted?

IgorIgnatevBolt · 2025-07-11T12:44:05Z

@IgorIgnatevBolt How will we know which resource should be deleted?

Maybe I misunderstood the question, but this PR is exactly about detection for such resources that were nominated by the controller manager for deletion but not deleted for some reason, eq blocked by finalizers

The controller managing that finalizer notices the update to the object setting the metadata.deletionTimestamp, indicating deletion of the object has been requested.

IgorIgnatevBolt · 2025-07-30T07:19:41Z

Hi @CatherineF-dev, do you need any more information about PR or anything else that can help you move forward?

dgrisonnet · 2025-08-07T16:59:33Z

/assign @CatherineF-dev
/triage accepted

CatherineF-dev · 2025-08-11T14:29:11Z

 | kube_deployment_labels                                      | Gauge       | Kubernetes labels converted to Prometheus labels controlled via [--metric-labels-allowlist](../../developer/cli-arguments.md)           | `deployment`=&lt;deployment-name&gt; <br> `namespace`=&lt;deployment-namespace&gt; <br> `label_DEPLOYMENT_LABEL`=&lt;DEPLOYMENT_LABEL&gt;                                   | STABLE       |
-| kube_deployment_created                                     | Gauge       |                                                                                                                           | `deployment`=&lt;deployment-name&gt; <br> `namespace`=&lt;deployment-namespace&gt;                                                                                          | STABLE       |
+| kube_deployment_created                                     | Gauge       |                                                                                                                                         | `deployment`=&lt;deployment-name&gt; <br> `namespace`=&lt;deployment-namespace&gt;                                                                                          | STABLE       |
+| kube_deployment_deletion_timestamp                          | Gauge       | Unix deletion timestamp                                                                                                                 | `deployment`=&lt;deployment-name&gt; <br> `namespace`=&lt;deployment-namespace&gt;                                                                                          | EXPIREMENTAL |


Should we use kube_deployment_deleted to align with kube_deployment_created?

I'd like to keep the pattern the same as for other resources like kube_node_deletion_timestamp or kube_pod_deletion_timestamp

@richabanker how do you think?

ah so its kube_deployment_created thats not following the _timestamp pattern :/

Looks like we have consciously made the switch to use _timestamp in the past so maybe kube_deployment_deletion_timestamp is the way to go.

Additionally does it make sense to rename the kube_deployment_created to also follow the same? How disruptive is that change?

I guess renaming any existing metrics would be a breaking change and require a longer release process. I'd like to keep the scope of the current PR only in the current state, with new metrics only.

IgorIgnatevBolt · 2025-08-15T06:38:18Z

rebased, conflicts solved

This commit adds the kube_*_deletion_timestamp metric for several Kubernetes resources: - Deployments - DaemonSets - StatefulSets - Services - PodDisruptionBudgets The deletion timestamp metric reports the Unix timestamp when a resource was marked for deletion. This helps with monitoring resource lifecycle and cleanup processes. All metrics follow the same pattern: - Help text: 'Unix deletion timestamp' - Type: gauge - Value: Unix timestamp in seconds when DeletionTimestamp is set, otherwise the metric is not emitted Updated documentation and tests are included for all affected resources.

IgorIgnatevBolt · 2025-08-18T05:58:21Z

fixed typo in the depl test

mrueg · 2025-08-20T17:55:19Z

/hold

for @CatherineF-dev to further comment.
Renamed the title to include all additional resources in there.

/lgtm

k8s-ci-robot · 2025-08-20T17:55:28Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: IgorIgnatevBolt, mrueg

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [mrueg]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

mrueg · 2025-08-25T05:39:46Z

/hold cancel

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 2, 2025

k8s-ci-robot requested review from logicalhan and mrueg June 2, 2025 11:42

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jun 2, 2025

IgorIgnatevBolt marked this pull request as ready for review June 2, 2025 11:43

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 2, 2025

k8s-ci-robot requested review from CatherineF-dev and dgrisonnet June 2, 2025 11:43

IgorIgnatevBolt force-pushed the feat-deletion-timestamp-resources branch from 63191f9 to d5bb362 Compare June 3, 2025 16:10

k8s-ci-robot assigned richabanker Jun 26, 2025

k8s-ci-robot assigned CatherineF-dev Aug 7, 2025

k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Aug 7, 2025

CatherineF-dev reviewed Aug 11, 2025

View reviewed changes

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 14, 2025

IgorIgnatevBolt force-pushed the feat-deletion-timestamp-resources branch from d5bb362 to b188b21 Compare August 15, 2025 06:37

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 15, 2025

mrueg added this to the v2.17.0 milestone Aug 15, 2025

IgorIgnatevBolt force-pushed the feat-deletion-timestamp-resources branch from b188b21 to 53ec1de Compare August 18, 2025 05:57

mrueg changed the title ~~feat: introduce deletion timestamp metric for multiple resources~~ feat: introduce deletion timestamp metric for daemonset, statefulset, deployment, service and pdb Aug 20, 2025

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 20, 2025

k8s-ci-robot assigned mrueg Aug 20, 2025

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 20, 2025

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 20, 2025

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 25, 2025

k8s-ci-robot merged commit 0641627 into kubernetes:main Aug 25, 2025
13 checks passed

Conversation

IgorIgnatevBolt commented Jun 2, 2025

Uh oh!

IgorIgnatevBolt commented Jun 3, 2025

Uh oh!

CatherineF-dev commented Jun 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

IgorIgnatevBolt commented Jun 20, 2025

Uh oh!

richabanker commented Jun 26, 2025

Uh oh!

CatherineF-dev commented Jul 11, 2025

Uh oh!

IgorIgnatevBolt commented Jul 11, 2025

Uh oh!

IgorIgnatevBolt commented Jul 30, 2025

Uh oh!

dgrisonnet commented Aug 7, 2025

Uh oh!

CatherineF-dev Aug 11, 2025

Choose a reason for hiding this comment

Uh oh!

IgorIgnatevBolt Aug 12, 2025

Choose a reason for hiding this comment

Uh oh!

CatherineF-dev Aug 13, 2025

Choose a reason for hiding this comment

Uh oh!

richabanker Aug 14, 2025

Choose a reason for hiding this comment

Uh oh!

IgorIgnatevBolt Aug 15, 2025

Choose a reason for hiding this comment

Uh oh!

IgorIgnatevBolt commented Aug 15, 2025

Uh oh!

IgorIgnatevBolt commented Aug 18, 2025

Uh oh!

mrueg commented Aug 20, 2025

Uh oh!

k8s-ci-robot commented Aug 20, 2025

Uh oh!

mrueg commented Aug 25, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

CatherineF-dev commented Jun 20, 2025 •

edited

Loading