consul-template as configured by webhook is unable to read vault token file

[forsberg]

Preflight Checklist

• I have searched the issue tracker for an issue that matches the one I want to file, without success.
• I am not looking for support or already pursued the available support channels without success.
• I agree to follow the Code of Conduct.

Vault Secrets Webhook Version

1.21.0

Installation Type

Official Helm chart

Bank-Vaults Version

No response

Kubernetes Version

1.23

Kubernetes Distribution/Provisioner

EKS

Expected Behavior

Deployment where annotations to enable consul-template were added should run consul template and retrieve secrets from vault, as it did on vault-secrets-webhook 0.17.0.

In 1.17.0, we saw the following two injected containers:

consul-template: hashicorp/consul-template:0.24.1-alpine
vault-agent: vault:1.6.2

Actual Behavior

gitlab-agent ran flawlessly, but consul-template only got 403 back from vault. On closer inspection, it turns out that /vault/.vault-token was owned by root, and could only be read by root.

As the new consul-template container used ran as uid 100, this made things not work.

On 1.21.0 we see the following two injected containers:

consul-template: hashicorp/consul-template:0.32.0
vault-agent: hashicorp/vault:1.14.1

Manual inspection shows that the consul-template container now runs as uid 100.

Steps To Reproduce

No response

Configuration

No response

Logs

│ consul-template 2024-01-19T15:51:27.589Z [WARN] (view) vault.read(database/creds/energycounter-timeperiod-review): vault.read(database/creds/energycounter- │
│ consul-template                                                                                                                                             │
│ consul-template URL: GET https://ext-vault.vault:8200/v1/database/creds/energycounter-timeperiod-review                                                     │
│ consul-template Code: 403. Errors:                                                                                                                          │
│ consul-template                                                                                                                                             │
│ consul-template * permission denied (retry attempt 10 after "1m0s")

Additional Information

Adding annotation:

vault.security.banzaicloud.io/run-as-user: "100"

Fixes the problem.

This should preferrably be part of the default configuration pushed by the webhook.

[ramizpolic]

Thanks for sharing this, but adding a default annotation for a specific use case could potentially block other users. I think if you could summarise these notes, we could add them to our docs page. @fekete-robert can assist here

[forsberg]

I understand the concern regarding changing defaults, but just to make things clear, with vault secrets webhook 1.21.0 and forward, due to changes in how the consul-template container runs (specifically, which USER the container is running under by default), using consul-template with vault secrets webhook is now broken out of the box, and requires setting that annotation in order to work at all.

I would lean towards the default settings for a functionality be such that it makes the functionality work.

[forsberg]

Still an issue.

[ramizpolic]

We currently dont have too much capacity due to work on this and other projects, but we will address it in the upcoming weeks. Thanks for your patience @forsberg!