Vault Agent - Self-DDOS due to failure on rendering 1 template causing all templates to be rerendered #26283

F21 · 2024-04-04T23:29:38Z

Describe the bug
We have Vault Agent configured to render multiple templates, with each template retrieving certain secrets from Vault using the {{- with secret "xxx" }} function. If one template fails to retrieve a secret (perhaps due to a permissions/policy issues), it triggers a retry. At the end of the retry, the runner restarts, forcing all secrets for all templates to be retrieved again and rerendered. This goes into a loop, which effective DDoses our Vault server.

To Reproduce
Steps to reproduce the behavior:

Set up vault agent's config to render multiple templates:

vault {
  address = "https://localhost:8200"
  retry {
    num_retries = 5
  }
}

auto_auth {
  method {
    type = "token_file"
    config = {
      token_file_path = "/home/username/.vault-token"
    }
  }
}

cache {}

template {
  source      = "/etc/vault.d/agent/template1.ctmpl"
  destination = "/template1.crt"
}

template {
  source      = "/etc/vault.d/agent/template2.ctmpl"
  destination = "/template2.crt"
}

Set up templates, with 1 pointing to a secret that is not accessible:

{{ with secret "lets-encrypt/certs/company.com" "common_name=company.com" }}
{{ .Data.cert }}
{{ end }}

{{ with secret "lets-encrypt/certs/doesnotexist.com" "common_name=company.com" }}
{{ .Data.cert }}
{{ end }}

Run Vault Agent and see that it restarts the runner and tries to rerequest all secrets and rerender all templates if one template fails.

Expected behavior
It would be nice if the runner does not restart and rerender all templates, but only retries the failed templates.

Environment:

Vault Server Version (retrieve with vault status): 1.16.0
Vault CLI Version (retrieve with vault version): 1.16.0
Server Operating System/Architecture: Ubuntu 22.04 amd_64

Vault server configuration file(s):
N/a

Additional context
N/a

The text was updated successfully, but these errors were encountered:

kingindanord · 2024-04-09T08:08:01Z

in my case, it keeps retry to render until my VM is out of memory.

VioletHynes · 2024-04-18T12:59:11Z

Hi there! What version of Agent are you using? There's a chance that this bug was fixed by this PR: #25497

Previous to that PR, there was no backoff between the retry when the template server had to be restarted. This adds exponential backoff. It'd be helpful if there were logs to show the time around the retry, as if the template server restarts due to a permissions issue as you claim, that'd be something we'd love to fix.

F21 · 2024-04-18T22:32:11Z

I just tested it using Vault v1.16.1 and can confirm this is still an issue.

Given the following templates:
Template 1:

{{- with secret "secret/secret1" }}
secret2={{ .Data.data.key }}
{{- end }}

Template 2:

{{- with secret "aws/static-creds/test" }}
secret1={{ .Data.data.key }}
{{- end }}

If the token used by vault agent has access to aws/static-creds/test but not secret/secret1 via its attached policies, the agent will retry exponentially and then restart once the threshold has been met:

==> Vault Agent started! Log data will stream in below:

==> Vault Agent configuration:

           Api Address 1: http://bufconn
                     Cgo: disabled
               Log Level:
                 Version: Vault v1.16.1, built 2024-04-03T12:35:53Z
             Version Sha: 6b5986790d7748100de77f7f127119c4a0f78946

2024-04-19T08:36:34.423+1000 [INFO]  agent.exec.server: starting exec server
2024-04-19T08:36:34.423+1000 [INFO]  agent.exec.server: no env templates or exec config, exiting
2024-04-19T08:36:34.423+1000 [INFO]  agent.auth.handler: starting auth handler
2024-04-19T08:36:34.423+1000 [INFO]  agent.auth.handler: authenticating
2024-04-19T08:36:34.423+1000 [INFO]  agent.sink.server: starting sink server
2024-04-19T08:36:34.423+1000 [INFO]  agent.template.server: starting template server
2024-04-19T08:36:34.423+1000 [INFO]  agent: (runner) creating new runner (dry: false, once: false)
2024-04-19T08:36:34.423+1000 [INFO]  agent: (runner) creating watcher
2024-04-19T08:36:34.424+1000 [INFO]  agent.auth.handler: authentication successful, sending token to sinks
2024-04-19T08:36:34.424+1000 [INFO]  agent.auth.handler: starting renewal process
2024-04-19T08:36:34.424+1000 [INFO]  agent.template.server: template server received new token
2024-04-19T08:36:34.424+1000 [INFO]  agent: (runner) stopping
2024-04-19T08:36:34.424+1000 [INFO]  agent: (runner) creating new runner (dry: false, once: false)
2024-04-19T08:36:34.424+1000 [INFO]  agent: (runner) creating watcher
2024-04-19T08:36:34.424+1000 [INFO]  agent: (runner) starting
2024-04-19T08:36:34.424+1000 [INFO]  agent.auth.handler: renewed auth token
2024-04-19T08:36:34.425+1000 [WARN]  agent: (view) vault.read(secret/secret1): vault.read(secret/secret1): Error making API request.

URL: GET http://127.0.0.1:8200/v1/secret/data/secret1
Code: 403. Errors:

* 1 error occurred:
        * permission denied

 (retry attempt 1 after "250ms")
2024-04-19T08:36:34.427+1000 [INFO]  agent: (runner) rendered "./test2.tmpl" => "./test2.cfg"
2024-04-19T08:36:34.681+1000 [WARN]  agent: (view) vault.read(secret/secret1): vault.read(secret/secret1): Error making API request.

URL: GET http://127.0.0.1:8200/v1/secret/data/secret1
Code: 403. Errors:

* 1 error occurred:
        * permission denied

 (retry attempt 2 after "500ms")
2024-04-19T08:36:35.184+1000 [WARN]  agent: (view) vault.read(secret/secret1): vault.read(secret/secret1): Error making API request.

URL: GET http://127.0.0.1:8200/v1/secret/data/secret1
Code: 403. Errors:

* 1 error occurred:
        * permission denied

 (retry attempt 3 after "1s")
2024-04-19T08:36:36.188+1000 [WARN]  agent: (view) vault.read(secret/secret1): vault.read(secret/secret1): Error making API request.

URL: GET http://127.0.0.1:8200/v1/secret/data/secret1
Code: 403. Errors:

* 1 error occurred:
        * permission denied

 (retry attempt 4 after "2s")
2024-04-19T08:36:38.192+1000 [WARN]  agent: (view) vault.read(secret/secret1): vault.read(secret/secret1): Error making API request.

URL: GET http://127.0.0.1:8200/v1/secret/data/secret1
Code: 403. Errors:

* 1 error occurred:
        * permission denied

 (retry attempt 5 after "4s")
2024-04-19T08:36:42.197+1000 [WARN]  agent: (view) vault.read(secret/secret1): vault.read(secret/secret1): Error making API request.

URL: GET http://127.0.0.1:8200/v1/secret/data/secret1
Code: 403. Errors:

* 1 error occurred:
        * permission denied

 (retry attempt 6 after "8s")
2024-04-19T08:36:50.200+1000 [WARN]  agent: (view) vault.read(secret/secret1): vault.read(secret/secret1): Error making API request.

URL: GET http://127.0.0.1:8200/v1/secret/data/secret1
Code: 403. Errors:

* 1 error occurred:
        * permission denied

 (retry attempt 7 after "16s")
2024-04-19T08:37:06.201+1000 [WARN]  agent: (view) vault.read(secret/secret1): vault.read(secret/secret1): Error making API request.

URL: GET http://127.0.0.1:8200/v1/secret/data/secret1
Code: 403. Errors:

* 1 error occurred:
        * permission denied

 (retry attempt 8 after "32s")
2024-04-19T08:37:38.207+1000 [WARN]  agent: (view) vault.read(secret/secret1): vault.read(secret/secret1): Error making API request.

URL: GET http://127.0.0.1:8200/v1/secret/data/secret1
Code: 403. Errors:

* 1 error occurred:
        * permission denied

 (retry attempt 9 after "1m0s")
2024-04-19T08:38:38.209+1000 [WARN]  agent: (view) vault.read(secret/secret1): vault.read(secret/secret1): Error making API request.

URL: GET http://127.0.0.1:8200/v1/secret/data/secret1
Code: 403. Errors:

* 1 error occurred:
        * permission denied

 (retry attempt 10 after "1m0s")
2024-04-19T08:39:38.210+1000 [WARN]  agent: (view) vault.read(secret/secret1): vault.read(secret/secret1): Error making API request.

URL: GET http://127.0.0.1:8200/v1/secret/data/secret1
Code: 403. Errors:

* 1 error occurred:
        * permission denied

 (retry attempt 11 after "1m0s")
2024-04-19T08:40:38.213+1000 [WARN]  agent: (view) vault.read(secret/secret1): vault.read(secret/secret1): Error making API request.

URL: GET http://127.0.0.1:8200/v1/secret/data/secret1
Code: 403. Errors:

* 1 error occurred:
        * permission denied

 (retry attempt 12 after "1m0s")
2024-04-19T08:41:38.219+1000 [ERROR] agent: (view) vault.read(secret/secret1): vault.read(secret/secret1): Error making API request.

URL: GET http://127.0.0.1:8200/v1/secret/data/secret1
Code: 403. Errors:

* 1 error occurred:
        * permission denied

 (exceeded maximum retries)
2024-04-19T08:41:38.219+1000 [ERROR] agent: (runner) watcher reported error: vault.read(secret/secret1): vault.read(secret/secret1): Error making API request.

URL: GET http://127.0.0.1:8200/v1/secret/data/secret1
Code: 403. Errors:

* 1 error occurred:
        * permission denied
2024-04-19T08:41:38.219+1000 [ERROR] agent.template.server: template server error:
  error=
  | vault.read(secret/secret1): vault.read(secret/secret1): Error making API request.
  |
  | URL: GET http://127.0.0.1:8200/v1/secret/data/secret1
  | Code: 403. Errors:
  |
  | * 1 error occurred:
  | \t* permission denied
  |

2024-04-19T08:41:38.220+1000 [INFO]  agent: (runner) stopping
2024-04-19T08:41:38.220+1000 [INFO]  agent: (runner) creating new runner (dry: false, once: false)
2024-04-19T08:41:38.220+1000 [INFO]  agent: (runner) creating watcher
2024-04-19T08:41:38.220+1000 [INFO]  agent: (runner) starting
2024-04-19T08:41:38.225+1000 [WARN]  agent: (view) vault.read(secret/secret1): vault.read(secret/secret1): Error making API request.

URL: GET http://127.0.0.1:8200/v1/secret/data/secret1
Code: 403. Errors:

* 1 error occurred:
        * permission denied

 (retry attempt 1 after "250ms")
2024-04-19T08:41:38.480+1000 [WARN]  agent: (view) vault.read(secret/secret1): vault.read(secret/secret1): Error making API request.

URL: GET http://127.0.0.1:8200/v1/secret/data/secret1
Code: 403. Errors:

* 1 error occurred:
        * permission denied

 (retry attempt 2 after "500ms")
2024-04-19T08:41:38.984+1000 [WARN]  agent: (view) vault.read(secret/secret1): vault.read(secret/secret1): Error making API request.

URL: GET http://127.0.0.1:8200/v1/secret/data/secret1
Code: 403. Errors:

* 1 error occurred:
        * permission denied

 (retry attempt 3 after "1s")
2024-04-19T08:41:39.990+1000 [WARN]  agent: (view) vault.read(secret/secret1): vault.read(secret/secret1): Error making API request.

URL: GET http://127.0.0.1:8200/v1/secret/data/secret1
Code: 403. Errors:

* 1 error occurred:
        * permission denied

 (retry attempt 4 after "2s")
2024-04-19T08:41:41.996+1000 [WARN]  agent: (view) vault.read(secret/secret1): vault.read(secret/secret1): Error making API request.

URL: GET http://127.0.0.1:8200/v1/secret/data/secret1
Code: 403. Errors:

* 1 error occurred:
        * permission denied

 (retry attempt 5 after "4s")
2024-04-19T08:41:46.000+1000 [WARN]  agent: (view) vault.read(secret/secret1): vault.read(secret/secret1): Error making API request.

URL: GET http://127.0.0.1:8200/v1/secret/data/secret1
Code: 403. Errors:

* 1 error occurred:
        * permission denied

 (retry attempt 6 after "8s")
2024-04-19T08:41:54.006+1000 [WARN]  agent: (view) vault.read(secret/secret1): vault.read(secret/secret1): Error making API request.

URL: GET http://127.0.0.1:8200/v1/secret/data/secret1
Code: 403. Errors:

* 1 error occurred:
        * permission denied

 (retry attempt 7 after "16s")
2024-04-19T08:42:10.012+1000 [WARN]  agent: (view) vault.read(secret/secret1): vault.read(secret/secret1): Error making API request.

URL: GET http://127.0.0.1:8200/v1/secret/data/secret1
Code: 403. Errors:

* 1 error occurred:
        * permission denied

 (retry attempt 8 after "32s")
2024-04-19T08:42:42.017+1000 [WARN]  agent: (view) vault.read(secret/secret1): vault.read(secret/secret1): Error making API request.

URL: GET http://127.0.0.1:8200/v1/secret/data/secret1
Code: 403. Errors:

* 1 error occurred:
        * permission denied

 (retry attempt 9 after "1m0s")
2024-04-19T08:43:42.023+1000 [WARN]  agent: (view) vault.read(secret/secret1): vault.read(secret/secret1): Error making API request.

URL: GET http://127.0.0.1:8200/v1/secret/data/secret1
Code: 403. Errors:

* 1 error occurred:
        * permission denied

 (retry attempt 10 after "1m0s")
2024-04-19T08:44:42.029+1000 [WARN]  agent: (view) vault.read(secret/secret1): vault.read(secret/secret1): Error making API request.

URL: GET http://127.0.0.1:8200/v1/secret/data/secret1
Code: 403. Errors:

* 1 error occurred:
        * permission denied

 (retry attempt 11 after "1m0s")
^C==> Vault Agent shutdown triggered
2024-04-19T08:44:59.769+1000 [INFO]  agent: (runner) stopping
2024-04-19T08:44:59.769+1000 [INFO]  agent.exec.server: exec server stopped
2024-04-19T08:44:59.769+1000 [INFO]  agent.sink.server: sink server stopped
2024-04-19T08:44:59.769+1000 [INFO]  agent: sinks finished, exiting
2024-04-19T08:44:59.769+1000 [INFO]  agent.template.server: template server stopped
2024-04-19T08:44:59.769+1000 [INFO]  agent.auth.handler: shutdown triggered, stopping lifetime watcher
2024-04-19T08:44:59.769+1000 [INFO]  agent.auth.handler: auth handler stopped

Because the read to aws/static-creds/test works, everytime the template server restarts, new credentials are issued again, causing AWS creds to be issued indefinitely, due to the failure to read another secret (secret/secret1). If the source backing the secret engine has a limit to the number of available calls per day, this is quickly exhausted.

divyaac · 2024-08-19T20:44:35Z

Hi @F21 , thank you so much for your response. Unfortunately, the way that a new consul-template runner is initialize prevents us from "removing" and "adding" a new template, because there are dependencies that are created between templates that allow it to be accessible from other templates. For that reason, we have to restart the template with all of the configs.

heatherezell added bug Used to indicate a potential bug agent labels Apr 17, 2024

divyaac added the waiting-for-response label Apr 18, 2024

divyaac closed this as completed Aug 19, 2024

divyaac added the wontfix label Aug 19, 2024

github-actions bot removed the waiting-for-response label Aug 19, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Vault Agent - Self-DDOS due to failure on rendering 1 template causing all templates to be rerendered #26283

Vault Agent - Self-DDOS due to failure on rendering 1 template causing all templates to be rerendered #26283

F21 commented Apr 4, 2024

kingindanord commented Apr 9, 2024

VioletHynes commented Apr 18, 2024

F21 commented Apr 18, 2024 •

edited

Loading

divyaac commented Aug 19, 2024

Vault Agent - Self-DDOS due to failure on rendering 1 template causing all templates to be rerendered #26283

Vault Agent - Self-DDOS due to failure on rendering 1 template causing all templates to be rerendered #26283

Comments

F21 commented Apr 4, 2024

kingindanord commented Apr 9, 2024

VioletHynes commented Apr 18, 2024

F21 commented Apr 18, 2024 • edited Loading

divyaac commented Aug 19, 2024

F21 commented Apr 18, 2024 •

edited

Loading