GCP: Vault 1.4 crashes on start-up with KMS unseal

[jpds]

I use Vault on Google Cloud Platform with KMS auto-unseal and a cloud storage bucket for the backend. I also use the Vault Docker image.

This ran fine with Vault 1.3.3, I upgraded to 1.4.0 and my Vault containers fail with the below log output.

I downgraded back to 1.3.4 and my containers were able to start again.

Environment: GCP

• Vault Version: 1.4.0

Vault Config File:

{"api_addr":"http://vault.company","listener":{"tcp":{"address":"[::]:8200","cluster_address":"[::]:8201","tls_disable":true}},"storage":{"gcs":{"bucket":"vault-store-string","ha_enabled":"true"}},"ui":true}

      extraEnv:
      - name: GOOGLE_APPLICATION_CREDENTIALS
        value: "/vault/sa/key.json"
      - name: GOOGLE_PROJECT
        value: "project-name"
      - name: GOOGLE_REGION
        value: "us-east1"
      - name: VAULT_GCPCKMS_SEAL_KEY_RING
        value: "company-keyring"
      - name: VAULT_GCPCKMS_SEAL_CRYPTO_KEY
        value: "company-key"
      - name: VAULT_SEAL_TYPE
        value: "gcpckms"

Startup Log Output:

production-vault/vault-6ccdc569d-nsmnc[vault]: goroutine 1 [running]:
production-vault/vault-6ccdc569d-nsmnc[vault]: github.com/hashicorp/vault/command/server/seal.configureGCPCKMSSeal(0xc0006a7280, 0xc0006a7260, 0xc00000ead0, 0x3e3c240, 0xc000860840, 0x3e3e100, 0xc0006a72c0, 0x305aae0, 0x305aae0, 0xc000d51258, ...)
production-vault/vault-6ccdc569d-nsmnc[vault]:  /go/src/github.com/hashicorp/vault/command/server/seal/server_seal_gcpckms.go:16 +0x86
production-vault/vault-6ccdc569d-nsmnc[vault]: github.com/hashicorp/vault/command/server/seal.configureSeal(0xc0006a7280, 0xc0006a7260, 0xc00000ead0, 0x3e3c240, 0xc000860840, 0x3e3e100, 0xc0006a72c0, 0x1, 0x0, 0x0, ...)
production-vault/vault-6ccdc569d-nsmnc[vault]:  /go/src/github.com/hashicorp/vault/command/server/seal/server_seal.go:28 +0x4ca
production-vault/vault-6ccdc569d-nsmnc[vault]: github.com/hashicorp/vault/command.(*ServerCommand).Run(0xc0000c6160, 0xc00004c0e0, 0x2, 0x2, 0x0)
production-vault/vault-6ccdc569d-nsmnc[vault]:  /go/src/github.com/hashicorp/vault/command/server.go:1039 +0x139f
production-vault/vault-6ccdc569d-nsmnc[vault]: github.com/hashicorp/vault/vendor/github.com/mitchellh/cli.(*CLI).Run(0xc000362a00, 0xc000362a00, 0xc0002f7ea0, 0xc0002f7da0)
production-vault/vault-6ccdc569d-nsmnc[vault]:  /go/src/github.com/hashicorp/vault/vendor/github.com/mitchellh/cli/cli.go:255 +0x1da
production-vault/vault-6ccdc569d-nsmnc[vault]: github.com/hashicorp/vault/command.RunCustom(0xc00004c0d0, 0x3, 0x3, 0xc0000b1900, 0xc00008e058)
production-vault/vault-6ccdc569d-nsmnc[vault]:  /go/src/github.com/hashicorp/vault/command/main.go:180 +0x858
production-vault/vault-6ccdc569d-nsmnc[vault]: github.com/hashicorp/vault/command.Run(...)
production-vault/vault-6ccdc569d-nsmnc[vault]:  /go/src/github.com/hashicorp/vault/command/main.go:88
production-vault/vault-6ccdc569d-nsmnc[vault]: main.main()
production-vault/vault-6ccdc569d-nsmnc[vault]:  /go/src/github.com/hashicorp/vault/main.go:10 +0x71

[corbesero]

I believe I am having the same problem but with AWSKMS as the unsealer. I am running in an AWS EKS cluster using the Hashicorp vault helm chart (0.5.0). I started with Shamir, but when I add the seal stanza to use the KMS to do a migrate, the vault pods launch but the server never come up. In addition, there are no log messages. If I comment the seal stanza, it comes up. I am using raft as my storage. It's a two node cluster.

[calvn]

@jpds thanks for the report! Is it possible for you to share the full log, including the startup log entries?

[corbesero]

I would. But I see nothing. Once the container has been created, kubectl logs -n vault vault-0 displays absolutely nothing. As I said, this is in an AWS EKS cluster. The worker node is in a IAM role that gives it access to kms for the key. We have also set up an OIDC role in the service account, but with the previous helm chart (0.4.0) and its default vault version (1.3.?), that didn't seem to work and that's why we just added the kms actions to the worker node's role.

This is the config from my helm values.yml

...
    raft:
      enabled: true
      config: |
        ui = true
        log_level = "trace"

        listener "tcp" {
          tls_disable = 1
          address = "[::]:8200"
          cluster_address = "[::]:8201"
        }

        storage "raft" {
          path = "/vault/data"

          retry_join {
            leader_api_addr = "http://vault-0.vault-internal:8200"
          }

          retry_join {
            leader_api_addr = "http://vault-1.vault-internal:8200"
          }
        }
        seal "awskms" {
         region     = "us-east-1"
         kms_key_id = "<key identifier>"
        }
        service_registration "kubernetes" {}

When the seal stanza is there: nothing. No seal stanza, vault comes up (and I can manually unseal the master and standby)

[calvn]

@jpds I was able to find the source of the panic and opened a PR for a fix. In the meantime, can you try setting the seal type via the config (instead of the env var) and see if you're able to get around this panic? The other gcpkms values should still be properly set via environment variables.

Remove this line:

      - name: VAULT_SEAL_TYPE
        value: "gcpckms"

Add this to your config:

{...,"ui":true, "seal": {"gcpkms":{}}}

@corbesero Can you provide more information on your set up? This issue seems to be specific to gcpkms autoseal so it might be unrelated to your problem. Edit: I missed your latest update. I'll need to investigate on this, but your config seems to be fine.

[corbesero]

aws eks cluster (v1.15) in us-east-1
I am attaching my complete values.yaml for helm.
I am using the hashicorp helm chart for vault at tag 0.5.0
What else would you like?

values.yaml.txt

[calvn]

@corbesero it's a bit odd that you're not getting any logs at all, so it could be that the pods are failing to start. Can you give get pod and/or describe pod a try to see if it's an issue on the deployment side of things?

Since this issue is not related to your report, can we also open a separate GH issue to track this going forward? Thanks!