Post-unseal set up failed: failed to set up mount table

[ghost]

Environment:

• Vault Version: Vault v0.7.3 ('0b20ae0b9b7a748d607082b1add3663a28e31b68')
• Operating System/Architecture: Linux 4.9.0-3-amd64 Initial Website Import #1 SMP Debian 4.9.30-2+deb9u5 (2017-09-19) x86_64 GNU/Linux

Vault Config File:

{
  "backend": {
    "consul": {
      "address": "127.0.0.1:8500",
      "path": "vault",
      "token": "xxx"
    }
  },
  "listener": {
    "tcp": {
      "address": "0.0.0.0:8200"
    }
  },
  "telemetry": {
    "disable_hostname": true,
    "statsd_address": "127.0.0.1:8125"
  }
}

Expected Behavior:

Vault should start up post unseal.

Actual Behavior:

==> Vault server started! Log data will stream in below:

2017/10/04 13:21:59.711731 [WARN ] physical/consul: appending trailing forward slash to path
2017/10/04 13:22:35.828339 [INFO ] core: vault is unsealed
2017/10/04 13:22:35.828419 [INFO ] core: entering standby mode
2017/10/04 13:22:35.842449 [INFO ] core: acquired lock, enabling active operation
2017/10/04 13:22:35.886476 [INFO ] core: post-unseal setup starting
2017/10/04 13:22:35.887805 [INFO ] core: loaded wrapping token key
2017/10/04 13:22:35.889660 [ERROR] core: failed to create mount entry: path=secret/ error=unknown backend type: kv
2017/10/04 13:22:35.889680 [INFO ] core: pre-seal teardown starting
2017/10/04 13:22:35.889685 [INFO ] core: cluster listeners not running
2017/10/04 13:22:35.889723 [INFO ] core: pre-seal teardown complete
2017/10/04 13:22:35.889738 [ERROR] core: post-unseal setup failed: error=failed to setup mount table
2017/10/04 13:22:35.892957 [WARN ] grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: Error while dialing dial tcp 10.132.0.3:8201: getsockopt: connection refused"; Reconnecting to {10.132.0.3:8201 <nil>}
2017/10/04 13:22:35.909750 [INFO ] core: acquired lock, enabling active operation
2017/10/04 13:22:35.956017 [INFO ] core: post-unseal setup starting
2017/10/04 13:22:35.956510 [WARN ] Failed to dial 10.132.0.3:8201: context canceled; please retry.
2017/10/04 13:22:35.957487 [INFO ] core: loaded wrapping token key
2017/10/04 13:22:35.959333 [ERROR] core: failed to create mount entry: path=secret/ error=unknown backend type: kv
2017/10/04 13:22:35.959351 [INFO ] core: pre-seal teardown starting
2017/10/04 13:22:35.959356 [INFO ] core: cluster listeners not running
2017/10/04 13:22:35.959397 [INFO ] core: pre-seal teardown complete
2017/10/04 13:22:35.959412 [ERROR] core: post-unseal setup failed: error=failed to setup mount table
2017/10/04 13:22:35.962073 [WARN ] grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: Error while dialing dial tcp 10.132.0.3:8201: getsockopt: connection refused"; Reconnecting to {10.132.0.3:8201 <nil>}

It then attempts to acquire leadership in a loop (create session, acquire lock, delete session). Each attempt writes a new key to vault/core/leader/ but the old keys are not deleted.

curl -sH "X-Consul-Token: xxx" http://127.0.0.1:8500/v1/kv/vault/core/leader/?keys | jq '. | length'
14934

Steps to Reproduce:

Unknown. This is a new cluster for a new project.

[chrishoffman]

You mention that you are using 0.7.3. The kv backend type was not introduced until 0.8.3. Could you have initialized vault with 0.8.3 and are now trying to start up with 0.7.3?

[ghost]

Hello,

That was the issue! Thanks so much. I'll close the ticket.