Error reading aws/creds/my-role: Get http://127.0.0.1:8200/v1/aws/creds/my-role: net/http: request canceled (Client.Timeout exceeded while awaiting headers) #5878

pawarrchetan · 2018-11-30T11:37:25Z

Describe the bug
After using the AWS Secrets Backend in VAULT I am getting below error after using the correct credentials stored in VAULT.

[root@ip-100-66-12-90 tf]# vault read aws/creds/my-role

Error reading aws/creds/my-role: Get http://127.0.0.1:8200/v1/aws/creds/my-role: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

To Reproduce
Steps to reproduce the behavior:

vault write aws/config/root
access_key=ABCDEFGHIJKL
secret_key=AcVFGCbhjksluyCGD/JHGDPbps2X
region=eu-central-1

-- Above entry is dummy keys.

vault write aws/roles/my-role
credential_type=iam_user
policy_document=-<<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"iam:DeleteAccessKey",
"iam:GetUserPolicy",
"iam:GetAccessKeyLastUsed",
"iam:UpdateAccessKey",
"iam:GetUser",
"iam:ListRolePolicies",
"iam:CreateAccessKey",
"iam:ListAccessKeys"
],
"Resource": "arn:aws:iam::80527613****:user/ark-key-rotate"
},
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": [
"iam:GenerateCredentialReport",
"iam:GetContextKeysForCustomPolicy",
"iam:ListPolicies",
"iam:ListUsers",
"iam:GetAccountAuthorizationDetails",
"iam:GetCredentialReport",
"iam:DeleteAccessKey",
"iam:GetUserPolicy",
"iam:GetAccessKeyLastUsed",
"iam:UpdateAccessKey",
"iam:GetUser",
"iam:ListRolePolicies",
"iam:CreateAccessKey",
"iam:ListAccessKeys"
],
"Resource": "*"
}
]
}
EOF
Success! Data written to: aws/roles/my-role

vault read aws/creds/my-role
Error reading aws/creds/my-role: Get http://127.0.0.1:8200/v1/aws/creds/my-role: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

Expected behavior
The output should contain the new temporary IAM credentials which can be used to login to AWS CLI.

Environment:

Vault Server Version (retrieve with vault status):
vault status
Key Value

Seal Type shamir
Initialized true
Sealed false
Total Shares 3
Threshold 2
Version 0.11.4
Cluster Name vault-cluster-fb591cdc
Cluster ID a924d375-d667-0303-2d1d-4ecfaed1f4de
HA Enabled false

Vault CLI Version (retrieve with vault version):
vault version
Vault v0.11.4 ('612120e76de651ef669c9af5e77b27a749b0dba3')
Server Operating System/Architecture:
Amazon Linux
Linux ip-100-66-12-**.eu-central-1.compute.internal 4.14.42-61.37.amzn2.x86_64 Initial Website Import #1 SMP Mon May 21 23:43:11 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Vault server configuration file(s):

backend "file" {
        path = "/opt/SP/apps/VAULT/data"
}

listener "tcp" {
        tls_disable = 1
}

Additional context
I would like to know what could be the problem here.
I have also added the credentials for the master user in the .aws/credentials file.
but the error still persists.

The text was updated successfully, but these errors were encountered:

pawarrchetan · 2018-11-30T12:40:03Z

In the /var/log/messages file I can see below errors when I try to execute vault read aws/creds/my-role

Nov 30 12:33:04 ip-100-66-12-90 amazon-ssm-agent: For verbose messaging see aws.Config.CredentialsChainVerboseErrors
Nov 30 12:34:22 ip-100-66-12-90 dhclient[3289]: XMT: Solicit on eth0, interval 111680ms.
Nov 30 12:35:20 ip-100-66-12-90 amazon-ssm-agent: 2018-11-30 12:35:20 INFO [instanceID=i-0eb6e4936f0377681] [HealthCheck] HealthCheck reporting agent health.
Nov 30 12:35:20 ip-100-66-12-90 amazon-ssm-agent: 2018-11-30 12:35:20 ERROR [instanceID=i-0eb6e4936f0377681] [HealthCheck] error when calling AWS APIs. error details - NoCredentialProviders: no valid providers in chain. Deprecated.
Nov 30 12:35:20 ip-100-66-12-90 amazon-ssm-agent: For verbose messaging see aws.Config.CredentialsChainVerboseErrors
Nov 30 12:35:21 ip-100-66-12-90 amazon-ssm-agent: 2018-11-30 12:35:20 INFO [instanceID=i-0eb6e4936f0377681] [HealthCheck] increasing error count by 1
Nov 30 12:35:21 ip-100-66-12-90 amazon-ssm-agent: 2018-11-30 12:35:20 ERROR [instanceID=i-0eb6e4936f0377681] [HealthCheck] error when calling AWS APIs. error details - NoCredentialProviders: no valid providers in chain. Deprecated.
Nov 30 12:35:21 ip-100-66-12-90 amazon-ssm-agent: For verbose messaging see aws.Config.CredentialsChainVerboseErrors
Nov 30 12:35:21 ip-100-66-12-90 amazon-ssm-agent: 2018-11-30 12:35:20 INFO [instanceID=i-0eb6e4936f0377681] [HealthCheck] increasing error count by 1

joelthompson · 2018-11-30T14:28:28Z

Hi @pawarrchetan -- that error most likely means your box doesn't have network connectivity to be able to talk to the AWS IAM endpoints. Does your machine have outbound internet access (e.g., in a subnet with routes to either an IGW or NAT, a security group that permits outbound access to 0.0.0.0/0, no iptables rules on the machine blocking outbound internet access)?

pawarrchetan · 2018-11-30T14:43:15Z

@joelthompson
Thanks for your reply.

however my machine has access to the AWS API endpoints as I can easily use aws cli to communicate with the AWS resources.
Please see below :
$ nslookup www.google.com
Server: 100.66.12.2
Address: 100.66.12.2#53

Non-authoritative answer:
Name: www.google.com
Address: 172.217.22.4

I can list the S3 buckets in my account using aws cli.

aws s3api list-buckets

{
"Owner": {
"ID": "**************************a115287808afc4a02fb33f4ef4"
},
"Buckets": [
{
"CreationDate": "2018-09-10T07:59:04.000Z",
"Name": "2gj9i-g8s-access-logs"
},
{
"CreationDate": "2018-11-28T16:12:51.000Z",
"Name": "2mpte-g8s-access-logs"
},

Regards...
Chetan

joelthompson · 2018-11-30T14:48:18Z

The ListBucket command could be going through an S3 VPC Endpoint, while the IAM calls wouldn't. Can you use the awscli to make an IAM call, e.g., aws iam get-user?

pawarrchetan · 2018-11-30T14:53:40Z

@joelthompson
Yes I can do IAM calls using aws cli

aws iam get-user

{
"User": {
"UserName": "terraform-crp",
"Path": "/",
"CreateDate": "2018-11-08T11:20:01Z",
"UserId": "AIDAJJENJGUOS********",
"Arn": "arn:aws:iam::80527613****:user/terraform-crp"
}
}

pawarrchetan · 2018-12-11T09:28:22Z

@joelthompson

ANy update on the request ?

Regards...
Chetan

gauthamatic · 2019-07-25T16:28:23Z

I'm facing a similar issue but for Assume role credentials.
Any update on this?

mdasilva · 2019-12-13T19:55:12Z

I'm facing the same issue. Is this a confirmed bug?

TomHowarth · 2019-12-18T16:32:09Z

this is my code sanitized for here.

provider "vault" {
address = "${var.vault_addr}"
token = "${var.vault_token}"
}

resource "vault_aws_secret_backend" "aws" {
access_key = "${var.access_key}"
secret_key = "${var.secret_key}"
region = "us-east-1"

default_lease_ttl_seconds = "120"
max_lease_ttl_seconds = "240"
}

resource "vault_aws_secret_backend_role" "ec2-admin" {
backend = "${vault_aws_secret_backend.aws.path}"
name = "my-role"
credential_type = "assumed_role"

policy_document=<<EOT
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "",
"Resource": ""
}
]
}
EOT
}

my vault is running and showing as initialized.

When I run the above code it is successful.

however when I run the following: vault read /aws/creds/my-role
I receive the following error

Error reading aws/creds/ec2-admin-role: Error making API request.

URL: GET http://127.0.0.1:8200/v1/aws/creds/my-role
Code: 400. Errors:

did not supply a role_arn parameter and unable to determine one

spangenberg · 2020-01-28T14:01:05Z

We have two different issues here.

@TomHowarth your vault_aws_secret_backend is missing the allowed AWS roles to assume, please take a look here: https://www.terraform.io/docs/providers/vault/r/aws_secret_backend_role.html#role_arns

Regarding the first issue from @pawarrchetan, it looks networking related to me. If this still persists, I'd like to know more about the network configuration you're using, how the VPC is set up, which region you're running.

spangenberg · 2020-02-21T18:47:55Z

We haven't heard back regarding this issue in over 24 days. To try and keep our GitHub issues current, we'll be closing this issue in approximately seven days if we do not hear back regarding this issue. Please let us know if you can still reproduce this issue, and if there is any more information you could share, otherwise we'll be closing this issue.

spangenberg · 2020-03-02T16:03:31Z

Closing for now. Let us know if you have any more information!

michelvocks added bug Used to indicate a potential bug secret/aws labels Nov 12, 2019

catsby added the version/0.11.x label Nov 18, 2019

spangenberg self-assigned this Jan 27, 2020

spangenberg added the waiting-for-response label Jan 28, 2020

spangenberg closed this as completed Mar 2, 2020

heatherezell removed the waiting-for-response label Oct 19, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error reading aws/creds/my-role: Get http://127.0.0.1:8200/v1/aws/creds/my-role: net/http: request canceled (Client.Timeout exceeded while awaiting headers) #5878

Error reading aws/creds/my-role: Get http://127.0.0.1:8200/v1/aws/creds/my-role: net/http: request canceled (Client.Timeout exceeded while awaiting headers) #5878

pawarrchetan commented Nov 30, 2018

pawarrchetan commented Nov 30, 2018 •

edited by spangenberg

Loading

joelthompson commented Nov 30, 2018

pawarrchetan commented Nov 30, 2018

joelthompson commented Nov 30, 2018

pawarrchetan commented Nov 30, 2018

pawarrchetan commented Dec 11, 2018

gauthamatic commented Jul 25, 2019

mdasilva commented Dec 13, 2019

TomHowarth commented Dec 18, 2019

spangenberg commented Jan 28, 2020

spangenberg commented Feb 21, 2020

spangenberg commented Mar 2, 2020

Error reading aws/creds/my-role: Get http://127.0.0.1:8200/v1/aws/creds/my-role: net/http: request canceled (Client.Timeout exceeded while awaiting headers) #5878

Error reading aws/creds/my-role: Get http://127.0.0.1:8200/v1/aws/creds/my-role: net/http: request canceled (Client.Timeout exceeded while awaiting headers) #5878

Comments

pawarrchetan commented Nov 30, 2018

pawarrchetan commented Nov 30, 2018 • edited by spangenberg Loading

joelthompson commented Nov 30, 2018

pawarrchetan commented Nov 30, 2018

aws s3api list-buckets

joelthompson commented Nov 30, 2018

pawarrchetan commented Nov 30, 2018

aws iam get-user

pawarrchetan commented Dec 11, 2018

gauthamatic commented Jul 25, 2019

mdasilva commented Dec 13, 2019

TomHowarth commented Dec 18, 2019

spangenberg commented Jan 28, 2020

spangenberg commented Feb 21, 2020

spangenberg commented Mar 2, 2020

pawarrchetan commented Nov 30, 2018 •

edited by spangenberg

Loading