Vault 7916 user group support

[maxcoulombe]

Overview

Adding support for elasticache user groups.

A user only has access to the ElastiCache replication group or cluster only if it is part of the User Group associated with a particular cluster.

We are using a best effort approach to bootstrap the configuration for the user if needed while respecting any prior configuration that may be already in place:

If the replication group already has an associated user group, we use it
If not and a default user group already exists, we associate it
If not and a default group does not exist, we create and associate it

Modifying a user group is an extremely slow operation so we do our best to return as soon as it is safe and keep the number of modifications to a minimum.

Locking is handled AWS side. Whenever a resource is modified is can be in one of the 4 following states:
active, creating, deleting, modifying
A resource can only be modified when it is active and a non-active resource will always either disappear or return active once the on-going operation is completed. For this reason we block during entry but not during exit as AWS is responsible to "release" the lock, or return to an active state.
For these reasons waiting for the active state before returning is simply slowing things down.
We also need to be defensive and block until the active state is achieved before any operations because the cluster could be modified by a system different than vault or a user via a manual operation; so only waiting for an operation to complete before returning does not guarantee a new subsequent request is safe to proceed without blocking upon entry.

Contributor Checklist

• Add relevant docs to upstream Vault repository, or sufficient reasoning why docs won’t be added yet
• Add output for any tests not ran in CI to the PR description (eg, acceptance tests)

/usr/local/go/bin/go tool test2json -t /tmp/GoLand/___redis_elasticache_client_test_go__1_.test -test.v -test.paniconexit0 -test.run ^\QTest_redisElastiCacheDB_Initialize\E|\QTest_redisElastiCacheDB_NewUser\E|\QTest_redisElastiCacheDB_UpdateUser\E|\QTest_redisElastiCacheDB_DeleteUser\E|\QTest_parseCreationCommands\E|\QTest_normaliseId\E$
=== RUN   Test_redisElastiCacheDB_Initialize
--- PASS: Test_redisElastiCacheDB_Initialize (1.19s)
=== RUN   Test_redisElastiCacheDB_Initialize/initialize_and_verify_connection_succeeds
{"@level":"debug","@message":"initializing AWS ElastiCache Redis client","@timestamp":"2022-08-26T11:47:05.280060-04:00"}
{"@level":"debug","@message":"Verifying connection to instance","@timestamp":"2022-08-26T11:47:05.280212-04:00","url":"master.vault-plugin-elasticache-test.6ylpiw.use1.cache.amazonaws.com:6379"}
    --- PASS: Test_redisElastiCacheDB_Initialize/initialize_and_verify_connection_succeeds (0.30s)
=== RUN   Test_redisElastiCacheDB_Initialize/initialize_with_invalid_config_fails
{"@level":"debug","@message":"initializing AWS ElastiCache Redis client","@timestamp":"2022-08-26T11:47:05.584075-04:00"}
{"@level":"debug","@message":"Verifying connection to instance","@timestamp":"2022-08-26T11:47:05.584205-04:00","url":"wrong"}
    --- PASS: Test_redisElastiCacheDB_Initialize/initialize_with_invalid_config_fails (0.88s)
=== RUN   Test_redisElastiCacheDB_NewUser
{"@level":"debug","@message":"initializing AWS ElastiCache Redis client","@timestamp":"2022-08-26T11:47:06.465695-04:00"}
{"@level":"debug","@message":"Verifying connection to instance","@timestamp":"2022-08-26T11:47:06.465819-04:00","url":"master.vault-plugin-elasticache-test.6ylpiw.use1.cache.amazonaws.com:6379"}
--- PASS: Test_redisElastiCacheDB_NewUser (109.99s)
=== RUN   Test_redisElastiCacheDB_NewUser/create_new_valid_user_succeeds
{"@level":"debug","@message":"creating new AWS ElastiCache Redis user","@timestamp":"2022-08-26T11:47:06.535975-04:00","role":"role"}
{"@level":"debug","@message":"deleting AWS ElastiCache Redis user","@timestamp":"2022-08-26T11:47:07.186186-04:00","username":"v_displ_role_4JC160t3FaS6fsZRUb5P_1661528826"}
    --- PASS: Test_redisElastiCacheDB_NewUser/create_new_valid_user_succeeds (1.02s)
=== RUN   Test_redisElastiCacheDB_NewUser/create_user_truncates_username
{"@level":"debug","@message":"creating new AWS ElastiCache Redis user","@timestamp":"2022-08-26T11:47:07.555327-04:00","role":"iAmEvenLongerTheApiWillDefinitelyRejectUsIfWeArePassedAsIsWithoutAnyModifications"}
{"@level":"debug","@message":"deleting AWS ElastiCache Redis user","@timestamp":"2022-08-26T11:48:56.098695-04:00","username":"v_iAmSu_iAmEvenLongerTheApiWillDefinitelyReject_xKJ9hlLoIjf10pz9peAR_1661528827"}
    --- PASS: Test_redisElastiCacheDB_NewUser/create_user_truncates_username (108.82s)
=== RUN   Test_redisElastiCacheDB_NewUser/create_user_with_invalid_password_fails
{"@level":"debug","@message":"creating new AWS ElastiCache Redis user","@timestamp":"2022-08-26T11:48:56.371597-04:00","role":"role"}
    --- PASS: Test_redisElastiCacheDB_NewUser/create_user_with_invalid_password_fails (0.08s)
=== RUN   Test_redisElastiCacheDB_NewUser/create_user_with_invalid_statements_fails
{"@level":"debug","@message":"creating new AWS ElastiCache Redis user","@timestamp":"2022-08-26T11:48:56.454186-04:00","role":"role"}
    --- PASS: Test_redisElastiCacheDB_NewUser/create_user_with_invalid_statements_fails (0.00s)
=== RUN   Test_redisElastiCacheDB_UpdateUser
{"@level":"debug","@message":"initializing AWS ElastiCache Redis client","@timestamp":"2022-08-26T11:48:56.454265-04:00"}
{"@level":"debug","@message":"Verifying connection to instance","@timestamp":"2022-08-26T11:48:56.454379-04:00","url":"master.vault-plugin-elasticache-test.6ylpiw.use1.cache.amazonaws.com:6379"}
{"@level":"debug","@message":"creating new AWS ElastiCache Redis user","@timestamp":"2022-08-26T11:48:56.716891-04:00","role":"role"}
--- PASS: Test_redisElastiCacheDB_UpdateUser (184.24s)
=== RUN   Test_redisElastiCacheDB_UpdateUser/update_password_of_existing_user_succeeds
{"@level":"debug","@message":"updating AWS ElastiCache Redis user","@timestamp":"2022-08-26T11:50:57.956213-04:00","username":"v_displ_role_PYzRdlo3DrpqA3JFrogc_1661528936"}
    --- PASS: Test_redisElastiCacheDB_UpdateUser/update_password_of_existing_user_succeeds (0.21s)
=== RUN   Test_redisElastiCacheDB_UpdateUser/update_password_of_non-existing_user_fails
{"@level":"debug","@message":"updating AWS ElastiCache Redis user","@timestamp":"2022-08-26T11:50:58.169767-04:00","username":"I do not exist"}
    --- PASS: Test_redisElastiCacheDB_UpdateUser/update_password_of_non-existing_user_fails (0.08s)
=== RUN   Test_redisElastiCacheDB_UpdateUser/update_to_invalid_password_fails
{"@level":"debug","@message":"updating AWS ElastiCache Redis user","@timestamp":"2022-08-26T11:50:58.251704-04:00","username":"v_displ_role_PYzRdlo3DrpqA3JFrogc_1661528936"}
{"@level":"debug","@message":"deleting AWS ElastiCache Redis user","@timestamp":"2022-08-26T11:52:00.270318-04:00","username":"v_displ_role_PYzRdlo3DrpqA3JFrogc_1661528936"}
    --- PASS: Test_redisElastiCacheDB_UpdateUser/update_to_invalid_password_fails (62.02s)
=== RUN   Test_redisElastiCacheDB_DeleteUser
{"@level":"debug","@message":"initializing AWS ElastiCache Redis client","@timestamp":"2022-08-26T11:52:00.689718-04:00"}
{"@level":"debug","@message":"Verifying connection to instance","@timestamp":"2022-08-26T11:52:00.690217-04:00","url":"master.vault-plugin-elasticache-test.6ylpiw.use1.cache.amazonaws.com:6379"}
{"@level":"debug","@message":"creating new AWS ElastiCache Redis user","@timestamp":"2022-08-26T11:52:00.764490-04:00","role":"role"}
--- PASS: Test_redisElastiCacheDB_DeleteUser (62.83s)
=== RUN   Test_redisElastiCacheDB_DeleteUser/delete_existing_user_succeeds
{"@level":"debug","@message":"deleting AWS ElastiCache Redis user","@timestamp":"2022-08-26T11:53:03.076053-04:00","username":"v_displ_role_D5YmqKYiGnzmBTKUTDNM_1661529120"}
    --- PASS: Test_redisElastiCacheDB_DeleteUser/delete_existing_user_succeeds (0.32s)
=== RUN   Test_redisElastiCacheDB_DeleteUser/delete_non-existing_user_is_lenient
{"@level":"debug","@message":"deleting AWS ElastiCache Redis user","@timestamp":"2022-08-26T11:53:03.394341-04:00","username":"I do not exist"}
{"@level":"debug","@message":"user does not exist or is being deleted, considering deletion successful","@timestamp":"2022-08-26T11:53:03.516500-04:00","user id":"Idonotexist"}
    --- PASS: Test_redisElastiCacheDB_DeleteUser/delete_non-existing_user_is_lenient (0.12s)
=== RUN   Test_parseCreationCommands
--- PASS: Test_parseCreationCommands (0.00s)
=== RUN   Test_parseCreationCommands/empty_command_returns_read-only_user
    --- PASS: Test_parseCreationCommands/empty_command_returns_read-only_user (0.00s)
=== RUN   Test_parseCreationCommands/single_command_with_multiple_rules_parses_correctly
    --- PASS: Test_parseCreationCommands/single_command_with_multiple_rules_parses_correctly (0.00s)
=== RUN   Test_parseCreationCommands/multiple_commands_with_multiple_rules_parses_correctly
    --- PASS: Test_parseCreationCommands/multiple_commands_with_multiple_rules_parses_correctly (0.00s)
=== RUN   Test_parseCreationCommands/empty_commands_are_tolerated
    --- PASS: Test_parseCreationCommands/empty_commands_are_tolerated (0.00s)
=== RUN   Test_parseCreationCommands/'on'_is_added_if_missing_for_convenience
    --- PASS: Test_parseCreationCommands/'on'_is_added_if_missing_for_convenience (0.00s)
=== RUN   Test_parseCreationCommands/'on'_is_ignored_if_passed_at_the_beginning
    --- PASS: Test_parseCreationCommands/'on'_is_ignored_if_passed_at_the_beginning (0.00s)
=== RUN   Test_parseCreationCommands/'on'_is_ignored_if_passed_explicitly_within_the_rules
    --- PASS: Test_parseCreationCommands/'on'_is_ignored_if_passed_explicitly_within_the_rules (0.00s)
=== RUN   Test_parseCreationCommands/'on'_is_ignored_if_passed_explicitly_at_the_end
    --- PASS: Test_parseCreationCommands/'on'_is_ignored_if_passed_explicitly_at_the_end (0.00s)
=== RUN   Test_parseCreationCommands/parsing_invalid_command_format_fails
    --- PASS: Test_parseCreationCommands/parsing_invalid_command_format_fails (0.00s)
=== RUN   Test_parseCreationCommands/creation_of_disabled_users_is_forbidden
    --- PASS: Test_parseCreationCommands/creation_of_disabled_users_is_forbidden (0.00s)
=== RUN   Test_normaliseId
--- PASS: Test_normaliseId (0.00s)
=== RUN   Test_normaliseId/compliant_username
    --- PASS: Test_normaliseId/compliant_username (0.00s)
=== RUN   Test_normaliseId/short_username
    --- PASS: Test_normaliseId/short_username (0.00s)
=== RUN   Test_normaliseId/username_too_long
    --- PASS: Test_normaliseId/username_too_long (0.00s)
=== RUN   Test_normaliseId/username_with_non-alphanumeric_characters
    --- PASS: Test_normaliseId/username_with_non-alphanumeric_characters (0.00s)
=== RUN   Test_normaliseId/username_starting_with_a_number
    --- PASS: Test_normaliseId/username_starting_with_a_number (0.00s)
PASS

Process finished with the exit code 0

• Backwards compatible