Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[3006.x] Fix x509_v2 state without changes can trigger file module to run in test mode #64269

Closed
wants to merge 5 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog/64195.fixed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Fixed x509_v2 state without changes can trigger file module to run in test mode
21 changes: 19 additions & 2 deletions salt/states/x509_v2.py
Original file line number Diff line number Diff line change
Expand Up @@ -186,6 +186,7 @@
import logging
import os.path

import salt.utils.context
import salt.utils.files
from salt.exceptions import CommandExecutionError, SaltInvocationError
from salt.features import features
Expand Down Expand Up @@ -1578,8 +1579,24 @@ def _file_managed(name, test=None, **kwargs):
raise SaltInvocationError("test param can only be None or True")
# work around https://github.com/saltstack/salt/issues/62590
test = test or __opts__["test"]
res = __salt__["state.single"]("file.managed", name, test=test, **kwargs)
return res[next(iter(res))]
file_managed = __states__["file.managed"]
if test:
# calls via __salt__["state.single"](..., test=test)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't the fix be in state.single if that call is what is causing the overwrite in __opts__?

Copy link
Contributor Author

@lkubb lkubb Jun 22, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suspect the problem is somewhere in the loader, though I'm neither qualified nor motivated to find out tbh. I have tried to debug a very similar problem with overriding __opts__ not being reversed in the Vault runner before. Also not sure if this workaround can be applied to state.single easily.

This specific bug breaks the x509_v2 state module pretty hard though and I suspect it will take a bit of time until someone figures out the root cause, if at all (since it's not a huge deal for most of Salt).

Edit: Note that in addition, by avoiding state.single this should run quite a bit more performant.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Ch3LL can we merge it as is? It's a huge issue in 3006.x since there is a problem installing mcrypto to onedir and the only other way to use x509 state is to use x509_v2 which has that issue.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lkubb Mind checking the file.directory issue reported here #64195 (comment) ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Oloremo Without having much to go on regarding the report, I fail to see how file.directory would behave differently with this patch atm. If anything, I would expect it the other way around (test to not be respected). Using file.directory instead of file.managed for the reproduction test in this PR succeeds as well. So my current assumption is that it's an unrelated issue or the patch was not applied correctly, barring any future (and more detailed) reports.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This bug is nicely distilled in this issue #62590. I think this should be addressed between salt/state.py and salt/loader/lazy.py which would fix it for all cases where we could run into this, not just x509_v2.

  • I've done some preliminary investigation and think this is something we can address in a 3006.x maintenance release.

  • We want all of Salt's dunders to be managed by the loader. Using func_globals_inject should be considered an anti-pattern.

  • We'd like to get to a place where opts is no longer mutable. This means in the long run we'll have to figure how a better way to handle __opts__["test"]. Currently the loader adds __opts__ to the modules it imports (at import time). The loader adds a copy of opts, not opts itself. This means updating __opts__["test"] or loader.opts after a loader has imported a module won't have any affect.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This bug is nicely distilled in this issue #62590.

Do you mean that bug has the same source as the sticky __opts__ causing this issue with state.single or fixing that bug will remove the necessity for using state.single instead of __states__ here in the first place?

The problem is that the x509_v2 state module also overrides test for early checks to avoid signing certificates that are discarded directly after, e.g. by a missing parent dir without makedirs. So if it does not have the same cause, a restructuring of the x509_v2 state module would still be necessary (not the end of the world though).

Copy link
Contributor

@dwoz dwoz Jul 7, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This bug is nicely distilled in this issue #62590.

fixing that bug will remove the necessity for using state.single instead of __states__ here in the first place?

I don't think we should need to refactor x509_v2 after fixing the bug. I will pull in your tests to the PR where this gets fixed so we can validate the issue is resolved.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dwoz
It's been a month, any updates on this issue? I believe it block literally everyone who wants to use 3006.x and x509

Copy link
Contributor

@dwoz dwoz Aug 11, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#64960 addresses this issue. I think this PR can be closed.

# can overwrite __opts__["test"] permanently. Workaround:
opts = __opts__
if not __opts__["test"]:
opts = copy.copy(__opts__)
opts["test"] = test
with salt.utils.context.func_globals_inject(file_managed, __opts__=opts):
# The file execution module accesses __opts__["test"] as well
with salt.utils.context.func_globals_inject(
__salt__["file.check_perms"], __opts__=opts
):
with salt.utils.context.func_globals_inject(
__salt__["file.manage_file"], __opts__=opts
):
return file_managed(name, **kwargs)
return file_managed(name, **kwargs)


def _check_file_ret(fret, ret, current):
Expand Down
61 changes: 61 additions & 0 deletions tests/pytests/integration/states/test_x509_v2.py
Original file line number Diff line number Diff line change
Expand Up @@ -502,6 +502,17 @@ def existing_cert(x509_salt_call_cli, cert_args, ca_key, rsa_privkey, request):
yield cert_args["name"]


@pytest.fixture(params=[{}])
def existing_privkey(x509_salt_call_cli, request, tmp_path):
pk_file = tmp_path / "priv.key"
pk_args = {"name": str(pk_file)}
pk_args.update(request.param)
ret = x509_salt_call_cli.run("state.single", "x509.private_key_managed", **pk_args)
assert ret.returncode == 0
assert pk_file.exists()
yield pk_args["name"]


@pytest.fixture(scope="module")
def x509_salt_run_cli(x509_salt_master):
return x509_salt_master.salt_run_cli()
Expand Down Expand Up @@ -626,6 +637,56 @@ def test_privkey_new_with_prereq_pkcs12(x509_salt_call_cli, tmp_path):
assert not _belongs_to(cert_new, pk_cur)


def test_file_managed_does_not_run_in_test_mode_after_x509_v2_invocation_without_changes(
x509_salt_master, x509_salt_call_cli, tmp_path, existing_privkey
):
"""
The x509_v2 state module tries to workaround issue #62590 (Test mode does
not propagate to __states__ when using prereq) by invoking the ``state.single``
execution module with an explicit test parameter. In some cases, this seems
to trigger another bug: The file module always runs in test mode afterwards.
This seems to be the case when the x509_v2 state module does not report changes
after having been invoked at least once before, until another x509_v2 call results
in a ``file.managed`` call without test mode.
Issue #64195.
"""
new_privkey = tmp_path / "new_privkey"
new_file = tmp_path / "new_file"
assert not new_file.exists()
state = f"""\
# The result of this call is irrelevant, just that it exists
Some private key is present:
x509.private_key_managed:
- name: {new_privkey}

# This single call without changes does not trigger the bug on its own
Another private key is (already) present:
x509.private_key_managed:
- name: {existing_privkey}

Subsequent file.managed call should not run in test mode:
file.managed:
- name: {new_file}
- contents: foo
- require:
- Another private key is (already) present
"""
with x509_salt_master.state_tree.base.temp_file("file_managed_test.sls", state):
ret = x509_salt_call_cli.run("state.apply", "file_managed_test")
assert ret.returncode == 0
assert ret.data
x509_res = next(ret.data[x] for x in ret.data if x.startswith("x509_|-Another"))
assert x509_res["result"] is True
assert not x509_res["changes"]
file_res = next(
ret.data[x] for x in ret.data if x.startswith("file_|-Subsequent")
)
assert file_res["result"] is True
assert file_res["changes"]
assert new_file.exists()
assert new_file.read_text() == "foo\n"


def _belongs_to(cert_or_pubkey, privkey):
if isinstance(cert_or_pubkey, cx509.Certificate):
cert_or_pubkey = cert_or_pubkey.public_key()
Expand Down