[BUG] file_roots to relative path seems broken

[dtzampanakis]

Description
Upgrading from salt-ssh version 3007.0 (Chlorine)
to 3007.1 (Chlorine), seems to broke the file_roots that i use in my master file.

root_dir: "."
conf_file: ./master
cachedir: /tmp/salt
gpg_keydir: /etc/salt/gpgkeys
file_roots:
  base:
    - ./states
pillar_roots:
  base:
    - ./pillar

Pillars are working, but when i try to state.apply i am unable to apply any state with

serverx:
    - No matching sls found for 'ntp' in env 'base'

Working only if i change the file_roots with an absolute path.

I installed 3006.8(Sulfur) to make it work again with relative path since i am unable to go back to 3007.0.

[petrows]

I have the same issue, all my scripts are broken now after update. I also have downgraded to previous version.

[bendikro]

b0e7c62 by @hurzhurz is what caused this regression. @s0undt3ch

[hartwork]

This is crazy, it makes 3007.1 unusable. Could this be fixed please?

[hurzhurz]

@dtzampanakis Can you tell a bit more about how you run the master with this example config?

I wanted to try it, but I can't get it working, even with 3006.8.

I made a file structure like this in a test VM:

/root/master-test/
/root/master-test/master (your mentioned config)
/root/master-test/states/
/root/master-test/states/abc (dummy file)
/root/master-test/pillar/

And I started the master like this:
cd /root/master-test/ && salt-master -c .

The master actually uses the file structure in /root/master-test/ for minion keys, cache, etc.
But a cp.list_master against a connected minion doesn't list anything.

It looks like the master worker threads have / as CWD. So with "./states" they actually look in "/states".
And if I create "/states/" and a file "/states/abc", cp.list_master actually shows the file "abc".

Am I missing a step? Or do you use anything special to run the master, like docker or so?

[hartwork]

@hurzhurz this is about salt-ssh. Have you tried salt-ssh?

[hurzhurz]

@hartwork Thanks for the hint! I somehow overlooked and also didn't expect that...

[dwoz]

@hurzhurz Assigned this to you assuming you are up for the task. :)

[vincent-olivert-riera]

@hurzhurz this is about salt-ssh. Have you tried salt-ssh?

No, it is NOT about salt-ssh. This is an error in the salt-common package.

---

I have the same issue using a masterless minion. After updating to 3007.1, salt-call -c . state.apply fails because I set the file_roots with relative paths.

For instance, this is my directory structure:

/home/user/foo/
/home/user/foo/pillar/*.sls
/home/user/foo/salt/*.sls
/home/user/foo/minion

And this is the content of the minion file:

file_client: local
file_roots:
  base:
    - salt/
pillar_roots:
  base:
    - pillar/

This was working just fine when running salt-call -c . state.apply inside the foo directory. After updating to 3007.1, it breaks with the following error:

$ salt-call -c . state.apply 
local:
----------
          ID: states
    Function: no.None
      Result: False
     Comment: No Top file or master_tops data matches found. Please see master log for details.
     Changes:   

Summary for local
------------
Succeeded: 0
Failed:    1
------------
Total states run:     1
Total run time:   0.000 ms

---

I can confirm that the changes from #66690 fix the issue. I have manually modified the verify.py file, and started to work again:

$ diff -up \
  /opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/verify.py.orig \
  /opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/verify.py
--- /opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/verify.py.orig	2024-08-10 17:38:28.388211789 +0900
+++ /opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/verify.py	2024-08-10 17:33:52.276321252 +0900
@@ -532,7 +532,7 @@ def clean_path(root, path, subdir=False,
     Pass realpath=False if filesystem links should not be resolved.
     """
     if not os.path.isabs(root):
-        root = ""
+        root = os.path.join(os.getcwd(), root)
     root = os.path.normpath(root)
     if not os.path.isabs(path):
         path = os.path.join(root, path)

---

3006.9 includes the fix, but 3007.1 is still broken. I hope a new point release for the 3007 branch gets published soon.

[bbinet]

Same issue here using 3007.1 release. When do you think a new point release will be published?

[hartwork]

I'm still getting the same "not found in saltenv 'base'" even with 3006.9 including the patch.

[hurzhurz]

@hartwork If you still have the issue with 3006.9, there might be something special about your setup. So maybe you can provide some details?
Though the error can have different causes, for example incorrect file permissions (keep in mind the salt-master is running as user "salt" by default).

[hartwork]

Hi @hurzhurz there is no salt master for me, it's only salt-ssh here, the salt_state.tgz ending up on the server does contain the files that salt-ssh started complaining they could no longer be found. The roster has user: root for all related target hosts.

My setup is simple (and super close to what I described in https://blog.hartwork.org/posts/replacing-ansible-with-salt-ssh-for-speed-and-for-good/ in 2020):

# cat Saltfile 
salt-ssh:
  roster_file: ./roster
  config_dir: .
  ssh_log_file: ./log.txt


# cat master 
root_dir: .

cachedir: ./cachedir

file_roots:
  base:
    - ./salt

pillar_roots:
  base:
    - ./pillar

state_output_diff: True


# cat pillar/top.sls 
base:
  '*':
    - data

The files that stopped being found are all in salt/files/ and my salt:// URLs all start with ``salt://files/(two slashes, not three), e.g.salt://files/restart-as-needed.sh` was previously found fine and came from on the local machine `salt/files/restart-as-needed.sh`, originally.

I would go back further in time trying to find the last version of salt that worked but the target host has Python 3.13 by now and so e.g. 3005.5 is crashing from import cgi today. All I know is the same set of flies worked well with some version of Salt in the past.

[hurzhurz]

@hartwork I have just tested it with your config and 3006.9. Seems to work so far.
Does salt-ssh xxx cp.list_master list your files?
Can you show what exact CLI command you try to execute and what the complete output is?

Well, maybe you could also try to wipe the temporary salt files on the target system.
Either by running salt-ssh once with "-w" or by manually deleting /var/tmp/.root_*_salt

[hartwork]

Hi @hurzhurz,

thanks for your reply and the hint about cp.list_master.

I have re-installed 3006.9 now and applied match_hostname.patch on top so that 3006.9 doesn't crash any more. I then checked cp.list_master output…

# salt-ssh -w -t 'docker*' cp.list_master
docker2:
    - files/authorized-keys-root.txt
[..]
    - files/restart-as-needed.sh
[..]

…which looks healthy to me to then find, that Salt can still not find those exact files in practice:

# salt-ssh -w -t 'docker*' state.apply setup
docker2:
----------
          ID: ssh-daemon
    Function: ssh_auth.present
      Result: False
     Comment: Failed to add the ssh key. Source file salt://files/authorized-keys-root.txt is missing
     Started: 18:17:48.051438
    Duration: 8.851 ms
     Changes:   
----------
[..]
----------
          ID: restart-as-needed-sh
    Function: file.managed
        Name: /usr/local/bin/restart-as-needed.sh
      Result: False
     Comment: Source file salt://files/restart-as-needed.sh not found in saltenv 'base'
     Started: 18:17:57.696776
    Duration: 5.563 ms
     Changes:   
----------
[..]

What do you think?

[hurzhurz]

@hartwork ok, so I have tested a state with a file.managed and in worked in my test setup. So I have no real clue what's wrong at your setup.
Though you mentioned Python 3.13 ... maybe something about that. My test system has 3.10.

Well, maybe another test you could do is: salt-ssh xyz cp.get_file salt://files/xyz /tmp/
If it works, salt-ssh is able to transfer the file. If not... maybe check if debug (-l debug) shows something.

[lkubb]

Since Python 3.13 was mentioned to be the target host Python, maybe it's related to #66898, so you could try the fix in #66899. After applying the patch, don't forget to replace the thin archive on the target.

[hartwork]

Well, maybe another test you could do is: salt-ssh xyz cp.get_file salt://files/xyz /tmp/

@hurzhurz that worked, I get output…

# salt-ssh -w -t 'docker*' cp.get_file salt://files/restart-as-needed.sh /tmp/
docker2:
    True
docker1:
    True

…and file /tmp/restart-as-needed.sh appeared on host docker2, just checked. What can we conclude from that part working?

[hartwork]

@lkubb if I can trust my eyes, applying #66899 indeed solves the issue for 3006.9. (There is still red in the output because I'm also hitting #64532 now but that's another story). Thank you! 🙏

[hartwork]

For anyone coming here to find a working solution based on Salt 3007.1: it took these two pull requests on top for me:

• [3006.x] Fix relative file_roots paths #66690
• [master] Fix Python3.13 compatibility regarding urllib.parse module #66899

[bbinet]

Is there any ETA for Salt 3007.2 with these patchs included?