Brew tests on mac failing

[Ch3LL]

The following brew tests are failing on mac:

integration.modules.test_mac_brew.BrewModuleTest.test_brew_install
integration.modules.test_mac_brew.BrewModuleTest.test_info_installed
integration.modules.test_mac_brew.BrewModuleTest.test_latest_version
integration.modules.test_mac_brew.BrewModuleTest.test_list_upgrades
integration.modules.test_mac_brew.BrewModuleTest.test_version

Here is the stack trace:

Traceback (most recent call last):
  File "/testing/tests/integration/modules/test_mac_brew.py", line 44, in test_brew_install
    self.assertIn(ADD_PKG, pkg_list)
AssertionError: u'algol68g' not found in u'ERROR: Brew command failed. Additional info follows:\n\nresult:\n    ----------\n    pid:\n        30165\n    retcode:\n        1\n    stderr:\n        Error: Running Homebrew as root is extremely dangerous and no longer supported.\n        As Homebrew does not drop privileges on installation you would be giving all\n        build scripts full access to your system.\n    stdout:'

If i run pkg.install outside of the test runner I see this:

salt-call --local pkg.install vim
[ERROR   ] stderr: -bash: brew: command not found
[ERROR   ] retcode: 127
Error running 'pkg.install': Brew command failed. Additional info follows:

result:
    ----------
    pid:
        55236
    retcode:
        127
    stderr:
        -bash: brew: command not found
    stdout:

If i run it in our test runner we get the "homebrew can't be run by root" error.

[ERROR   ] stderr: -bash: brew: command not found
[ERROR   ] retcode: 127
Error running 'pkg.install': Brew command failed. Additional info follows:

result:
    ----------
    pid:
        55236
    retcode:
        127
    stderr:
        -bash: brew: command not found
    stdout:

I wrote a quick python script to quickly replicate the issue as well:

import salt.utils.timed_subprocess
kwargs = {u'timeout': None,
          u'with_communicate': True,
          u'shell': False,
          u'bg': False,
          u'stderr': -1,
          u'env': {
                   'USER': 'jenkins',
                   'HOME': '/Users/jenkins/',
                   'PATH': '/testing/tests/integration/mockbin:/opt/salt/bin/:/opt/salt/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/salt/bin:/usr/local/sbin',
}}

cmd = [u'su', u'-l', u'jenkins', u'-c', u"brew list --versions"]
#cmd = [u'su -l jenkins -c "brew list --versions"']
proc = salt.utils.timed_subprocess.TimedProc(cmd, **kwargs)
proc.run()
out, err = proc.stdout, proc.stderr
print(out)
print(err)

which returns:

jk-sierra-base-2:~ root# python ~/test.py 
None
Error: Running Homebrew as root is extremely dangerous and no longer supported.
As Homebrew does not drop privileges on installation you would be giving all
build scripts full access to your system.

jk-sierra-base-2:~ root# 

I bisected this issue to this change: saltstack/salt#47212

If i revert this PR tests pass

ping @weswhet

[weswhet]

@Ch3LL are these test macs logged in as a user, and which OS version are they running?

[Ch3LL]

we are seeing the failures on:

sierra, high sierra, and el capitan

We login as root as you can see below:

'ssh', '-oStrictHostKeyChecking=no', '-oUserKnownHostsFile=/dev/null', '-oControlPath=none', '-oPasswordAuthentication=no', '-oChallengeResponseAuthentication=no', '-oPubkeyAuthentication=yes', '-oKbdInteractiveAuthentication=no', '-oIdentityFile=/private/var/jenkins_key', '-t', '-t', '[email protected]', "'source /etc/profile ; /opt/salt/bin/python /testing/tests/runtests.py -v --run-destructive --sysinfo --transport=zeromq --output-columns=80 --ssh --coverage-xml=/tmp/coverage.xml --xml=/tmp/xml-unittests-output'"

and here is the /etc/profile that we source:

jksierrabase2:~ jenkins$ cat /etc/profile
# System-wide .profile for sh(1)

if [ -x /usr/libexec/path_helper ]; then
        eval `/usr/libexec/path_helper -s`
fi

if [ "${BASH-no}" != "no" ]; then
        [ -r /etc/bashrc ] && . /etc/bashrc
fi
# ----- envs for salt-testing -----
[[ ${PATH} =~ "^/opt/salt/bin" ]] || export PATH="/opt/salt/bin:${PATH}"
unset TMPDIR
# ---------------------------------
export PATH=/opt/salt/bin/:$PATH
#------ start locale zone ------
export LANG=en_US.UTF-8
#------ endlocale zone ------

[weswhet]

I can replicate this issue

[ERROR   ] stderr: -bash: brew: command not found
[ERROR   ] retcode: 127
Error running 'pkg.install': Brew command failed. Additional info follows:

which we can fix by doing a which brew and passing the full path.

But I'm still trying to reproduce this bit

Error: Running Homebrew as root is extremely dangerous and no longer supported.
As Homebrew does not drop privileges on installation you would be giving all
build scripts full access to your system.

[Ch3LL]

okay got a step further. I was able to get the script i wrote above to now fail with "brew command not found" instead of "Running Homebrew as root is exteremely dangerous" by changing the PATHS in the environment.

<                    'PATH': '/testing/scripts/:/testing/salt/tests/:/opt/salt/bin/:/opt/salt/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/salt/bin:/usr/local/sbin',
---
>                    'PATH': '/testing/tests/integration/mockbin:/opt/salt/bin/:/opt/salt/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/salt/bin:/usr/local/sbin',

mostly just updating here so I can start off where i ended today. will look into how to fix for tests

[Ch3LL]

also just tested hardcoding that PATH in salt/modules/cmdmod.py and getting the "brew command not found" expected error when running tests. will look at cleaning up the paths tomorrow and figuring out why this is occurring.

[Ch3LL]

k its the fake su we add to our integration test:

jk-sierra-base-2:~ root# /testing/tests/integration/mockbin/su -l jenkins -c 'brew list'
Error: Running Homebrew as root is extremely dangerous and no longer supported.
As Homebrew does not drop privileges on installation you would be giving all
build scripts full access to your system.

jk-sierra-base-2:~ root# cat /testing/tests/integration/mockbin/su 
#!/bin/bash
# "Fake" su command
#
# Just executes the command without changing effective uid. Used in integration
# tests.
while true; do
    shift
    test "x$1" == "x-c" && break
    test "x$1" == "x" && break
done
shift
exec /bin/bash -c "$@"

[weswhet]

Oh, that’s interesting. Nice find.

[weswhet]

I suppose this could be fixed by changing the cmdmod.py to use to /usr/bin/su vs just su

[Ch3LL]

@weswhet just closed we ended up just removing the su path in the integration tests