[BUG] Unable to connect pusher: Stream is closed #62334

nicholasmhughes · 2022-07-14T20:34:05Z

Description
For a certain configuration where a master is configured on minion in order to host SLS and other files but the long-running minion service is stopped, the following error is thrown when the salt-call command is run on the minion:

[ERROR   ] Unable to connect pusher: Stream is closed

It seems that this configuration, where salt-call is run manually or via scheduled (cron) job, should be valid. Everything seems to work just fine, but the errors cause user confusion and the perception that something critical has failed.

Setup
Normal master/minion setup, but stop the salt-minion service.

Steps to Reproduce the behavior
Run any command and view the error message as the minion code spins up.

# salt-call cp.list_master
[ERROR   ] Unable to connect pusher: Stream is closed
local:
    - test.sls

Expected behavior
It would be nice to suppress this error message in certain scenarios where the connection to a master is desired without the configuration of long-running minion service.

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Salt Version:
          Salt: 3005+0na.b18281e
 
Dependency Versions:
          cffi: Not Installed
      cherrypy: Not Installed
      dateutil: Not Installed
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.2
       libgit2: Not Installed
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.3
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: 2.6.1
  pycryptodome: 3.14.1
        pygit2: Not Installed
        Python: 3.7.3 (default, Jan 22 2021, 20:04:44)
  python-gnupg: Not Installed
        PyYAML: 6.0
         PyZMQ: 23.0.0
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: debian 10 buster
        locale: UTF-8
       machine: x86_64
       release: 4.19.0-21-amd64
        system: Linux
       version: Debian GNU/Linux 10 buster

Additional context
Versions report is from recent pre-3005 master clone, but is reproducible on previous versions.

The text was updated successfully, but these errors were encountered:

bdrx312 · 2022-09-07T22:25:19Z

I also get this error message when using salt-call --local with saltutil sync methods like salt-call --local saltutil.sync_all

fra-iesus · 2022-09-15T10:33:39Z

I have the same errors here:

apt upgrade command ends up with not configured packages openmediavault and openmediavault-backup
manual execution of dpkg --configure openmediavault ends up with these errors:

# dpkg --configure openmediavault
Setting up openmediavault (6.0.39-1) ...
Creating configuration database ...
Migrating configuration database ...
Setting up Salt environment ...
[ERROR   ] Unable to connect pusher: Stream is closed
[ERROR   ] Unable to connect pusher: Stream is closed
[ERROR   ] Unable to connect pusher: Stream is closed
[ERROR   ] Unable to connect pusher: Stream is closed
[ERROR   ] Unable to connect pusher: Stream is closed
[ERROR   ] Unable to connect pusher: Stream is closed
[ERROR   ] Unable to connect pusher: Stream is closed
[ERROR   ] Unable to connect pusher: Stream is closed
[ERROR   ] Unable to connect pusher: Stream is closed
[ERROR   ] Unable to connect pusher: Stream is closed
[ERROR   ] Unable to connect pusher: Stream is closed
[ERROR   ] Unable to connect pusher: Stream is closed
Processing system modifications ...
Processing triggers for openmediavault (6.0.39-1) ...
Restarting engine daemon ...
Updating workbench configuration files ...

tjyang · 2022-09-22T23:19:28Z

Another salt-minion report

root@test13:~# salt-minion --versions-report
Salt Version:
Salt: 3005

Dependency Versions:
cffi: 1.11.5
cherrypy: Not Installed
dateutil: 2.6.1
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 2.10
libgit2: Not Installed
M2Crypto: Not Installed
Mako: 1.0.7
msgpack: 0.5.6
msgpack-pure: Not Installed
mysql-python: Not Installed
pycparser: 2.18
pycrypto: Not Installed
pycryptodome: 3.4.7
pygit2: Not Installed
Python: 3.6.9 (default, Jun 29 2022, 11:45:57)
python-gnupg: 0.4.1
PyYAML: 3.12
PyZMQ: 17.1.2
smmap: Not Installed
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.2.5

System Versions:
dist: ubuntu 18.04 Bionic Beaver
locale: UTF-8
machine: x86_64
release: 4.15.0-193-generic
system: Linux
version: Ubuntu 18.04 Bionic Beaver

root@test13:~#

- update openssl and ca-certificate deb

apt-get install openssl
apt-get install ca-certificates

Different error message this time after openssal and ca-cerficate updated.

SaltReqTimeoutError: Message timed out
Traceback (most recent call last):
File "/usr/bin/salt-call", line 11, in
load_entry_point('salt==3005', 'console_scripts', 'salt-call')()
File "/usr/lib/python3/dist-packages/salt/scripts.py", line 441, in salt_call
client.run()
File "/usr/lib/python3/dist-packages/salt/cli/call.py", line 50, in run
caller.run()
File "/usr/lib/python3/dist-packages/salt/cli/caller.py", line 95, in run
ret = self.call()
File "/usr/lib/python3/dist-packages/salt/cli/caller.py", line 203, in call
self.opts, data, func, args, kwargs
File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 149, in call
return self.loader.run(run_func, *args, **kwargs)
File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 1228, in run
return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
File "/usr/lib/python3/dist-packages/contextvars/init.py", line 38, in run
return callable(*args, **kwargs)
File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 1243, in _run_as
return _func_or_method(*args, **kwargs)
File "/usr/lib/python3/dist-packages/salt/executors/direct_call.py", line 10, in execute
return func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 149, in call
return self.loader.run(run_func, *args, **kwargs)
File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 1228, in run
return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
File "/usr/lib/python3/dist-packages/contextvars/init.py", line 38, in run
return callable(*args, **kwargs)
File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 1243, in _run_as
return _func_or_method(*args, **kwargs)
File "/usr/lib/python3/dist-packages/salt/modules/cp.py", line 756, in list_master
return _client().file_list(saltenv, prefix)
File "/usr/lib/python3/dist-packages/salt/fileclient.py", line 1363, in file_list
return self.channel.send(load)
File "/usr/lib/python3/dist-packages/salt/utils/asynchronous.py", line 125, in wrap
raise exc_info[1].with_traceback(exc_info[2])
File "/usr/lib/python3/dist-packages/salt/utils/asynchronous.py", line 131, in _target
result = io_loop.run_sync(lambda: getattr(self.obj, key)(*args, **kwargs))
File "/usr/lib/python3/dist-packages/salt/ext/tornado/ioloop.py", line 459, in run_sync
return future_cell[0].result()
File "/usr/lib/python3/dist-packages/salt/ext/tornado/concurrent.py", line 249, in result
raise_exc_info(self._exc_info)
File "", line 4, in raise_exc_info
File "/usr/lib/python3/dist-packages/salt/ext/tornado/gen.py", line 1064, in run
yielded = self.gen.throw(*exc_info)
File "/usr/lib/python3/dist-packages/salt/channel/client.py", line 295, in send
ret = yield self._crypted_transfer(load, timeout=timeout, raw=raw)
File "/usr/lib/python3/dist-packages/salt/ext/tornado/gen.py", line 1056, in run
value = future.result()
File "/usr/lib/python3/dist-packages/salt/ext/tornado/concurrent.py", line 249, in result
raise_exc_info(self._exc_info)
File "", line 4, in raise_exc_info
File "/usr/lib/python3/dist-packages/salt/ext/tornado/gen.py", line 1064, in run
yielded = self.gen.throw(*exc_info)
File "/usr/lib/python3/dist-packages/salt/channel/client.py", line 252, in _crypted_transfer
ret = yield _do_transfer()
File "/usr/lib/python3/dist-packages/salt/ext/tornado/gen.py", line 1056, in run
value = future.result()
File "/usr/lib/python3/dist-packages/salt/ext/tornado/concurrent.py", line 249, in result
raise_exc_info(self._exc_info)
File "", line 4, in raise_exc_info
File "/usr/lib/python3/dist-packages/salt/ext/tornado/gen.py", line 1064, in run
yielded = self.gen.throw(*exc_info)
File "/usr/lib/python3/dist-packages/salt/channel/client.py", line 235, in _do_transfer
timeout=timeout,
File "/usr/lib/python3/dist-packages/salt/ext/tornado/gen.py", line 1056, in run
value = future.result()
File "/usr/lib/python3/dist-packages/salt/ext/tornado/concurrent.py", line 249, in result
raise_exc_info(self._exc_info)
File "", line 4, in raise_exc_info
File "/usr/lib/python3/dist-packages/salt/ext/tornado/gen.py", line 1064, in run
yielded = self.gen.throw(*exc_info)
File "/usr/lib/python3/dist-packages/salt/transport/zeromq.py", line 914, in send
ret = yield self.message_client.send(load, timeout=timeout)
File "/usr/lib/python3/dist-packages/salt/ext/tornado/gen.py", line 1056, in run
value = future.result()
File "/usr/lib/python3/dist-packages/salt/ext/tornado/concurrent.py", line 249, in result
raise_exc_info(self._exc_info)
File "", line 4, in raise_exc_info
File "/usr/lib/python3/dist-packages/salt/ext/tornado/gen.py", line 1064, in run
yielded = self.gen.throw(*exc_info)
File "/usr/lib/python3/dist-packages/salt/transport/zeromq.py", line 624, in send
recv = yield future
File "/usr/lib/python3/dist-packages/salt/ext/tornado/gen.py", line 1056, in run
value = future.result()
File "/usr/lib/python3/dist-packages/salt/ext/tornado/concurrent.py", line 249, in result
raise_exc_info(self._exc_info)
File "", line 4, in raise_exc_info
salt.exceptions.SaltReqTimeoutError: Message timed out
Traceback (most recent call last):
File "/usr/bin/salt-call", line 11, in
load_entry_point('salt==3005', 'console_scripts', 'salt-call')()
File "/usr/lib/python3/dist-packages/salt/scripts.py", line 441, in salt_call
client.run()
File "/usr/lib/python3/dist-packages/salt/cli/call.py", line 50, in run
caller.run()
File "/usr/lib/python3/dist-packages/salt/cli/caller.py", line 95, in run
ret = self.call()
File "/usr/lib/python3/dist-packages/salt/cli/caller.py", line 203, in call
self.opts, data, func, args, kwargs
File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 149, in call
return self.loader.run(run_func, *args, **kwargs)
File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 1228, in run
return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
File "/usr/lib/python3/dist-packages/contextvars/init.py", line 38, in run
return callable(*args, **kwargs)
File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 1243, in _run_as
return _func_or_method(*args, **kwargs)
File "/usr/lib/python3/dist-packages/salt/executors/direct_call.py", line 10, in execute
return func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 149, in call
return self.loader.run(run_func, *args, **kwargs)
File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 1228, in run
return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
File "/usr/lib/python3/dist-packages/contextvars/init.py", line 38, in run
return callable(*args, **kwargs)
File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 1243, in _run_as
return _func_or_method(*args, **kwargs)
File "/usr/lib/python3/dist-packages/salt/modules/cp.py", line 756, in list_master
return _client().file_list(saltenv, prefix)
File "/usr/lib/python3/dist-packages/salt/fileclient.py", line 1363, in file_list
return self.channel.send(load)
File "/usr/lib/python3/dist-packages/salt/utils/asynchronous.py", line 125, in wrap
raise exc_info[1].with_traceback(exc_info[2])
File "/usr/lib/python3/dist-packages/salt/utils/asynchronous.py", line 131, in _target
result = io_loop.run_sync(lambda: getattr(self.obj, key)(*args, **kwargs))
File "/usr/lib/python3/dist-packages/salt/ext/tornado/ioloop.py", line 459, in run_sync
return future_cell[0].result()
File "/usr/lib/python3/dist-packages/salt/ext/tornado/concurrent.py", line 249, in result
raise_exc_info(self._exc_info)
File "", line 4, in raise_exc_info
File "/usr/lib/python3/dist-packages/salt/ext/tornado/gen.py", line 1064, in run
yielded = self.gen.throw(*exc_info)
File "/usr/lib/python3/dist-packages/salt/channel/client.py", line 295, in send
ret = yield self._crypted_transfer(load, timeout=timeout, raw=raw)
File "/usr/lib/python3/dist-packages/salt/ext/tornado/gen.py", line 1056, in run
value = future.result()
File "/usr/lib/python3/dist-packages/salt/ext/tornado/concurrent.py", line 249, in result
raise_exc_info(self._exc_info)
File "", line 4, in raise_exc_info
File "/usr/lib/python3/dist-packages/salt/ext/tornado/gen.py", line 1064, in run
yielded = self.gen.throw(*exc_info)
File "/usr/lib/python3/dist-packages/salt/channel/client.py", line 252, in _crypted_transfer
ret = yield _do_transfer()
File "/usr/lib/python3/dist-packages/salt/ext/tornado/gen.py", line 1056, in run
value = future.result()
File "/usr/lib/python3/dist-packages/salt/ext/tornado/concurrent.py", line 249, in result
raise_exc_info(self._exc_info)
File "", line 4, in raise_exc_info
File "/usr/lib/python3/dist-packages/salt/ext/tornado/gen.py", line 1064, in run
yielded = self.gen.throw(*exc_info)
File "/usr/lib/python3/dist-packages/salt/channel/client.py", line 235, in _do_transfer
timeout=timeout,
File "/usr/lib/python3/dist-packages/salt/ext/tornado/gen.py", line 1056, in run
value = future.result()
File "/usr/lib/python3/dist-packages/salt/ext/tornado/concurrent.py", line 249, in result
raise_exc_info(self._exc_info)
File "", line 4, in raise_exc_info
File "/usr/lib/python3/dist-packages/salt/ext/tornado/gen.py", line 1064, in run
yielded = self.gen.throw(*exc_info)
File "/usr/lib/python3/dist-packages/salt/transport/zeromq.py", line 914, in send
ret = yield self.message_client.send(load, timeout=timeout)
File "/usr/lib/python3/dist-packages/salt/ext/tornado/gen.py", line 1056, in run
value = future.result()
File "/usr/lib/python3/dist-packages/salt/ext/tornado/concurrent.py", line 249, in result
raise_exc_info(self._exc_info)
File "", line 4, in raise_exc_info
File "/usr/lib/python3/dist-packages/salt/ext/tornado/gen.py", line 1064, in run
yielded = self.gen.throw(*exc_info)
File "/usr/lib/python3/dist-packages/salt/transport/zeromq.py", line 624, in send
recv = yield future
File "/usr/lib/python3/dist-packages/salt/ext/tornado/gen.py", line 1056, in run
value = future.result()
File "/usr/lib/python3/dist-packages/salt/ext/tornado/concurrent.py", line 249, in result
raise_exc_info(self._exc_info)
File "", line 4, in raise_exc_info
salt.exceptions.SaltReqTimeoutError: Message timed out

leon0306 · 2022-10-20T03:37:15Z

I also get this error message. any solution for this?

salt-call --local status.ping_master localhost
[ERROR   ] Unable to connect pusher: Stream is closed
local:
    True

twangboy · 2022-10-25T15:01:21Z

Here's the actual traceback that is being caught:

(venv) PS C:\src\salt> salt-call --local saltutil.sync_grains
Traceback (most recent call last):
  File "c:\src\salt\salt\utils\event.py", line 427, in connect_pull
    self.pusher.connect(timeout=timeout)
  File "c:\src\salt\salt\utils\asynchronous.py", line 125, in wrap
    raise exc_info[1].with_traceback(exc_info[2])
  File "c:\src\salt\salt\utils\asynchronous.py", line 131, in _target
    result = io_loop.run_sync(lambda: getattr(self.obj, key)(*args, **kwargs))
  File "c:\src\salt\salt\ext\tornado\ioloop.py", line 459, in run_sync
    return future_cell[0].result()
  File "c:\src\salt\salt\ext\tornado\concurrent.py", line 249, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "c:\src\salt\salt\transport\ipc.py", line 345, in _connect
    yield self.stream.connect(sock_addr)
  File "c:\src\salt\salt\ext\tornado\gen.py", line 1056, in run
    value = future.result()
  File "c:\src\salt\salt\ext\tornado\concurrent.py", line 249, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
salt.ext.tornado.iostream.StreamClosedError: Stream is closed
None
[ERROR   ] Unable to connect pusher: Stream is closed
local:

golimix · 2022-12-07T14:36:00Z

sudo salt-call --local schedule.add job4 function='test.ping' job_args='[]'
==============================
job4
{'function': 'test.ping', 'job_args': [], '__pub_fun': 'schedule.add', '__pub_pid': 141641, '__pub_jid': '20221207130814457623', '__pub_tgt': 'salt-call'}
==============================
[ERROR   ] Encountered StreamClosedException
[ERROR   ] Unable to connect pusher: Stream is closed
[ERROR   ] Encountered StreamClosedException
[ERROR   ] Unable to connect pusher: Stream is closed
local:
    ----------
    changes:
        ----------
    comment:
        Failed to add job job4 to schedule.
    result:
        False

chnrxn · 2023-07-11T02:17:27Z

Why is it necessary to connect to the minion/pusher when doing a local salt-call? I run tests using salt-call --local and this message interferes with my CI scripts. Can we at least have a way to turn off / disable this attempt to contact the pusher?

nicholasmhughes added Bug broken, incorrect, or confusing behavior severity-low 4th level, cosemtic problems, work around exists salt-call labels Jul 14, 2022

twangboy mentioned this issue Oct 25, 2022

Use debug instead of error for logging pusher failure #62952

Merged

3 tasks

garethgreenaway closed this as completed in #62952 Oct 26, 2022

votdev mentioned this issue Nov 25, 2022

Upgrade to Salt 3005.1 openmediavault/openmediavault#1117

Merged

2 tasks

cdalvaro mentioned this issue Apr 25, 2023

Error setting schedule on macOS #64153

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] Unable to connect pusher: Stream is closed #62334

[BUG] Unable to connect pusher: Stream is closed #62334

nicholasmhughes commented Jul 14, 2022

bdrx312 commented Sep 7, 2022

fra-iesus commented Sep 15, 2022

tjyang commented Sep 22, 2022

leon0306 commented Oct 20, 2022

twangboy commented Oct 25, 2022

golimix commented Dec 7, 2022

chnrxn commented Jul 11, 2023 •

edited

Loading

[BUG] Unable to connect pusher: Stream is closed #62334

[BUG] Unable to connect pusher: Stream is closed #62334

Comments

nicholasmhughes commented Jul 14, 2022

bdrx312 commented Sep 7, 2022

fra-iesus commented Sep 15, 2022

tjyang commented Sep 22, 2022

leon0306 commented Oct 20, 2022

twangboy commented Oct 25, 2022

golimix commented Dec 7, 2022

chnrxn commented Jul 11, 2023 • edited Loading

chnrxn commented Jul 11, 2023 •

edited

Loading