[BUG] incorrect metric connected_peers with istio service mesh

[timofeevmd]

OS and Environment

Linux, K8s

GIT commit hash

eca8569

Minimum working example / Steps to reproduce

1. Raise five peers in Kubernetes
2. Istio must be turned off
3. Metric connected_peers = 4 on all peers
4. Turn on istio, restart the pods. Metric connected_peers = 4
5. Restart the pods again.
6. Check the metric for peer0, peer1, peer4 connected_peers = 4,
7. Check the connected_peers metric for peer2 and peer3. It should be equal to connected_peers = 3.

Peculiarities

1. There are no errors in the logs related to the connection between peers
2. Go to peer 3 (where connected_peers = 3), make a transaction - it should receive commit status.
3. We go to another peer, for example peer1, where connected_peers = 4 - the transaction should be displayed in this peer

Actual result

The metric for peer0, peer1, peer4 connected_peers = 4,
The connected_peers metric for peer2 and peer3. It should be equal to connected_peers = 3.

Expected result

The metric connected_peers should be equal 4 in all peers

Logs in JSON format

Log contents

see attachments

Who can help to reproduce?

@BAStos525

Notes

No response

[BAStos525]

We also can see this metric dashboard in Grafana for internal usage (DM me).

[Erigara]

Let's check first that it's metric issue and not smt with connectivity.
Provided instructions to @BAStos525 how to check this.

[BAStos525]

1. Check on iroha environment, where there are four peers in docker containers on different machines. Correct connected_peers metric value. I substitute the domain addresses of machines from trusted peer config file, the message is not sent to nc. It is also not sent via the IP addresses of the machines.
2. At another stand, where all four peers are in a container on one machine. Correct connected_peers metric value. They are on the same docker network, so I use the names of the containers, and the message comes.
3. k8s deployment, where istio is enabled. Wrong connected_peers metric value. Responses via nc come through the FQDN of the pods, as well as through the local peer addresses or peer service addresses.