gRPC connection drop in fabric-nodejs

[tittuvarghese]

When I use fabric node sdk for invokes, at times getting the below connection drop errors and during retry it's getting executed without any issues.

2023-01-12T12:25:49.557Z - error: [Endorser]: sendProposal[peer2-org1.example.com:30002] - Received error response from: grpcs://peer2-org1.example.com:30002 error: Error: 14 UNAVAILABLE: Connection dropped
2023-01-12T12:25:49.557Z - error: [Endorser]: sendProposal[peer2-org1.example.com:30002] - rejecting with: Error: 14 UNAVAILABLE: Connection dropped
2023-01-12T12:25:49.558Z - error: [Endorser]: sendProposal[peer3-org0.example.com:30002] - Received error response from: grpcs://peer3-org0.example.com:30002 error: Error: 14 UNAVAILABLE: Connection dropped
2023-01-12T12:25:49.558Z - error: [Endorser]: sendProposal[peer3-org0.example.com:30002] - rejecting with: Error: 14 UNAVAILABLE: Connection dropped
2023-01-12T12:25:49.558Z - error: [Endorser]: sendProposal[peer3-org2.example.com:30002] - Received error response from: grpcs://peer3-org2.example.com:30002 error: Error: 14 UNAVAILABLE: Connection dropped
2023-01-12T12:25:49.558Z - error: [Endorser]: sendProposal[peer3-org2.example.com:30002] - rejecting with: Error: 14 UNAVAILABLE: Connection dropped
2023-01-12T12:25:49.558Z - error: [Endorser]: sendProposal[peer3-org1.example.com:30002] - Received error response from: grpcs://peer3-org1.example.com:30002 error: Error: 14 UNAVAILABLE: Connection dropped
2023-01-12T12:25:49.558Z - error: [Endorser]: sendProposal[peer3-org1.example.com:30002] - rejecting with: Error: 14 UNAVAILABLE: Connection dropped
2023-01-12T12:25:49.559Z - error: [Endorser]: sendProposal[peer2-org0.example.com:30002] - Received error response from: grpcs://peer2-org0.example.com:30002 error: Error: 14 UNAVAILABLE: Connection dropped
2023-01-12T12:25:49.559Z - error: [Endorser]: sendProposal[peer2-org0.example.com:30002] - rejecting with: Error: 14 UNAVAILABLE: Connection dropped
2023-01-12T12:25:49.559Z - error: [Endorser]: sendProposal[peer4-org2.example.com:30002] - Received error response from: grpcs://peer4-org2.example.com:30002 error: Error: 14 UNAVAILABLE: Connection dropped
2023-01-12T12:25:49.559Z - error: [Endorser]: sendProposal[peer4-org2.example.com:30002] - rejecting with: Error: 14 UNAVAILABLE: Connection dropped
peer=peer2-org0.example.com:30002, status=grpc, message=14 UNAVAILABLE: Connection dropped
peer=peer3-org0.example.com:30002, status=grpc, message=14 UNAVAILABLE: Connection dropped
peer=peer3-org1.example.com:30002, status=grpc, message=14 UNAVAILABLE: Connection dropped
peer=peer2-org1.example.com:30002, status=grpc, message=14 UNAVAILABLE: Connection dropped
peer=peer3-org2.example.com:30002, status=grpc, message=14 UNAVAILABLE: Connection dropped
peer=peer4-org2.example.com:30002, status=grpc, message=14 UNAVAILABLE: Connection dropped
peer=peer2-org0.example.com:30002, status=grpc, message=14 UNAVAILABLE: Connection dropped
peer=peer3-org0.example.com:30002, status=grpc, message=14 UNAVAILABLE: Connection dropped
peer=peer3-org1.example.com:30002, status=grpc, message=14 UNAVAILABLE: Connection dropped
peer=peer2-org1.example.com:30002, status=grpc, message=14 UNAVAILABLE: Connection dropped
peer=peer3-org2.example.com:30002, status=grpc, message=14 UNAVAILABLE: Connection dropped
peer=peer4-org2.example.com:30002, status=grpc, message=14 UNAVAILABLE: Connection dropped

[bestbeforetoday]

It might be the connections to the peers are being closed by something along the network path (like an ingress controller or firewall) if they are idle too long. If this is the case, configure appropriate gRPC keep-alive settings for your environment:

• https://hyperledger.github.io/fabric-sdk-node/release-2.2/tutorial-grpc-settings.html
• https://github.com/grpc/grpc/blob/master/doc/keepalive.md

[tittuvarghese]

it is passing through HAProxy and Nginx Ingress controller. Whenever connection drop happens, it drops for all peers in same time.

[tittuvarghese]

it is passing through HAProxy and Nginx Ingress controller. Whenever connection drop happens, it drops for all peers in same time.

[tittuvarghese]

it is passing through HAProxy and Nginx Ingress controller. Whenever connection drop happens, it drops for all peers in same time.

[Ethereumx]

I am facing the same issue and no firewall is used

[bestbeforetoday]

Is this an issue that has only started occurring after picking up a newer Node SDK version? If so, do you know the versions that worked and the one that is demonstrating problems? If not, do configuring appropriate keepalive settings solve it?

[bestbeforetoday]

@tittuvarghese Did you have any luck solving the disconnection issues by configuring appropriate keep-alive and related gRPC settings?

[tittuvarghese]

@bestbeforetoday I started getting stable connection after removing Haproxy from the connection path (tried different configs in haproxy but nothing worked out).

But sill I observe some drop connections / commit failures randomly at the orderer level. (This started coming when we distributed the orderer geographically. (between two different region with in the country))

[filipeandre]

Are there any settings for HTTP/2 connections, such as load balancer sticky sessions?

Regarding Haproxy, you have to adjust your "timeout client" and "timeout server" as well as the keep-alive settings.