Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

No TLSHANDSHAKETIMESHIFT for intermediate Ca Certificates #3238

Closed
afrancoc2000 opened this issue Feb 18, 2022 · 13 comments
Closed

No TLSHANDSHAKETIMESHIFT for intermediate Ca Certificates #3238

afrancoc2000 opened this issue Feb 18, 2022 · 13 comments

Comments

@afrancoc2000
Copy link
Contributor

afrancoc2000 commented Feb 18, 2022 •
edited
Loading

Hy everyone,

I'm having a problem with expired certificates, I'm running a cluster with several channels and I updated the certificates on time for most of them, but I forgot about a channel that was created and used only during the first initialization of the cluster.

So today in my test environment the orderers got restarted and now they don't start, my production environment has the same problem but as they haven't been restarted they are running as if nothing has changed, for now.

So, I did as it is said, in this issue and changed these environment variables:

  • ORDERER_GENERAL_AUTHENTICATION_NOEXPIRATIONCHECKS: "true"
  • ORDERER_GENERAL_CLUSTER_TLSHANDSHAKETIMESHIFT: 96h
  • ORDERER_GENERAL_TLS_TLSHANDSHAKETIMESHIFT: 96h
  • ORDERER_OPERATIONS_TLS_TLSHANDSHAKETIMESHIFT: 96h
  • ORDERER_ADMIN_TLS_TLSHANDSHAKETIMESHIFT: 96h

The only handshake time shift still missing is the one from kafta that I don't use, I'm using Raft, but still, I'm getting this error trying to run the orderers:

PANI 00d Error creating ledger resources: error creating channelconfig bundle: initializing channelconfig failed: could not create channel Orderer sub-group config: setting up the MSP manager failed: CA Certificate is not valid, (SN: 512917139581241201287378615618194486727802882): could not obtain certification chain: the supplied identity is not valid: x509: certificate has expired or is not yet valid: current time 2022-02-17T14:28:20Z is after 2022-02-16T16:41:01Z

I published the issue in the message channel and with the help of @yacovm found that there's no TLS Handshake time shift for the CA certificates.

Here's the stack trace:

Thu, Feb 17 2022 9:28:20 am	2022-02-17 14:28:20.681 UTC [localconfig] completeInitialization -> WARN 001 General.GenesisFile should be replaced by General.BootstrapFile
Thu, Feb 17 2022 9:28:20 am	2022-02-17 14:28:20.681 UTC [localconfig] completeInitialization -> INFO 002 Kafka.Version unset, setting to 0.10.2.0
Thu, Feb 17 2022 9:28:20 am	2022-02-17 14:28:20.681 UTC [orderer.common.server] prettyPrintStruct -> INFO 003 Orderer config values:
Thu, Feb 17 2022 9:28:20 am	General.ListenAddress = "0.0.0.0"
Thu, Feb 17 2022 9:28:20 am	General.ListenPort = 7050
Thu, Feb 17 2022 9:28:20 am	General.TLS.Enabled = true
Thu, Feb 17 2022 9:28:20 am	General.TLS.PrivateKey = "/var/hyperledger/orderer/tls/server.key"
Thu, Feb 17 2022 9:28:20 am	General.TLS.Certificate = "/var/hyperledger/orderer/tls/server.crt"
Thu, Feb 17 2022 9:28:20 am	General.TLS.RootCAs = [/var/hyperledger/orderer/tls/chain.crt]
Thu, Feb 17 2022 9:28:20 am	General.TLS.ClientAuthRequired = true
Thu, Feb 17 2022 9:28:20 am	General.TLS.ClientRootCAs = [/var/hyperledger/orderer/tls/chain.crt]
Thu, Feb 17 2022 9:28:20 am	General.TLS.TLSHandshakeTimeShift = 96h0m0s
Thu, Feb 17 2022 9:28:20 am	General.Cluster.ListenAddress = ""
Thu, Feb 17 2022 9:28:20 am	General.Cluster.ListenPort = 0
Thu, Feb 17 2022 9:28:20 am	General.Cluster.ServerCertificate = ""
Thu, Feb 17 2022 9:28:20 am	General.Cluster.ServerPrivateKey = ""
Thu, Feb 17 2022 9:28:20 am	General.Cluster.ClientCertificate = "/var/hyperledger/orderer/tls/server.crt"
Thu, Feb 17 2022 9:28:20 am	General.Cluster.ClientPrivateKey = "/var/hyperledger/orderer/tls/server.key"
Thu, Feb 17 2022 9:28:20 am	General.Cluster.RootCAs = [/var/hyperledger/orderer/tls/chain.crt]
Thu, Feb 17 2022 9:28:20 am	General.Cluster.DialTimeout = 5s
Thu, Feb 17 2022 9:28:20 am	General.Cluster.RPCTimeout = 7s
Thu, Feb 17 2022 9:28:20 am	General.Cluster.ReplicationBufferSize = 20971520
Thu, Feb 17 2022 9:28:20 am	General.Cluster.ReplicationPullTimeout = 5s
Thu, Feb 17 2022 9:28:20 am	General.Cluster.ReplicationRetryTimeout = 5s
Thu, Feb 17 2022 9:28:20 am	General.Cluster.ReplicationBackgroundRefreshInterval = 5m0s
Thu, Feb 17 2022 9:28:20 am	General.Cluster.ReplicationMaxRetries = 12
Thu, Feb 17 2022 9:28:20 am	General.Cluster.SendBufferSize = 10
Thu, Feb 17 2022 9:28:20 am	General.Cluster.CertExpirationWarningThreshold = 168h0m0s
Thu, Feb 17 2022 9:28:20 am	General.Cluster.TLSHandshakeTimeShift = 96h0m0s
Thu, Feb 17 2022 9:28:20 am	General.Keepalive.ServerMinInterval = 1m0s
Thu, Feb 17 2022 9:28:20 am	General.Keepalive.ServerInterval = 2h0m0s
Thu, Feb 17 2022 9:28:20 am	General.Keepalive.ServerTimeout = 20s
Thu, Feb 17 2022 9:28:20 am	General.ConnectionTimeout = 0s
Thu, Feb 17 2022 9:28:20 am	General.GenesisMethod = "file"
Thu, Feb 17 2022 9:28:20 am	General.GenesisFile = "/var/hyperledger/orderer/genesis.block"
Thu, Feb 17 2022 9:28:20 am	General.BootstrapMethod = "file"
Thu, Feb 17 2022 9:28:20 am	General.BootstrapFile = "/var/hyperledger/orderer/genesis.block"
Thu, Feb 17 2022 9:28:20 am	General.Profile.Enabled = false
Thu, Feb 17 2022 9:28:20 am	General.Profile.Address = "0.0.0.0:6060"
Thu, Feb 17 2022 9:28:20 am	General.LocalMSPDir = "/var/hyperledger/orderer/msp"
Thu, Feb 17 2022 9:28:20 am	General.LocalMSPID = "company"
Thu, Feb 17 2022 9:28:20 am	General.BCCSP.Default = "SW"
Thu, Feb 17 2022 9:28:20 am	General.BCCSP.SW.Security = 256
Thu, Feb 17 2022 9:28:20 am	General.BCCSP.SW.Hash = "SHA2"
Thu, Feb 17 2022 9:28:20 am	General.BCCSP.SW.FileKeystore.KeyStorePath = ""
Thu, Feb 17 2022 9:28:20 am	General.Authentication.TimeWindow = 15m0s
Thu, Feb 17 2022 9:28:20 am	General.Authentication.NoExpirationChecks = true
Thu, Feb 17 2022 9:28:20 am	General.MaxRecvMsgSize = 104857600
Thu, Feb 17 2022 9:28:20 am	General.MaxSendMsgSize = 104857600
Thu, Feb 17 2022 9:28:20 am	FileLedger.Location = "/var/hyperledger/production/orderer"
Thu, Feb 17 2022 9:28:20 am	FileLedger.Prefix = ""
Thu, Feb 17 2022 9:28:20 am	Kafka.Retry.ShortInterval = 5s
Thu, Feb 17 2022 9:28:20 am	Kafka.Retry.ShortTotal = 10m0s
Thu, Feb 17 2022 9:28:20 am	Kafka.Retry.LongInterval = 5m0s
Thu, Feb 17 2022 9:28:20 am	Kafka.Retry.LongTotal = 12h0m0s
Thu, Feb 17 2022 9:28:20 am	Kafka.Retry.NetworkTimeouts.DialTimeout = 10s
Thu, Feb 17 2022 9:28:20 am	Kafka.Retry.NetworkTimeouts.ReadTimeout = 10s
Thu, Feb 17 2022 9:28:20 am	Kafka.Retry.NetworkTimeouts.WriteTimeout = 10s
Thu, Feb 17 2022 9:28:20 am	Kafka.Retry.Metadata.RetryMax = 3
Thu, Feb 17 2022 9:28:20 am	Kafka.Retry.Metadata.RetryBackoff = 250ms
Thu, Feb 17 2022 9:28:20 am	Kafka.Retry.Producer.RetryMax = 3
Thu, Feb 17 2022 9:28:20 am	Kafka.Retry.Producer.RetryBackoff = 100ms
Thu, Feb 17 2022 9:28:20 am	Kafka.Retry.Consumer.RetryBackoff = 2s
Thu, Feb 17 2022 9:28:20 am	Kafka.Verbose = false
Thu, Feb 17 2022 9:28:20 am	Kafka.Version = 0.10.2.0
Thu, Feb 17 2022 9:28:20 am	Kafka.TLS.Enabled = false
Thu, Feb 17 2022 9:28:20 am	Kafka.TLS.PrivateKey = ""
Thu, Feb 17 2022 9:28:20 am	Kafka.TLS.Certificate = ""
Thu, Feb 17 2022 9:28:20 am	Kafka.TLS.RootCAs = []
Thu, Feb 17 2022 9:28:20 am	Kafka.TLS.ClientAuthRequired = false
Thu, Feb 17 2022 9:28:20 am	Kafka.TLS.ClientRootCAs = []
Thu, Feb 17 2022 9:28:20 am	Kafka.TLS.TLSHandshakeTimeShift = 0s
Thu, Feb 17 2022 9:28:20 am	Kafka.SASLPlain.Enabled = false
Thu, Feb 17 2022 9:28:20 am	Kafka.SASLPlain.User = ""
Thu, Feb 17 2022 9:28:20 am	Kafka.SASLPlain.Password = ""
Thu, Feb 17 2022 9:28:20 am	Kafka.Topic.ReplicationFactor = 3
Thu, Feb 17 2022 9:28:20 am	Debug.BroadcastTraceDir = ""
Thu, Feb 17 2022 9:28:20 am	Debug.DeliverTraceDir = ""
Thu, Feb 17 2022 9:28:20 am	Consensus = map[SnapDir:/var/hyperledger/production/orderer/etcdraft/snapshot WALDir:/var/hyperledger/production/orderer/etcdraft/wal]
Thu, Feb 17 2022 9:28:20 am	Operations.ListenAddress = ":8443"
Thu, Feb 17 2022 9:28:20 am	Operations.TLS.Enabled = false
Thu, Feb 17 2022 9:28:20 am	Operations.TLS.PrivateKey = ""
Thu, Feb 17 2022 9:28:20 am	Operations.TLS.Certificate = ""
Thu, Feb 17 2022 9:28:20 am	Operations.TLS.RootCAs = []
Thu, Feb 17 2022 9:28:20 am	Operations.TLS.ClientAuthRequired = false
Thu, Feb 17 2022 9:28:20 am	Operations.TLS.ClientRootCAs = []
Thu, Feb 17 2022 9:28:20 am	Operations.TLS.TLSHandshakeTimeShift = 96h0m0s
Thu, Feb 17 2022 9:28:20 am	Metrics.Provider = "prometheus"
Thu, Feb 17 2022 9:28:20 am	Metrics.Statsd.Network = "udp"
Thu, Feb 17 2022 9:28:20 am	Metrics.Statsd.Address = "127.0.0.1:8125"
Thu, Feb 17 2022 9:28:20 am	Metrics.Statsd.WriteInterval = 30s
Thu, Feb 17 2022 9:28:20 am	Metrics.Statsd.Prefix = ""
Thu, Feb 17 2022 9:28:20 am	ChannelParticipation.Enabled = false
Thu, Feb 17 2022 9:28:20 am	ChannelParticipation.MaxRequestBodySize = 1048576
Thu, Feb 17 2022 9:28:20 am	Admin.ListenAddress = "127.0.0.1:9443"
Thu, Feb 17 2022 9:28:20 am	Admin.TLS.Enabled = false
Thu, Feb 17 2022 9:28:20 am	Admin.TLS.PrivateKey = ""
Thu, Feb 17 2022 9:28:20 am	Admin.TLS.Certificate = ""
Thu, Feb 17 2022 9:28:20 am	Admin.TLS.RootCAs = []
Thu, Feb 17 2022 9:28:20 am	Admin.TLS.ClientAuthRequired = true
Thu, Feb 17 2022 9:28:20 am	Admin.TLS.ClientRootCAs = []
Thu, Feb 17 2022 9:28:20 am	Admin.TLS.TLSHandshakeTimeShift = 96h0m0s
Thu, Feb 17 2022 9:28:20 am	2022-02-17 14:28:20.698 UTC [orderer.common.server] initializeServerConfig -> INFO 004 Starting orderer with mutual TLS enabled
Thu, Feb 17 2022 9:28:20 am	2022-02-17 14:28:20.730 UTC [orderer.common.server] Main -> INFO 005 Not bootstrapping the system channel because of existing channels
Thu, Feb 17 2022 9:28:20 am	2022-02-17 14:28:20.800 UTC [orderer.common.server] Main -> INFO 006 Starting without a system channel
Thu, Feb 17 2022 9:28:20 am	2022-02-17 14:28:20.800 UTC [orderer.common.server] Main -> INFO 007 Setting up cluster
Thu, Feb 17 2022 9:28:20 am	2022-02-17 14:28:20.800 UTC [orderer.common.server] reuseListener -> INFO 008 Cluster listener is not configured, defaulting to use the general listener on port 7050
Thu, Feb 17 2022 9:28:20 am	2022-02-17 14:28:20.800 UTC [orderer.common.server] reuseListener -> INFO 009 Cluster listener is not configured, defaulting to use the general listener on port 7050
Thu, Feb 17 2022 9:28:20 am	2022-02-17 14:28:20.800 UTC [certmonitor] trackCertExpiration -> INFO 00a The enrollment certificate will expire on 2023-01-25 23:25:48 +0000 UTC
Thu, Feb 17 2022 9:28:20 am	2022-02-17 14:28:20.800 UTC [certmonitor] trackCertExpiration -> INFO 00b The server TLS certificate will expire on 2023-01-25 23:25:16 +0000 UTC
Thu, Feb 17 2022 9:28:20 am	2022-02-17 14:28:20.801 UTC [certmonitor] trackCertExpiration -> INFO 00c The client TLS certificate will expire on 2023-01-25 23:25:16 +0000 UTC
Thu, Feb 17 2022 9:28:20 am	2022-02-17 14:28:20.825 UTC [orderer.commmon.multichannel] initSystemChannel -> PANI 00d Error creating ledger resources: error creating channelconfig bundle: initializing channelconfig failed: could not create channel Orderer sub-group config: setting up the MSP manager failed: CA Certificate is not valid, (SN: 512917139581241201287378615618194486727802882): could not obtain certification chain: the supplied identity is not valid: x509: certificate has expired or is not yet valid: current time 2022-02-17T14:28:20Z is after 2022-02-16T16:41:01Z
Thu, Feb 17 2022 9:28:20 am	panic: Error creating ledger resources: error creating channelconfig bundle: initializing channelconfig failed: could not create channel Orderer sub-group config: setting up the MSP manager failed: CA Certificate is not valid, (SN: 512917139581241201287378615618194486727802882): could not obtain certification chain: the supplied identity is not valid: x509: certificate has expired or is not yet valid: current time 2022-02-17T14:28:20Z is after 2022-02-16T16:41:01Z
Thu, Feb 17 2022 9:28:20 am	
Thu, Feb 17 2022 9:28:20 am	goroutine 1 [running]:
Thu, Feb 17 2022 9:28:20 am	go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc0002620b0, 0x0, 0x0, 0x0)
Thu, Feb 17 2022 9:28:20 am	/go/src/github.com/hyperledger/fabric/vendor/go.uber.org/zap/zapcore/entry.go:230 +0x565
Thu, Feb 17 2022 9:28:20 am	go.uber.org/zap.(*SugaredLogger).log(0xc000130328, 0xc00069c104, 0xfdab86, 0x23, 0xc0002d0188, 0x1, 0x1, 0x0, 0x0, 0x0)
Thu, Feb 17 2022 9:28:20 am	/go/src/github.com/hyperledger/fabric/vendor/go.uber.org/zap/sugar.go:234 +0xf6
Thu, Feb 17 2022 9:28:20 am	go.uber.org/zap.(*SugaredLogger).Panicf(...)
Thu, Feb 17 2022 9:28:20 am	/go/src/github.com/hyperledger/fabric/vendor/go.uber.org/zap/sugar.go:159
Thu, Feb 17 2022 9:28:20 am	github.com/hyperledger/fabric/common/flogging.(*FabricLogger).Panicf(...)
Thu, Feb 17 2022 9:28:20 am	/go/src/github.com/hyperledger/fabric/common/flogging/zap.go:74
Thu, Feb 17 2022 9:28:20 am	github.com/hyperledger/fabric/orderer/common/multichannel.(*Registrar).initSystemChannel(0xc000703700, 0xc0007ac040, 0x4, 0x4)
Thu, Feb 17 2022 9:28:20 am	/go/src/github.com/hyperledger/fabric/orderer/common/multichannel/registrar.go:233 +0x22e
Thu, Feb 17 2022 9:28:20 am	github.com/hyperledger/fabric/orderer/common/multichannel.(*Registrar).init(0xc000703700, 0xc0007c1890)
Thu, Feb 17 2022 9:28:20 am	/go/src/github.com/hyperledger/fabric/orderer/common/multichannel/registrar.go:163 +0xa5
Thu, Feb 17 2022 9:28:20 am	github.com/hyperledger/fabric/orderer/common/multichannel.(*Registrar).Initialize(0xc000703700, 0xc0007c1890)
Thu, Feb 17 2022 9:28:20 am	/go/src/github.com/hyperledger/fabric/orderer/common/multichannel/registrar.go:143 +0x46
Thu, Feb 17 2022 9:28:20 am	github.com/hyperledger/fabric/orderer/common/server.initializeMultichannelRegistrar(0x0, 0x0, 0xc00070aa50, 0x0, 0x0, 0xc0000c8900, 0x852, 0x853, 0xc000070c00, 0xf1, ...)
Thu, Feb 17 2022 9:28:20 am	/go/src/github.com/hyperledger/fabric/orderer/common/server/main.go:828 +0x485
Thu, Feb 17 2022 9:28:20 am	github.com/hyperledger/fabric/orderer/common/server.Main()
Thu, Feb 17 2022 9:28:20 am	/go/src/github.com/hyperledger/fabric/orderer/common/server/main.go:245 +0x10ef
Thu, Feb 17 2022 9:28:20 am	main.main()
Thu, Feb 17 2022 9:28:20 am	/go/src/github.com/hyperledger/fabric/cmd/orderer/main.go:15 +0x25

Thanks
@yacovm
Copy link
Contributor

yacovm commented Feb 18, 2022

The problem is not that the TLS handshake timeshift is not honored by the channel config, but rather that the MSP setup of TLS CA certificates (or the setup of enrollment CAs) doesn't use the same trick that is used for identities.

This means that if any CA certificate in the network (present in the channel config) expired, all nodes (orderers and peers alike) will forever be in a crash loop.

I think we should use the same time override that we do for validating identities in the TLS (and regular) CA setup in the MSP.

Thoughts? @denyeart @adecaro @ale-linux

@yacovm
Copy link
Contributor

yacovm commented Feb 18, 2022

@afrancoc2000 do you mind editing your issue and putting the entire stack trace so it will be more evident what the problem is?

@afrancoc2000
Copy link
Contributor Author

afrancoc2000 commented Feb 18, 2022 •
edited
Loading

Sure @yacovm, here's the entire stack trace:

Thu, Feb 17 2022 9:28:20 am	2022-02-17 14:28:20.681 UTC [localconfig] completeInitialization -> WARN 001 General.GenesisFile should be replaced by General.BootstrapFile
Thu, Feb 17 2022 9:28:20 am	2022-02-17 14:28:20.681 UTC [localconfig] completeInitialization -> INFO 002 Kafka.Version unset, setting to 0.10.2.0
Thu, Feb 17 2022 9:28:20 am	2022-02-17 14:28:20.681 UTC [orderer.common.server] prettyPrintStruct -> INFO 003 Orderer config values:
Thu, Feb 17 2022 9:28:20 am	General.ListenAddress = "0.0.0.0"
Thu, Feb 17 2022 9:28:20 am	General.ListenPort = 7050
Thu, Feb 17 2022 9:28:20 am	General.TLS.Enabled = true
Thu, Feb 17 2022 9:28:20 am	General.TLS.PrivateKey = "/var/hyperledger/orderer/tls/server.key"
Thu, Feb 17 2022 9:28:20 am	General.TLS.Certificate = "/var/hyperledger/orderer/tls/server.crt"
Thu, Feb 17 2022 9:28:20 am	General.TLS.RootCAs = [/var/hyperledger/orderer/tls/chain.crt]
Thu, Feb 17 2022 9:28:20 am	General.TLS.ClientAuthRequired = true
Thu, Feb 17 2022 9:28:20 am	General.TLS.ClientRootCAs = [/var/hyperledger/orderer/tls/chain.crt]
Thu, Feb 17 2022 9:28:20 am	General.TLS.TLSHandshakeTimeShift = 96h0m0s
Thu, Feb 17 2022 9:28:20 am	General.Cluster.ListenAddress = ""
Thu, Feb 17 2022 9:28:20 am	General.Cluster.ListenPort = 0
Thu, Feb 17 2022 9:28:20 am	General.Cluster.ServerCertificate = ""
Thu, Feb 17 2022 9:28:20 am	General.Cluster.ServerPrivateKey = ""
Thu, Feb 17 2022 9:28:20 am	General.Cluster.ClientCertificate = "/var/hyperledger/orderer/tls/server.crt"
Thu, Feb 17 2022 9:28:20 am	General.Cluster.ClientPrivateKey = "/var/hyperledger/orderer/tls/server.key"
Thu, Feb 17 2022 9:28:20 am	General.Cluster.RootCAs = [/var/hyperledger/orderer/tls/chain.crt]
Thu, Feb 17 2022 9:28:20 am	General.Cluster.DialTimeout = 5s
Thu, Feb 17 2022 9:28:20 am	General.Cluster.RPCTimeout = 7s
Thu, Feb 17 2022 9:28:20 am	General.Cluster.ReplicationBufferSize = 20971520
Thu, Feb 17 2022 9:28:20 am	General.Cluster.ReplicationPullTimeout = 5s
Thu, Feb 17 2022 9:28:20 am	General.Cluster.ReplicationRetryTimeout = 5s
Thu, Feb 17 2022 9:28:20 am	General.Cluster.ReplicationBackgroundRefreshInterval = 5m0s
Thu, Feb 17 2022 9:28:20 am	General.Cluster.ReplicationMaxRetries = 12
Thu, Feb 17 2022 9:28:20 am	General.Cluster.SendBufferSize = 10
Thu, Feb 17 2022 9:28:20 am	General.Cluster.CertExpirationWarningThreshold = 168h0m0s
Thu, Feb 17 2022 9:28:20 am	General.Cluster.TLSHandshakeTimeShift = 96h0m0s
Thu, Feb 17 2022 9:28:20 am	General.Keepalive.ServerMinInterval = 1m0s
Thu, Feb 17 2022 9:28:20 am	General.Keepalive.ServerInterval = 2h0m0s
Thu, Feb 17 2022 9:28:20 am	General.Keepalive.ServerTimeout = 20s
Thu, Feb 17 2022 9:28:20 am	General.ConnectionTimeout = 0s
Thu, Feb 17 2022 9:28:20 am	General.GenesisMethod = "file"
Thu, Feb 17 2022 9:28:20 am	General.GenesisFile = "/var/hyperledger/orderer/genesis.block"
Thu, Feb 17 2022 9:28:20 am	General.BootstrapMethod = "file"
Thu, Feb 17 2022 9:28:20 am	General.BootstrapFile = "/var/hyperledger/orderer/genesis.block"
Thu, Feb 17 2022 9:28:20 am	General.Profile.Enabled = false
Thu, Feb 17 2022 9:28:20 am	General.Profile.Address = "0.0.0.0:6060"
Thu, Feb 17 2022 9:28:20 am	General.LocalMSPDir = "/var/hyperledger/orderer/msp"
Thu, Feb 17 2022 9:28:20 am	General.LocalMSPID = "company"
Thu, Feb 17 2022 9:28:20 am	General.BCCSP.Default = "SW"
Thu, Feb 17 2022 9:28:20 am	General.BCCSP.SW.Security = 256
Thu, Feb 17 2022 9:28:20 am	General.BCCSP.SW.Hash = "SHA2"
Thu, Feb 17 2022 9:28:20 am	General.BCCSP.SW.FileKeystore.KeyStorePath = ""
Thu, Feb 17 2022 9:28:20 am	General.Authentication.TimeWindow = 15m0s
Thu, Feb 17 2022 9:28:20 am	General.Authentication.NoExpirationChecks = true
Thu, Feb 17 2022 9:28:20 am	General.MaxRecvMsgSize = 104857600
Thu, Feb 17 2022 9:28:20 am	General.MaxSendMsgSize = 104857600
Thu, Feb 17 2022 9:28:20 am	FileLedger.Location = "/var/hyperledger/production/orderer"
Thu, Feb 17 2022 9:28:20 am	FileLedger.Prefix = ""
Thu, Feb 17 2022 9:28:20 am	Kafka.Retry.ShortInterval = 5s
Thu, Feb 17 2022 9:28:20 am	Kafka.Retry.ShortTotal = 10m0s
Thu, Feb 17 2022 9:28:20 am	Kafka.Retry.LongInterval = 5m0s
Thu, Feb 17 2022 9:28:20 am	Kafka.Retry.LongTotal = 12h0m0s
Thu, Feb 17 2022 9:28:20 am	Kafka.Retry.NetworkTimeouts.DialTimeout = 10s
Thu, Feb 17 2022 9:28:20 am	Kafka.Retry.NetworkTimeouts.ReadTimeout = 10s
Thu, Feb 17 2022 9:28:20 am	Kafka.Retry.NetworkTimeouts.WriteTimeout = 10s
Thu, Feb 17 2022 9:28:20 am	Kafka.Retry.Metadata.RetryMax = 3
Thu, Feb 17 2022 9:28:20 am	Kafka.Retry.Metadata.RetryBackoff = 250ms
Thu, Feb 17 2022 9:28:20 am	Kafka.Retry.Producer.RetryMax = 3
Thu, Feb 17 2022 9:28:20 am	Kafka.Retry.Producer.RetryBackoff = 100ms
Thu, Feb 17 2022 9:28:20 am	Kafka.Retry.Consumer.RetryBackoff = 2s
Thu, Feb 17 2022 9:28:20 am	Kafka.Verbose = false
Thu, Feb 17 2022 9:28:20 am	Kafka.Version = 0.10.2.0
Thu, Feb 17 2022 9:28:20 am	Kafka.TLS.Enabled = false
Thu, Feb 17 2022 9:28:20 am	Kafka.TLS.PrivateKey = ""
Thu, Feb 17 2022 9:28:20 am	Kafka.TLS.Certificate = ""
Thu, Feb 17 2022 9:28:20 am	Kafka.TLS.RootCAs = []
Thu, Feb 17 2022 9:28:20 am	Kafka.TLS.ClientAuthRequired = false
Thu, Feb 17 2022 9:28:20 am	Kafka.TLS.ClientRootCAs = []
Thu, Feb 17 2022 9:28:20 am	Kafka.TLS.TLSHandshakeTimeShift = 0s
Thu, Feb 17 2022 9:28:20 am	Kafka.SASLPlain.Enabled = false
Thu, Feb 17 2022 9:28:20 am	Kafka.SASLPlain.User = ""
Thu, Feb 17 2022 9:28:20 am	Kafka.SASLPlain.Password = ""
Thu, Feb 17 2022 9:28:20 am	Kafka.Topic.ReplicationFactor = 3
Thu, Feb 17 2022 9:28:20 am	Debug.BroadcastTraceDir = ""
Thu, Feb 17 2022 9:28:20 am	Debug.DeliverTraceDir = ""
Thu, Feb 17 2022 9:28:20 am	Consensus = map[SnapDir:/var/hyperledger/production/orderer/etcdraft/snapshot WALDir:/var/hyperledger/production/orderer/etcdraft/wal]
Thu, Feb 17 2022 9:28:20 am	Operations.ListenAddress = ":8443"
Thu, Feb 17 2022 9:28:20 am	Operations.TLS.Enabled = false
Thu, Feb 17 2022 9:28:20 am	Operations.TLS.PrivateKey = ""
Thu, Feb 17 2022 9:28:20 am	Operations.TLS.Certificate = ""
Thu, Feb 17 2022 9:28:20 am	Operations.TLS.RootCAs = []
Thu, Feb 17 2022 9:28:20 am	Operations.TLS.ClientAuthRequired = false
Thu, Feb 17 2022 9:28:20 am	Operations.TLS.ClientRootCAs = []
Thu, Feb 17 2022 9:28:20 am	Operations.TLS.TLSHandshakeTimeShift = 96h0m0s
Thu, Feb 17 2022 9:28:20 am	Metrics.Provider = "prometheus"
Thu, Feb 17 2022 9:28:20 am	Metrics.Statsd.Network = "udp"
Thu, Feb 17 2022 9:28:20 am	Metrics.Statsd.Address = "127.0.0.1:8125"
Thu, Feb 17 2022 9:28:20 am	Metrics.Statsd.WriteInterval = 30s
Thu, Feb 17 2022 9:28:20 am	Metrics.Statsd.Prefix = ""
Thu, Feb 17 2022 9:28:20 am	ChannelParticipation.Enabled = false
Thu, Feb 17 2022 9:28:20 am	ChannelParticipation.MaxRequestBodySize = 1048576
Thu, Feb 17 2022 9:28:20 am	Admin.ListenAddress = "127.0.0.1:9443"
Thu, Feb 17 2022 9:28:20 am	Admin.TLS.Enabled = false
Thu, Feb 17 2022 9:28:20 am	Admin.TLS.PrivateKey = ""
Thu, Feb 17 2022 9:28:20 am	Admin.TLS.Certificate = ""
Thu, Feb 17 2022 9:28:20 am	Admin.TLS.RootCAs = []
Thu, Feb 17 2022 9:28:20 am	Admin.TLS.ClientAuthRequired = true
Thu, Feb 17 2022 9:28:20 am	Admin.TLS.ClientRootCAs = []
Thu, Feb 17 2022 9:28:20 am	Admin.TLS.TLSHandshakeTimeShift = 96h0m0s
Thu, Feb 17 2022 9:28:20 am	2022-02-17 14:28:20.698 UTC [orderer.common.server] initializeServerConfig -> INFO 004 Starting orderer with mutual TLS enabled
Thu, Feb 17 2022 9:28:20 am	2022-02-17 14:28:20.730 UTC [orderer.common.server] Main -> INFO 005 Not bootstrapping the system channel because of existing channels
Thu, Feb 17 2022 9:28:20 am	2022-02-17 14:28:20.800 UTC [orderer.common.server] Main -> INFO 006 Starting without a system channel
Thu, Feb 17 2022 9:28:20 am	2022-02-17 14:28:20.800 UTC [orderer.common.server] Main -> INFO 007 Setting up cluster
Thu, Feb 17 2022 9:28:20 am	2022-02-17 14:28:20.800 UTC [orderer.common.server] reuseListener -> INFO 008 Cluster listener is not configured, defaulting to use the general listener on port 7050
Thu, Feb 17 2022 9:28:20 am	2022-02-17 14:28:20.800 UTC [orderer.common.server] reuseListener -> INFO 009 Cluster listener is not configured, defaulting to use the general listener on port 7050
Thu, Feb 17 2022 9:28:20 am	2022-02-17 14:28:20.800 UTC [certmonitor] trackCertExpiration -> INFO 00a The enrollment certificate will expire on 2023-01-25 23:25:48 +0000 UTC
Thu, Feb 17 2022 9:28:20 am	2022-02-17 14:28:20.800 UTC [certmonitor] trackCertExpiration -> INFO 00b The server TLS certificate will expire on 2023-01-25 23:25:16 +0000 UTC
Thu, Feb 17 2022 9:28:20 am	2022-02-17 14:28:20.801 UTC [certmonitor] trackCertExpiration -> INFO 00c The client TLS certificate will expire on 2023-01-25 23:25:16 +0000 UTC
Thu, Feb 17 2022 9:28:20 am	2022-02-17 14:28:20.825 UTC [orderer.commmon.multichannel] initSystemChannel -> PANI 00d Error creating ledger resources: error creating channelconfig bundle: initializing channelconfig failed: could not create channel Orderer sub-group config: setting up the MSP manager failed: CA Certificate is not valid, (SN: 512917139581241201287378615618194486727802882): could not obtain certification chain: the supplied identity is not valid: x509: certificate has expired or is not yet valid: current time 2022-02-17T14:28:20Z is after 2022-02-16T16:41:01Z
Thu, Feb 17 2022 9:28:20 am	panic: Error creating ledger resources: error creating channelconfig bundle: initializing channelconfig failed: could not create channel Orderer sub-group config: setting up the MSP manager failed: CA Certificate is not valid, (SN: 512917139581241201287378615618194486727802882): could not obtain certification chain: the supplied identity is not valid: x509: certificate has expired or is not yet valid: current time 2022-02-17T14:28:20Z is after 2022-02-16T16:41:01Z
Thu, Feb 17 2022 9:28:20 am	
Thu, Feb 17 2022 9:28:20 am	goroutine 1 [running]:
Thu, Feb 17 2022 9:28:20 am	go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc0002620b0, 0x0, 0x0, 0x0)
Thu, Feb 17 2022 9:28:20 am	/go/src/github.com/hyperledger/fabric/vendor/go.uber.org/zap/zapcore/entry.go:230 +0x565
Thu, Feb 17 2022 9:28:20 am	go.uber.org/zap.(*SugaredLogger).log(0xc000130328, 0xc00069c104, 0xfdab86, 0x23, 0xc0002d0188, 0x1, 0x1, 0x0, 0x0, 0x0)
Thu, Feb 17 2022 9:28:20 am	/go/src/github.com/hyperledger/fabric/vendor/go.uber.org/zap/sugar.go:234 +0xf6
Thu, Feb 17 2022 9:28:20 am	go.uber.org/zap.(*SugaredLogger).Panicf(...)
Thu, Feb 17 2022 9:28:20 am	/go/src/github.com/hyperledger/fabric/vendor/go.uber.org/zap/sugar.go:159
Thu, Feb 17 2022 9:28:20 am	github.com/hyperledger/fabric/common/flogging.(*FabricLogger).Panicf(...)
Thu, Feb 17 2022 9:28:20 am	/go/src/github.com/hyperledger/fabric/common/flogging/zap.go:74
Thu, Feb 17 2022 9:28:20 am	github.com/hyperledger/fabric/orderer/common/multichannel.(*Registrar).initSystemChannel(0xc000703700, 0xc0007ac040, 0x4, 0x4)
Thu, Feb 17 2022 9:28:20 am	/go/src/github.com/hyperledger/fabric/orderer/common/multichannel/registrar.go:233 +0x22e
Thu, Feb 17 2022 9:28:20 am	github.com/hyperledger/fabric/orderer/common/multichannel.(*Registrar).init(0xc000703700, 0xc0007c1890)
Thu, Feb 17 2022 9:28:20 am	/go/src/github.com/hyperledger/fabric/orderer/common/multichannel/registrar.go:163 +0xa5
Thu, Feb 17 2022 9:28:20 am	github.com/hyperledger/fabric/orderer/common/multichannel.(*Registrar).Initialize(0xc000703700, 0xc0007c1890)
Thu, Feb 17 2022 9:28:20 am	/go/src/github.com/hyperledger/fabric/orderer/common/multichannel/registrar.go:143 +0x46
Thu, Feb 17 2022 9:28:20 am	github.com/hyperledger/fabric/orderer/common/server.initializeMultichannelRegistrar(0x0, 0x0, 0xc00070aa50, 0x0, 0x0, 0xc0000c8900, 0x852, 0x853, 0xc000070c00, 0xf1, ...)
Thu, Feb 17 2022 9:28:20 am	/go/src/github.com/hyperledger/fabric/orderer/common/server/main.go:828 +0x485
Thu, Feb 17 2022 9:28:20 am	github.com/hyperledger/fabric/orderer/common/server.Main()
Thu, Feb 17 2022 9:28:20 am	/go/src/github.com/hyperledger/fabric/orderer/common/server/main.go:245 +0x10ef
Thu, Feb 17 2022 9:28:20 am	main.main()
Thu, Feb 17 2022 9:28:20 am	/go/src/github.com/hyperledger/fabric/cmd/orderer/main.go:15 +0x25

Also I added it to the description

@denyeart
Copy link
Contributor

@yacovm Yes, I think we need a timeshift for regular CA and TLS CA verification. Instead of all of these different timeshift options, what would you think of a single global timeshift that would apply to all of them? Users have a difficult time keeping track of all the different certs and which overrides apply to which ones.

@yacovm
Copy link
Contributor

yacovm commented Feb 18, 2022

@yacovm Yes, I think we need a timeshift for regular CA and TLS CA verification. Instead of all of these different timeshift options, what would you think of a single global timeshift that would apply to all of them? Users have a difficult time keeping track of all the different certs and which overrides apply to which ones.

Please click on the links in my comment and read the code and understand why a "global time shift" or TLSHANDSHAKETIMESHIFT is irrelevant to the problem at hand.

@denyeart
Copy link
Contributor

denyeart commented Feb 18, 2022 •
edited
Loading

@yacovm
Ah, when you said "use the time override" I thought you were talking about the timeshift override.
Rather, you are suggesting to override the check time based on the CA cert NotBefore time so that it ALWAYS passes when validating CA and TLS CA certs.
Are you suggesting this be hardcoded to always pass (like the identity checks that you referenced), or based on a new orderer/peer config option like NoCAExpirationChecks?

@yacovm
Copy link
Contributor

yacovm commented Feb 18, 2022

Are you suggesting this be hardcoded to always pass (like the identity checks that you referenced), or based on a new orderer/peer config option like NoCAExpirationChecks?

The former, of course. The no expiration checks is a check for validation in the admission path of transactions.
Here we're dealing with code path that initializes the channel config bundle.

@denyeart
Copy link
Contributor

Ok, I agree with the proposal.
To take an extreme example... say you shut down a peer for 30 years and then start it back up... it shouldn't fail to start simply because the old CA certs in the channel config have expired. It should be allowed to start and process blocks so that it can catch up to the latest channel config containing the latest CA certs. (Of course, you will likely need to update the peer's own local certs, but that is a different story and under control of the peer admin).

afrancoc2000 pushed a commit to afrancoc2000/fabric that referenced this issue Feb 20, 2022
Ignore expired CA/TLS CA certs on msp init (hyperledger#3238)
e1b8725 
Signed-off-by: Ana Maria Franco <[email protected]>
afrancoc2000 pushed a commit to afrancoc2000/fabric that referenced this issue Feb 20, 2022
Ignore expired CA/TLS CA certs on msp init (hyperledger#3238)
f5238be 
Signed-off-by: Ana Maria Franco <[email protected]>
@afrancoc2000 afrancoc2000 mentioned this issue Feb 20, 2022
Closed
afrancoc2000 pushed a commit to afrancoc2000/fabric that referenced this issue Feb 21, 2022
Ignore expired CA/TLS CA certs on msp init (hyperledger#3238)
d4f3fdb 
Signed-off-by: Ana Maria Franco <[email protected]>
afrancoc2000 pushed a commit to afrancoc2000/fabric that referenced this issue Feb 21, 2022
Ignore expired CA/TLS CA certs on msp init (hyperledger#3238)
e2513cb 
Signed-off-by: Ana Maria Franco <[email protected]>
Signed-off-by: Ana Maria Franco Cuesta <[email protected]>
afrancoc2000 pushed a commit to afrancoc2000/fabric that referenced this issue Feb 21, 2022
@afrancoc2000
Ignore expired CA/TLS CA certs on msp init (hyperledger#3238)
f4c6f67 
Signed-off-by: Ana Maria Franco <[email protected]>
afrancoc2000 pushed a commit to afrancoc2000/fabric that referenced this issue Feb 21, 2022
@afrancoc2000
Ignore expired TLS CA certs on msp init (hyperledger#3238)
8147757 
Signed-off-by: Ana Maria Franco <[email protected]>
afrancoc2000 pushed a commit to afrancoc2000/fabric that referenced this issue Feb 22, 2022
@afrancoc2000
Ignore expired CA/TLS CA certs on msp init (hyperledger#3238)
b0012c9 
Signed-off-by: Ana Maria Franco <[email protected]>
@afrancoc2000 afrancoc2000 mentioned this issue Feb 22, 2022
Closed
afrancoc2000 added a commit to afrancoc2000/fabric that referenced this issue Feb 22, 2022
@afrancoc2000
Deleting not needed test (hyperledger#3238)
ac38ebb 
Signed-off-by: Ana Maria Franco <[email protected]>
afrancoc2000 pushed a commit to afrancoc2000/fabric that referenced this issue Feb 22, 2022
@afrancoc2000
Ignore expired CA/TLS CA certs on msp init (hyperledger#3238)
89196ee 
Signed-off-by: Ana Maria Franco <[email protected]>
afrancoc2000 added a commit to afrancoc2000/fabric that referenced this issue Feb 22, 2022
@afrancoc2000
Deleting not needed test (hyperledger#3238)
d56ef09 
Signed-off-by: Ana Maria Franco <[email protected]>
afrancoc2000 added a commit to afrancoc2000/fabric that referenced this issue Feb 22, 2022
@afrancoc2000
Deleting unnecessary test (hyperledger#3238)
b7d5466 
Signed-off-by: Ana Maria Franco <[email protected]>
afrancoc2000 added a commit to afrancoc2000/fabric that referenced this issue Feb 22, 2022
@afrancoc2000
Deleting unnecessary test (hyperledger#3238)
3ea47f8 
Signed-off-by: Ana Maria Franco <[email protected]>
afrancoc2000 added a commit to afrancoc2000/fabric that referenced this issue Feb 22, 2022
@afrancoc2000
Ignore expired CA/TLS CA certs on msp init (hyperledger#3238)
272986b 
Signed-off-by: Ana Maria Franco <[email protected]>
@afrancoc2000 afrancoc2000 mentioned this issue Feb 22, 2022
Closed
afrancoc2000 added a commit to afrancoc2000/fabric that referenced this issue Feb 22, 2022
@afrancoc2000
Ignore expired CA/TLS CA certs on msp init (hyperledger#3238)
2afcad7 
Signed-off-by: Ana Maria Franco <[email protected]>
afrancoc2000 added a commit to afrancoc2000/fabric that referenced this issue Feb 22, 2022
@afrancoc2000
Ignore expired CA/TLS CA certs on msp init (hyperledger#3238)
10158ef 
Signed-off-by: Ana Maria Franco <[email protected]>
afrancoc2000 added a commit to afrancoc2000/fabric that referenced this issue Feb 22, 2022
@afrancoc2000
Ignore expired CA/TLS CA certs on msp init (hyperledger#3238)
553d0b5 
Signed-off-by: Ana Maria Franco <[email protected]>
afrancoc2000 added a commit to afrancoc2000/fabric that referenced this issue Feb 22, 2022
@afrancoc2000
Ignore expired CA/TLS CA certs on msp init (hyperledger#3238)
a7c6353 
Signed-off-by: Ana Maria Franco <[email protected]>
afrancoc2000 added a commit to afrancoc2000/fabric that referenced this issue Feb 22, 2022
@afrancoc2000
Ignore expired CA/TLS CA certs on msp init (hyperledger#3238)
3f9bbfa 
Signed-off-by: Ana Maria Franco <[email protected]>
afrancoc2000 added a commit to afrancoc2000/fabric that referenced this issue Feb 22, 2022
@afrancoc2000
Ignore expired CA/TLS CA certs on msp init (hyperledger#3238)
fee81f9 
Signed-off-by: Ana Maria Franco <[email protected]>
@afrancoc2000 afrancoc2000 mentioned this issue Feb 22, 2022
Merged
@adecaro
Copy link
Contributor

adecaro commented Feb 23, 2022

Hi All,

Just a comment here to make sure everything is in context.
When a peer restarts, the peer should rescan the ledger and apply the configuration blocks in the order given by the ledger. It is clear that, the peer should not panic because a configuration block in the middle contains expired certificates because that block refers to a previous time period. For this reason, I agree that we must fix this and align the validation of TLS CA certs to what we do for the others CA certs.

Having said that, it must be clear that the administrators of the Fabric network must ensure that all CA certificates used in the current epoch are not expired. Fabric cannot check that, it must be done manually or with other automated processes.

@yacovm
Copy link
Contributor

yacovm commented Feb 23, 2022

Having said that, it must be clear that the administrators of the Fabric network must ensure that all CA certificates used in the current epoch are not expired. Fabric cannot check that, it must be done manually or with other automated processes.

But the TLS CA certificates are not going to be used by the channel config at all.
If anything, the expired ones will not be used to validate certificates at all, because the only place that uses them is the TLS configuration which does take into account their expiration.

yacovm pushed a commit that referenced this issue Feb 23, 2022
@afrancoc2000
Ignore expired CA/TLS CA certs on msp init (#3238) (#3249)
0e6c8d4 
Signed-off-by: Ana Maria Franco <[email protected]>
mergify bot pushed a commit that referenced this issue Feb 23, 2022
@afrancoc2000
Ignore expired CA/TLS CA certs on msp init (#3238) (#3249)
3494516 
Signed-off-by: Ana Maria Franco <[email protected]>
(cherry picked from commit 0e6c8d4)
mergify bot pushed a commit that referenced this issue Feb 23, 2022
@afrancoc2000
Ignore expired CA/TLS CA certs on msp init (#3238) (#3249)
e563421 
Signed-off-by: Ana Maria Franco <[email protected]>
(cherry picked from commit 0e6c8d4)
yacovm pushed a commit that referenced this issue Feb 23, 2022
@mergify @afrancoc2000
Ignore expired CA/TLS CA certs on msp init (#3238) (#3249) (#3253)
7cc33bb 
Signed-off-by: Ana Maria Franco <[email protected]>
(cherry picked from commit 0e6c8d4)

Co-authored-by: Ana Maria Franco <[email protected]>
yacovm pushed a commit that referenced this issue Feb 23, 2022
@mergify @afrancoc2000
Ignore expired CA/TLS CA certs on msp init (#3238) (#3249) (#3252)
47dd17a 
Signed-off-by: Ana Maria Franco <[email protected]>
(cherry picked from commit 0e6c8d4)

Co-authored-by: Ana Maria Franco <[email protected]>
@yacovm yacovm closed this as completed Feb 23, 2022
mergify bot pushed a commit that referenced this issue Feb 24, 2022
@afrancoc2000
Ignore expired CA/TLS CA certs on msp init (#3238) (#3249)
efd6bdb 
Signed-off-by: Ana Maria Franco <[email protected]>
(cherry picked from commit 0e6c8d4)
@denyeart
Copy link
Contributor

Fix backported to release-2.4, release-2.3, release-2.2.

yacovm pushed a commit that referenced this issue Feb 24, 2022
@mergify @afrancoc2000
Ignore expired CA/TLS CA certs on msp init (#3238) (#3249) (#3255)
e2f05e6 
Signed-off-by: Ana Maria Franco <[email protected]>
(cherry picked from commit 0e6c8d4)

Co-authored-by: Ana Maria Franco <[email protected]>
@rodolfoleal
Copy link
Contributor

It also affects Fabric 1.4.x ? Looks like the shift gets nor applied on the for the root CA.

@denyeart
Copy link
Contributor

@rodolfoleal Fabric v1.4 is out of maintenance. There are many issues with v1.4 that have been fixed only in v2.x. Users must upgrade to v2.x to get these fixes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@adecaro @afrancoc2000 @yacovm @rodolfoleal @denyeart