Plugin runs into an exception on invalid/unparseable HTTP response body

[mgrafl]

The rabbitmq-trust-store plugin crashes if the JSON object of the HTTP response cannot be parsed.

Stack trace:

2019-11-15 13:30:17.172 [error] <0.599.0> ** Generic server trust_store terminating
** Last message in was refresh
** When Server state == {state,[{rabbit_trust_store_http_provider,{http_state,"https://url.to.truststore/path/to/trust_store",[],[{"If-Modified-Since","Fri, 15 Nov 2019 12:29:15 GMT"}]}}],30000}
** Reason for termination ==
** {function_clause,[{rabbit_trust_store,terminate,[{badarg,[{jsx_decoder,incomplete,6,[{file,"src/jsx_decoder.erl"},{line,141}]},{rabbit_trust_store_http_provider,decode_cert_list,1,[{file,"src/rabbit_trust_store_http_provider.erl"},{line,76}]},{rabbit_trust_store_http_provider,list_certs,2,[{file,"src/rabbit_trust_store_http_provider.erl"},{line,28}]},{rabbit_trust_store,refresh_provider_certs,3,[{file,"src/rabbit_trust_store.erl"},{line,228}]},{rabbit_trust_store,'-refresh_certs/2-fun-0-',4,[{file,"src/rabbit_trust_store.erl"},{line,217}]},{lists,foldl,3,[{file,"lists.erl"},{line,1263}]},{rabbit_trust_store,handle_info,2,[{file,"src/rabbit_trust_store.erl"},{line,169}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,637}]}]},{state,[{rabbit_trust_store_http_provider,{http_state,"https://url.to.truststore/path/to/trust_store",[],[{"If-Modified-Since","Fri, 15 Nov 2019 12:29:15 GMT"}]}}],30000}],[{file,"src/rabbit_trust_store.erl"},{line,175}]},{gen_server,try_terminate,3,[{file,"gen_server.erl"},{line,673}]},{gen_server,terminate,10,[{file,"gen_server.erl"},{line,858}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}
2019-11-15 13:30:17.173 [error] <0.599.0> CRASH REPORT Process trust_store with 0 neighbours crashed with reason: no function clause matching rabbit_trust_store:terminate({badarg,[{jsx_decoder,incomplete,6,[{file,"src/jsx_decoder.erl"},{line,141}]},{rabbit_trust_store_http_provider,...},...]}, {state,[{rabbit_trust_store_http_provider,{http_state,"https://url.to.truststore/...",...}}],...}) line 175
2019-11-15 13:30:17.173 [error] <0.598.0> Supervisor rabbit_trust_store_sup had child trust_store started with rabbit_trust_store:start_link() at <0.599.0> exit with reason no function clause matching rabbit_trust_store:terminate({badarg,[{jsx_decoder,incomplete,6,[{file,"src/jsx_decoder.erl"},{line,141}]},{rabbit_trust_store_http_provider,...},...]}, {state,[{rabbit_trust_store_http_provider,{http_state,"https://url.to.truststore/...",...}}],...}) line 175 in context child_terminated
2019-11-15 13:30:17.255 [error] <0.30984.12> CRASH REPORT Process <0.30984.12> with 0 neighbours crashed with reason: bad argument in jsx_decoder:incomplete/6 line 141
2019-11-15 13:30:17.256 [error] <0.598.0> Supervisor rabbit_trust_store_sup had child trust_store started with rabbit_trust_store:start_link() at <0.599.0> exit with reason bad argument in jsx_decoder:incomplete/6 line 141 in context start_error
2019-11-15 13:30:17.256 [error] <0.598.0> Supervisor rabbit_trust_store_sup had child trust_store started with rabbit_trust_store:start_link() at {restarting,<0.599.0>} exit with reason reached_max_restart_intensity in context shutdown
2019-11-15 13:30:17.257 [info] <0.42.0> Application rabbitmq_trust_store exited with reason: shutdown

Reproduction:

• Environment: RabbitMQ 3.7.16 on Erlang 21.0.9 running on CentOS 7.
• Unfortunately, the offending invalid HTTP response body is not available.

[michaelklishin]

What does "crashes" mean exactly? A mere exception in the logs from the JSON parser (jsx) or the availability of the plugin or node's ability to accept new connections is somehow affected?

[mgrafl]

The rabbitmq-trust-store plugin stops working (as indicated by the last lines of the stack trace).
It no longer polls the configured HTTP server for the certificate whitelist.
RabbitMQ can therefore not accept connections with client certificates for accounts that are created/changed after the plugin crashed.

[mgrafl]

The problem still appears in RabbitMQ version 3.7.24.

Reproduction steps:

1. Configure rabbitmq-truststore-plugin with an HTTP provider URL that returns either valid JSON or just a 404 HTTP status code.
2. Start RabbitMQ.
3. Change the endpoint to return HTTP status code 200 OK and invalid JSON.
4. The rabbitmq-truststore-plugin crashes.

Stack trace (with endpoint initially returning 404 status code):

2020-02-18 13:37:36.901 [error] <0.534.0> Unable to load certificate list for provider rabbit_trust_store_http_provider, reason: {http_error,404,<<"<title>Error</title>404 - Not Found">>}
2020-02-18 13:37:41.904 [error] <0.534.0> Unable to load certificate list for provider rabbit_trust_store_http_provider, reason: {http_error,404,<<"<title>Error</title>404 - Not Found">>}
2020-02-18 13:37:46.911 [error] <0.534.0> Unable to load certificate list for provider rabbit_trust_store_http_provider, reason: {http_error,404,<<"<title>Error</title>404 - Not Found">>}
2020-02-18 13:37:52.138 [error] <0.534.0> ** Generic server trust_store terminating
** Last message in was refresh
** When Server state == {state,[{rabbit_trust_store_http_provider,nostate}],5000}
** Reason for termination ==
** {function_clause,[{rabbit_trust_store,terminate,[{badarg,[{jsx_decoder,value,4,[{file,"src/jsx_decoder.erl"},{line,234}]},{rabbit_trust_store_http_provider,decode_cert_list,1,[{file,"src/rabbit_trust_store_http_provider.erl"},{line,76}]},{rabbit_trust_store_http_provider,list_certs,2,[{file,"src/rabbit_trust_store_http_provider.erl"},{line,28}]},{rabbit_trust_store,refresh_provider_certs,3,[{file,"src/rabbit_trust_store.erl"},{line,228}]},{rabbit_trust_store,'-refresh_certs/2-fun-0-',4,[{file,"src/rabbit_trust_store.erl"},{line,217}]},{lists,foldl,3,[{file,"lists.erl"},{line,1263}]},{rabbit_trust_store,handle_info,2,[{file,"src/rabbit_trust_store.erl"},{line,169}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,637}]}]},{state,[{rabbit_trust_store_http_provider,nostate}],5000}],[{file,"src/rabbit_trust_store.erl"},{line,175}]},{gen_server,try_terminate,3,[{file,"gen_server.erl"},{line,673}]},{gen_server,terminate,10,[{file,"gen_server.erl"},{line,858}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}
2020-02-18 13:37:52.139 [error] <0.534.0> CRASH REPORT Process trust_store with 0 neighbours crashed with reason: no function clause matching rabbit_trust_store:terminate({badarg,[{jsx_decoder,value,4,[{file,"src/jsx_decoder.erl"},{line,234}]},{rabbit_trust_store_http_provider,...},...]}, {state,[{rabbit_trust_store_http_provider,nostate}],5000}) line 175
2020-02-18 13:37:52.139 [error] <0.533.0> Supervisor rabbit_trust_store_sup had child trust_store started with rabbit_trust_store:start_link() at <0.534.0> exit with reason no function clause matching rabbit_trust_store:terminate({badarg,[{jsx_decoder,value,4,[{file,"src/jsx_decoder.erl"},{line,234}]},{rabbit_trust_store_http_provider,...},...]}, {state,[{rabbit_trust_store_http_provider,nostate}],5000}) line 175 in context child_terminated
2020-02-18 13:37:52.147 [error] <0.893.0> CRASH REPORT Process <0.893.0> with 0 neighbours crashed with reason: bad argument in jsx_decoder:value/4 line 234
2020-02-18 13:37:52.148 [error] <0.533.0> Supervisor rabbit_trust_store_sup had child trust_store started with rabbit_trust_store:start_link() at <0.534.0> exit with reason bad argument in jsx_decoder:value/4 line 234 in context start_error
2020-02-18 13:37:52.149 [error] <0.533.0> Supervisor rabbit_trust_store_sup had child trust_store started with rabbit_trust_store:start_link() at {restarting,<0.534.0>} exit with reason reached_max_restart_intensity in context shutdown
2020-02-18 13:37:52.150 [info] <0.43.0> Application rabbitmq_trust_store exited with reason: shutdown

[michaelklishin]

#74 was not backported to v3.7.x.

[michaelklishin]

Once we have a new 3.7 alpha that includes an updated build of this plugin we will post a link here so you can try it.

[michaelklishin]

3.7.25-alpha.2 should include #74.