The terminate-while-queueing test added for #4428 failed spuriously
indicating that sometimes WebSocket clients receive code 1000 instead of 1001.
This happens if the ixwebsocket server is shutdown before the reply thread had a
chance to process queued close messages.
Fix by signaling and waiting for the dispatcher's reply thread to terminate
before returning from Terminate().
Terminate() is called when Zeek shuts down. If WebSocket client threads
were blocked in QueueForProcessing() due to reaching queue limits, these
previously would not exit QueueForProcessing() and instead block
indefinitely, resulting in the ixwebsocket library blocking and its
garbage collection thread running at 100%. Not great.
Closing the onloop instance will unblock the WebSocket client threads
for a timely shutdown.
Closes#4420
Limit the number WebSocket events queued from external clients to
dispatcher instances to produce back pressure to the clients if
Zeek's IO loop is overloaded.
When Cluster::backend is configured with CLUSTER_BACKEND_BROKER, switch
WebSocketClients to CLUSTER_BACKEND_BROKER_WEBSOCKET_SHIM instead.
Instead of the special case, we could also add something to Backend
called NewWebSocketBackend(), but if it only affects broker, I think
the special case is okay for now.
WebSocket clients that connected with the wrong URL do not have
a backend attached. If a dispatcher is terminated while these
clients are still connected, a null deref would happen.
This was found while running all cluster/websocket tests in a loop
for a long time, tickling a segfault during the bad-url test.
