exhibitor opens/leaks file descriptors under heavy load
When exhibitor is placed under heavy load, the number of open files descriptors climbs and eventually it hits whatever ulimit was specified.
We should make the following changes to the Jetty 6.1.22 Server() we run:
In Jetty 6.1.22 the default connector is the jetty.bio.SocketConnector. The SocketConnector performs blocking I/O and so suffers from spawning a new thread per connection. To improve performance and limit the number of threads we should switch to the jetty.nio.SelectChannelConnector. This is a non-blocking I/O connector.
This by itself will improve performance and so will alleviate the problem but is not enough to solve it outright. To do that we first need to explain the issue more clearly.
When a new connection is accepted (whether by the SocketConnector or the SelectChannelConnector) it is dispatched to the Server's threadPool for processing. The default threadPool implementation is unbounded. This means that when new requests are made at a higher rate than they are served, the queue of pending requests grows steadily. Any request has an active TCP connection underlying it which Jetty uses to send the response back to the client. Every connection is represented by a socket, which is identified by an open file descriptor. This means that the more connections that are active, the more open file descriptors are open. Linux has a limited number of file descriptors and so it is standard to limit the number of file descriptors a process may have open at any given time.
When exhibitor receives many requests, these are placed on a queue, and the number of open connections / file descriptors gets higher and higher until the limit is reached.
The solution to this problem is to configure a maximum number of pending requests (ie. limit the length of the request queue) and refuse any new request until one of pending requests has been served.
Jetty 6.1.22 doesn't have a component that implements a limited request queue, so we should backport the ExecutorThreadPool and plug a LinkedBlockingQueue into it as its custom queue implementation.
This should solve the "many open file descriptors" issue.