Lockup in jetty-client under low threads / slow destination condition

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Lockup in jetty-client under low threads / slow destination condition

John Gardiner Myers
While investigating reports of requests being starved in the presence of
other requests to slow destinations, I managed to get embedded
jetty-client 9.3.10 to lock up pretty hard.

My test case uses two HttpClient instances sharing the same Executor
(with 4 threads), Scheduler, and ByteBufferPool. One HttpClient makes
requests to a working plain-HTTP destination, the other to a plain-HTTP
destination that always times out connects.

After a while, all threads block on a Selector.select() with zero keys,
out of ManagedSelector$SelectorProducer.select(), out of
ManagedSelector$SelectorProducer.produce(), out of
ExecuteProduceConsume.produceExecuteConsume(), out of
ExecuteProduceConsume.produceConsume(). At this point, Jetty will not
cleanly shut down.

org.eclipse.jetty.client.HttpClient@78dc4696 - STARTED
  += org.eclipse.jetty.util.thread.Sweeper@1d03459d - STARTED
  += SslContextFactory@106c710e(null,null) - STARTED
  +~ http-client-shared{STARTED,4<=4<=4,i=0,q=7} - STARTED
  +- org.eclipse.jetty.io.MappedByteBufferPool@794e4b46
  +~ org.eclipse.jetty.util.thread.ScheduledExecutorScheduler@529e10f0 -
STARTED
  += org.eclipse.jetty.client.http.HttpClientTransportOverHTTP@71aff321
- STARTED
  |   +=
org.eclipse.jetty.client.AbstractHttpClientTransport$ClientSelectorManager@5ee8e234
- STARTED
  |       += org.eclipse.jetty.io.ManagedSelector@59406bd9 id=0 keys=0
selected=0 id=0
  |       |   +- sun.nio.ch.KQueueSelectorImpl@65abf03e keys=0
  |       += org.eclipse.jetty.io.ManagedSelector@2a5795fe id=1 keys=0
selected=0 id=1
  |           +- sun.nio.ch.KQueueSelectorImpl@483695b6 keys=0
  +- org.eclipse.jetty.util.SocketAddressResolver$Async@246f5d6e
  +=
HttpDestination[http://localhost:4111]@3d8fe295,queue=2,pool=DuplexConnectionPool[c=2/20,a=0,i=0]
+- DuplexConnectionPool[c=2/20,a=0,i=0]

org.eclipse.jetty.client.HttpClient@52169758 - STARTED
  += org.eclipse.jetty.util.thread.Sweeper@17f3461a - STARTED
  += SslContextFactory@51a5c5e4(null,null) - STARTED
  +~ http-client-shared{STARTED,4<=4<=4,i=0,q=7} - STARTED
  +- org.eclipse.jetty.io.MappedByteBufferPool@794e4b46
  +~ org.eclipse.jetty.util.thread.ScheduledExecutorScheduler@529e10f0 -
STARTED
  += org.eclipse.jetty.client.http.HttpClientTransportOverHTTP@5f505f2f
- STARTED
  |   +=
org.eclipse.jetty.client.AbstractHttpClientTransport$ClientSelectorManager@20a52bc4
- STARTED
  |       += org.eclipse.jetty.io.ManagedSelector@2b3a50ed id=0 keys=0
selected=0 id=0
  |       |   +- sun.nio.ch.KQueueSelectorImpl@70b111fe keys=0
  |       += org.eclipse.jetty.io.ManagedSelector@24c337f6 id=1 keys=0
selected=0 id=1
  |           +- sun.nio.ch.KQueueSelectorImpl@56dd5f46 keys=0
  +- org.eclipse.jetty.util.SocketAddressResolver$Async@23225908
  +=
HttpDestination[http://10.2.45.41:8080]@5db775ed,queue=3,pool=DuplexConnectionPool[c=3/20,a=0,i=0]
      +- DuplexConnectionPool[c=3/20,a=0,i=0]

Is there anything else I could investigate or produce to make a useful
bug report?


_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Lockup in jetty-client under low threads / slow destination condition

Simone Bordet-3
Hi,

On Thu, Jul 28, 2016 at 1:35 AM, John Gardiner Myers
<[hidden email]> wrote:

> While investigating reports of requests being starved in the presence of
> other requests to slow destinations, I managed to get embedded jetty-client
> 9.3.10 to lock up pretty hard.
>
> My test case uses two HttpClient instances sharing the same Executor (with 4
> threads), Scheduler, and ByteBufferPool. One HttpClient makes requests to a
> working plain-HTTP destination, the other to a plain-HTTP destination that
> always times out connects.
>
> After a while, all threads block on a Selector.select() with zero keys, out
> of ManagedSelector$SelectorProducer.select(), out of
> ManagedSelector$SelectorProducer.produce(), out of
> ExecuteProduceConsume.produceExecuteConsume(), out of
> ExecuteProduceConsume.produceConsume(). At this point, Jetty will not
> cleanly shut down.
>
> org.eclipse.jetty.client.HttpClient@78dc4696 - STARTED
>  += org.eclipse.jetty.util.thread.Sweeper@1d03459d - STARTED
>  += SslContextFactory@106c710e(null,null) - STARTED
>  +~ http-client-shared{STARTED,4<=4<=4,i=0,q=7} - STARTED
>  +- org.eclipse.jetty.io.MappedByteBufferPool@794e4b46
>  +~ org.eclipse.jetty.util.thread.ScheduledExecutorScheduler@529e10f0 -
> STARTED
>  += org.eclipse.jetty.client.http.HttpClientTransportOverHTTP@71aff321 -
> STARTED
>  |   +=
> org.eclipse.jetty.client.AbstractHttpClientTransport$ClientSelectorManager@5ee8e234
> - STARTED
>  |       += org.eclipse.jetty.io.ManagedSelector@59406bd9 id=0 keys=0
> selected=0 id=0
>  |       |   +- sun.nio.ch.KQueueSelectorImpl@65abf03e keys=0
>  |       += org.eclipse.jetty.io.ManagedSelector@2a5795fe id=1 keys=0
> selected=0 id=1
>  |           +- sun.nio.ch.KQueueSelectorImpl@483695b6 keys=0
>  +- org.eclipse.jetty.util.SocketAddressResolver$Async@246f5d6e
>  +=
> HttpDestination[http://localhost:4111]@3d8fe295,queue=2,pool=DuplexConnectionPool[c=2/20,a=0,i=0]
> +- DuplexConnectionPool[c=2/20,a=0,i=0]
>
> org.eclipse.jetty.client.HttpClient@52169758 - STARTED
>  += org.eclipse.jetty.util.thread.Sweeper@17f3461a - STARTED
>  += SslContextFactory@51a5c5e4(null,null) - STARTED
>  +~ http-client-shared{STARTED,4<=4<=4,i=0,q=7} - STARTED
>  +- org.eclipse.jetty.io.MappedByteBufferPool@794e4b46
>  +~ org.eclipse.jetty.util.thread.ScheduledExecutorScheduler@529e10f0 -
> STARTED
>  += org.eclipse.jetty.client.http.HttpClientTransportOverHTTP@5f505f2f -
> STARTED
>  |   +=
> org.eclipse.jetty.client.AbstractHttpClientTransport$ClientSelectorManager@20a52bc4
> - STARTED
>  |       += org.eclipse.jetty.io.ManagedSelector@2b3a50ed id=0 keys=0
> selected=0 id=0
>  |       |   +- sun.nio.ch.KQueueSelectorImpl@70b111fe keys=0
>  |       += org.eclipse.jetty.io.ManagedSelector@24c337f6 id=1 keys=0
> selected=0 id=1
>  |           +- sun.nio.ch.KQueueSelectorImpl@56dd5f46 keys=0
>  +- org.eclipse.jetty.util.SocketAddressResolver$Async@23225908
>  +=
> HttpDestination[http://10.2.45.41:8080]@5db775ed,queue=3,pool=DuplexConnectionPool[c=3/20,a=0,i=0]
>      +- DuplexConnectionPool[c=3/20,a=0,i=0]
>
> Is there anything else I could investigate or produce to make a useful bug
> report?

JVM thread dump ?

Why do you run with only 4 threads ?
It will limit the HttpClient operativity by a lot under load.

You can also try to use different ExecutionStrategies, by setting them
in the transport:
HttpClientTransportOverHTTP.setExecutionStrategyFactory(...).

In summary, running with a thread pool with 4 threads is discouraged,
unless you have very good reasons to do so.
If you really, really need to run with these small number of threads,
you have to configure HttpClient to support this mode of operation:
reduce connection pool size, change the ExecutionStrategy, be prepared
to a major loss of performance.

--
Simone Bordet
----
http://cometd.org
http://webtide.com
Developer advice, training, services and support
from the Jetty & CometD experts.
_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Lockup in jetty-client under low threads / slow destination condition

John Gardiner Myers


On 7/28/16 2:11 AM, Simone Bordet wrote:
> JVM thread dump ?
For a different run than the previous dumps:

2016-07-28 10:53:52
Full thread dump OpenJDK 64-Bit Server VM (25.92-b15 mixed mode):

"http-admin-worker-49" #49 prio=5 os_prio=31 tid=0x00007feab9afa800
nid=0x3d17 waiting on condition [0x0000700002402000]
    java.lang.Thread.State: TIMED_WAITING (parking)
     at sun.misc.Unsafe.park(Native Method)
     - parking to wait for  <0x00000007704ea3c8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
     at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
     at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
     at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
     at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
     at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
     at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
     at java.lang.Thread.run(Thread.java:745)

"http-worker-48" #48 prio=5 os_prio=31 tid=0x00007feab9b35000 nid=0x780f
waiting on condition [0x00007000022ff000]
    java.lang.Thread.State: TIMED_WAITING (parking)
     at sun.misc.Unsafe.park(Native Method)
     - parking to wait for  <0x00000007704eac20> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
     at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
     at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
     at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
     at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
     at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
     at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
     at java.lang.Thread.run(Thread.java:745)

"Attach Listener" #39 daemon prio=9 os_prio=31 tid=0x00007feaba4fb800
nid=0x1307 waiting on condition [0x0000000000000000]
    java.lang.Thread.State: RUNNABLE

"DestroyJavaVM" #38 prio=5 os_prio=31 tid=0x00007feab95a0000 nid=0x1703
waiting on condition [0x0000000000000000]
    java.lang.Thread.State: RUNNABLE

"http-admin-worker-34" #34 prio=5 os_prio=31 tid=0x00007feab3e0b800
nid=0x8803 runnable [0x0000700002d1d000]
    java.lang.Thread.State: RUNNABLE
     at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
     at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
     at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
     at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
     - locked <0x000000076ab2f060> (a sun.nio.ch.Util$2)
     - locked <0x000000076ab2f050> (a java.util.Collections$UnmodifiableSet)
     - locked <0x000000076ab2ef30> (a sun.nio.ch.KQueueSelectorImpl)
     at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
     at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
     at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)
     at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
     at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)
     at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
     at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)
     at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147)
     at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
     at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
     at java.lang.Thread.run(Thread.java:745)

"http-admin-worker-33" #33 prio=5 os_prio=31 tid=0x00007feab7b13000
nid=0x8603 runnable [0x0000700002c1a000]
    java.lang.Thread.State: RUNNABLE
     at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
     at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
     at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
     at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
     - locked <0x000000076ab2ee28> (a sun.nio.ch.Util$2)
     - locked <0x000000076ab2ee18> (a java.util.Collections$UnmodifiableSet)
     - locked <0x000000076ab2ecf8> (a sun.nio.ch.KQueueSelectorImpl)
     at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
     at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
     at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)
     at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
     at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)
     at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
     at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)
     at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147)
     at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
     at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
     at java.lang.Thread.run(Thread.java:745)

"http-admin-worker-32" #32 prio=5 os_prio=31 tid=0x00007feab7b11000
nid=0x8403 runnable [0x0000700002b17000]
    java.lang.Thread.State: RUNNABLE
     at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
     at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
     at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
     at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
     - locked <0x000000076ab2ec10> (a sun.nio.ch.Util$2)
     - locked <0x000000076ab2ec00> (a java.util.Collections$UnmodifiableSet)
     - locked <0x000000076ab2eae0> (a sun.nio.ch.KQueueSelectorImpl)
     at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
     at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
     at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)
     at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
     at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)
     at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
     at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)
     at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147)
     at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
     at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
     at java.lang.Thread.run(Thread.java:745)

"http-admin-worker-31" #31 prio=5 os_prio=31 tid=0x00007feab566b800
nid=0x8203 runnable [0x0000700002a14000]
    java.lang.Thread.State: RUNNABLE
     at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
     at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
     at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
     at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
     - locked <0x000000076ab2e9d8> (a sun.nio.ch.Util$2)
     - locked <0x000000076ab2e9c8> (a java.util.Collections$UnmodifiableSet)
     - locked <0x000000076ab2e8a8> (a sun.nio.ch.KQueueSelectorImpl)
     at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
     at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
     at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)
     at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
     at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)
     at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
     at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)
     at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147)
     at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
     at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
     at java.lang.Thread.run(Thread.java:745)

"http-worker-29-acceptor-0@4eb45fec-http@2e54db99{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}"
#29 prio=3 os_prio=31 tid=0x00007feab7904800 nid=0x8003 runnable
[0x0000700002911000]
    java.lang.Thread.State: RUNNABLE
     at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
     at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
     at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
     - locked <0x00000007704ec168> (a java.lang.Object)
     at
org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:373)
     at
org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:593)
     at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
     at com.proofpoint.http.server.HttpServer$1.runJob(HttpServer.java:137)
     at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
     at java.lang.Thread.run(Thread.java:745)

"http-worker-27" #27 prio=5 os_prio=31 tid=0x00007feab8157000 nid=0x7c03
runnable [0x000070000270b000]
    java.lang.Thread.State: RUNNABLE
     at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
     at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
     at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
     at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
     - locked <0x00000007704eb580> (a sun.nio.ch.Util$2)
     - locked <0x00000007704eb570> (a java.util.Collections$UnmodifiableSet)
     - locked <0x00000007704eb450> (a sun.nio.ch.KQueueSelectorImpl)
     at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
     at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
     at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)
     at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
     at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)
     at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
     at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)
     at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147)
     at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
     at com.proofpoint.http.server.HttpServer$1.runJob(HttpServer.java:137)
     at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
     at java.lang.Thread.run(Thread.java:745)

"http-worker-26" #26 prio=5 os_prio=31 tid=0x00007feab4a51800 nid=0x7a03
runnable [0x0000700002608000]
    java.lang.Thread.State: RUNNABLE
     at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
     at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
     at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
     at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
     - locked <0x00000007704eb1b0> (a sun.nio.ch.Util$2)
     - locked <0x00000007704eb1a0> (a java.util.Collections$UnmodifiableSet)
     - locked <0x00000007704eb080> (a sun.nio.ch.KQueueSelectorImpl)
     at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
     at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
     at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)
     at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
     at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)
     at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
     at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)
     at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147)
     at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
     at com.proofpoint.http.server.HttpServer$1.runJob(HttpServer.java:137)
     at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
     at java.lang.Thread.run(Thread.java:745)

"http-worker-22" #22 prio=5 os_prio=31 tid=0x00007feab7972000 nid=0x7203
runnable [0x00007000021fc000]
    java.lang.Thread.State: RUNNABLE
     at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
     at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
     at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
     at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
     - locked <0x00000007704eb970> (a sun.nio.ch.Util$2)
     - locked <0x00000007704eb960> (a java.util.Collections$UnmodifiableSet)
     - locked <0x00000007704eb840> (a sun.nio.ch.KQueueSelectorImpl)
     at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
     at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
     at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)
     at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
     at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)
     at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
     at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)
     at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147)
     at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
     at com.proofpoint.http.server.HttpServer$1.runJob(HttpServer.java:137)
     at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
     at java.lang.Thread.run(Thread.java:745)

"http-worker-21" #21 prio=5 os_prio=31 tid=0x00007feab633e800 nid=0x7003
runnable [0x00007000020f9000]
    java.lang.Thread.State: RUNNABLE
     at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
     at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
     at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
     at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
     - locked <0x00000007704ebd90> (a sun.nio.ch.Util$2)
     - locked <0x00000007704ebd80> (a java.util.Collections$UnmodifiableSet)
     - locked <0x00000007704ebc60> (a sun.nio.ch.KQueueSelectorImpl)
     at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
     at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
     at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)
     at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
     at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)
     at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
     at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)
     at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147)
     at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
     at com.proofpoint.http.server.HttpServer$1.runJob(HttpServer.java:137)
     at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
     at java.lang.Thread.run(Thread.java:745)

"reporting-client-0" #20 daemon prio=5 os_prio=31 tid=0x00007feab624a000
nid=0x6e03 waiting on condition [0x0000700001ff6000]
    java.lang.Thread.State: TIMED_WAITING (parking)
     at sun.misc.Unsafe.park(Native Method)
     - parking to wait for  <0x000000076c28aba0> (a
java.util.concurrent.CountDownLatch$Sync)
     at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
     at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
     at
java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
     at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
     at
org.eclipse.jetty.client.util.InputStreamResponseListener.get(InputStreamResponseListener.java:222)
     at
com.proofpoint.http.client.jetty.JettyHttpClient.execute(JettyHttpClient.java:362)
     at
com.proofpoint.http.client.balancing.BalancingHttpClient.execute(BalancingHttpClient.java:96)
     at com.proofpoint.reporting.ReportClient.report(ReportClient.java:88)
     at
com.proofpoint.reporting.ReportCollector.lambda$collectData$1(ReportCollector.java:109)
     at
com.proofpoint.reporting.ReportCollector$$Lambda$72/487717827.run(Unknown
Source)
     at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
     at java.util.concurrent.FutureTask.run(FutureTask.java:266)
     at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
     at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
     at java.lang.Thread.run(Thread.java:745)

"reporting-collector-0" #19 daemon prio=5 os_prio=31
tid=0x00007feab488a800 nid=0x6c03 waiting on condition [0x0000700001ef3000]
    java.lang.Thread.State: TIMED_WAITING (parking)
     at sun.misc.Unsafe.park(Native Method)
     - parking to wait for  <0x00000007704e8528> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
     at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
     at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
     at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
     at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
     at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
     at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
     at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
     at java.lang.Thread.run(Thread.java:745)

"Discovery-1" #18 daemon prio=5 os_prio=31 tid=0x00007feab429b000
nid=0x6a03 waiting on condition [0x0000700001df0000]
    java.lang.Thread.State: WAITING (parking)
     at sun.misc.Unsafe.park(Native Method)
     - parking to wait for  <0x000000077042e630> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
     at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
     at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
     at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
     at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
     at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
     at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
     at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
     at java.lang.Thread.run(Thread.java:745)

"Discovery-0" #17 daemon prio=5 os_prio=31 tid=0x00007feab52dd000
nid=0x6803 waiting on condition [0x0000700001ced000]
    java.lang.Thread.State: WAITING (parking)
     at sun.misc.Unsafe.park(Native Method)
     - parking to wait for  <0x000000077042e630> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
     at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
     at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
     at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
     at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
     at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
     at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
     at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
     at java.lang.Thread.run(Thread.java:745)

"http-client-shared-scheduler" #16 daemon prio=5 os_prio=31
tid=0x00007feab62fc800 nid=0x6603 waiting on condition [0x0000700001bea000]
    java.lang.Thread.State: TIMED_WAITING (parking)
     at sun.misc.Unsafe.park(Native Method)
     - parking to wait for  <0x00000006c000e900> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
     at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
     at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
     at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
     at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
     at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
     at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
     at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
     at java.lang.Thread.run(Thread.java:745)

"http-client-shared-15" #15 daemon prio=5 os_prio=31
tid=0x00007feab70f2000 nid=0x6403 runnable [0x0000700001ae7000]
    java.lang.Thread.State: RUNNABLE
     at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
     at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
     at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
     at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
     - locked <0x000000077053f438> (a sun.nio.ch.Util$2)
     - locked <0x000000077053f428> (a java.util.Collections$UnmodifiableSet)
     - locked <0x000000077053f228> (a sun.nio.ch.KQueueSelectorImpl)
     at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
     at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
     at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)
     at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
     at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceExecuteConsume(ExecuteProduceConsume.java:169)
     at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:145)
     at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
     at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
     at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
     at java.lang.Thread.run(Thread.java:745)

"http-client-shared-14" #14 daemon prio=5 os_prio=31
tid=0x00007feab62b3800 nid=0x6203 runnable [0x00007000019e4000]
    java.lang.Thread.State: RUNNABLE
     at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
     at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
     at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
     at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
     - locked <0x00000007704016f0> (a sun.nio.ch.Util$2)
     - locked <0x00000007704016e0> (a java.util.Collections$UnmodifiableSet)
     - locked <0x00000007704015b0> (a sun.nio.ch.KQueueSelectorImpl)
     at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
     at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
     at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)
     at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
     at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceExecuteConsume(ExecuteProduceConsume.java:169)
     at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:145)
     at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)
     at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147)
     at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
     at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
     at java.lang.Thread.run(Thread.java:745)

"http-client-shared-13" #13 daemon prio=5 os_prio=31
tid=0x00007feab62b6800 nid=0x6003 runnable [0x00007000018e1000]
    java.lang.Thread.State: RUNNABLE
     at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
     at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
     at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
     at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
     - locked <0x000000077053fcf8> (a sun.nio.ch.Util$2)
     - locked <0x000000077053fce8> (a java.util.Collections$UnmodifiableSet)
     - locked <0x000000077053fbb8> (a sun.nio.ch.KQueueSelectorImpl)
     at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
     at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
     at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)
     at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
     at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)
     at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
     at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)
     at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147)
     at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
     at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
     at java.lang.Thread.run(Thread.java:745)

"http-client-shared-12" #12 daemon prio=5 os_prio=31
tid=0x00007feab39a0000 nid=0x5e07 runnable [0x00007000017de000]
    java.lang.Thread.State: RUNNABLE
     at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
     at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
     at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
     at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
     - locked <0x00000007703a60c8> (a sun.nio.ch.Util$2)
     - locked <0x00000007703a60b8> (a java.util.Collections$UnmodifiableSet)
     - locked <0x00000007703a5f88> (a sun.nio.ch.KQueueSelectorImpl)
     at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
     at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
     at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)
     at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
     at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceExecuteConsume(ExecuteProduceConsume.java:169)
     at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:145)
     at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)
     at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147)
     at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
     at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
     at java.lang.Thread.run(Thread.java:745)

"Monitor Ctrl-Break" #10 daemon prio=5 os_prio=31 tid=0x00007feab50c2800
nid=0x5803 runnable [0x00007000016db000]
    java.lang.Thread.State: RUNNABLE
     at java.net.PlainSocketImpl.socketAccept(Native Method)
     at
java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
     at java.net.ServerSocket.implAccept(ServerSocket.java:545)
     at java.net.ServerSocket.accept(ServerSocket.java:513)
     at com.intellij.rt.execution.application.AppMain$1.run(AppMain.java:79)
     at java.lang.Thread.run(Thread.java:745)

"Service Thread" #9 daemon prio=9 os_prio=31 tid=0x00007feab502e000
nid=0x5403 runnable [0x0000000000000000]
    java.lang.Thread.State: RUNNABLE

"C1 CompilerThread3" #8 daemon prio=9 os_prio=31 tid=0x00007feab6006800
nid=0x5203 waiting on condition [0x0000000000000000]
    java.lang.Thread.State: RUNNABLE

"C2 CompilerThread2" #7 daemon prio=9 os_prio=31 tid=0x00007feab405c000
nid=0x5003 waiting on condition [0x0000000000000000]
    java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" #6 daemon prio=9 os_prio=31 tid=0x00007feab405b800
nid=0x4e03 waiting on condition [0x0000000000000000]
    java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #5 daemon prio=9 os_prio=31 tid=0x00007feab502d800
nid=0x4c03 waiting on condition [0x0000000000000000]
    java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" #4 daemon prio=9 os_prio=31 tid=0x00007feab500f800
nid=0x4913 runnable [0x0000000000000000]
    java.lang.Thread.State: RUNNABLE

"Finalizer" #3 daemon prio=8 os_prio=31 tid=0x00007feab3826800
nid=0x3803 in Object.wait() [0x0000700000d3a000]
    java.lang.Thread.State: WAITING (on object monitor)
     at java.lang.Object.wait(Native Method)
     - waiting on <0x00000006c0043df0> (a java.lang.ref.ReferenceQueue$Lock)
     at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
     - locked <0x00000006c0043df0> (a java.lang.ref.ReferenceQueue$Lock)
     at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
     at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

"Reference Handler" #2 daemon prio=10 os_prio=31 tid=0x00007feab6002000
nid=0x3603 in Object.wait() [0x0000700000c37000]
    java.lang.Thread.State: WAITING (on object monitor)
     at java.lang.Object.wait(Native Method)
     - waiting on <0x00000006c003b768> (a java.lang.ref.Reference$Lock)
     at java.lang.Object.wait(Object.java:502)
     at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
     - locked <0x00000006c003b768> (a java.lang.ref.Reference$Lock)
     at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)

"VM Thread" os_prio=31 tid=0x00007feab3826000 nid=0x3403 runnable

"GC task thread#0 (ParallelGC)" os_prio=31 tid=0x00007feab3812800
nid=0x2403 runnable

"GC task thread#1 (ParallelGC)" os_prio=31 tid=0x00007feab3813800
nid=0x2603 runnable

"GC task thread#2 (ParallelGC)" os_prio=31 tid=0x00007feab3814000
nid=0x2803 runnable

"GC task thread#3 (ParallelGC)" os_prio=31 tid=0x00007feab3814800
nid=0x2a03 runnable

"GC task thread#4 (ParallelGC)" os_prio=31 tid=0x00007feab3815000
nid=0x2c03 runnable

"GC task thread#5 (ParallelGC)" os_prio=31 tid=0x00007feab3816000
nid=0x2e03 runnable

"GC task thread#6 (ParallelGC)" os_prio=31 tid=0x00007feab3816800
nid=0x3003 runnable

"GC task thread#7 (ParallelGC)" os_prio=31 tid=0x00007feab3817000
nid=0x3203 runnable

"VM Periodic Task Thread" os_prio=31 tid=0x00007feab3838000 nid=0x5603
waiting on condition

JNI global references: 494

> Why do you run with only 4 threads ?
Because this is a test intended to more easily reproduce request
starvation problems found when running a longer amount of time, in a
more complex environment, with more threads.


_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Lockup in jetty-client under low threads / slow destination condition

Simone Bordet-3
John,

On Thu, Jul 28, 2016 at 8:04 PM, John Gardiner Myers
<[hidden email]> wrote:
> For a different run than the previous dumps:

[snip]

So the 4 threads are all inside select(), so there is no threads left
to process IO events.
Do you really need 4 selectors for the client ?
You can configure the client transport to use only 1 selector, which
is typically enough for few thousands connections.

This particular dump seems just wrong configuration to me.
Configuring the client with N selectors and the thread pool with N
threads, there is none left for processing IO events, but I doubt this
is what happens in your more complex environment.

If HttpClient is used always in non-blocking mode, one thread left for
processing should be enough to continue processing normally.

Thanks !

--
Simone Bordet
----
http://cometd.org
http://webtide.com
Developer advice, training, services and support
from the Jetty & CometD experts.
_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Lockup in jetty-client under low threads / slow destination condition

John Gardiner Myers
Thank you for the information on the constraint; I will increase threads
to 6 and continue attempts to reproduce the original problem report.

In the scenario there are two HttpClient instances each sharing the same
pool. Each HttpClient has a transport configured with 2 selectors. The
commit that put the selector configuration in there appears to have been
intended to reduce the number of selectors from Jetty's default of half
the available processors.

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users