Jetty EatWhatYouKill - Deadlock or something!

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Jetty EatWhatYouKill - Deadlock or something!

Sarath Prabath Redlapalli Jaya
Hi,

We've been running a java server with Jetty(Dropwizard) and the load of system per node would be ~ 10k QPS as of now. 

We ran Load Test, Connection Test and Peak Test(No Stress test yet.), but couldn't reproduce what's happening on production when we chose to go live

Below is the error that spits out continuosly

[2018-01-05 16:05:00,874] [] [WARN] MSG - o.e.j.u.t.QueuedThreadPool org.eclipse.jetty.util.thread.QueuedThreadPool@dw{STARTED,4000<=4000<=4000,i=0,q=10000} rejected CEP:SocketChannelEndPoint@62241285{/127.0.0.1:49625<->/127.0.0.1:7000,OPEN,fill=FI,flush=-,to=0/30000}{io=1/0,kio=1,kro=1}->HttpConnection@23c692fe[p=HttpParser{s=START,0 of 0},g=HttpGenerator@544fccad{s=START}]=>HttpChannelOverHttp@2c630bfe{r=0,c=false,a=IDLE,uri=null}:runFillable:BLOCKING
[2018-01-05 16:05:00,874] [] [WARN] MSG - o.e.j.u.t.s.EatWhatYouKill 
java.util.concurrent.RejectedExecutionException: CEP:SocketChannelEndPoint@62241285{/127.0.0.1:49625<->/127.0.0.1:7000,OPEN,fill=FI,flush=-,to=0/30000}{io=1/0,kio=1,kro=1}->HttpConnection@23c692fe[p=HttpParser{s=START,0 of 0},g=HttpGenerator@544fccad{s=START}]=>HttpChannelOverHttp@2c630bfe{r=0,c=false,a=IDLE,uri=null}:runFillable:BLOCKING
        at org.eclipse.jetty.util.thread.QueuedThreadPool.execute(QueuedThreadPool.java:385)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:249)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:243)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:679)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:597)
        at java.lang.Thread.run(Thread.java:745)

Tried to find an answer, but couldn't. Can anyone help. At this point the server stops accepting new connections and doesn't serve new http requests. 

But Jetty QueuedThreadPool Metrics show that thread Utilization is 100%

Thanks

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Jetty EatWhatYouKill - Deadlock or something!

Simone Bordet-3
Hi,

On Fri, Jan 5, 2018 at 6:50 PM, Sarath Prabath Redlapalli Jaya
<[hidden email]> wrote:

> Hi,
>
> We've been running a java server with Jetty(Dropwizard) and the load of
> system per node would be ~ 10k QPS as of now.
>
> We ran Load Test, Connection Test and Peak Test(No Stress test yet.), but
> couldn't reproduce what's happening on production when we chose to go live
>
> Below is the error that spits out continuosly
>
> [2018-01-05 16:05:00,874] [] [WARN] MSG - o.e.j.u.t.QueuedThreadPool
> org.eclipse.jetty.util.thread.QueuedThreadPool@dw{STARTED,4000<=4000<=4000,i=0,q=10000}
> rejected
> CEP:SocketChannelEndPoint@62241285{/127.0.0.1:49625<->/127.0.0.1:7000,OPEN,fill=FI,flush=-,to=0/30000}{io=1/0,kio=1,kro=1}->HttpConnection@23c692fe[p=HttpParser{s=START,0
> of
> 0},g=HttpGenerator@544fccad{s=START}]=>HttpChannelOverHttp@2c630bfe{r=0,c=false,a=IDLE,uri=null}:runFillable:BLOCKING
> [2018-01-05 16:05:00,874] [] [WARN] MSG - o.e.j.u.t.s.EatWhatYouKill
> java.util.concurrent.RejectedExecutionException:
> CEP:SocketChannelEndPoint@62241285{/127.0.0.1:49625<->/127.0.0.1:7000,OPEN,fill=FI,flush=-,to=0/30000}{io=1/0,kio=1,kro=1}->HttpConnection@23c692fe[p=HttpParser{s=START,0
> of
> 0},g=HttpGenerator@544fccad{s=START}]=>HttpChannelOverHttp@2c630bfe{r=0,c=false,a=IDLE,uri=null}:runFillable:BLOCKING
>         at
> org.eclipse.jetty.util.thread.QueuedThreadPool.execute(QueuedThreadPool.java:385)
>         at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:249)
>         at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
>         at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
>         at
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:243)
>         at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:679)
>         at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:597)
>         at java.lang.Thread.run(Thread.java:745)
>
> Tried to find an answer, but couldn't. Can anyone help. At this point the
> server stops accepting new connections and doesn't serve new http requests.
>
> But Jetty QueuedThreadPool Metrics show that thread Utilization is 100%

What Jetty version ?
Have you tried 9.4.8 ?

Are you using a bounded thread pool ?

If it replicates with 9.4.8, can you take a server dump as explained here:
https://www.eclipse.org/jetty/documentation/9.4.x/jetty-dump-tool.html

--
Simone Bordet
----
http://cometd.org
http://webtide.com
Developer advice, training, services and support
from the Jetty & CometD experts.
_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Jetty EatWhatYouKill - Deadlock or something!

Sarath Prabath Redlapalli Jaya
Hi,

I'm using Dropwizard 1.2.0 -> Jetty 9.4.7.v20170914. 

Dropwizard has a maxThreads setting for jetty. Following is the config i used
 
maxThreads: 4000
minThreads: 4000

Connectors:
acceptorThreads: 500
selectorThreads: 100. Although i believe these counts are high, i was just tweaking them.

Also, i'll try to take the dump

Thanks

On Sat, Jan 6, 2018 at 12:17 AM, Simone Bordet <[hidden email]> wrote:
Hi,

On Fri, Jan 5, 2018 at 6:50 PM, Sarath Prabath Redlapalli Jaya
<[hidden email]> wrote:
> Hi,
>
> We've been running a java server with Jetty(Dropwizard) and the load of
> system per node would be ~ 10k QPS as of now.
>
> We ran Load Test, Connection Test and Peak Test(No Stress test yet.), but
> couldn't reproduce what's happening on production when we chose to go live
>
> Below is the error that spits out continuosly
>
> [2018-01-05 16:05:00,874] [] [WARN] MSG - o.e.j.u.t.QueuedThreadPool
> org.eclipse.jetty.util.thread.QueuedThreadPool@dw{STARTED,4000<=4000<=4000,i=0,q=10000}
> rejected
> CEP:SocketChannelEndPoint@62241285{/127.0.0.1:49625<->/127.0.0.1:7000,OPEN,fill=FI,flush=-,to=0/30000}{io=1/0,kio=1,kro=1}->HttpConnection@23c692fe[p=HttpParser{s=START,0
> of
> 0},g=HttpGenerator@544fccad{s=START}]=>HttpChannelOverHttp@2c630bfe{r=0,c=false,a=IDLE,uri=null}:runFillable:BLOCKING
> [2018-01-05 16:05:00,874] [] [WARN] MSG - o.e.j.u.t.s.EatWhatYouKill
> java.util.concurrent.RejectedExecutionException:
> CEP:SocketChannelEndPoint@62241285{/127.0.0.1:49625<->/127.0.0.1:7000,OPEN,fill=FI,flush=-,to=0/30000}{io=1/0,kio=1,kro=1}->HttpConnection@23c692fe[p=HttpParser{s=START,0
> of
> 0},g=HttpGenerator@544fccad{s=START}]=>HttpChannelOverHttp@2c630bfe{r=0,c=false,a=IDLE,uri=null}:runFillable:BLOCKING
>         at
> org.eclipse.jetty.util.thread.QueuedThreadPool.execute(QueuedThreadPool.java:385)
>         at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:249)
>         at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
>         at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
>         at
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:243)
>         at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:679)
>         at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:597)
>         at java.lang.Thread.run(Thread.java:745)
>
> Tried to find an answer, but couldn't. Can anyone help. At this point the
> server stops accepting new connections and doesn't serve new http requests.
>
> But Jetty QueuedThreadPool Metrics show that thread Utilization is 100%

What Jetty version ?
Have you tried 9.4.8 ?

Are you using a bounded thread pool ?

If it replicates with 9.4.8, can you take a server dump as explained here:
https://www.eclipse.org/jetty/documentation/9.4.x/jetty-dump-tool.html

--
Simone Bordet
----
http://cometd.org
http://webtide.com
Developer advice, training, services and support
from the Jetty & CometD experts.
_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users


_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Jetty EatWhatYouKill - Deadlock or something!

Simone Bordet-3
Hi,

On Sat, Jan 6, 2018 at 6:18 AM, Sarath Prabath Redlapalli Jaya
<[hidden email]> wrote:

> Hi,
>
> I'm using Dropwizard 1.2.0 -> Jetty 9.4.7.v20170914.
>
> Dropwizard has a maxThreads setting for jetty. Following is the config i
> used
>
> maxThreads: 4000
> minThreads: 4000
>
> Connectors:
> acceptorThreads: 500

Wow, no.
You don't need 500 acceptor threads. Typically 1 is more than enough.

> selectorThreads: 100. Although i believe these counts are high, i was just
> tweaking them.

I am almost sure that you or Dropwizard set a limit to the number of
tasks of the thread pool queue.
I would double check that, and possibly leave it unbounded.

> Also, i'll try to take the dump

Yes, that is typically key to understand what's going on.

If you can, set QueuedThreadPool.setDetailedDump=true before dumping.

--
Simone Bordet
----
http://cometd.org
http://webtide.com
Developer advice, training, services and support
from the Jetty & CometD experts.
_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Jetty EatWhatYouKill - Deadlock or something!

Sarath Prabath Redlapalli Jaya
Hi Simone,

I took the thread dump, and after initial observation, it was some rogue heavy blocking I/O call that's keeping the thread busy. 

Jetty's perfect :). Thanks for the help

Thanks

On Sat, Jan 6, 2018 at 9:42 PM, Simone Bordet <[hidden email]> wrote:
Hi,

On Sat, Jan 6, 2018 at 6:18 AM, Sarath Prabath Redlapalli Jaya
<[hidden email]> wrote:
> Hi,
>
> I'm using Dropwizard 1.2.0 -> Jetty 9.4.7.v20170914.
>
> Dropwizard has a maxThreads setting for jetty. Following is the config i
> used
>
> maxThreads: 4000
> minThreads: 4000
>
> Connectors:
> acceptorThreads: 500

Wow, no.
You don't need 500 acceptor threads. Typically 1 is more than enough.

> selectorThreads: 100. Although i believe these counts are high, i was just
> tweaking them.

I am almost sure that you or Dropwizard set a limit to the number of
tasks of the thread pool queue.
I would double check that, and possibly leave it unbounded.

> Also, i'll try to take the dump

Yes, that is typically key to understand what's going on.

If you can, set QueuedThreadPool.setDetailedDump=true before dumping.

--
Simone Bordet
----
http://cometd.org
http://webtide.com
Developer advice, training, services and support
from the Jetty & CometD experts.
_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users


_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users