How to log/monitor/troubleshoot the number of Jetty active threads in threadingParameters

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

How to log/monitor/troubleshoot the number of Jetty active threads in threadingParameters

lekkie
This post has NOT been accepted by the mailing list yet.
Hi guys,

I will like to monitor the performance of my jetty server. I had configured it to use certain number of minimum and maximum threads in threadingParameters:

 <httpj:threadingParameters minThreads="${minimum.jetty.request.thread}" maxThreads="${maximum.jetty.request.thread}" />

I will like to know how I can monitor how much threads are being used (during request processing) and how much are left. It will be nice if I can keep this historical data.

I have been having some performance issues lately which I am guessing it may be caused by overloading. See exception below:

13:15:16,741 | WARN  | tp523013841-1326 | PhaseInterceptorChain            | 169 - org.apache.cxf.cxf-api - 2.6.0.fuse-71-047 | Interceptor for {http://resources.consumer.rest.service.com/}RestRequestResource has thrown exception, unwinding now
java.lang.RuntimeException: java.lang.IllegalStateException: DISPATCHED,initial
        at org.apache.cxf.jaxrs.interceptor.JAXRSOutInterceptor.writeResponseToStream(JAXRSOutInterceptor.java:499)[183:org.apache.cxf.cxf-rt-frontend-jaxrs:2.6.0.fuse-71-047]
        at org.apache.cxf.jaxrs.interceptor.JAXRSOutInterceptor.serializeMessage(JAXRSOutInterceptor.java:199)[183:org.apache.cxf.cxf-rt-frontend-jaxrs:2.6.0.fuse-71-047]
        at org.apache.cxf.jaxrs.interceptor.JAXRSOutInterceptor.processResponse(JAXRSOutInterceptor.java:146)[183:org.apache.cxf.cxf-rt-frontend-jaxrs:2.6.0.fuse-71-047]
        at org.apache.cxf.jaxrs.interceptor.JAXRSOutInterceptor.handleMessage(JAXRSOutInterceptor.java:84)[183:org.apache.cxf.cxf-rt-frontend-jaxrs:2.6.0.fuse-71-047]
        at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:262)[169:org.apache.cxf.cxf-api:2.6.0.fuse-71-047]
        at org.apache.cxf.interceptor.OutgoingChainInterceptor.handleMessage(OutgoingChainInterceptor.java:77)[169:org.apache.cxf.cxf-api:2.6.0.fuse-71-047]
        at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:262)[169:org.apache.cxf.cxf-api:2.6.0.fuse-71-047]
        at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)[169:org.apache.cxf.cxf-api:2.6.0.fuse-71-047]
        at org.apache.cxf.transport.http_jetty.JettyHTTPDestination.serviceRequest(JettyHTTPDestination.java:348)[192:org.apache.cxf.cxf-rt-transports-http-jetty:2.6.0.fuse-71-047]
        at org.apache.cxf.transport.http_jetty.JettyHTTPDestination.doService(JettyHTTPDestination.java:312)[192:org.apache.cxf.cxf-rt-transports-http-jetty:2.6.0.fuse-71-047]
        at org.apache.cxf.transport.http_jetty.JettyHTTPHandler.handle(JettyHTTPHandler.java:72)[192:org.apache.cxf.cxf-rt-transports-http-jetty:2.6.0.fuse-71-047]
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1040)[84:org.eclipse.jetty.server:7.6.7.v20120910]
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:976)[84:org.eclipse.jetty.server:7.6.7.v20120910]
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)[84:org.eclipse.jetty.server:7.6.7.v20120910]
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)[84:org.eclipse.jetty.server:7.6.7.v20120910]
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)[84:org.eclipse.jetty.server:7.6.7.v20120910]
        at org.eclipse.jetty.server.Server.handle(Server.java:363)[84:org.eclipse.jetty.server:7.6.7.v20120910]
        at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:483)[84:org.eclipse.jetty.server:7.6.7.v20120910]
        at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:920)[84:org.eclipse.jetty.server:7.6.7.v20120910]
        at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:982)[84:org.eclipse.jetty.server:7.6.7.v20120910]
        at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:635)[79:org.eclipse.jetty.http:7.6.7.v20120910]
        at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)[79:org.eclipse.jetty.http:7.6.7.v20120910]
        at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)[84:org.eclipse.jetty.server:7.6.7.v20120910]
        at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:627)[81:org.eclipse.jetty.io:7.6.7.v20120910]
        at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:51)[81:org.eclipse.jetty.io:7.6.7.v20120910]
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)[90:org.eclipse.jetty.util:7.6.7.v20120910]
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)[90:org.eclipse.jetty.util:7.6.7.v20120910]
        at java.lang.Thread.run(Thread.java:745)[:1.7.0_67]
Caused by: java.lang.IllegalStateException: DISPATCHED,initial
        at org.eclipse.jetty.server.AsyncContinuation.handling(AsyncContinuation.java:292)[84:org.eclipse.jetty.server:7.6.7.v20120910]
        at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:444)[84:org.eclipse.jetty.server:7.6.7.v20120910]
        at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:920)[84:org.eclipse.jetty.server:7.6.7.v20120910]
        at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:982)[84:org.eclipse.jetty.server:7.6.7.v20120910]
        at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:635)[79:org.eclipse.jetty.http:7.6.7.v20120910]
        at org.eclipse.jetty.http.HttpParser.available(HttpParser.java:1225)[79:org.eclipse.jetty.http:7.6.7.v20120910]
        at org.eclipse.jetty.server.HttpInput.available(HttpInput.java:73)[84:org.eclipse.jetty.server:7.6.7.v20120910]
        at org.apache.cxf.helpers.IOUtils.copy(IOUtils.java:126)[169:org.apache.cxf.cxf-api:2.6.0.fuse-71-047]
        at org.apache.cxf.helpers.IOUtils.copy(IOUtils.java:103)[169:org.apache.cxf.cxf-api:2.6.0.fuse-71-047]
        at org.apache.cxf.io.DelegatingInputStream.cacheInput(DelegatingInputStream.java:54)[169:org.apache.cxf.cxf-api:2.6.0.fuse-71-047]
        at org.apache.cxf.transport.http.AbstractHTTPDestination$1.cacheInput(AbstractHTTPDestination.java:275)[178:org.apache.cxf.cxf-rt-transports-http:2.6.0.fuse-71-047]
        at org.apache.cxf.transport.http.AbstractHTTPDestination.cacheInput(AbstractHTTPDestination.java:524)[178:org.apache.cxf.cxf-rt-transports-http:2.6.0.fuse-71-047]
        at org.apache.cxf.transport.http.AbstractHTTPDestination.flushHeaders(AbstractHTTPDestination.java:536)[178:org.apache.cxf.cxf-rt-transports-http:2.6.0.fuse-71-047]
        at org.apache.cxf.transport.http.AbstractHTTPDestination.flushHeaders(AbstractHTTPDestination.java:529)[178:org.apache.cxf.cxf-rt-transports-http:2.6.0.fuse-71-047]
        at org.apache.cxf.transport.http.AbstractHTTPDestination$WrappedOutputStream.onFirstWrite(AbstractHTTPDestination.java:693)[178:org.apache.cxf.cxf-rt-transports-http:2.6.0.fuse-71-047]
        at org.apache.cxf.io.AbstractWrappedOutputStream.write(AbstractWrappedOutputStream.java:47)[169:org.apache.cxf.cxf-api:2.6.0.fuse-71-047]
        at org.apache.cxf.io.CacheAndWriteOutputStream.write(CacheAndWriteOutputStream.java:71)[169:org.apache.cxf.cxf-api:2.6.0.fuse-71-047]
        at org.apache.cxf.jaxrs.interceptor.JAXRSOutInterceptor.writeResponseToStream(JAXRSOutInterceptor.java:495)[183:org.apache.cxf.cxf-rt-frontend-jaxrs:2.6.0.fuse-71-047]
        ... 27 more
13:15:16,741 | ERROR | tp523013841-1368 | JAXRSOutInterceptor              | 183 - org.apache.cxf.cxf-rt-frontend-jaxrs - 2.6.0.fuse-71-047 | Problem with writing the data to the output stream
13:15:16,741 | ERROR | tp523013841-1368 | JAXRSOutInterceptor              | 183 - org.apache.cxf.cxf-rt-frontend-jaxrs - 2.6.0.fuse-71-047 | Problem with writing the data to the output stream

Regards.