Hanging threads when using Jetty 6 with IE8 & IE9

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Hanging threads when using Jetty 6 with IE8 & IE9

Lior Yaffe

Greetings,


Jetty is great, we love it so much, you are all doing such a wonderful work !

 

We are experiencing a high priority problem at one of our customer sites.

The symptoms we are seeing are, that simple browser GET requests for images or static resources just hang for up 50 seconds with no apparent reason.

 

We are using:

Jetty 6.x (latest) with BIO connector running on Windows 2008 32 bit with Java 1.6.x (latest)

 

We are using the following thread pool settings in jetty.xml:

    <Set name="ThreadPool">

      <New class="org.mortbay.thread.QueuedThreadPool">

        <Set name="minThreads">10</Set>

        <Set name="maxThreads">200</Set>

        <Set name="lowThreads">20</Set>

        <Set name="SpawnOrShrinkAt">2</Set>

      </New>

    </Set>

 

On the client side the problem only occurs when using IE8 and IE9 but not with IE7 (probably since IE7 is limited to two concurrent requests but we are not sure)

 

When the problem occurs we can see in the server logs the following events:

1.       A new web page is sent is to the browser as a response to a client request.

2.       The client browser then submits 10 concurrent requests at once, for loading static resources such as images, JavaScripts, CSS etc.

3.       Jetty dispatches all 10 threads in its thread pool to serve these resource.

4.       All Jetty threads are now hang, with the following stack trace waiting for something:

"9543875@qtp-23978087-3" prio=6 tid=0x07b07c00 nid=0x1f6c runnable [0x0908f000..0x0908fbe8]

   java.lang.Thread.State: RUNNABLE

        at java.net.SocketInputStream.socketRead0(Native Method)

        at java.net.SocketInputStream.read(SocketInputStream.java:129)

        at org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:382)

        at org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:107)

        at org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:198)

        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:290)

        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)

        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)

        at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)

        at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:520)

 

5.       At some point around 50 seconds later we receive the following exception in the log:

java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:150)
        at java.net.SocketInputStream.read(SocketInputStream.java:121)
        at org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:382)
        at org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:107)
        at org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:198)
        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:290)
        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
        at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)

 

6.       After this exception the Jetty threads are operational again freeing the client browser to finish rendering the page.

We are able to reproduce this problem fairly easily in our test environment even with a single session.


We have two questions:

1.       Why are the Jetty threads hanging on socket read after serving a static resources ?

2.       Why doesn’t the Jetty thread pool spawn more threads up to 200 when all initial 10 threads are locked ?


We already made the following observation:

1.       If we change SpawnOrShrinkAt to 0 then the thread pool is able to spawn more threads and this improves the situation and delays the occurrence of the problem.

 

Any advice on this would be much appreciated,

 

Best regards,

Lior