[jetty-users] Jetty refuses to accept new request

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

[jetty-users] Jetty refuses to accept new request

Peter Henzler

Hi everyone

 

We have Jetty embedded in our eclipse osgi server application.

Everything worked fine until we upgraded to using Jetty 7.5.4.v20111024 and Java 7.

 

Now we have serious troubles with our application.

From one moment to the other the web server does not accept new requests.
That means users connecting new to the server do not get a response from the server. It seems as if the call is hanging at the server side.
First we thought this could be a memory issue.

But we found out that this is not the case.

Users that to have their sessions can happily work and the server handles their request without problems.

It looks as if the web server cannot accept requests from clients that are new.

 

We are booting our application every morning now.

Sometimes the server is up for 2-3 days until this issue occurs.
But sometimes this happens already 2 hours after starting.

So it seems that it has nothing to do with workload, memory usage or computer resources.

 

We are totally desperate and hope that somebody can give us a hint, what might be the problem.

 

Best regards from Switzerland

Peter Henzler


_______________________________________________
jetty-users mailing list
[hidden email]
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: [jetty-users] Jetty refuses to accept new request

Joakim Erdfelt-9
Without stacktraces to go on, I would have to speculate.

My first guess would be that you are relying on /tmp for your temporary & work directories.
The problem you describe is a behavior seen on many unix systems where the contents of the /tmp directory are cleared out occasionally by various cronjobs.
Using /tmp is a behavior seen on many unix systems as the System.getProperty("java.io.tmpdir") often returns this location (but not on all unix systems and configurations).
The "java.io.tmpdir" directory is used by jetty to unpack war files into and run the war file from, if that directory gets cleared out from under jetty then jetty stops processing requests to that context.

You can solve this in one of two ways.

Way 1:
Create a ${jetty.home}/work directory.
By default jetty will use this directory for unpacking war files into (if it exists).

Way 2:
Specify the temp directory.
Using the DeploymentManager + WebAppProvider you can specify the "tempDir" property to somewhere more stable.
Example of this configuration can be seen in the test cases in jetty.

--
Joakim Erdfelt

(the people behind jetty and cometd)



On Fri, Jan 6, 2012 at 8:25 AM, Peter Henzler <[hidden email]> wrote:

Hi everyone

 

We have Jetty embedded in our eclipse osgi server application.

Everything worked fine until we upgraded to using Jetty 7.5.4.v20111024 and Java 7.

 

Now we have serious troubles with our application.

From one moment to the other the web server does not accept new requests.
That means users connecting new to the server do not get a response from the server. It seems as if the call is hanging at the server side.
First we thought this could be a memory issue.

But we found out that this is not the case.

Users that to have their sessions can happily work and the server handles their request without problems.

It looks as if the web server cannot accept requests from clients that are new.

 

We are booting our application every morning now.

Sometimes the server is up for 2-3 days until this issue occurs.
But sometimes this happens already 2 hours after starting.

So it seems that it has nothing to do with workload, memory usage or computer resources.

 

We are totally desperate and hope that somebody can give us a hint, what might be the problem.

 

Best regards from Switzerland

Peter Henzler


_______________________________________________
jetty-users mailing list
[hidden email]
https://dev.eclipse.org/mailman/listinfo/jetty-users



_______________________________________________
jetty-users mailing list
[hidden email]
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: [jetty-users] Jetty refuses to accept new request

Peter Henzler

Hi Joakim

 

Thank you for your answer.

The problem is that there are no stacktraces.
I do not get a single error message and all the people that do have a connection can happily work with the context while the new request do not find the way to our application. It seems as if those new requests are hanging in the jetty webserver part and do not go get processed.

We are using Windows not Unix.

 

Even static html content is not delivered any more to those new requests.

 

Best regards

Peter

 

Von: [hidden email] [mailto:[hidden email]] Im Auftrag von Joakim Erdfelt
Gesendet: Freitag, 6. Januar 2012 16:55
An: JETTY user mailing list
Betreff: Re: [jetty-users] Jetty refuses to accept new request

 

Without stacktraces to go on, I would have to speculate.

 

My first guess would be that you are relying on /tmp for your temporary & work directories.

The problem you describe is a behavior seen on many unix systems where the contents of the /tmp directory are cleared out occasionally by various cronjobs.

Using /tmp is a behavior seen on many unix systems as the System.getProperty("java.io.tmpdir") often returns this location (but not on all unix systems and configurations).

The "java.io.tmpdir" directory is used by jetty to unpack war files into and run the war file from, if that directory gets cleared out from under jetty then jetty stops processing requests to that context.

 

You can solve this in one of two ways.

 

Way 1:

Create a ${jetty.home}/work directory.

By default jetty will use this directory for unpacking war files into (if it exists).

 

Way 2:

Specify the temp directory.

Using the DeploymentManager + WebAppProvider you can specify the "tempDir" property to somewhere more stable.

Example of this configuration can be seen in the test cases in jetty.

 

--

Joakim Erdfelt

 

(the people behind jetty and cometd)



On Fri, Jan 6, 2012 at 8:25 AM, Peter Henzler <[hidden email]> wrote:

Hi everyone

 

We have Jetty embedded in our eclipse osgi server application.

Everything worked fine until we upgraded to using Jetty 7.5.4.v20111024 and Java 7.

 

Now we have serious troubles with our application.

From one moment to the other the web server does not accept new requests.
That means users connecting new to the server do not get a response from the server. It seems as if the call is hanging at the server side.
First we thought this could be a memory issue.

But we found out that this is not the case.

Users that to have their sessions can happily work and the server handles their request without problems.

It looks as if the web server cannot accept requests from clients that are new.

 

We are booting our application every morning now.

Sometimes the server is up for 2-3 days until this issue occurs.
But sometimes this happens already 2 hours after starting.

So it seems that it has nothing to do with workload, memory usage or computer resources.

 

We are totally desperate and hope that somebody can give us a hint, what might be the problem.

 

Best regards from Switzerland

Peter Henzler


_______________________________________________
jetty-users mailing list
[hidden email]
https://dev.eclipse.org/mailman/listinfo/jetty-users

 


_______________________________________________
jetty-users mailing list
[hidden email]
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: [jetty-users] Jetty refuses to accept new request

Simone Bordet-2
Hi,

On Sat, Jan 7, 2012 at 07:53, Peter Henzler <[hidden email]> wrote:
> Hi Joakim
>
> The problem is that there are no stacktraces.

You can take a stack trace by using the JDK tool "jstack", see
http://docs.oracle.com/javase/7/docs/webnotes/tsg/TSG-VM/html/tooldescr.html#gblfh.

That should give some hint on what's going on.

Do connection land directly to Jetty, or they pass through some other
network element (such as Apache, nginx, firewalls) ?

Have you double checked that Windows is not "protecting" you by
limiting the number of concurrent connections ?

Simon
--
http://cometd.org
http://intalio.com
http://bordet.blogspot.com
----
Finally, no matter how good the architecture and design are,
to deliver bug-free software with optimal performance and reliability,
the implementation technique must be flawless.   Victoria Livschitz
_______________________________________________
jetty-users mailing list
[hidden email]
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: [jetty-users] Jetty refuses to accept new request

Peter Henzler
Hi

Ok, I will try to use jstack the next time the issue occurs.

I definitely can say that it is not a network nor a windows problem.
The application has run without problems when we where using Jetty 6 and Java 6. And there where the same number of users and usage at that time.

Peter

-----Ursprüngliche Nachricht-----
Von: [hidden email] [mailto:[hidden email]] Im Auftrag von Simone Bordet
Gesendet: Sonntag, 8. Januar 2012 15:04
An: JETTY user mailing list
Betreff: Re: [jetty-users] Jetty refuses to accept new request

Hi,

On Sat, Jan 7, 2012 at 07:53, Peter Henzler <[hidden email]> wrote:
> Hi Joakim
>
> The problem is that there are no stacktraces.

You can take a stack trace by using the JDK tool "jstack", see http://docs.oracle.com/javase/7/docs/webnotes/tsg/TSG-VM/html/tooldescr.html#gblfh.

That should give some hint on what's going on.

Do connection land directly to Jetty, or they pass through some other network element (such as Apache, nginx, firewalls) ?

Have you double checked that Windows is not "protecting" you by limiting the number of concurrent connections ?

Simon
--
http://cometd.org
http://intalio.com
http://bordet.blogspot.com
----
Finally, no matter how good the architecture and design are, to deliver bug-free software with optimal performance and reliability, the implementation technique must be flawless.   Victoria Livschitz _______________________________________________
jetty-users mailing list
[hidden email]
https://dev.eclipse.org/mailman/listinfo/jetty-users
_______________________________________________
jetty-users mailing list
[hidden email]
https://dev.eclipse.org/mailman/listinfo/jetty-users