Connections being refused

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Connections being refused

Tony Seebregts
Hi,

I'm busy load testing the Jetty server and am getting a high number of
connections refused.

The load test is 250 simultaneous connections and it varies, but typically
around 50 connections are refused. I've done the obvious things:

- maxThreads is 512
- minThreads is 128
- the machine has more than enough memory
- all the servlet is doing is sending an HTTP OK response.

Would appreciate any suggestions ...





-------------------------------------------------------
SF.Net email is sponsored by: Tell us your software development plans!
Take this survey and enter to win a one-year sub to SourceForge.net
Plus IDC's 2005 look-ahead and a copy of this survey
Click here to start!  http://www.idcswdc.com/cgi-bin/survey?id=105hix
_______________________________________________
Jetty-support mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jetty-support
Reply | Threaded
Open this post in threaded view
|

Re: Connections being refused

Chris Haynes
Can you tell if its the OS that's refusing them, or is the Jetty SocketServer
accepting them but then not being able to allocate a Thread?  (a packet-sniffer
might be needed).

I've not tried this myself: I wonder if it is possible to trace the ThreadPool
size at the start/end of every connection, or at least be alerted to the point
at which Jetty switches over its connection persistance strategy..

Chris Haynes

----- Original Message -----
From: "Tony Seebregts" <[hidden email]>
To: <[hidden email]>
Sent: Thursday, April 28, 2005 3:05 PM
Subject: [Jetty-support] Connections being refused


> Hi,
>
> I'm busy load testing the Jetty server and am getting a high number of
> connections refused.
>
> The load test is 250 simultaneous connections and it varies, but typically
> around 50 connections are refused. I've done the obvious things:
>
> - maxThreads is 512
> - minThreads is 128
> - the machine has more than enough memory
> - all the servlet is doing is sending an HTTP OK response.
>
> Would appreciate any suggestions ...
>
>
>
>
>
> -------------------------------------------------------
> SF.Net email is sponsored by: Tell us your software development plans!
> Take this survey and enter to win a one-year sub to SourceForge.net
> Plus IDC's 2005 look-ahead and a copy of this survey
> Click here to start!  http://www.idcswdc.com/cgi-bin/survey?id=105hix
> _______________________________________________
> Jetty-support mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/jetty-support
>
>




-------------------------------------------------------
SF.Net email is sponsored by: Tell us your software development plans!
Take this survey and enter to win a one-year sub to SourceForge.net
Plus IDC's 2005 look-ahead and a copy of this survey
Click here to start!  http://www.idcswdc.com/cgi-bin/survey?id=105hix
_______________________________________________
Jetty-support mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jetty-support
Reply | Threaded
Open this post in threaded view
|

Re: Connections being refused

Tony Seebregts
Hi Chris,

The OS limit is supposedly 65534 connections though this being Windows XP
I don't know if that's even vaguely meaningful.

And have been (briefly) through the Jetty code and can't see anything
obvious there - the only limit on accepting connections seems to be
in the SSL listener.

Jetty does post a log message if its running low on threads - but it
doesn't seem to be that.

regards

Tony


 




-------------------------------------------------------
SF.Net email is sponsored by: Tell us your software development plans!
Take this survey and enter to win a one-year sub to SourceForge.net
Plus IDC's 2005 look-ahead and a copy of this survey
Click here to start!  http://www.idcswdc.com/cgi-bin/survey?id=105hix
_______________________________________________
Jetty-support mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jetty-support
Reply | Threaded
Open this post in threaded view
|

Re: Connections being refused

Metal Fatigue
>>>>> "T" == Tony Seebregts <[hidden email]> writes:

T> Hi Chris, The OS limit is supposedly 65534 connections though this
T> being Windows XP I don't know if that's even vaguely meaningful.

Windows client systems (including Windows XP both Home and Pro,
excluding Windows Server 2003) have a limit of 10 simultaneous inbound
connections, IIRC; the 65534-connection limit is for the sum of
inbound and outbound connections.

Hope this helps.

--
Seth Blumberg, a.k.a. Metal Fatigue, a.k.a. Nodens <[hidden email]> O-
-  --  ---  -----  --------  -------------  --------  -----  ---  --  -
"Stan went to a group to try to accept that Sheila was gone.  It was a
 group for people whose unrequited love had ended in some kind of
 surrealist moment.  There is a group for everything in California."
         -- Benjamin Rosenbaum, "The Ant King: A California Fairy Tale"


-------------------------------------------------------
SF.Net email is sponsored by: Tell us your software development plans!
Take this survey and enter to win a one-year sub to SourceForge.net
Plus IDC's 2005 look-ahead and a copy of this survey
Click here to start!  http://www.idcswdc.com/cgi-bin/survey?id=105hix
_______________________________________________
Jetty-support mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jetty-support
Reply | Threaded
Open this post in threaded view
|

Re: Connections being refused

Tony Seebregts
Done some more research & testing ...

The connection limit of 10 on Windows XP seems to apply only to
incoming client connections - at a guess this means to system
services like file sharing.

Using a simple server socket I can mostly (not always) get 1000
connections (setting soLinger to false helps enormously). With
Jetty I mostly don't get up to 200.

The number of refused connections jumps radically if I run two
or more load tests back to back ... which leads me to wonder if
its not the same problem I remember having way back when with
Windows 'leaking' socket handles.

Think it took something like a minute or ten to recover a socket
handle for a socket that had been closed - which as far as I
remember was fixed by using a socket pool.

Would welcome any other ideas though.

regards

Tony





-------------------------------------------------------
SF.Net email is sponsored by: Tell us your software development plans!
Take this survey and enter to win a one-year sub to SourceForge.net
Plus IDC's 2005 look-ahead and a copy of this survey
Click here to start!  http://www.idcswdc.com/cgi-bin/survey?id=105hix
_______________________________________________
Jetty-support mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jetty-support
Reply | Threaded
Open this post in threaded view
|

Re: Connections being refused

Greg Wilkins-5

How is your load tester creating the connections?  all at once or
over a period of time?

If it is all at once, it is conceivable that you are overflowing the
connection accept queue?

Try setting a large number (say 500?) for the listeners setAcceptQueueSize
and see if this makes a difference.
You could also try having more than 1 acceptor thread: setAcceptorThreads(5)


Tony Seebregts wrote:

> Done some more research & testing ...
>
> The connection limit of 10 on Windows XP seems to apply only to
> incoming client connections - at a guess this means to system
> services like file sharing.
>
> Using a simple server socket I can mostly (not always) get 1000
> connections (setting soLinger to false helps enormously). With
> Jetty I mostly don't get up to 200.
>
> The number of refused connections jumps radically if I run two
> or more load tests back to back ... which leads me to wonder if
> its not the same problem I remember having way back when with
> Windows 'leaking' socket handles.
>
> Think it took something like a minute or ten to recover a socket
> handle for a socket that had been closed - which as far as I
> remember was fixed by using a socket pool.
>
> Would welcome any other ideas though.
>
> regards
>
> Tony
>
>
>
>
>
> -------------------------------------------------------
> SF.Net email is sponsored by: Tell us your software development plans!
> Take this survey and enter to win a one-year sub to SourceForge.net
> Plus IDC's 2005 look-ahead and a copy of this survey
> Click here to start!  http://www.idcswdc.com/cgi-bin/survey?id=105hix
> _______________________________________________
> Jetty-support mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/jetty-support
>



-------------------------------------------------------
This SF.Net email is sponsored by Oracle Space Sweepstakes
Want to be the first software developer in space?
Enter now for the Oracle Space Sweepstakes!
http://ads.osdn.com/?ad_id=7412&alloc_id=16344&op=click
_______________________________________________
Jetty-support mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jetty-support
Reply | Threaded
Open this post in threaded view
|

RE: Re: Connections being refused

Tony Seebregts
Hi Greg,

The load tester was creating the connections all at once, since this was a
very possible scenario for us.

Changing the AcceptQueueSize helps enormously and gets past the immediate
problem - thanks !

FYI though, did notice that after running the test back-to-back several
times it started refusing connections again. At a guess the sockets aren't
being reclaimed fast enough, which is a problem I've had on Windows before.
I'll have a look into it when I have time again.

Thanks again.

Tony

 
-----Original Message-----
From: [hidden email]
[mailto:[hidden email]] On Behalf Of Greg Wilkins
Sent: 18 May 2005 11:07
To: [hidden email]
Subject: [Jetty-support] Re: Connections being refused


How is your load tester creating the connections?  all at once or
over a period of time?

If it is all at once, it is conceivable that you are overflowing the
connection accept queue?

Try setting a large number (say 500?) for the listeners setAcceptQueueSize
and see if this makes a difference.
You could also try having more than 1 acceptor thread: setAcceptorThreads(5)


Tony Seebregts wrote:

> Done some more research & testing ...
>
> The connection limit of 10 on Windows XP seems to apply only to
> incoming client connections - at a guess this means to system
> services like file sharing.
>
> Using a simple server socket I can mostly (not always) get 1000
> connections (setting soLinger to false helps enormously). With
> Jetty I mostly don't get up to 200.
>
> The number of refused connections jumps radically if I run two
> or more load tests back to back ... which leads me to wonder if
> its not the same problem I remember having way back when with
> Windows 'leaking' socket handles.
>
> Think it took something like a minute or ten to recover a socket
> handle for a socket that had been closed - which as far as I
> remember was fixed by using a socket pool.
>
> Would welcome any other ideas though.
>
> regards
>
> Tony
>
>
>
>
>
> -------------------------------------------------------
> SF.Net email is sponsored by: Tell us your software development plans!
> Take this survey and enter to win a one-year sub to SourceForge.net
> Plus IDC's 2005 look-ahead and a copy of this survey
> Click here to start!  http://www.idcswdc.com/cgi-bin/survey?id=105hix
> _______________________________________________
> Jetty-support mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/jetty-support
>



-------------------------------------------------------
This SF.Net email is sponsored by Oracle Space Sweepstakes
Want to be the first software developer in space?
Enter now for the Oracle Space Sweepstakes!
http://ads.osdn.com/?ad_id=7412&alloc_id=16344&op=click
_______________________________________________
Jetty-support mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jetty-support



-------------------------------------------------------
This SF.Net email is sponsored by Oracle Space Sweepstakes
Want to be the first software developer in space?
Enter now for the Oracle Space Sweepstakes!
http://ads.osdn.com/?ad_id=7412&alloc_id=16344&op=click
_______________________________________________
Jetty-support mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jetty-support
Reply | Threaded
Open this post in threaded view
|

SSL Redirecting

Tony Seebregts
In reply to this post by Greg Wilkins-5
Hi,

If I connect to Jetty using SSL on port 8443 it generates a
"HTTP/1.1 302 Moved Temporarily" response.

Any idea why ?

Regards

Tony Seebregts




-------------------------------------------------------
This SF.Net email is sponsored by: NEC IT Guy Games.  How far can you shotput
a projector? How fast can you ride your desk chair down the office luge track?
If you want to score the big prize, get to know the little guy.  
Play to win an NEC 61" plasma display: http://www.necitguy.com/?r=20
_______________________________________________
Jetty-support mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jetty-support
Reply | Threaded
Open this post in threaded view
|

Re: SSL Redirecting

Greg Wilkins-5

Tony,

I need more info.

Jetty could be doing a 302 to move the request from /dirname to /dirname/
or it could be a security data constraint thang?

cheers


Tony Seebregts wrote:

> Hi,
>
> If I connect to Jetty using SSL on port 8443 it generates a
> "HTTP/1.1 302 Moved Temporarily" response.
>
> Any idea why ?
>
> Regards
>
> Tony Seebregts
>
>
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by: NEC IT Guy Games.  How far can you shotput
> a projector? How fast can you ride your desk chair down the office luge track?
> If you want to score the big prize, get to know the little guy.  
> Play to win an NEC 61" plasma display: http://www.necitguy.com/?r=20
> _______________________________________________
> Jetty-support mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/jetty-support
>



-------------------------------------------------------
SF.Net email is Sponsored by the Better Software Conference & EXPO
September 19-22, 2005 * San Francisco, CA * Development Lifecycle Practices
Agile & Plan-Driven Development * Managing Projects & Teams * Testing & QA
Security * Process Improvement & Measurement * http://www.sqe.com/bsce5sf
_______________________________________________
Jetty-support mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jetty-support
Reply | Threaded
Open this post in threaded view
|

Re: Re: SSL Redirecting

Tony Seebregts
Hi Greg,

Apologies for the delay in replying ... been busy.

re. SSL redirecting. Either I'm doing something less stupid (most
likely) or the RC1 release has fixed it, but the problem seems to have
gone away.

regards

Tony

Greg Wilkins wrote:

>Tony,
>
>I need more info.
>
>Jetty could be doing a 302 to move the request from /dirname to /dirname/
>or it could be a security data constraint thang?
>
>cheers
>
>
>Tony Seebregts wrote:
>  
>
>>Hi,
>>
>>If I connect to Jetty using SSL on port 8443 it generates a
>>"HTTP/1.1 302 Moved Temporarily" response.
>>
>>Any idea why ?
>>
>>Regards
>>
>>Tony Seebregts
>>    
>>



-------------------------------------------------------
SF.Net email is Sponsored by the Better Software Conference & EXPO
September 19-22, 2005 * San Francisco, CA * Development Lifecycle Practices
Agile & Plan-Driven Development * Managing Projects & Teams * Testing & QA
Security * Process Improvement & Measurement * http://www.sqe.com/bsce5sf
_______________________________________________
Jetty-support mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jetty-support