[jetty-users] How to get Jetty to degrade gracefully under load

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

[jetty-users] How to get Jetty to degrade gracefully under load

ccleve
We're seeing an odd problem when load testing our site. Something's
probably misconfigured, but I don't know where to look.

As we ramp up the number of users with our load testing software, the
number of requests Jetty can handle tops out at about 100/second. (Our
app is doing some work to process each request, so the low rate isn't
Jetty's fault). As more and more requests come in, average page load
time creeps up to 6 or 7 seconds, max load time becomes very high, and
then all at once the system starts sending 503s. Any new requests get a
503, and the load testing software reports that it starts getting about
500 503s per second, presumably from earlier requests that were queued
up. This continues for about a minute, and then the system starts
accepting requests again.

So when a lot of users come in, some get serviced, some have to wait a
long time, and then whole system shuts down for a minute while it recovers.

I'd prefer not to have a catastrophic failure like this. Perhaps the
system should send 503s earlier so the queue never gets too big. Or
maybe it should never send a 503, and just make the wait times uniformly
longer for all users.

We're on Amazon, so we can use monitoring software to detect when page
load time is > X seconds and spin up more instances.

What parameters can we twiddle to control how Jetty degrades? Or have we
misconfigured things?

_______________________________________________
jetty-users mailing list
[hidden email]
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: [jetty-users] How to get Jetty to degrade gracefully under load

Simone Bordet-2
Hi,

On Fri, Jun 8, 2012 at 6:21 PM, ccleve <[hidden email]> wrote:

> We're seeing an odd problem when load testing our site. Something's probably
> misconfigured, but I don't know where to look.
>
> As we ramp up the number of users with our load testing software, the number
> of requests Jetty can handle tops out at about 100/second. (Our app is doing
> some work to process each request, so the low rate isn't Jetty's fault). As
> more and more requests come in, average page load time creeps up to 6 or 7
> seconds, max load time becomes very high, and then all at once the system
> starts sending 503s. Any new requests get a 503, and the load testing
> software reports that it starts getting about 500 503s per second,
> presumably from earlier requests that were queued up. This continues for
> about a minute, and then the system starts accepting requests again.
>
> So when a lot of users come in, some get serviced, some have to wait a long
> time, and then whole system shuts down for a minute while it recovers.
>
> I'd prefer not to have a catastrophic failure like this. Perhaps the system
> should send 503s earlier so the queue never gets too big. Or maybe it should
> never send a 503, and just make the wait times uniformly longer for all
> users.
>
> We're on Amazon, so we can use monitoring software to detect when page load
> time is > X seconds and spin up more instances.
>
> What parameters can we twiddle to control how Jetty degrades? Or have we
> misconfigured things?

Are you sure that's Jetty sending the 503 and not the application or
some other network component (e.g. Apache) in front of Jetty ?
Are you using the QoSFilter ?

I ask because Jetty does not send, to my knowledge, 503s when the
system is overloaded.

Simon
--
www.webtide.com
Developer advice, services and support
from the Jetty & CometD experts.
----
Finally, no matter how good the architecture and design are,
to deliver bug-free software with optimal performance and reliability,
the implementation technique must be flawless.   Victoria Livschitz
_______________________________________________
jetty-users mailing list
[hidden email]
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: [jetty-users] How to get Jetty to degrade gracefully under load

ccleve
On 6/8/2012 11:47 AM, Simone Bordet wrote:

> Hi,
>
> On Fri, Jun 8, 2012 at 6:21 PM, ccleve<[hidden email]>  wrote:
>> We're seeing an odd problem when load testing our site. Something's probably
>> misconfigured, but I don't know where to look.
>>
>> As we ramp up the number of users with our load testing software, the number
>> of requests Jetty can handle tops out at about 100/second. (Our app is doing
>> some work to process each request, so the low rate isn't Jetty's fault). As
>> more and more requests come in, average page load time creeps up to 6 or 7
>> seconds, max load time becomes very high, and then all at once the system
>> starts sending 503s. Any new requests get a 503, and the load testing
>> software reports that it starts getting about 500 503s per second,
>> presumably from earlier requests that were queued up. This continues for
>> about a minute, and then the system starts accepting requests again.
>>
>> So when a lot of users come in, some get serviced, some have to wait a long
>> time, and then whole system shuts down for a minute while it recovers.
>>
>> I'd prefer not to have a catastrophic failure like this. Perhaps the system
>> should send 503s earlier so the queue never gets too big. Or maybe it should
>> never send a 503, and just make the wait times uniformly longer for all
>> users.
>>
>> We're on Amazon, so we can use monitoring software to detect when page load
>> time is>  X seconds and spin up more instances.
>>
>> What parameters can we twiddle to control how Jetty degrades? Or have we
>> misconfigured things?
>
> Are you sure that's Jetty sending the 503 and not the application or
> some other network component (e.g. Apache) in front of Jetty ?
> Are you using the QoSFilter ?
>
> I ask because Jetty does not send, to my knowledge, 503s when the
> system is overloaded.
>
> Simon

Thank you for the clue. Turns out that there was a load balancer between
the loadtester and our site that I didn't know about. That was throwing
the 503s. I will check out QoSFilter.


_______________________________________________
jetty-users mailing list
[hidden email]
https://dev.eclipse.org/mailman/listinfo/jetty-users