[jetty-dev] Maximum allowed time to proceed request

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

[jetty-dev] Maximum allowed time to proceed request

Игорь Периков
Hello everyone, 
While I was looking at https://grpc.io/blog/deadlines the other day, I was thinking about optimizing resource utilization and better availability in my applications. And here what I came up with - it always makes sense to handle some sort of deadlines, especially in io-intensive applications where one business function might consist of N different http/database calls, likely with retries. Since all external party timings can vary (for a bunch of different reasons), we can't force exact timeouts for every party. Without concept like that our load can be significantly amplified, because client (as an application or as an user with almighty F5) would come in a moment, asking us to do the same thing once again (and very likely to start queuing, then forgetting about this request and retrying once again and so on). 

Which bring us to the first question - since Jetty itself handles all worker threads, do you think it would make sense to control this deadlines concept from inside? Is it possible to handle it with already existing abstractions? If so, would you mind mentioning them to me? 

Next, we should somehow define this point of time, after which we can execute our termination operation (throw exception, stop the thread, etc). I see a few possibilities here: 
1) Since it's our application, we know better what is our expected sla, so basically we can have a Map URI -> Timeout, get the timeout, sum it up with current timestamp, store in ThreadLocal and verify at stoppoints, terminate on exceeding. 
2) We can ask client to send header with timestamp value, when he will stop waiting for response, read this in inbound filter and then follow option 1 mechanism. 
3) We can stamp request with timestamp on balancer/reverse proxy and then follow option 2 mechanism. 

Option 1 seems to be the most convenient from architectural and security purposes, but there is one nasty problem - Filter executes after request being popped from the queue, and we don't know for how many time it was held in queue. If server suffers under increased load, most likely these tail requests (from queue) are 'dead' as well - there is no such client who need them to be completed, and it would be extremely convenient to just drop those and proceed fresh ones. 

This lead us to the second question - is there any abstraction which executes before queue insertion or immediately after successful insertion (so we are sure it doesn't surpass pool capacity and still have reference to it) and/or augment request or whatever else is suitable so we can retrieve it later in filter?

--
Best regards,
Igor Perikov

_______________________________________________
jetty-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-dev
Reply | Threaded
Open this post in threaded view
|

Re: [jetty-dev] Maximum allowed time to proceed request

Greg Wilkins

Many decades ago, Jetty did have a built in mechanism to limit the time that a thread could work on a response. However it is now: against the servlet spec; impossible to generally stop a busy thread; async processing can make deadlines non thread dependent.

We do carry a timestamp on the request that is set moderately early in request processing.  It is set as the parsing of the request header is completed.   However, there can still be some delays before this.  To get the earliest timestamp for a request it would be necessary to grab a timestamp as the selector wakes up to dispatch and/or parse a connection that will likely have a request on it.     I have done this as a special case in the past, but as we have done a lot of reworking in the scheduling for http2, there is no general way to do that.    However, if this is something of interest, we can have a look again at how this can be done (now that the scheduling code is more stable).    My caution is that I don't want to put a burden on code that does not need this precise timing.

If you have a load balancer that can add a timestamp, then that is probably a simpler way to base a deadline on.   The challenge is of course how to stop a thread or cancel an async action when the deadline is exceeded.  There is no general way to do that in java!  But you may be able to do so in an application specific way.

Thus the best solution is probably a filter that implements deadlines in an application specific way based of timestamps put in the request by a load balancer.

regards







On Sat, 12 Jan 2019 at 00:13, Игорь Периков <[hidden email]> wrote:
Hello everyone, 
While I was looking at https://grpc.io/blog/deadlines the other day, I was thinking about optimizing resource utilization and better availability in my applications. And here what I came up with - it always makes sense to handle some sort of deadlines, especially in io-intensive applications where one business function might consist of N different http/database calls, likely with retries. Since all external party timings can vary (for a bunch of different reasons), we can't force exact timeouts for every party. Without concept like that our load can be significantly amplified, because client (as an application or as an user with almighty F5) would come in a moment, asking us to do the same thing once again (and very likely to start queuing, then forgetting about this request and retrying once again and so on). 

Which bring us to the first question - since Jetty itself handles all worker threads, do you think it would make sense to control this deadlines concept from inside? Is it possible to handle it with already existing abstractions? If so, would you mind mentioning them to me? 

Next, we should somehow define this point of time, after which we can execute our termination operation (throw exception, stop the thread, etc). I see a few possibilities here: 
1) Since it's our application, we know better what is our expected sla, so basically we can have a Map URI -> Timeout, get the timeout, sum it up with current timestamp, store in ThreadLocal and verify at stoppoints, terminate on exceeding. 
2) We can ask client to send header with timestamp value, when he will stop waiting for response, read this in inbound filter and then follow option 1 mechanism. 
3) We can stamp request with timestamp on balancer/reverse proxy and then follow option 2 mechanism. 

Option 1 seems to be the most convenient from architectural and security purposes, but there is one nasty problem - Filter executes after request being popped from the queue, and we don't know for how many time it was held in queue. If server suffers under increased load, most likely these tail requests (from queue) are 'dead' as well - there is no such client who need them to be completed, and it would be extremely convenient to just drop those and proceed fresh ones. 

This lead us to the second question - is there any abstraction which executes before queue insertion or immediately after successful insertion (so we are sure it doesn't surpass pool capacity and still have reference to it) and/or augment request or whatever else is suitable so we can retrieve it later in filter?

--
Best regards,
Igor Perikov
_______________________________________________
jetty-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-dev


--

_______________________________________________
jetty-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-dev
Reply | Threaded
Open this post in threaded view
|

Re: [jetty-dev] Maximum allowed time to proceed request

Игорь Периков
Thanks for the response, Greg. Yes, application-specific way looks good for me. Bad thing about balancers is that you don't always have a precise control over them, so I'd like to stick to the application-only way. 

We do carry a timestamp on the request that is set moderately early in request processing.  It is set as the parsing of the request header is completed.   However, there can still be some delays before this.
Would you mind explaining what type of delays you are talking about and how do I obtain this timestamp value? I'm not very good with jetty internals so any help would be useful. 

сб, 12 янв. 2019 г. в 00:04, Greg Wilkins <[hidden email]>:

Many decades ago, Jetty did have a built in mechanism to limit the time that a thread could work on a response. However it is now: against the servlet spec; impossible to generally stop a busy thread; async processing can make deadlines non thread dependent.

We do carry a timestamp on the request that is set moderately early in request processing.  It is set as the parsing of the request header is completed.   However, there can still be some delays before this.  To get the earliest timestamp for a request it would be necessary to grab a timestamp as the selector wakes up to dispatch and/or parse a connection that will likely have a request on it.     I have done this as a special case in the past, but as we have done a lot of reworking in the scheduling for http2, there is no general way to do that.    However, if this is something of interest, we can have a look again at how this can be done (now that the scheduling code is more stable).    My caution is that I don't want to put a burden on code that does not need this precise timing.

If you have a load balancer that can add a timestamp, then that is probably a simpler way to base a deadline on.   The challenge is of course how to stop a thread or cancel an async action when the deadline is exceeded.  There is no general way to do that in java!  But you may be able to do so in an application specific way.

Thus the best solution is probably a filter that implements deadlines in an application specific way based of timestamps put in the request by a load balancer.

regards







On Sat, 12 Jan 2019 at 00:13, Игорь Периков <[hidden email]> wrote:
Hello everyone, 
While I was looking at https://grpc.io/blog/deadlines the other day, I was thinking about optimizing resource utilization and better availability in my applications. And here what I came up with - it always makes sense to handle some sort of deadlines, especially in io-intensive applications where one business function might consist of N different http/database calls, likely with retries. Since all external party timings can vary (for a bunch of different reasons), we can't force exact timeouts for every party. Without concept like that our load can be significantly amplified, because client (as an application or as an user with almighty F5) would come in a moment, asking us to do the same thing once again (and very likely to start queuing, then forgetting about this request and retrying once again and so on). 

Which bring us to the first question - since Jetty itself handles all worker threads, do you think it would make sense to control this deadlines concept from inside? Is it possible to handle it with already existing abstractions? If so, would you mind mentioning them to me? 

Next, we should somehow define this point of time, after which we can execute our termination operation (throw exception, stop the thread, etc). I see a few possibilities here: 
1) Since it's our application, we know better what is our expected sla, so basically we can have a Map URI -> Timeout, get the timeout, sum it up with current timestamp, store in ThreadLocal and verify at stoppoints, terminate on exceeding. 
2) We can ask client to send header with timestamp value, when he will stop waiting for response, read this in inbound filter and then follow option 1 mechanism. 
3) We can stamp request with timestamp on balancer/reverse proxy and then follow option 2 mechanism. 

Option 1 seems to be the most convenient from architectural and security purposes, but there is one nasty problem - Filter executes after request being popped from the queue, and we don't know for how many time it was held in queue. If server suffers under increased load, most likely these tail requests (from queue) are 'dead' as well - there is no such client who need them to be completed, and it would be extremely convenient to just drop those and proceed fresh ones. 

This lead us to the second question - is there any abstraction which executes before queue insertion or immediately after successful insertion (so we are sure it doesn't surpass pool capacity and still have reference to it) and/or augment request or whatever else is suitable so we can retrieve it later in filter?

--
Best regards,
Igor Perikov
_______________________________________________
jetty-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-dev


--
_______________________________________________
jetty-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-dev


--
Best regards,
Igor Perikov

_______________________________________________
jetty-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-dev