Preventing queuing of requests when all connections exhausted

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

Preventing queuing of requests when all connections exhausted

John Gardiner Myers
I want to configure Jetty client so that if all
MaxConnectionsPerDestination are in use then additional requests to that
destination immediately fail rather than queue. If a destination is
hung, better to immediately fail over to another instance of the service
than sit in the request queue for some multiple of the idle timeout.

Unfortunately, setting MaxRequestsQueuedPerDestination to zero prevents
initiating any requests whatsoever. Setting it to the same value as
MaxConnectionsPerDestination allows that many requests to be initiated,
but once those requests are assigned to and consume connections a second
batch of that many requests can be initiated and will queue waiting for
the first batch to complete.

Any suggestions? If this required a new feature, what would the feature
look like?

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Preventing queuing of requests when all connections exhausted

Simone Bordet-2
Hi,

On Thu, Dec 11, 2014 at 11:19 PM, John Gardiner Myers
<[hidden email]> wrote:

> I want to configure Jetty client so that if all MaxConnectionsPerDestination
> are in use then additional requests to that destination immediately fail
> rather than queue. If a destination is hung, better to immediately fail over
> to another instance of the service than sit in the request queue for some
> multiple of the idle timeout.
>
> Unfortunately, setting MaxRequestsQueuedPerDestination to zero prevents
> initiating any requests whatsoever. Setting it to the same value as
> MaxConnectionsPerDestination allows that many requests to be initiated, but
> once those requests are assigned to and consume connections a second batch
> of that many requests can be initiated and will queue waiting for the first
> batch to complete.
>
> Any suggestions? If this required a new feature, what would the feature look
> like?

This would require a refactoring of the code to reject the offer() to
the queue in case you have all connections active.
Can you please file an issue about this ?

If it is possible to generalize this feature also to a multiplexed
connection, I think it would be a boolean in HttpClient.
Otherwise you will have a method to override to return true/false
based on your conditions.

--
Simone Bordet
----
http://cometd.org
http://webtide.com
http://intalio.com
Developer advice, training, services and support
from the Jetty & CometD experts.
Intalio, the modern way to build business applications.
_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Preventing queuing of requests when all connections exhausted

John Gardiner Myers
On 1/31/15 9:38 AM, Simone Bordet wrote:
> This would require a refactoring of the code to reject the offer() to
> the queue in case you have all connections active.
> Can you please file an issue about this ?
Will do.
> If it is possible to generalize this feature also to a multiplexed
> connection, I think it would be a boolean in HttpClient.
> Otherwise you will have a method to override to return true/false
> based on your conditions.
>
My idea for an interface would be to exempt unused/idle
MaxConnectionsPerDestination from the MaxRequestsQueuedPerDestination
limit. Then I could set MaxRequestsQueuedPerDestination to zero. I can't
think of a use case that would require the current method of accounting.

I don't see how multiplexed connections affects it. Doesn't
MaxConnectionsPerDestination apply to virtual connections?

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Preventing queuing of requests when all connections exhausted

Simone Bordet-2
Hi,

On Sat, Jan 31, 2015 at 9:04 PM, John Gardiner Myers
<[hidden email]> wrote:

> On 1/31/15 9:38 AM, Simone Bordet wrote:
>>
>> This would require a refactoring of the code to reject the offer() to
>> the queue in case you have all connections active.
>> Can you please file an issue about this ?
>
> Will do.
>>
>> If it is possible to generalize this feature also to a multiplexed
>> connection, I think it would be a boolean in HttpClient.
>> Otherwise you will have a method to override to return true/false
>> based on your conditions.
>>
> My idea for an interface would be to exempt unused/idle
> MaxConnectionsPerDestination from the MaxRequestsQueuedPerDestination limit.

Now I am confused. You said your case was for when all connections are in use.
Why you want to exempt idle connections from a queue limit (which,
incidentally, is not even there) ?

> Then I could set MaxRequestsQueuedPerDestination to zero. I can't think of a
> use case that would require the current method of accounting.
>
> I don't see how multiplexed connections affects it. Doesn't
> MaxConnectionsPerDestination apply to virtual connections?

There are no virtual connections, whatever you mean.
There is one physical connection, so MaxConnectionsPerDestination is
implicitly overridden to 1.

Let's figure out what you exactly want to do before filing the issue.
I evidently did not understand.

--
Simone Bordet
----
http://cometd.org
http://webtide.com
http://intalio.com
Developer advice, training, services and support
from the Jetty & CometD experts.
Intalio, the modern way to build business applications.
_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Preventing queuing of requests when all connections exhausted

John Gardiner Myers
On 1/31/15 3:16 PM, Simone Bordet wrote:
>
>> My idea for an interface would be to exempt unused/idle
>> MaxConnectionsPerDestination from the MaxRequestsQueuedPerDestination limit.
> Now I am confused. You said your case was for when all connections are in use.
> Why you want to exempt idle connections from a queue limit (which,
> incidentally, is not even there) ?
In my case, I do not want any requests to sit in
HttpDestination.exchanges waiting for another, pending request to
complete. If a request would have to wait for another request to
complete before being started, then it should immediately fail,
presumably with a RejectedExecutionException.

An idle connection will soon pick up the request, as will a connection
that does not exist but can be opened promptly.

My proposal is to define HttpClient.maxRequestsQueuedPerDestination as
the limit of requests per destination that are permitted to wait for
another request to complete.

>> Then I could set MaxRequestsQueuedPerDestination to zero. I can't think of a
>> use case that would require the current method of accounting.
>>
>> I don't see how multiplexed connections affects it. Doesn't
>> MaxConnectionsPerDestination apply to virtual connections?
> There are no virtual connections, whatever you mean.
> There is one physical connection, so MaxConnectionsPerDestination is
> implicitly overridden to 1.
By "connection" I mean a unit of request concurrency. If five requests
are simultaneously being communicated with a destination, then there are
five connections to that destination. It does not matter if the
underlying implementation uses five TCP connections or multiplexes them
over one TCP connection.

I would expect that with the multiplexing protocols such as SPDY
HttpClient.maxConnectionsPerDestination would limit the number of
requests that would simultaneously be sent multiplexed over the TCP
connection. One would not want to initiate an unbound number of
simultaneous requests.

But if one did, or if one wanted to apply a higher concurrency limit to
multiplexed connections, then my proposal would still work--it would
just mean the code would have to re-apply the
MaxRequestsQueuedPerDestination limit when it finds out a destination
doesn't support a multiplexed protocol.

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Preventing queuing of requests when all connections exhausted

Simone Bordet-2
Hi,

On Sun, Feb 1, 2015 at 6:35 AM, John Gardiner Myers
<[hidden email]> wrote:

> On 1/31/15 3:16 PM, Simone Bordet wrote:
>>
>>
>>> My idea for an interface would be to exempt unused/idle
>>> MaxConnectionsPerDestination from the MaxRequestsQueuedPerDestination
>>> limit.
>>
>> Now I am confused. You said your case was for when all connections are in
>> use.
>> Why you want to exempt idle connections from a queue limit (which,
>> incidentally, is not even there) ?
>
> In my case, I do not want any requests to sit in HttpDestination.exchanges
> waiting for another, pending request to complete. If a request would have to
> wait for another request to complete before being started, then it should
> immediately fail, presumably with a RejectedExecutionException.
>
> An idle connection will soon pick up the request, as will a connection that
> does not exist but can be opened promptly.
>
> My proposal is to define HttpClient.maxRequestsQueuedPerDestination as the
> limit of requests per destination that are permitted to wait for another
> request to complete.

Let me rephrase it to see if I understand.

Let cx = maxConnectionsPerDestination, qx = maxRequestsQueuedPerDestination.

You propose to change the BlockingQueue capacity from qx to qx+cx.

This is needed in case a flow of requests arrives and establishing the
first connection is slow.
With qx=0 and cx=8, for example, I need to be able to queue at least 8
exchanges, and fail the 9th.
With qx=0 and cx=1, I basically only allow serialized requests.

Since now the queue capacity is larger than qx I cannot rely on the
queue to tell me if I have exceeded the queuing capacity.
I must now implement the limitation manually, along with the actual
connectionCount from the ConnectionPool.
That is, an exchange is allowed to be queued, if the already queued
exchanges are less than qx + cx - connectionCount

This may lead to spurious queuing or or spurious failures, as
connectionCount varies concurrently.
I'm not keen adding a synchronization point to handle the enqueuing precision.

For multiplexed destinations, the logic is the same, with the
connectionCount replaced by the inflight exchanges.

I may refactor the code to allow you to do all of the above, if the
spuriousness is something you can tolerate.
If so, please file an issue.

Other thoughts ?

--
Simone Bordet
----
http://cometd.org
http://webtide.com
http://intalio.com
Developer advice, training, services and support
from the Jetty & CometD experts.
Intalio, the modern way to build business applications.
_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Preventing queuing of requests when all connections exhausted

John Gardiner Myers
On 2/3/2015 1:16 AM, Simone Bordet wrote:
> Let me rephrase it to see if I understand.
>
> Let cx = maxConnectionsPerDestination, qx = maxRequestsQueuedPerDestination.
>
> You propose to change the BlockingQueue capacity from qx to qx+cx.
>
> This is needed in case a flow of requests arrives and establishing the
> first connection is slow.
This is needed in case the destination is persistently slow (or hung).
The delay could be in connection establishment, data transfer, or
request processing.
> With qx=0 and cx=8, for example, I need to be able to queue at least 8
> exchanges, and fail the 9th.
> With qx=0 and cx=1, I basically only allow serialized requests.
Correct.

I only care about the case of qx=0. I proposed an API that makes sense
for other values of qx, but can tolerate an API where specifying qx=0
(or specifying qx=cx and enabling some other flag) invokes a special mode.
> Since now the queue capacity is larger than qx I cannot rely on the
> queue to tell me if I have exceeded the queuing capacity.
> I must now implement the limitation manually, along with the actual
> connectionCount from the ConnectionPool.
> That is, an exchange is allowed to be queued, if the already queued
> exchanges are less than qx + cx - connectionCount
I can tolerate the failure being delayed by the time it takes to
dispatch the request to a worker thread. Milliseconds, not seconds or hours.
> This may lead to spurious queuing or or spurious failures, as
> connectionCount varies concurrently.
> I'm not keen adding a synchronization point to handle the enqueuing precision.
I believe I can tolerate spurious failures as long as the time window is
reasonably short. Spurious queuing when it results in a request waiting
twice the connection idle timeout much less so.
> For multiplexed destinations, the logic is the same, with the
> connectionCount replaced by the inflight exchanges.
>
> I may refactor the code to allow you to do all of the above, if the
> spuriousness is something you can tolerate.
> If so, please file an issue.
>
> Other thoughts ?
>

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Preventing queuing of requests when all connections exhausted

Simone Bordet-2
Hi,

On Sat, Feb 7, 2015 at 12:13 AM, John Gardiner Myers
<[hidden email]> wrote:
>> I may refactor the code to allow you to do all of the above, if the
>> spuriousness is something you can tolerate.
>> If so, please file an issue.

I pushed a couple of changes that I think would allow you to obtain
the behavior your want.
Override HttpDestination.enqueue() and HttpDestination.newExchangeQueue().

Mind to try and report back if you can achieve what you want ?

--
Simone Bordet
----
http://cometd.org
http://webtide.com
http://intalio.com
Developer advice, training, services and support
from the Jetty & CometD experts.
Intalio, the modern way to build business applications.
_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Preventing queuing of requests when all connections exhausted

John Gardiner Myers
On 2/6/2015 3:33 PM, Simone Bordet wrote:
> I pushed a couple of changes that I think would allow you to obtain
> the behavior your want. Override HttpDestination.enqueue() and
> HttpDestination.newExchangeQueue(). Mind to try and report back if you
> can achieve what you want ?

Let me see if I've figured out what you're thinking:

I'd override HttpDestination.newExchangeQueue() to return an object that
keeps an outstanding request count. That returned object would override
Queue.offer() to fail if the count is too high, increment the count, and
decorate the HttpExchange so that HttpExchange.getResponseListeners()
returns an extra completion listener to decrement the count.

I don't see why it would be necessary to override HttpDestination.enqueue().

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Preventing queuing of requests when all connections exhausted

Simone Bordet-2
Hi,

On Tue, Feb 10, 2015 at 11:49 PM, John Gardiner Myers
<[hidden email]> wrote:

> On 2/6/2015 3:33 PM, Simone Bordet wrote:
>>
>> I pushed a couple of changes that I think would allow you to obtain the
>> behavior your want. Override HttpDestination.enqueue() and
>> HttpDestination.newExchangeQueue(). Mind to try and report back if you can
>> achieve what you want ?
>
>
> Let me see if I've figured out what you're thinking:
>
> I'd override HttpDestination.newExchangeQueue() to return an object that
> keeps an outstanding request count. That returned object would override
> Queue.offer() to fail if the count is too high, increment the count, and
> decorate the HttpExchange so that HttpExchange.getResponseListeners()
> returns an extra completion listener to decrement the count.

Creative ! :)
But no.

> I don't see why it would be necessary to override HttpDestination.enqueue().

You override newExchangeQueue() to provide a Queue with additional room.
Then you override enqueue() to return true/false depending on whether
you have enqueued or you want to reject enqueuing.

Overriding enqueue() only won't be enough for your case, since the
queue created by default is bounded at qx, while you need room for at
least qx+cx.

--
Simone Bordet
----
http://cometd.org
http://webtide.com
http://intalio.com
Developer advice, training, services and support
from the Jetty & CometD experts.
Intalio, the modern way to build business applications.
_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Preventing queuing of requests when all connections exhausted

John Gardiner Myers
On 2/10/2015 3:16 PM, Simone Bordet wrote:
> You override newExchangeQueue() to provide a Queue with additional
> room. Then you override enqueue() to return true/false depending on
> whether you have enqueued or you want to reject enqueuing.
The part I haven't figured out is how to tell how much of cx is
currently in use, so I can know when to reject enqueuing. If not by
splicing in some sort of listener into the HttpExchange, that is.

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Preventing queuing of requests when all connections exhausted

Simone Bordet-2
Hi,

On Wed, Feb 11, 2015 at 12:33 AM, John Gardiner Myers
<[hidden email]> wrote:
> On 2/10/2015 3:16 PM, Simone Bordet wrote:
>>
>> You override newExchangeQueue() to provide a Queue with additional room.
>> Then you override enqueue() to return true/false depending on whether you
>> have enqueued or you want to reject enqueuing.
>
> The part I haven't figured out is how to tell how much of cx is currently in
> use, so I can know when to reject enqueuing. If not by splicing in some sort
> of listener into the HttpExchange, that is.

You are subclassing HttpDestinationOverHTTP already so:

@Override
boolean enqueue(...)
{
    int cx = getConnectionPool().getConnectionCount().
    ...
}

--
Simone Bordet
----
http://cometd.org
http://webtide.com
http://intalio.com
Developer advice, training, services and support
from the Jetty & CometD experts.
Intalio, the modern way to build business applications.
_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Preventing queuing of requests when all connections exhausted

John Gardiner Myers
In reply to this post by Simone Bordet-2
On 2/6/2015 3:33 PM, Simone Bordet wrote:
> Mind to try and report back if you can achieve what you want ?
I finally got some time to try this out and believe I can. If you're
interested, look at
https://github.com/johngmyers/platform/commit/bc3341a5885c99fc800b8567b09592e08fbee569
_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users