[jetty-users] Jetty 7.4.2 to 7.6.1 upgrade: Losing final packets in large responses

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

[jetty-users] Jetty 7.4.2 to 7.6.1 upgrade: Losing final packets in large responses

Michael Henderson-2
I've been working on a REST solution that makes use of an embedded
Jetty instance (specifically Restlet). Currently, the framework's
built in extension for Jetty support only up to version 7.4.2 due to
some class name and package changes (HttpConnection ->
AbstractHttpConnection, the change of the preferred SslContextFactory
to being that in org.eclipse.jetty.util.ssl rather than
org.eclipse.jetty.http.ssl).

Therefore, I have been looking to create a connector based off the
existing that I can keep up to date with newest Jetty versions.

I encountered little trouble until I began wider testing, and found
that with either Jetty 7.6.1 or Jetty 8.1.1 in use, certain requests
whose responses were large enough to exceed a single packet/buffer
size, would fail to ever receive the final response packet. The
responses would be cut off at the same point each time in these larger
responses. (to the exact same byte each time)

To further confuse this issue, the cut off would occur differently on
different machines, and for a few machines it would not occur
whatsoever. It does not appear to be browser or operating system
dependent (though they only ones I test with at this early stage are
IE8 and Google Chrome, and Windows 7 varieties).

I am trying to determine what could have changed in the response
behavior of Jetty between versions 7.4 and 7.6 that could cause such a
failure. The connection appears to simply idle out rather than
providing the final packet needed. The behavior for the REST
application interacting with the Jetty instance has not changed
outside of the renaming mentioned above, so I expect there is
something new that needs to be done to properly provide a response.

Any insight that could be provided would be appreciated, as my
knowledge of Jetty is quite limited.
_______________________________________________
jetty-users mailing list
[hidden email]
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: [jetty-users] Jetty 7.4.2 to 7.6.1 upgrade: Losing final packets in large responses

Joakim Erdfelt-9
Can you tell us a bit about the conditions of this?
We'd like to setup a unit test to attempt to replicate it.

What kind of client?
What is the size of the content?
Is the response gzip'd?
Is this large content part of a HTTP/1.1 pipeline request chain?
What speed is the network connection? (such as wired 1gbit / wifi g / 4g / 3g / edge / 14.4k modem, etc...)

--
Joakim Erdfelt

(the people behind jetty and cometd)



On Thu, Feb 23, 2012 at 8:02 PM, Michael Henderson <[hidden email]> wrote:
I've been working on a REST solution that makes use of an embedded
Jetty instance (specifically Restlet). Currently, the framework's
built in extension for Jetty support only up to version 7.4.2 due to
some class name and package changes (HttpConnection ->
AbstractHttpConnection, the change of the preferred SslContextFactory
to being that in org.eclipse.jetty.util.ssl rather than
org.eclipse.jetty.http.ssl).

Therefore, I have been looking to create a connector based off the
existing that I can keep up to date with newest Jetty versions.

I encountered little trouble until I began wider testing, and found
that with either Jetty 7.6.1 or Jetty 8.1.1 in use, certain requests
whose responses were large enough to exceed a single packet/buffer
size, would fail to ever receive the final response packet. The
responses would be cut off at the same point each time in these larger
responses. (to the exact same byte each time)

To further confuse this issue, the cut off would occur differently on
different machines, and for a few machines it would not occur
whatsoever. It does not appear to be browser or operating system
dependent (though they only ones I test with at this early stage are
IE8 and Google Chrome, and Windows 7 varieties).

I am trying to determine what could have changed in the response
behavior of Jetty between versions 7.4 and 7.6 that could cause such a
failure. The connection appears to simply idle out rather than
providing the final packet needed. The behavior for the REST
application interacting with the Jetty instance has not changed
outside of the renaming mentioned above, so I expect there is
something new that needs to be done to properly provide a response.

Any insight that could be provided would be appreciated, as my
knowledge of Jetty is quite limited.
_______________________________________________
jetty-users mailing list
[hidden email]
https://dev.eclipse.org/mailman/listinfo/jetty-users


_______________________________________________
jetty-users mailing list
[hidden email]
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: [jetty-users] Jetty 7.4.2 to 7.6.1 upgrade: Losing final packets in large responses

Michael Henderson-2
Clients:

Internet Explorer 8, Google Chrome 17.0.963, Silverlight 4.0 in ClientHttp mode

Rough content information:

HTTP/1.1 200 OK
Accept-Ranges: bytes
Vary: Accept-Charset, Accept-Encoding, Accept-Language, Accept
Content-Length: 30636
Content-Type: text/plain; charset=UTF-8

In this case, data received before stopping was around 16421. (looking
only at body)

HTTP/1.1 200 OK
Accept-Ranges: bytes
Vary: Accept-Charset, Accept-Encoding, Accept-Language, Accept
Content-Length: 191395
Content-Type: text/plain; charset=UTF-8

In this case, rough data received before stopping was around 183019.
(looking only at body)

The response is raw XML data. It is not compressed in any way.

It is a single REST style HTTP/1.1 GET call, though a favicon download
occurs following making this call through internet explorer or chrome
and Silverlight performs a preceding download of XML files for client
access policy and cross domain policy.

Multiple tests were performed between different sets of machines, with
connections between the two including:
22/5 Cable connection (with VPN overhead)
12/3 Cable connection (with VPN overhead)
Loopback (performed GET from the same server hosting the running Jetty)

Tomorrow I will have the chance to perform some tests over a 10/100
LAN in place of the VPN.

On Thu, Feb 23, 2012 at 23:02, Joakim Erdfelt <[hidden email]> wrote:

> Can you tell us a bit about the conditions of this?
> We'd like to setup a unit test to attempt to replicate it.
>
> What kind of client?
> What is the size of the content?
> Is the response gzip'd?
> Is this large content part of a HTTP/1.1 pipeline request chain?
> What speed is the network connection? (such as wired 1gbit / wifi g / 4g /
> 3g / edge / 14.4k modem, etc...)
>
> --
> Joakim Erdfelt
> [hidden email]
>
> http://webtide.com | http://intalio.com
> (the people behind jetty and cometd)
>
>
>
> On Thu, Feb 23, 2012 at 8:02 PM, Michael Henderson <[hidden email]>
> wrote:
>>
>> I've been working on a REST solution that makes use of an embedded
>> Jetty instance (specifically Restlet). Currently, the framework's
>> built in extension for Jetty support only up to version 7.4.2 due to
>> some class name and package changes (HttpConnection ->
>> AbstractHttpConnection, the change of the preferred SslContextFactory
>> to being that in org.eclipse.jetty.util.ssl rather than
>> org.eclipse.jetty.http.ssl).
>>
>> Therefore, I have been looking to create a connector based off the
>> existing that I can keep up to date with newest Jetty versions.
>>
>> I encountered little trouble until I began wider testing, and found
>> that with either Jetty 7.6.1 or Jetty 8.1.1 in use, certain requests
>> whose responses were large enough to exceed a single packet/buffer
>> size, would fail to ever receive the final response packet. The
>> responses would be cut off at the same point each time in these larger
>> responses. (to the exact same byte each time)
>>
>> To further confuse this issue, the cut off would occur differently on
>> different machines, and for a few machines it would not occur
>> whatsoever. It does not appear to be browser or operating system
>> dependent (though they only ones I test with at this early stage are
>> IE8 and Google Chrome, and Windows 7 varieties).
>>
>> I am trying to determine what could have changed in the response
>> behavior of Jetty between versions 7.4 and 7.6 that could cause such a
>> failure. The connection appears to simply idle out rather than
>> providing the final packet needed. The behavior for the REST
>> application interacting with the Jetty instance has not changed
>> outside of the renaming mentioned above, so I expect there is
>> something new that needs to be done to properly provide a response.
>>
>> Any insight that could be provided would be appreciated, as my
>> knowledge of Jetty is quite limited.
>> _______________________________________________
>> jetty-users mailing list
>> [hidden email]
>> https://dev.eclipse.org/mailman/listinfo/jetty-users
>
>
>
> _______________________________________________
> jetty-users mailing list
> [hidden email]
> https://dev.eclipse.org/mailman/listinfo/jetty-users
>
_______________________________________________
jetty-users mailing list
[hidden email]
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: [jetty-users] Jetty 7.4.2 to 7.6.1 upgrade: Losing final packets in large responses

Simone Bordet-2
Michael,

On Fri, Feb 24, 2012 at 05:31, Michael Henderson <[hidden email]> wrote:

> Clients:
>
> Internet Explorer 8, Google Chrome 17.0.963, Silverlight 4.0 in ClientHttp mode
>
> Rough content information:
>
> HTTP/1.1 200 OK
> Accept-Ranges: bytes
> Vary: Accept-Charset, Accept-Encoding, Accept-Language, Accept
> Content-Length: 30636
> Content-Type: text/plain; charset=UTF-8
>
> In this case, data received before stopping was around 16421. (looking
> only at body)

We have fixed an issue that may hit you also: see
https://bugs.eclipse.org/bugs/show_bug.cgi?id=372457.

We have committed a fix and deployed Jetty 7.6.2-SNAPSHOT here:
https://oss.sonatype.org/content/repositories/jetty-snapshots/org/eclipse/jetty/jetty-distribution/7.6.2-SNAPSHOT/

Can you please try it out and tell us if your problem is solved ?

Simon
--
http://cometd.org
http://intalio.com
http://bordet.blogspot.com
----
Finally, no matter how good the architecture and design are,
to deliver bug-free software with optimal performance and reliability,
the implementation technique must be flawless.   Victoria Livschitz
_______________________________________________
jetty-users mailing list
[hidden email]
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: [jetty-users] Jetty 7.4.2 to 7.6.1 upgrade: Losing final packets in large responses

Michael Henderson-2
Simone,

I have attempted to run the application with 7.6.2-20120224.113908
Snapshot and the results have not changed.

Though this did seem to remove another issue I was encountering where
the first request would fail each time I brought the application up.


On Fri, Feb 24, 2012 at 07:58, Simone Bordet <[hidden email]> wrote:

> Michael,
>
> On Fri, Feb 24, 2012 at 05:31, Michael Henderson <[hidden email]> wrote:
>> Clients:
>>
>> Internet Explorer 8, Google Chrome 17.0.963, Silverlight 4.0 in ClientHttp mode
>>
>> Rough content information:
>>
>> HTTP/1.1 200 OK
>> Accept-Ranges: bytes
>> Vary: Accept-Charset, Accept-Encoding, Accept-Language, Accept
>> Content-Length: 30636
>> Content-Type: text/plain; charset=UTF-8
>>
>> In this case, data received before stopping was around 16421. (looking
>> only at body)
>
> We have fixed an issue that may hit you also: see
> https://bugs.eclipse.org/bugs/show_bug.cgi?id=372457.
>
> We have committed a fix and deployed Jetty 7.6.2-SNAPSHOT here:
> https://oss.sonatype.org/content/repositories/jetty-snapshots/org/eclipse/jetty/jetty-distribution/7.6.2-SNAPSHOT/
>
> Can you please try it out and tell us if your problem is solved ?
>
> Simon
> --
> http://cometd.org
> http://intalio.com
> http://bordet.blogspot.com
> ----
> Finally, no matter how good the architecture and design are,
> to deliver bug-free software with optimal performance and reliability,
> the implementation technique must be flawless.   Victoria Livschitz
> _______________________________________________
> jetty-users mailing list
> [hidden email]
> https://dev.eclipse.org/mailman/listinfo/jetty-users
_______________________________________________
jetty-users mailing list
[hidden email]
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: [jetty-users] Jetty 7.4.2 to 7.6.1 upgrade: Losing final packets in large responses

Simone Bordet-2
Hi,

On Fri, Feb 24, 2012 at 14:42, Michael Henderson <[hidden email]> wrote:
> Simone,
>
> I have attempted to run the application with 7.6.2-20120224.113908
> Snapshot and the results have not changed.

Do you have (or can you build) a reproducible test case that we can try ?

Simon
--
http://cometd.org
http://intalio.com
http://bordet.blogspot.com
----
Finally, no matter how good the architecture and design are,
to deliver bug-free software with optimal performance and reliability,
the implementation technique must be flawless.   Victoria Livschitz
_______________________________________________
jetty-users mailing list
[hidden email]
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: [jetty-users] Jetty 7.4.2 to 7.6.1 upgrade: Losing final packets in large responses

Michael Henderson-2
I do not currently have a test separated from the overall application.
I will attempt to create one today.


On Fri, Feb 24, 2012 at 08:52, Simone Bordet <[hidden email]> wrote:

> Hi,
>
> On Fri, Feb 24, 2012 at 14:42, Michael Henderson <[hidden email]> wrote:
>> Simone,
>>
>> I have attempted to run the application with 7.6.2-20120224.113908
>> Snapshot and the results have not changed.
>
> Do you have (or can you build) a reproducible test case that we can try ?
>
> Simon
> --
> http://cometd.org
> http://intalio.com
> http://bordet.blogspot.com
> ----
> Finally, no matter how good the architecture and design are,
> to deliver bug-free software with optimal performance and reliability,
> the implementation technique must be flawless.   Victoria Livschitz
> _______________________________________________
> jetty-users mailing list
> [hidden email]
> https://dev.eclipse.org/mailman/listinfo/jetty-users
_______________________________________________
jetty-users mailing list
[hidden email]
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: [jetty-users] Jetty 7.4.2 to 7.6.1 upgrade: Losing final packets in large responses

Michael Henderson-2
In building an independent test, I have found that the issue
originated with the interaction between Restlet and Jetty and was not
a Jetty issue in itself. I have now resolved the issue without any
necessary changes to the release version of Jetty 7.6.1.

Thank you all for your help!


On Fri, Feb 24, 2012 at 08:56, Michael Henderson <[hidden email]> wrote:

> I do not currently have a test separated from the overall application.
> I will attempt to create one today.
>
>
> On Fri, Feb 24, 2012 at 08:52, Simone Bordet <[hidden email]> wrote:
>> Hi,
>>
>> On Fri, Feb 24, 2012 at 14:42, Michael Henderson <[hidden email]> wrote:
>>> Simone,
>>>
>>> I have attempted to run the application with 7.6.2-20120224.113908
>>> Snapshot and the results have not changed.
>>
>> Do you have (or can you build) a reproducible test case that we can try ?
>>
>> Simon
>> --
>> http://cometd.org
>> http://intalio.com
>> http://bordet.blogspot.com
>> ----
>> Finally, no matter how good the architecture and design are,
>> to deliver bug-free software with optimal performance and reliability,
>> the implementation technique must be flawless.   Victoria Livschitz
>> _______________________________________________
>> jetty-users mailing list
>> [hidden email]
>> https://dev.eclipse.org/mailman/listinfo/jetty-users
_______________________________________________
jetty-users mailing list
[hidden email]
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: [jetty-users] Jetty 7.4.2 to 7.6.1 upgrade: Losing final packets in large responses

Joakim Erdfelt-9
Cool.
Can you explain what the resolution was?
It might help others in a similar situation.

--
Joakim Erdfelt

(the people behind jetty and cometd)



On Fri, Feb 24, 2012 at 9:02 AM, Michael Henderson <[hidden email]> wrote:
In building an independent test, I have found that the issue
originated with the interaction between Restlet and Jetty and was not
a Jetty issue in itself. I have now resolved the issue without any
necessary changes to the release version of Jetty 7.6.1.

Thank you all for your help!


On Fri, Feb 24, 2012 at 08:56, Michael Henderson <[hidden email]> wrote:
> I do not currently have a test separated from the overall application.
> I will attempt to create one today.
>
>
> On Fri, Feb 24, 2012 at 08:52, Simone Bordet <[hidden email]> wrote:
>> Hi,
>>
>> On Fri, Feb 24, 2012 at 14:42, Michael Henderson <[hidden email]> wrote:
>>> Simone,
>>>
>>> I have attempted to run the application with 7.6.2-20120224.113908
>>> Snapshot and the results have not changed.
>>
>> Do you have (or can you build) a reproducible test case that we can try ?
>>
>> Simon
>> --
>> http://cometd.org
>> http://intalio.com
>> http://bordet.blogspot.com
>> ----
>> Finally, no matter how good the architecture and design are,
>> to deliver bug-free software with optimal performance and reliability,
>> the implementation technique must be flawless.   Victoria Livschitz
>> _______________________________________________
>> jetty-users mailing list
>> [hidden email]
>> https://dev.eclipse.org/mailman/listinfo/jetty-users
_______________________________________________
jetty-users mailing list
[hidden email]
https://dev.eclipse.org/mailman/listinfo/jetty-users


_______________________________________________
jetty-users mailing list
[hidden email]
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: [jetty-users] Jetty 7.4.2 to 7.6.1 upgrade: Losing final packets in large responses

Michael Henderson-2
It is actually quite ridiculous.

Configuration of Restlet involves the including any number of
"Connection Helpers" for Server setup. These can include Jetty,
Grizzly, Simple, and others, which allow you to embed your hosting
server directly in the application rather than configuring Restlet as
a servlet/war and simply dropping it in on a running server.

Apparently, the "Simple" server helpers are always available as
defaults, and for some reason the framework was choosing to make use
of them rather than the Jetty helpers when I switched the app to
HTTPS.

To resolve the issue, I was forced to manually wipe out all of the
registered server helpers on startup then insert Jetty's exclusively.



On Fri, Feb 24, 2012 at 11:12, Joakim Erdfelt <[hidden email]> wrote:

> Cool.
> Can you explain what the resolution was?
> It might help others in a similar situation.
>
> --
> Joakim Erdfelt
> [hidden email]
>
> http://webtide.com | http://intalio.com
> (the people behind jetty and cometd)
>
>
>
> On Fri, Feb 24, 2012 at 9:02 AM, Michael Henderson <[hidden email]>
> wrote:
>>
>> In building an independent test, I have found that the issue
>> originated with the interaction between Restlet and Jetty and was not
>> a Jetty issue in itself. I have now resolved the issue without any
>> necessary changes to the release version of Jetty 7.6.1.
>>
>> Thank you all for your help!
>>
>>
>> On Fri, Feb 24, 2012 at 08:56, Michael Henderson <[hidden email]>
>> wrote:
>> > I do not currently have a test separated from the overall application.
>> > I will attempt to create one today.
>> >
>> >
>> > On Fri, Feb 24, 2012 at 08:52, Simone Bordet <[hidden email]>
>> > wrote:
>> >> Hi,
>> >>
>> >> On Fri, Feb 24, 2012 at 14:42, Michael Henderson <[hidden email]>
>> >> wrote:
>> >>> Simone,
>> >>>
>> >>> I have attempted to run the application with 7.6.2-20120224.113908
>> >>> Snapshot and the results have not changed.
>> >>
>> >> Do you have (or can you build) a reproducible test case that we can try
>> >> ?
>> >>
>> >> Simon
>> >> --
>> >> http://cometd.org
>> >> http://intalio.com
>> >> http://bordet.blogspot.com
>> >> ----
>> >> Finally, no matter how good the architecture and design are,
>> >> to deliver bug-free software with optimal performance and reliability,
>> >> the implementation technique must be flawless.   Victoria Livschitz
>> >> _______________________________________________
>> >> jetty-users mailing list
>> >> [hidden email]
>> >> https://dev.eclipse.org/mailman/listinfo/jetty-users
>> _______________________________________________
>> jetty-users mailing list
>> [hidden email]
>> https://dev.eclipse.org/mailman/listinfo/jetty-users
>
>
>
> _______________________________________________
> jetty-users mailing list
> [hidden email]
> https://dev.eclipse.org/mailman/listinfo/jetty-users
>
_______________________________________________
jetty-users mailing list
[hidden email]
https://dev.eclipse.org/mailman/listinfo/jetty-users