"early EOF" warning when https call from old application in 9.2.9.v2015022er

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

"early EOF" warning when https call from old application in 9.2.9.v2015022er

lamphus
Hello,

Am running an older application with dependencies on jetty 8.1.16v20140903 (jars: jetty-client, jetty-continuation, jetty-http, jetty-io, jetty-security, jetty-server, jetty-servlet, jetty-util, jetty-xml). It works with 8.1.21.v20160908 jars too. It runs inside a jetty 9.2.9.v20150224 without issues using only http.

When using only https however the application with the jetty 8 deps tries to httprequest another application in the same container (jetty9) and the issue reported is :

     [java] 11031 [HttpClient-31] WARN  org.eclipse.jetty.client.HttpExchange  -
 EXCEPTION adapter1@34ff2b7c=GET//domain:443/application2/rest#WAITING
(1ms)->EXCEPTED(1ms)sent=4ms
     [java] org.eclipse.jetty.io.EofException: early EOF
     [java]     at org.eclipse.jetty.client.AbstractHttpConnection$Handler.early
EOF(AbstractHttpConnection.java:352)
     [java]     at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:3
23)
     [java]     at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.j
ava:235)
     [java]     at org.eclipse.jetty.client.AsyncHttpConnection.handle(AsyncHttp
Connection.java:135)
     [java]     at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.j
ava:196)
     [java]     at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectC
hannelEndPoint.java:696)
     [java]     at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectCh
annelEndPoint.java:53)
     [java]     at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedT
hreadPool.java:608)
     [java]     at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedTh
readPool.java:543)
     [java]     at java.lang.Thread.run(Thread.java:745)

and it yields a functional issue.

I've seen something close to it here: https://bugs.eclipse.org/bugs/show_bug.cgi?id=376273 but that issue should have been vanquished in 8.10+ .

The older application does not start with 9.2.9.v20150224 jars.

Am not in a position to change the older applications dependencies and am obviously a newbie with jetty, is there reason to believe the application needs to be updated with newer dependencies to work well in jetty9 or that this error would not occur if all applications were living inside a jetty 8 (using https)?


Thanks for any hints.
Reply | Threaded
Open this post in threaded view
|

Re: "early EOF" warning when https call from old application in 9.2.9.v2015022er

Greg Wilkins
lamphus,

you have a situation that is rather: a) special; b) complex; c) dated

It it going to be hard to help via open source channels, but we will attempt.

You have an application using a jetty-8.1.16 client to talk to a jetty-9.2.9 server and logs an EarlyEoF exception on the server and a "functional issue" in the application.

EarlyEOF is just an indication that the server was parsing a request and the EOF arrived. Typically this is because the client gave up - either it timed out or some other problem happened.

Does this happen on all requests, or just some requests.

Do you know the time the request was issued vs the time the "functional issue" occurs and the earlyEOF logged? Is there a delay.

Can you run the client stand alone talking to the server rather than talking to itself?

regards


--

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: "early EOF" warning when https call from old application in 9.2.9.v2015022er

lamphus
Hi, yep, this was understood, and I knew I ran the risk of 0 replies, so I'm glad for yours : )

Thanks for the info regarding EOF.

This seems to happen for a specific request. The application instantiates a http://www.eclipse.org/jetty/documentation/current/http-client.html and issues a get request with an url argument that goes "https://domain.topdomain/path" but what is sent according to the jetty log appears to be "http://domain.topdomain:443" (I think- log says "GET//domain.topdomain:443/geoserver/rest" and the way our jetty9 is configured one can't reach it like this. It has to be http://domain.topdomain/path (for a 302) or https://domain.topdomain/path or https://domain.topdomain:443 . Am currently trying to instantiate an SSLContextFactory for the httpclient constructor but am so far not seeing an effect. It's obviously less "I know what I'm doing" and more "poke this and see if that part moves" but I am picking up a few things..



On Fri, Jan 27, 2017 at 5:25 AM, Greg Wilkins <[hidden email]> wrote:
lamphus,

you have a situation that is rather: a) special; b) complex; c) dated

It it going to be hard to help via open source channels, but we will attempt.

You have an application using a jetty-8.1.16 client to talk to a jetty-9.2.9 server and logs an EarlyEoF exception on the server and a "functional issue" in the application.

EarlyEOF is just an indication that the server was parsing a request and the EOF arrived. Typically this is because the client gave up - either it timed out or some other problem happened.

Does this happen on all requests, or just some requests.

Do you know the time the request was issued vs the time the "functional issue" occurs and the earlyEOF logged? Is there a delay.

Can you run the client stand alone talking to the server rather than talking to itself?

regards


--

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users


_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: "early EOF" warning when https call from old application in 9.2.9.v2015022er

Simone Bordet-3
Hi,

On Mon, Jan 30, 2017 at 6:45 PM, David Persson <[hidden email]> wrote:

> Hi, yep, this was understood, and I knew I ran the risk of 0 replies, so I'm
> glad for yours : )
>
> Thanks for the info regarding EOF.
>
> This seems to happen for a specific request. The application instantiates a
> http://www.eclipse.org/jetty/documentation/current/http-client.html and
> issues a get request with an url argument that goes
> "https://domain.topdomain/path" but what is sent according to the jetty log
> appears to be "http://domain.topdomain:443" (I think- log says
> "GET//domain.topdomain:443/geoserver/rest" and the way our jetty9 is
> configured one can't reach it like this.

Mmm, that would be a gross bug that would have been reported ages ago.
Perhaps it's a logging bug, rather than an actual network protocol bug.

> It has to be
> http://domain.topdomain/path (for a 302) or https://domain.topdomain/path or
> https://domain.topdomain:443 . Am currently trying to instantiate an
> SSLContextFactory for the httpclient constructor but am so far not seeing an
> effect. It's obviously less "I know what I'm doing" and more "poke this and
> see if that part moves" but I am picking up a few things..

You should look at the server logs and try to understand why the
server rejects that request,
as the client has sent it (it is waiting for the response, but gets a
connection closed instead).

Especially, take a close look at what JDK you run on client and
server, what ciphers they have in common, what version of the TLS
protocol you have configured, etc.
My guess is that you have some TLS incompatibility between client and server.

See also: http://docs.oracle.com/javase/7/docs/technotes/guides/security/jsse/ReadDebug.html

--
Simone Bordet
----
http://cometd.org
http://webtide.com
Developer advice, training, services and support
from the Jetty & CometD experts.
_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: "early EOF" warning when https call from old application in 9.2.9.v2015022er

lamphus
Hi, thanks for your reply.

The EOF call that yields a functional issue for the end user appears to be made from the 'server side' of this application to its neighbour application in the same Jetty9(and as such the same jre, 8), Geoserver. I don't see it when looking at traffic between an end user and the application. The application uses the jars mentioned in the original email, among others (8.1.16v20140903) jetty-client in https://github.com/boundlessgeo/GeoExplorer/blob/master/app/httpclient.js which in turn is used by https://github.com/boundlessgeo/GeoExplorer/blob/master/app/auth.js :

The troubled part:

exports.getStatus = function(request) {
    var url = getAuthUrl(request);
    var status = 401;
    var headers = new Headers(request.headers);
    var token = headers.get("Cookie");
    var exchange = clientRequest({
        url: url,
        method: "GET",
        async: false,
        headers: headers
    });
    exchange.wait();
    return exchange.status;   //Status is 0, exchange.content is null for https url
};


where url == https://domain/geoserver/rest but the logs report it as GET//domain:443/geoserver/rest and if that is http://domain:443/geoserver/rest then our jetty9 can't handle that request, it gets aborted whether you try it from the application server or a client machine. It may be that we should configure jetty9 to understand and redirect http://domain:443 but I don't think just that would solve the issue, it is probably a bit of a tls configuration issue as you say since the official doc http://www.eclipse.org/jetty/documentation/current/http-client.html says "

When you create a HttpClient instance using the parameterless constructor, you will only be able to perform plain HTTP requests and you will not be able to perform HTTPS requests.

In order to perform HTTPS requests, you should create first a SslContextFactory, configure it, and pass it to the HttpClient constructor. When created with a SslContextFactory, the HttpClient will be able to perform both HTTP and HTTPS requests to any domain."

so I tried changing httpclient.js to import the util.ssl package and then create the httpclient like so:

var sslcontxtfctry = new SslContextFactory(true);
    var client = new HttpClient(sslcontxtfctry);

but this makes no perceivable difference. The GET is still to :443 and it still EOFs.


It is entirely possible that this issue is outside of the scope of this forum, if so I won't pursue it further.

If there's any further assistance to be had I shall of course be happy to receive it :  )

Regards


On Mon, Jan 30, 2017 at 7:34 PM, Simone Bordet <[hidden email]> wrote:
Hi,

On Mon, Jan 30, 2017 at 6:45 PM, David Persson <[hidden email]> wrote:
> Hi, yep, this was understood, and I knew I ran the risk of 0 replies, so I'm
> glad for yours : )
>
> Thanks for the info regarding EOF.
>
> This seems to happen for a specific request. The application instantiates a
> http://www.eclipse.org/jetty/documentation/current/http-client.html and
> issues a get request with an url argument that goes
> "https://domain.topdomain/path" but what is sent according to the jetty log
> appears to be "http://domain.topdomain:443" (I think- log says
> "GET//domain.topdomain:443/geoserver/rest" and the way our jetty9 is
> configured one can't reach it like this.

Mmm, that would be a gross bug that would have been reported ages ago.
Perhaps it's a logging bug, rather than an actual network protocol bug.

> It has to be
> http://domain.topdomain/path (for a 302) or https://domain.topdomain/path or
> https://domain.topdomain:443 . Am currently trying to instantiate an
> SSLContextFactory for the httpclient constructor but am so far not seeing an
> effect. It's obviously less "I know what I'm doing" and more "poke this and
> see if that part moves" but I am picking up a few things..

You should look at the server logs and try to understand why the
server rejects that request,
as the client has sent it (it is waiting for the response, but gets a
connection closed instead).

Especially, take a close look at what JDK you run on client and
server, what ciphers they have in common, what version of the TLS
protocol you have configured, etc.
My guess is that you have some TLS incompatibility between client and server.

See also: http://docs.oracle.com/javase/7/docs/technotes/guides/security/jsse/ReadDebug.html

--
Simone Bordet
----
http://cometd.org
http://webtide.com
Developer advice, training, services and support
from the Jetty & CometD experts.
_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users


_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: "early EOF" warning when https call from old application in 9.2.9.v2015022er

Simone Bordet-3
Hi,

On Tue, Jan 31, 2017 at 11:11 AM, David Persson <[hidden email]> wrote:
> Hi, thanks for your reply.
>
> The EOF call that yields a functional issue for the end user appears to be
> made from the 'server side' of this application to its neighbour application
> in the same Jetty9(and as such the same jre, 8), Geoserver.

Okay. So have you enabled javax.net.debug=all ?
DEBUG level logs on the server ?
Why the server closes the connection ?

--
Simone Bordet
----
http://cometd.org
http://webtide.com
Developer advice, training, services and support
from the Jetty & CometD experts.
_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: "early EOF" warning when https call from old application in 9.2.9.v2015022er

lamphus
It isn't clear to me where to enable javax.net.debug=all.

The server jetty9 does not have slf4jlog or javautillog defined, the level is at debug when I open prunmgr//ES//nameofservice, the stack trace from this server log is in the first email.

I don't even know whether "GET//domain:443/geoserver/rest" means an http prefix or an https prefix but I suppose http since our Jetty9 says connection reset to those.

Will probably need to hire external help here but thanks for your time.

 


On Tue, Jan 31, 2017 at 11:48 AM, Simone Bordet <[hidden email]> wrote:
Hi,

On Tue, Jan 31, 2017 at 11:11 AM, David Persson <[hidden email]> wrote:
> Hi, thanks for your reply.
>
> The EOF call that yields a functional issue for the end user appears to be
> made from the 'server side' of this application to its neighbour application
> in the same Jetty9(and as such the same jre, 8), Geoserver.

Okay. So have you enabled javax.net.debug=all ?
DEBUG level logs on the server ?
Why the server closes the connection ?

--
Simone Bordet
----
http://cometd.org
http://webtide.com
Developer advice, training, services and support
from the Jetty & CometD experts.
_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users


_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: "early EOF" warning when https call from old application in 9.2.9.v2015022er

Brian Reichert
On Tue, Jan 31, 2017 at 06:08:12PM +0100, David Persson wrote:
> It isn't clear to me where to enable javax.net.debug=all.

Several ways, if you're just experiementing:

https://www.eclipse.org/jetty/documentation/9.4.x/start-jar.html

- You can add 'javax.net.debug=all' to your start.ini file.
- I think you can add '-Djavax.net.debug=all' on the command line.


> I don't even know whether "GET//domain:443/geoserver/rest" means an http
> prefix or an https prefix but I suppose http since our Jetty9 says
> connection reset to those.

Well, port 443 implies HTTPS, but I haven't followed this whole
thread, so I don't know the context.

--
Brian Reichert <[hidden email]>
BSD admin/developer at large
_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: "early EOF" warning when https call from old application in 9.2.9.v2015022er

Simone Bordet-3
In reply to this post by lamphus
Hi,

On Tue, Jan 31, 2017 at 6:08 PM, David Persson <[hidden email]> wrote:
> It isn't clear to me where to enable javax.net.debug=all.

On the server that closes the connection, and on the client that
receives the connection closed.

> The server jetty9 does not have slf4jlog or javautillog defined, the level
> is at debug when I open prunmgr//ES//nameofservice, the stack trace from
> this server log is in the first email.
>
> I don't even know whether "GET//domain:443/geoserver/rest" means an http
> prefix or an https prefix but I suppose http since our Jetty9 says
> connection reset to those.

As I said, this is very unlikely.
The issue is probably something different.

> Will probably need to hire external help here but thanks for your time.

You can hire us :)
https://webtide.com/

--
Simone Bordet
----
http://cometd.org
http://webtide.com
Developer advice, training, services and support
from the Jetty & CometD experts.
_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: "early EOF" warning when https call from old application in 9.2.9.v2015022er

lamphus
Thanks both. (And good to know Simone )

Jetty9 server: Bits of my start.ini (installed as windows service) are being ignored, like send server version, and now javax.net.debug=all. In prunmgr however adding -Djavax.net.debug=all made the logging very active indeed.

For every :443/../rest call this is repeated:


qtp999661724-87, fatal error: 10: General SSLEngine problem
javax.net.ssl.SSLHandshakeException: SSLv2Hello is disabled
qtp999661724-87, SEND TLSv1.2 ALERT:  fatal, description = unexpected_message
qtp999661724-87, WRITE: TLSv1.2 Alert, length = 2
qtp999661724-87, fatal: engine already closed.  Rethrowing javax.net.ssl.SSLHandshakeException: SSLv2Hello is disabled
HttpClient-332, called closeInbound()
HttpClient-332, fatal error: 80: Inbound closed before receiving peer's close_notify: possible truncation attack?
javax.net.ssl.SSLException: Inbound closed before receiving peer's close_notify: possible truncation attack?
HttpClient-332, SEND TLSv1.2 ALERT:  fatal, description = internal_error
HttpClient-332, WRITE: TLSv1.2 Alert, length = 2
[Raw write]: length = 7
0000: 15 03 03 00 02 02 50                               ......P
HttpClient-332, called closeInbound()
HttpClient-332, closeInboundInternal()
HttpClient-332, called closeInbound()
HttpClient-332, closeInboundInternal()
2017-02-01 17:30:30.515:WARN:oejc.HttpExchange:HttpClient-332: EXCEPTION adapter1@6787ac61=GET//domain:443/geoserver/rest#WAITING(0ms)->EXCEPTED(0ms)sent=0ms
org.eclipse.jetty.io.EofException: early EOF

Right now TLS 1.0, 1.1 and 1.2 are accepted, no SSL version.

Regards,
David

On Tue, Jan 31, 2017 at 7:51 PM, Simone Bordet <[hidden email]> wrote:
Hi,

On Tue, Jan 31, 2017 at 6:08 PM, David Persson <[hidden email]> wrote:
> It isn't clear to me where to enable javax.net.debug=all.

On the server that closes the connection, and on the client that
receives the connection closed.

> The server jetty9 does not have slf4jlog or javautillog defined, the level
> is at debug when I open prunmgr//ES//nameofservice, the stack trace from
> this server log is in the first email.
>
> I don't even know whether "GET//domain:443/geoserver/rest" means an http
> prefix or an https prefix but I suppose http since our Jetty9 says
> connection reset to those.

As I said, this is very unlikely.
The issue is probably something different.

> Will probably need to hire external help here but thanks for your time.

You can hire us :)
https://webtide.com/

--
Simone Bordet
----
http://cometd.org
http://webtide.com
Developer advice, training, services and support
from the Jetty & CometD experts.
_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users


_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users