Jetty 9.4.24 prints warning messages to stderr

classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|

Jetty 9.4.24 prints warning messages to stderr

Silvio Bierman
Hello all,

Ever since upgrading to 9.4.24 our stderr-log is filled with these messages:

2019-11-28 22:56:27.469:WARN:oejs.HttpChannelState:qtp1519100796-25:
org.eclipse.jetty.io.EofException: Reset cancel_stream_error

Can anyone tell me what this means? I take it the situation is not
critical because the application has worked flawlessly for years with
earlier Jetty versions without these messages. Can I turn this off?

Kind regards,

Silvio

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Jetty 9.4.24 prints warning messages to stderr

Greg Wilkins

Silvio,

I believe it is ignorable and you can turn the HttpChannelState logger level down to suppress them.
However, if there a stack trace associated with that warning then it is not what I think it is and you need to provide more information.

What I believe is happening is that while a request is being processed, the associated HTTP/2 stream is being reset (probably by the client?)
This asynchronous error is detected but because the request is not async, it cannot be delivered to the request and instead we warn.  This is probably over verbose as clients can do silly things like close mid request handling.

[hidden email]  what do you think?

cheers











On Fri, 29 Nov 2019 at 09:03, Silvio Bierman <[hidden email]> wrote:
Hello all,

Ever since upgrading to 9.4.24 our stderr-log is filled with these messages:

2019-11-28 22:56:27.469:WARN:oejs.HttpChannelState:qtp1519100796-25:
org.eclipse.jetty.io.EofException: Reset cancel_stream_error

Can anyone tell me what this means? I take it the situation is not
critical because the application has worked flawlessly for years with
earlier Jetty versions without these messages. Can I turn this off?

Kind regards,

Silvio

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users


--

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Jetty 9.4.24 prints warning messages to stderr

Silvio Bierman
Thank you Greg,

Everything indeed seems to work fine. I do not see any stack traces or exceptions in our logging so no worries. I am not sure I would like to lower the default logger level at this point. For now I can live with some warnings in our stderr log.

Kind regards,

Silvio

On 11/29/19 12:27 AM, Greg Wilkins wrote:

Silvio,

I believe it is ignorable and you can turn the HttpChannelState logger level down to suppress them.
However, if there a stack trace associated with that warning then it is not what I think it is and you need to provide more information.

What I believe is happening is that while a request is being processed, the associated HTTP/2 stream is being reset (probably by the client?)
This asynchronous error is detected but because the request is not async, it cannot be delivered to the request and instead we warn.  This is probably over verbose as clients can do silly things like close mid request handling.

[hidden email]  what do you think?

cheers











On Fri, 29 Nov 2019 at 09:03, Silvio Bierman <[hidden email]> wrote:
Hello all,

Ever since upgrading to 9.4.24 our stderr-log is filled with these messages:

2019-11-28 22:56:27.469:WARN:oejs.HttpChannelState:qtp1519100796-25:
org.eclipse.jetty.io.EofException: Reset cancel_stream_error

Can anyone tell me what this means? I take it the situation is not
critical because the application has worked flawlessly for years with
earlier Jetty versions without these messages. Can I turn this off?

Kind regards,

Silvio

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users


--

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users


_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Jetty 9.4.24 prints warning messages to stderr

Silvio Bierman
In reply to this post by Greg Wilkins
Hi Greg,

The last few days exceptions have started to come up in the logging. We can quite easily reproduce them by testing common parts of our web applications using Firefox. Using Chrome the same actions do not produce exceptions (or warnings).

Strangely enough this seems to intermittently fail mostly (but not exclusively) on plain GET requests for CSS resources that are requested by the browser as a result of an @import from another CSS resource.

We serve the files ourselves from a servlet. Perhaps we are doing something that triggers this? GET requests for which we serve the response content dynamically seem to work fine. The same goes for POSTs. Since it only happens via one of our code paths I suspect we are causing this in some way, although the code is extremely simple.

Kind regards,

Silvio


On 11/29/19 12:27 AM, Greg Wilkins wrote:

Silvio,

I believe it is ignorable and you can turn the HttpChannelState logger level down to suppress them.
However, if there a stack trace associated with that warning then it is not what I think it is and you need to provide more information.

What I believe is happening is that while a request is being processed, the associated HTTP/2 stream is being reset (probably by the client?)
This asynchronous error is detected but because the request is not async, it cannot be delivered to the request and instead we warn.  This is probably over verbose as clients can do silly things like close mid request handling.

[hidden email]  what do you think?

cheers











On Fri, 29 Nov 2019 at 09:03, Silvio Bierman <[hidden email]> wrote:
Hello all,

Ever since upgrading to 9.4.24 our stderr-log is filled with these messages:

2019-11-28 22:56:27.469:WARN:oejs.HttpChannelState:qtp1519100796-25:
org.eclipse.jetty.io.EofException: Reset cancel_stream_error

Can anyone tell me what this means? I take it the situation is not
critical because the application has worked flawlessly for years with
earlier Jetty versions without these messages. Can I turn this off?

Kind regards,

Silvio

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users


--

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users


_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Jetty 9.4.24 prints warning messages to stderr

Greg Wilkins
Silvio,

This is the second time I've heard about a problem fetching browser resources like CSS or js.  Can you attach the stacks you are seeing?

On Mon, 2 Dec 2019, 22:58 Silvio Bierman, <[hidden email]> wrote:
Hi Greg,

The last few days exceptions have started to come up in the logging. We can quite easily reproduce them by testing common parts of our web applications using Firefox. Using Chrome the same actions do not produce exceptions (or warnings).

Strangely enough this seems to intermittently fail mostly (but not exclusively) on plain GET requests for CSS resources that are requested by the browser as a result of an @import from another CSS resource.

We serve the files ourselves from a servlet. Perhaps we are doing something that triggers this? GET requests for which we serve the response content dynamically seem to work fine. The same goes for POSTs. Since it only happens via one of our code paths I suspect we are causing this in some way, although the code is extremely simple.

Kind regards,

Silvio


On 11/29/19 12:27 AM, Greg Wilkins wrote:

Silvio,

I believe it is ignorable and you can turn the HttpChannelState logger level down to suppress them.
However, if there a stack trace associated with that warning then it is not what I think it is and you need to provide more information.

What I believe is happening is that while a request is being processed, the associated HTTP/2 stream is being reset (probably by the client?)
This asynchronous error is detected but because the request is not async, it cannot be delivered to the request and instead we warn.  This is probably over verbose as clients can do silly things like close mid request handling.

[hidden email]  what do you think?

cheers











On Fri, 29 Nov 2019 at 09:03, Silvio Bierman <[hidden email]> wrote:
Hello all,

Ever since upgrading to 9.4.24 our stderr-log is filled with these messages:

2019-11-28 22:56:27.469:WARN:oejs.HttpChannelState:qtp1519100796-25:
org.eclipse.jetty.io.EofException: Reset cancel_stream_error

Can anyone tell me what this means? I take it the situation is not
critical because the application has worked flawlessly for years with
earlier Jetty versions without these messages. Can I turn this off?

Kind regards,

Silvio

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users


--

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Jetty 9.4.24 prints warning messages to stderr

Silvio Bierman
Hi Greg,

Sure. Below are two such exceptions triggered via Firefox 70.0.1. The first one was captured while running the server in Eclipse and the second one was taken from the stderr-logging on a server. Both use Jetty 9.4.24. My desktop is a Fedora31/OpenJDK13 while the server runs Ubuntu 19.04/OpenJDK12.

Cheers,

Silvio


2019-12-02 14:56:03.564:WARN:oejs.HttpChannelState:qtp1441577726-46: org.eclipse.jetty.io.EofException: Reset cancel_stream_error
org.eclipse.jetty.io.EofException: Reset cancel_stream_error
    at org.eclipse.jetty.http2.server.HTTP2ServerConnectionFactory$HTTPServerSessionListener.onReset(HTTP2ServerConnectionFactory.java:157)
    at org.eclipse.jetty.http2.HTTP2Stream.notifyReset(HTTP2Stream.java:574)
    at org.eclipse.jetty.http2.HTTP2Stream.onReset(HTTP2Stream.java:343)
    at org.eclipse.jetty.http2.HTTP2Stream.process(HTTP2Stream.java:252)
    at org.eclipse.jetty.http2.HTTP2Session.onReset(HTTP2Session.java:295)
    at org.eclipse.jetty.http2.parser.Parser$Listener$Wrapper.onReset(Parser.java:372)
    at org.eclipse.jetty.http2.parser.BodyParser.notifyReset(BodyParser.java:144)
    at org.eclipse.jetty.http2.parser.ResetBodyParser.onReset(ResetBodyParser.java:97)
    at org.eclipse.jetty.http2.parser.ResetBodyParser.parse(ResetBodyParser.java:66)
    at org.eclipse.jetty.http2.parser.Parser.parseBody(Parser.java:198)
    at org.eclipse.jetty.http2.parser.Parser.parse(Parser.java:127)
    at org.eclipse.jetty.http2.parser.ServerParser.parse(ServerParser.java:115)
    at org.eclipse.jetty.http2.HTTP2Connection$HTTP2Producer.produce(HTTP2Connection.java:248)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:360)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:184)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
    at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
    at java.base/java.lang.Thread.run(Thread.java:830)
    Suppressed: java.lang.Throwable: HttpInput failure
        at org.eclipse.jetty.server.HttpInput.failed(HttpInput.java:823)
        at org.eclipse.jetty.http2.server.HttpChannelOverHTTP2.onFailure(HttpChannelOverHTTP2.java:323)
        at org.eclipse.jetty.http2.server.HTTP2ServerConnection.onStreamFailure(HTTP2ServerConnection.java:221)
        ... 21 more


2019-12-02 15:05:50.984:WARN:oejs.HttpChannelState:qtp1608297024-1004: org.eclipse.jetty.io.EofException: Reset cancel_stream_error
org.eclipse.jetty.io.EofException: Reset cancel_stream_error
    at org.eclipse.jetty.http2.server.HTTP2ServerConnectionFactory$HTTPServerSessionListener.onReset(HTTP2ServerConnectionFactory.java:157)
    at org.eclipse.jetty.http2.HTTP2Stream.notifyReset(HTTP2Stream.java:574)
    at org.eclipse.jetty.http2.HTTP2Stream.onReset(HTTP2Stream.java:343)
    at org.eclipse.jetty.http2.HTTP2Stream.process(HTTP2Stream.java:252)
    at org.eclipse.jetty.http2.HTTP2Session.onReset(HTTP2Session.java:295)
    at org.eclipse.jetty.http2.parser.Parser$Listener$Wrapper.onReset(Parser.java:372)
    at org.eclipse.jetty.http2.parser.BodyParser.notifyReset(BodyParser.java:144)
    at org.eclipse.jetty.http2.parser.ResetBodyParser.onReset(ResetBodyParser.java:97)
    at org.eclipse.jetty.http2.parser.ResetBodyParser.parse(ResetBodyParser.java:66)
    at org.eclipse.jetty.http2.parser.Parser.parseBody(Parser.java:198)
    at org.eclipse.jetty.http2.parser.Parser.parse(Parser.java:127)
    at org.eclipse.jetty.http2.parser.ServerParser.parse(ServerParser.java:115)
    at org.eclipse.jetty.http2.HTTP2Connection$HTTP2Producer.produce(HTTP2Connection.java:248)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:360)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:184)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:135)
    at org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:170)
    at org.eclipse.jetty.http2.HTTP2Connection.onFillable(HTTP2Connection.java:125)
    at org.eclipse.jetty.http2.HTTP2Connection$FillableCallback.succeeded(HTTP2Connection.java:348)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
    at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:543)
    at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:398)
    at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
    at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
    at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
    at java.base/java.lang.Thread.run(Thread.java:835)
    Suppressed: java.lang.Throwable: HttpInput failure
        at org.eclipse.jetty.server.HttpInput.failed(HttpInput.java:823)
        at org.eclipse.jetty.http2.server.HttpChannelOverHTTP2.onFailure(HttpChannelOverHTTP2.java:323)
        at org.eclipse.jetty.http2.server.HTTP2ServerConnection.onStreamFailure(HTTP2ServerConnection.java:221)
        ... 34 more

On 12/2/19 2:42 PM, Greg Wilkins wrote:
Silvio,

This is the second time I've heard about a problem fetching browser resources like CSS or js.  Can you attach the stacks you are seeing?

On Mon, 2 Dec 2019, 22:58 Silvio Bierman, <[hidden email]> wrote:
Hi Greg,

The last few days exceptions have started to come up in the logging. We can quite easily reproduce them by testing common parts of our web applications using Firefox. Using Chrome the same actions do not produce exceptions (or warnings).

Strangely enough this seems to intermittently fail mostly (but not exclusively) on plain GET requests for CSS resources that are requested by the browser as a result of an @import from another CSS resource.

We serve the files ourselves from a servlet. Perhaps we are doing something that triggers this? GET requests for which we serve the response content dynamically seem to work fine. The same goes for POSTs. Since it only happens via one of our code paths I suspect we are causing this in some way, although the code is extremely simple.

Kind regards,

Silvio


On 11/29/19 12:27 AM, Greg Wilkins wrote:

Silvio,

I believe it is ignorable and you can turn the HttpChannelState logger level down to suppress them.
However, if there a stack trace associated with that warning then it is not what I think it is and you need to provide more information.

What I believe is happening is that while a request is being processed, the associated HTTP/2 stream is being reset (probably by the client?)
This asynchronous error is detected but because the request is not async, it cannot be delivered to the request and instead we warn.  This is probably over verbose as clients can do silly things like close mid request handling.

[hidden email]  what do you think?

cheers











On Fri, 29 Nov 2019 at 09:03, Silvio Bierman <[hidden email]> wrote:
Hello all,

Ever since upgrading to 9.4.24 our stderr-log is filled with these messages:

2019-11-28 22:56:27.469:WARN:oejs.HttpChannelState:qtp1519100796-25:
org.eclipse.jetty.io.EofException: Reset cancel_stream_error

Can anyone tell me what this means? I take it the situation is not
critical because the application has worked flawlessly for years with
earlier Jetty versions without these messages. Can I turn this off?

Kind regards,

Silvio

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users


--

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users


_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Jetty 9.4.24 prints warning messages to stderr

Silvio Bierman
In reply to this post by Greg Wilkins
Hi Greg,

At this moment we are receiving multiple error reports from users who suffer from malfunctioning user interfaces. We already had received some of those before the weekend but I did not link this to our move to 9.4.24. Now a pattern is emerging.
They are mostly from Firefox users but some come from Safari users. The symptoms are consistently similar: missing images, unstyled content, parts of content missing etc.

We will probably have to revert to the previous Jetty version we where running (9.4.20) to make sure we do not pick one that behaves the same. In the meantime I would be happy to do any testing if you would require so.

Kind regards,

Silvio

On 12/2/19 2:42 PM, Greg Wilkins wrote:
Silvio,

This is the second time I've heard about a problem fetching browser resources like CSS or js.  Can you attach the stacks you are seeing?

On Mon, 2 Dec 2019, 22:58 Silvio Bierman, <[hidden email]> wrote:
Hi Greg,

The last few days exceptions have started to come up in the logging. We can quite easily reproduce them by testing common parts of our web applications using Firefox. Using Chrome the same actions do not produce exceptions (or warnings).

Strangely enough this seems to intermittently fail mostly (but not exclusively) on plain GET requests for CSS resources that are requested by the browser as a result of an @import from another CSS resource.

We serve the files ourselves from a servlet. Perhaps we are doing something that triggers this? GET requests for which we serve the response content dynamically seem to work fine. The same goes for POSTs. Since it only happens via one of our code paths I suspect we are causing this in some way, although the code is extremely simple.

Kind regards,

Silvio


On 11/29/19 12:27 AM, Greg Wilkins wrote:

Silvio,

I believe it is ignorable and you can turn the HttpChannelState logger level down to suppress them.
However, if there a stack trace associated with that warning then it is not what I think it is and you need to provide more information.

What I believe is happening is that while a request is being processed, the associated HTTP/2 stream is being reset (probably by the client?)
This asynchronous error is detected but because the request is not async, it cannot be delivered to the request and instead we warn.  This is probably over verbose as clients can do silly things like close mid request handling.

[hidden email]  what do you think?

cheers











On Fri, 29 Nov 2019 at 09:03, Silvio Bierman <[hidden email]> wrote:
Hello all,

Ever since upgrading to 9.4.24 our stderr-log is filled with these messages:

2019-11-28 22:56:27.469:WARN:oejs.HttpChannelState:qtp1519100796-25:
org.eclipse.jetty.io.EofException: Reset cancel_stream_error

Can anyone tell me what this means? I take it the situation is not
critical because the application has worked flawlessly for years with
earlier Jetty versions without these messages. Can I turn this off?

Kind regards,

Silvio

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users


--

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users


_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Jetty 9.4.24 prints warning messages to stderr

Simone Bordet-3
Silvio,

On Tue, Dec 3, 2019 at 10:15 AM Silvio Bierman
<[hidden email]> wrote:
>
> Hi Greg,
>
> At this moment we are receiving multiple error reports from users who suffer from malfunctioning user interfaces. We already had received some of those before the weekend but I did not link this to our move to 9.4.24. Now a pattern is emerging.
> They are mostly from Firefox users but some come from Safari users. The symptoms are consistently similar: missing images, unstyled content, parts of content missing etc.
>
> We will probably have to revert to the previous Jetty version we where running (9.4.20) to make sure we do not pick one that behaves the same. In the meantime I would be happy to do any testing if you would require so.

The stack trace shows clearly that it's the client sending a reset to
the server, and that is logged at WARN level; we can certainly improve
the logging as EofException should not be logged at WARN level.

One thing that should mitigate the issue for now is to set

HttpConfiguration.notifyRemoteAsyncErrors = false

This can be done either in code or with an XML snippet, depending on
how you start Jetty.
This flag basically says whether async remote errors such as a client
resetting a HTTP/2 stream are reported to the server.
This flag is true by default, and the only thing that happens is the
the failure is logged, so functionality wise it's similar to a
no-operation.

Now, why the client resets the stream is what we should figure out.
Sometimes they do it because they attempt a request, but then they
don't care anymore, and so that is entirely ignorable by the server.

If possible, we would like to have more information.

A) Can you take some information on the client as to what is exactly
the failure for those missing images, missing CSS, etc? What does the
Firefox console say about those requests? They remain pending? You get
back an error? If so, what error?
B) Can you enable DEBUG logging on the server for few seconds/minutes
while this is happening? This can be done via JMX if you're using
Jetty logging, or via some other tool/facility depending on the
logging library that you're using.

I have Firefox 70.0.1 on Ubuntu and I don't see any problem with our
own websites that run Jetty (in particular https://webtide.com). Do
you see problems hitting webtide.com?

--
Simone Bordet
----
http://cometd.org
http://webtide.com
Developer advice, training, services and support
from the Jetty & CometD experts.
_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Jetty 9.4.24 prints warning messages to stderr

Simone Bordet-3
Silvio,

On Tue, Dec 3, 2019 at 12:26 PM Simone Bordet <[hidden email]> wrote:
> The stack trace shows clearly that it's the client sending a reset to
> the server, and that is logged at WARN level; we can certainly improve
> the logging as EofException should not be logged at WARN level.

Now that I look better at this, we do print a line a WARN level, but
the whole stack trace is only printed at DEBUG level
May your application have some printStackTrace() left around?

--
Simone Bordet
----
http://cometd.org
http://webtide.com
Developer advice, training, services and support
from the Jetty & CometD experts.
_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Jetty 9.4.24 prints warning messages to stderr

Silvio Bierman
Hi Simone,

I should have provided you with more information. Yes, the stack-trace
does come from a printStackTrace that is in our application. The
application is a generic scripting engine that consists of a single
Servlet. This servlet dispatches incoming requests to scripts which
implement our applications. Most error/exception handling is done inside
the scripts themselves but at the highest level uncaught exceptions are
printed to stderr using printStackTrace. That is how these stack traces
end up in the Eclipse console or the application logs.

Apart from the JVM stack traces we also have application stack traces in
terms of the script code. The exceptions all lead to a tiny script class
that implements a handler for static resources. Basically it does some
standard HTTP header handling (If-Modified-Since) and either returns a
304 or sets a Content-Length, Content-Type, Last-Modified, optional
Cache-Control (max-age), transfers all bytes from a BufferedInputStream
to a BufferedOutputStream(response.getOutputStream) (loosely translated
from scripting code) and finally closes the streams.

Please see my response to your other mail for more info on what happens
inside FF.

Cheers,

Silvio


On 12/3/19 12:45 PM, Simone Bordet wrote:

> Silvio,
>
> On Tue, Dec 3, 2019 at 12:26 PM Simone Bordet <[hidden email]> wrote:
>> The stack trace shows clearly that it's the client sending a reset to
>> the server, and that is logged at WARN level; we can certainly improve
>> the logging as EofException should not be logged at WARN level.
> Now that I look better at this, we do print a line a WARN level, but
> the whole stack trace is only printed at DEBUG level
> May your application have some printStackTrace() left around?
>

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Jetty 9.4.24 prints warning messages to stderr

Silvio Bierman
In reply to this post by Simone Bordet-3
Posted inline...

On 12/3/19 12:26 PM, Simone Bordet wrote:

> Silvio,
>
> On Tue, Dec 3, 2019 at 10:15 AM Silvio Bierman
> <[hidden email]> wrote:
>> Hi Greg,
>>
>> At this moment we are receiving multiple error reports from users who suffer from malfunctioning user interfaces. We already had received some of those before the weekend but I did not link this to our move to 9.4.24. Now a pattern is emerging.
>> They are mostly from Firefox users but some come from Safari users. The symptoms are consistently similar: missing images, unstyled content, parts of content missing etc.
>>
>> We will probably have to revert to the previous Jetty version we where running (9.4.20) to make sure we do not pick one that behaves the same. In the meantime I would be happy to do any testing if you would require so.
> The stack trace shows clearly that it's the client sending a reset to
> the server, and that is logged at WARN level; we can certainly improve
> the logging as EofException should not be logged at WARN level.
>
> One thing that should mitigate the issue for now is to set
>
> HttpConfiguration.notifyRemoteAsyncErrors = false
>
> This can be done either in code or with an XML snippet, depending on
> how you start Jetty.
> This flag basically says whether async remote errors such as a client
> resetting a HTTP/2 stream are reported to the server.
> This flag is true by default, and the only thing that happens is the
> the failure is logged, so functionality wise it's similar to a
> no-operation.
>
> Now, why the client resets the stream is what we should figure out.
> Sometimes they do it because they attempt a request, but then they
> don't care anymore, and so that is entirely ignorable by the server.
>
> If possible, we would like to have more information.
>
> A) Can you take some information on the client as to what is exactly
> the failure for those missing images, missing CSS, etc? What does the
> Firefox console say about those requests? They remain pending? You get
> back an error? If so, what error?
I added some debugging info to see which requests fail so I could
compare them with the network log in FF. The leads to a very interesting
pattern. For each failed request FF says in the Transfer column: xxxKB
(raced). It shows this for other requests as well but for the failed
requests it lists nothing in the Status column, whereas the rest shows
200 OK (Cached). If I look at the request headers remote address, status
code, version and referrer policy are empty. Nothing else indicates
errors of any sort. In fact on the FF side it looks like the request is
never sent and FF does show a response (mostly images when I just
tested). However, on the server sides the request are received and throw
an exception.
> B) Can you enable DEBUG logging on the server for few seconds/minutes
> while this is happening? This can be done via JMX if you're using
> Jetty logging, or via some other tool/facility depending on the
> logging library that you're using.
>
> I have Firefox 70.0.1 on Ubuntu and I don't see any problem with our
> own websites that run Jetty (in particular https://webtide.com). Do
> you see problems hitting webtide.com?
>
Well, going to webtide.com does show a number of similar entries in the
network listing (all of them are CSS resources) for which FF shows no
Status value and lists "No response data available for this request"
when I look at the response. I do not know how to interpret that. The
site looks to be OK but that is just my first impression.

Kind regards,

Silvio

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Jetty 9.4.24 prints warning messages to stderr

Silvio Bierman
In reply to this post by Simone Bordet-3
Hi Simone,

Sorry for the fragmented posts but I stumbled onto something that may be
of interest:

In my testing environment the script code that returns the static
resources in fact failed to use a buffered input stream on the files.
After changing that and testing with Chrome the warning messages have
gone away. The testing scenario I use now used to produce at least 4-8
warnings when using Chrome and now the log stays clear. With Firefox
exceptions still appear but the number has decreased significantly.

For what its worth: disabling http/2 makes all exceptions and warnings
go away but that will probably not surprise you.

Cheers,

Silvio

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Jetty 9.4.24 prints warning messages to stderr

Silvio Bierman
In reply to this post by Simone Bordet-3
Hello Simone,

FYI: I just downgraded my test-environment to 9.4.20 and the same
exceptions appear with FF. No warnings with Chrome btw. So the FF issue
is not something that appeared in a recent version. For some reason my
FF 70.0.1 intermittently stumbles. We have been running http/2 for a
couple of months now and these error reports are all very recent. Maybe
they changed something in FF that suddenly causes this.

I am considering reverting to HTTP 1.1 for now.

Kind regards,

Silvio


On 12/3/19 12:45 PM, Simone Bordet wrote:

> Silvio,
>
> On Tue, Dec 3, 2019 at 12:26 PM Simone Bordet <[hidden email]> wrote:
>> The stack trace shows clearly that it's the client sending a reset to
>> the server, and that is logged at WARN level; we can certainly improve
>> the logging as EofException should not be logged at WARN level.
> Now that I look better at this, we do print a line a WARN level, but
> the whole stack trace is only printed at DEBUG level
> May your application have some printStackTrace() left around?
>

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Jetty 9.4.24 prints warning messages to stderr

Bill Ross-2
In reply to this post by Silvio Bierman

For what it's worth, I also just down-versioned from 9.4.24.v20191120 because my server was using 300% CPU with no client activity. I can't rule out my own changes, and a couple of out-of-practice looks at thread dumps didn't give me an answer. But there's nothing I've added that would keep a thread busy like that after startup, and it happens after running a while. So far it hasn't happened on the down rev: 9.4.12.v20180830.

I wonder if there's a thread activity monitor one could add that would warn if a thread seemed runaway..

Bill

On 12/3/19 1:14 AM, Silvio Bierman wrote:
Hi Greg,

At this moment we are receiving multiple error reports from users who suffer from malfunctioning user interfaces. We already had received some of those before the weekend but I did not link this to our move to 9.4.24. Now a pattern is emerging.
They are mostly from Firefox users but some come from Safari users. The symptoms are consistently similar: missing images, unstyled content, parts of content missing etc.

We will probably have to revert to the previous Jetty version we where running (9.4.20) to make sure we do not pick one that behaves the same. In the meantime I would be happy to do any testing if you would require so.

Kind regards,

Silvio

On 12/2/19 2:42 PM, Greg Wilkins wrote:
Silvio,

This is the second time I've heard about a problem fetching browser resources like CSS or js.  Can you attach the stacks you are seeing?

On Mon, 2 Dec 2019, 22:58 Silvio Bierman, <[hidden email]> wrote:
Hi Greg,

The last few days exceptions have started to come up in the logging. We can quite easily reproduce them by testing common parts of our web applications using Firefox. Using Chrome the same actions do not produce exceptions (or warnings).

Strangely enough this seems to intermittently fail mostly (but not exclusively) on plain GET requests for CSS resources that are requested by the browser as a result of an @import from another CSS resource.

We serve the files ourselves from a servlet. Perhaps we are doing something that triggers this? GET requests for which we serve the response content dynamically seem to work fine. The same goes for POSTs. Since it only happens via one of our code paths I suspect we are causing this in some way, although the code is extremely simple.

Kind regards,

Silvio


On 11/29/19 12:27 AM, Greg Wilkins wrote:

Silvio,

I believe it is ignorable and you can turn the HttpChannelState logger level down to suppress them.
However, if there a stack trace associated with that warning then it is not what I think it is and you need to provide more information.

What I believe is happening is that while a request is being processed, the associated HTTP/2 stream is being reset (probably by the client?)
This asynchronous error is detected but because the request is not async, it cannot be delivered to the request and instead we warn.  This is probably over verbose as clients can do silly things like close mid request handling.

[hidden email]  what do you think?

cheers











On Fri, 29 Nov 2019 at 09:03, Silvio Bierman <[hidden email]> wrote:
Hello all,

Ever since upgrading to 9.4.24 our stderr-log is filled with these messages:

2019-11-28 22:56:27.469:WARN:oejs.HttpChannelState:qtp1519100796-25:
org.eclipse.jetty.io.EofException: Reset cancel_stream_error

Can anyone tell me what this means? I take it the situation is not
critical because the application has worked flawlessly for years with
earlier Jetty versions without these messages. Can I turn this off?

Kind regards,

Silvio

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users


--

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users


_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Jetty 9.4.24 prints warning messages to stderr

Joakim Erdfelt-8
9.4.12 is subject to many security issues.


Joakim Erdfelt / [hidden email]


On Tue, Dec 3, 2019 at 1:22 PM Bill Ross <[hidden email]> wrote:

For what it's worth, I also just down-versioned from 9.4.24.v20191120 because my server was using 300% CPU with no client activity. I can't rule out my own changes, and a couple of out-of-practice looks at thread dumps didn't give me an answer. But there's nothing I've added that would keep a thread busy like that after startup, and it happens after running a while. So far it hasn't happened on the down rev: 9.4.12.v20180830.

I wonder if there's a thread activity monitor one could add that would warn if a thread seemed runaway..

Bill

On 12/3/19 1:14 AM, Silvio Bierman wrote:
Hi Greg,

At this moment we are receiving multiple error reports from users who suffer from malfunctioning user interfaces. We already had received some of those before the weekend but I did not link this to our move to 9.4.24. Now a pattern is emerging.
They are mostly from Firefox users but some come from Safari users. The symptoms are consistently similar: missing images, unstyled content, parts of content missing etc.

We will probably have to revert to the previous Jetty version we where running (9.4.20) to make sure we do not pick one that behaves the same. In the meantime I would be happy to do any testing if you would require so.

Kind regards,

Silvio

On 12/2/19 2:42 PM, Greg Wilkins wrote:
Silvio,

This is the second time I've heard about a problem fetching browser resources like CSS or js.  Can you attach the stacks you are seeing?

On Mon, 2 Dec 2019, 22:58 Silvio Bierman, <[hidden email]> wrote:
Hi Greg,

The last few days exceptions have started to come up in the logging. We can quite easily reproduce them by testing common parts of our web applications using Firefox. Using Chrome the same actions do not produce exceptions (or warnings).

Strangely enough this seems to intermittently fail mostly (but not exclusively) on plain GET requests for CSS resources that are requested by the browser as a result of an @import from another CSS resource.

We serve the files ourselves from a servlet. Perhaps we are doing something that triggers this? GET requests for which we serve the response content dynamically seem to work fine. The same goes for POSTs. Since it only happens via one of our code paths I suspect we are causing this in some way, although the code is extremely simple.

Kind regards,

Silvio


On 11/29/19 12:27 AM, Greg Wilkins wrote:

Silvio,

I believe it is ignorable and you can turn the HttpChannelState logger level down to suppress them.
However, if there a stack trace associated with that warning then it is not what I think it is and you need to provide more information.

What I believe is happening is that while a request is being processed, the associated HTTP/2 stream is being reset (probably by the client?)
This asynchronous error is detected but because the request is not async, it cannot be delivered to the request and instead we warn.  This is probably over verbose as clients can do silly things like close mid request handling.

[hidden email]  what do you think?

cheers











On Fri, 29 Nov 2019 at 09:03, Silvio Bierman <[hidden email]> wrote:
Hello all,

Ever since upgrading to 9.4.24 our stderr-log is filled with these messages:

2019-11-28 22:56:27.469:WARN:oejs.HttpChannelState:qtp1519100796-25:
org.eclipse.jetty.io.EofException: Reset cancel_stream_error

Can anyone tell me what this means? I take it the situation is not
critical because the application has worked flawlessly for years with
earlier Jetty versions without these messages. Can I turn this off?

Kind regards,

Silvio

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users


--

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users


_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users
_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Jetty 9.4.24 prints warning messages to stderr

Bill Ross-2

Thanks, good to know - so far the downrev is just on the home server, but will have to consider whether to dig deeper when I'm ready to push again. DoS at least isn't much of a factor for my deliberately-avoided-by-all site. :-)

Here's a peek at threads with lots of cpu on them, in case it's on the jetty side:

"qtp1008925772-146" #146 prio=5 os_prio=0 cpu=648389.04ms elapsed=711.23s tid=0x
00007f1a1c04b000 nid=0x6eed runnable  [0x00007f1af52e9000]
   java.lang.Thread.State: RUNNABLE
        at java.util.WeakHashMap.get([hidden email])
        at com.mchange.v2.encounter.AbstractEncounterCounter.encounter(AbstractE
ncounterCounter.java:41)

"qtp1008925772-145" #145 prio=5 os_prio=0 cpu=546854.46ms elapsed=772.97s tid=0x
00007f1a98369800 nid=0x6ecf runnable  [0x00007f1a5b0f1000]
   java.lang.Thread.State: RUNNABLE
        at java.util.WeakHashMap.get([hidden email])
        at com.mchange.v2.encounter.AbstractEncounterCounter.encounter(AbstractE
ncounterCounter.java:41)

"qtp1008925772-140" #140 prio=5 os_prio=0 cpu=601687.05ms elapsed=1112.57s tid=0
x00007f1ab00ed800 nid=0x6e15 runnable  [0x00007f1a5b3f2000]
   java.lang.Thread.State: RUNNABLE
        at java.util.WeakHashMap.get([hidden email])

"qtp1008925772-102" #102 prio=5 os_prio=0 cpu=7268038.26ms elapsed=7496.99s tid=
0x00007f1a3412e000 nid=0x59eb runnable  [0x00007f1af55ee000]
   java.lang.Thread.State: RUNNABLE
        at java.util.WeakHashMap.get([hidden email])

Whereas this polite thread gives an idea of uptime (elapsed):

"VM Periodic Task Thread" os_prio=0 cpu=21755.03ms elapsed=100153.98s tid=0x0000
7f1b2c565000 nid=0x5016 waiting on condition

Bill

On 12/3/19 11:34 AM, Joakim Erdfelt wrote:
9.4.12 is subject to many security issues.


Joakim Erdfelt / [hidden email]


On Tue, Dec 3, 2019 at 1:22 PM Bill Ross <[hidden email]> wrote:

For what it's worth, I also just down-versioned from 9.4.24.v20191120 because my server was using 300% CPU with no client activity. I can't rule out my own changes, and a couple of out-of-practice looks at thread dumps didn't give me an answer. But there's nothing I've added that would keep a thread busy like that after startup, and it happens after running a while. So far it hasn't happened on the down rev: 9.4.12.v20180830.

I wonder if there's a thread activity monitor one could add that would warn if a thread seemed runaway..

Bill

On 12/3/19 1:14 AM, Silvio Bierman wrote:
Hi Greg,

At this moment we are receiving multiple error reports from users who suffer from malfunctioning user interfaces. We already had received some of those before the weekend but I did not link this to our move to 9.4.24. Now a pattern is emerging.
They are mostly from Firefox users but some come from Safari users. The symptoms are consistently similar: missing images, unstyled content, parts of content missing etc.

We will probably have to revert to the previous Jetty version we where running (9.4.20) to make sure we do not pick one that behaves the same. In the meantime I would be happy to do any testing if you would require so.

Kind regards,

Silvio

On 12/2/19 2:42 PM, Greg Wilkins wrote:
Silvio,

This is the second time I've heard about a problem fetching browser resources like CSS or js.  Can you attach the stacks you are seeing?

On Mon, 2 Dec 2019, 22:58 Silvio Bierman, <[hidden email]> wrote:
Hi Greg,

The last few days exceptions have started to come up in the logging. We can quite easily reproduce them by testing common parts of our web applications using Firefox. Using Chrome the same actions do not produce exceptions (or warnings).

Strangely enough this seems to intermittently fail mostly (but not exclusively) on plain GET requests for CSS resources that are requested by the browser as a result of an @import from another CSS resource.

We serve the files ourselves from a servlet. Perhaps we are doing something that triggers this? GET requests for which we serve the response content dynamically seem to work fine. The same goes for POSTs. Since it only happens via one of our code paths I suspect we are causing this in some way, although the code is extremely simple.

Kind regards,

Silvio


On 11/29/19 12:27 AM, Greg Wilkins wrote:

Silvio,

I believe it is ignorable and you can turn the HttpChannelState logger level down to suppress them.
However, if there a stack trace associated with that warning then it is not what I think it is and you need to provide more information.

What I believe is happening is that while a request is being processed, the associated HTTP/2 stream is being reset (probably by the client?)
This asynchronous error is detected but because the request is not async, it cannot be delivered to the request and instead we warn.  This is probably over verbose as clients can do silly things like close mid request handling.

[hidden email]  what do you think?

cheers











On Fri, 29 Nov 2019 at 09:03, Silvio Bierman <[hidden email]> wrote:
Hello all,

Ever since upgrading to 9.4.24 our stderr-log is filled with these messages:

2019-11-28 22:56:27.469:WARN:oejs.HttpChannelState:qtp1519100796-25:
org.eclipse.jetty.io.EofException: Reset cancel_stream_error

Can anyone tell me what this means? I take it the situation is not
critical because the application has worked flawlessly for years with
earlier Jetty versions without these messages. Can I turn this off?

Kind regards,

Silvio

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users


--

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users


_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users
_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Jetty 9.4.24 prints warning messages to stderr

Bill Ross-2

Right below in that stack (duh).. it's a thread waiting for a C3P0 db connection. Must have been more tired when I looked before. Any way it could be jetty-related?

        at com.mchange.v2.c3p0.impl.C3P0ImplUtils.allocateIdentityToken(C3P0Impl
Utils.java:192)
        at com.mchange.v2.c3p0.impl.DriverManagerDataSourceBase.<init>(DriverMan
agerDataSourceBase.java:205)
        at com.mchange.v2.c3p0.DriverManagerDataSource.<init>(DriverManagerDataS
ource.java:60)
        at com.mchange.v2.c3p0.DriverManagerDataSource.<init>(DriverManagerDataSource.java:56)
        at com.mchange.v2.c3p0.ComboPooledDataSource.<init>(ComboPooledDataSource.java:113)
...

       at org.eclipse.jetty.jndi.NamingContext.lookup(NamingContext.java:467)
        at org.eclipse.jetty.jndi.NamingContext.lookup(NamingContext.java:491)
        at org.eclipse.jetty.jndi.NamingContext.lookup(NamingContext.java:491)
        at org.eclipse.jetty.jndi.NamingContext.lookup(NamingContext.java:491)
        at org.eclipse.jetty.jndi.NamingContext.lookup(NamingContext.java:505)
        at org.eclipse.jetty.jndi.java.javaRootURLContext.lookup(javaRootURLContext.java:101)
        at javax.naming.InitialContext.lookup([hidden email])
        at com.priot.db.dao.DaoBase.getConn(DaoBase.java:30)

On 12/3/19 11:58 AM, Bill Ross wrote:

Thanks, good to know - so far the downrev is just on the home server, but will have to consider whether to dig deeper when I'm ready to push again. DoS at least isn't much of a factor for my deliberately-avoided-by-all site. :-)

Here's a peek at threads with lots of cpu on them, in case it's on the jetty side:

"qtp1008925772-146" #146 prio=5 os_prio=0 cpu=648389.04ms elapsed=711.23s tid=0x
00007f1a1c04b000 nid=0x6eed runnable  [0x00007f1af52e9000]
   java.lang.Thread.State: RUNNABLE
        at java.util.WeakHashMap.get([hidden email])
        at com.mchange.v2.encounter.AbstractEncounterCounter.encounter(AbstractE
ncounterCounter.java:41)

"qtp1008925772-145" #145 prio=5 os_prio=0 cpu=546854.46ms elapsed=772.97s tid=0x
00007f1a98369800 nid=0x6ecf runnable  [0x00007f1a5b0f1000]
   java.lang.Thread.State: RUNNABLE
        at java.util.WeakHashMap.get([hidden email])
        at com.mchange.v2.encounter.AbstractEncounterCounter.encounter(AbstractE
ncounterCounter.java:41)

"qtp1008925772-140" #140 prio=5 os_prio=0 cpu=601687.05ms elapsed=1112.57s tid=0
x00007f1ab00ed800 nid=0x6e15 runnable  [0x00007f1a5b3f2000]
   java.lang.Thread.State: RUNNABLE
        at java.util.WeakHashMap.get([hidden email])

"qtp1008925772-102" #102 prio=5 os_prio=0 cpu=7268038.26ms elapsed=7496.99s tid=
0x00007f1a3412e000 nid=0x59eb runnable  [0x00007f1af55ee000]
   java.lang.Thread.State: RUNNABLE
        at java.util.WeakHashMap.get([hidden email])

Whereas this polite thread gives an idea of uptime (elapsed):

"VM Periodic Task Thread" os_prio=0 cpu=21755.03ms elapsed=100153.98s tid=0x0000
7f1b2c565000 nid=0x5016 waiting on condition

Bill

On 12/3/19 11:34 AM, Joakim Erdfelt wrote:
9.4.12 is subject to many security issues.


Joakim Erdfelt / [hidden email]


On Tue, Dec 3, 2019 at 1:22 PM Bill Ross <[hidden email]> wrote:

For what it's worth, I also just down-versioned from 9.4.24.v20191120 because my server was using 300% CPU with no client activity. I can't rule out my own changes, and a couple of out-of-practice looks at thread dumps didn't give me an answer. But there's nothing I've added that would keep a thread busy like that after startup, and it happens after running a while. So far it hasn't happened on the down rev: 9.4.12.v20180830.

I wonder if there's a thread activity monitor one could add that would warn if a thread seemed runaway..

Bill

On 12/3/19 1:14 AM, Silvio Bierman wrote:
Hi Greg,

At this moment we are receiving multiple error reports from users who suffer from malfunctioning user interfaces. We already had received some of those before the weekend but I did not link this to our move to 9.4.24. Now a pattern is emerging.
They are mostly from Firefox users but some come from Safari users. The symptoms are consistently similar: missing images, unstyled content, parts of content missing etc.

We will probably have to revert to the previous Jetty version we where running (9.4.20) to make sure we do not pick one that behaves the same. In the meantime I would be happy to do any testing if you would require so.

Kind regards,

Silvio

On 12/2/19 2:42 PM, Greg Wilkins wrote:
Silvio,

This is the second time I've heard about a problem fetching browser resources like CSS or js.  Can you attach the stacks you are seeing?

On Mon, 2 Dec 2019, 22:58 Silvio Bierman, <[hidden email]> wrote:
Hi Greg,

The last few days exceptions have started to come up in the logging. We can quite easily reproduce them by testing common parts of our web applications using Firefox. Using Chrome the same actions do not produce exceptions (or warnings).

Strangely enough this seems to intermittently fail mostly (but not exclusively) on plain GET requests for CSS resources that are requested by the browser as a result of an @import from another CSS resource.

We serve the files ourselves from a servlet. Perhaps we are doing something that triggers this? GET requests for which we serve the response content dynamically seem to work fine. The same goes for POSTs. Since it only happens via one of our code paths I suspect we are causing this in some way, although the code is extremely simple.

Kind regards,

Silvio


On 11/29/19 12:27 AM, Greg Wilkins wrote:

Silvio,

I believe it is ignorable and you can turn the HttpChannelState logger level down to suppress them.
However, if there a stack trace associated with that warning then it is not what I think it is and you need to provide more information.

What I believe is happening is that while a request is being processed, the associated HTTP/2 stream is being reset (probably by the client?)
This asynchronous error is detected but because the request is not async, it cannot be delivered to the request and instead we warn.  This is probably over verbose as clients can do silly things like close mid request handling.

[hidden email]  what do you think?

cheers











On Fri, 29 Nov 2019 at 09:03, Silvio Bierman <[hidden email]> wrote:
Hello all,

Ever since upgrading to 9.4.24 our stderr-log is filled with these messages:

2019-11-28 22:56:27.469:WARN:oejs.HttpChannelState:qtp1519100796-25:
org.eclipse.jetty.io.EofException: Reset cancel_stream_error

Can anyone tell me what this means? I take it the situation is not
critical because the application has worked flawlessly for years with
earlier Jetty versions without these messages. Can I turn this off?

Kind regards,

Silvio

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users


--

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users


_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users
_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Jetty 9.4.24 prints warning messages to stderr

Bill Ross-2

I got one replication with new jetty via a big query that seemed to be fine otherwise,

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND   
 9798 phootri+  20   0   10.8g   1.5g  28012 S 100.0   2.5   9:14.86 java 

I updated c3p0, and will file a proper report if I get jetty version-related diff.

Bill

On 12/3/19 12:12 PM, Bill Ross wrote:

Right below in that stack (duh).. it's a thread waiting for a C3P0 db connection. Must have been more tired when I looked before. Any way it could be jetty-related?

        at com.mchange.v2.c3p0.impl.C3P0ImplUtils.allocateIdentityToken(C3P0Impl
Utils.java:192)
        at com.mchange.v2.c3p0.impl.DriverManagerDataSourceBase.<init>(DriverMan
agerDataSourceBase.java:205)
        at com.mchange.v2.c3p0.DriverManagerDataSource.<init>(DriverManagerDataS
ource.java:60)
        at com.mchange.v2.c3p0.DriverManagerDataSource.<init>(DriverManagerDataSource.java:56)
        at com.mchange.v2.c3p0.ComboPooledDataSource.<init>(ComboPooledDataSource.java:113)
...

       at org.eclipse.jetty.jndi.NamingContext.lookup(NamingContext.java:467)
        at org.eclipse.jetty.jndi.NamingContext.lookup(NamingContext.java:491)
        at org.eclipse.jetty.jndi.NamingContext.lookup(NamingContext.java:491)
        at org.eclipse.jetty.jndi.NamingContext.lookup(NamingContext.java:491)
        at org.eclipse.jetty.jndi.NamingContext.lookup(NamingContext.java:505)
        at org.eclipse.jetty.jndi.java.javaRootURLContext.lookup(javaRootURLContext.java:101)
        at javax.naming.InitialContext.lookup([hidden email])
        at com.priot.db.dao.DaoBase.getConn(DaoBase.java:30)

On 12/3/19 11:58 AM, Bill Ross wrote:

Thanks, good to know - so far the downrev is just on the home server, but will have to consider whether to dig deeper when I'm ready to push again. DoS at least isn't much of a factor for my deliberately-avoided-by-all site. :-)

Here's a peek at threads with lots of cpu on them, in case it's on the jetty side:

"qtp1008925772-146" #146 prio=5 os_prio=0 cpu=648389.04ms elapsed=711.23s tid=0x
00007f1a1c04b000 nid=0x6eed runnable  [0x00007f1af52e9000]
   java.lang.Thread.State: RUNNABLE
        at java.util.WeakHashMap.get([hidden email])
        at com.mchange.v2.encounter.AbstractEncounterCounter.encounter(AbstractE
ncounterCounter.java:41)

"qtp1008925772-145" #145 prio=5 os_prio=0 cpu=546854.46ms elapsed=772.97s tid=0x
00007f1a98369800 nid=0x6ecf runnable  [0x00007f1a5b0f1000]
   java.lang.Thread.State: RUNNABLE
        at java.util.WeakHashMap.get([hidden email])
        at com.mchange.v2.encounter.AbstractEncounterCounter.encounter(AbstractE
ncounterCounter.java:41)

"qtp1008925772-140" #140 prio=5 os_prio=0 cpu=601687.05ms elapsed=1112.57s tid=0
x00007f1ab00ed800 nid=0x6e15 runnable  [0x00007f1a5b3f2000]
   java.lang.Thread.State: RUNNABLE
        at java.util.WeakHashMap.get([hidden email])

"qtp1008925772-102" #102 prio=5 os_prio=0 cpu=7268038.26ms elapsed=7496.99s tid=
0x00007f1a3412e000 nid=0x59eb runnable  [0x00007f1af55ee000]
   java.lang.Thread.State: RUNNABLE
        at java.util.WeakHashMap.get([hidden email])

Whereas this polite thread gives an idea of uptime (elapsed):

"VM Periodic Task Thread" os_prio=0 cpu=21755.03ms elapsed=100153.98s tid=0x0000
7f1b2c565000 nid=0x5016 waiting on condition

Bill

On 12/3/19 11:34 AM, Joakim Erdfelt wrote:
9.4.12 is subject to many security issues.


Joakim Erdfelt / [hidden email]


On Tue, Dec 3, 2019 at 1:22 PM Bill Ross <[hidden email]> wrote:

For what it's worth, I also just down-versioned from 9.4.24.v20191120 because my server was using 300% CPU with no client activity. I can't rule out my own changes, and a couple of out-of-practice looks at thread dumps didn't give me an answer. But there's nothing I've added that would keep a thread busy like that after startup, and it happens after running a while. So far it hasn't happened on the down rev: 9.4.12.v20180830.

I wonder if there's a thread activity monitor one could add that would warn if a thread seemed runaway..

Bill

On 12/3/19 1:14 AM, Silvio Bierman wrote:
Hi Greg,

At this moment we are receiving multiple error reports from users who suffer from malfunctioning user interfaces. We already had received some of those before the weekend but I did not link this to our move to 9.4.24. Now a pattern is emerging.
They are mostly from Firefox users but some come from Safari users. The symptoms are consistently similar: missing images, unstyled content, parts of content missing etc.

We will probably have to revert to the previous Jetty version we where running (9.4.20) to make sure we do not pick one that behaves the same. In the meantime I would be happy to do any testing if you would require so.

Kind regards,

Silvio

On 12/2/19 2:42 PM, Greg Wilkins wrote:
Silvio,

This is the second time I've heard about a problem fetching browser resources like CSS or js.  Can you attach the stacks you are seeing?

On Mon, 2 Dec 2019, 22:58 Silvio Bierman, <[hidden email]> wrote:
Hi Greg,

The last few days exceptions have started to come up in the logging. We can quite easily reproduce them by testing common parts of our web applications using Firefox. Using Chrome the same actions do not produce exceptions (or warnings).

Strangely enough this seems to intermittently fail mostly (but not exclusively) on plain GET requests for CSS resources that are requested by the browser as a result of an @import from another CSS resource.

We serve the files ourselves from a servlet. Perhaps we are doing something that triggers this? GET requests for which we serve the response content dynamically seem to work fine. The same goes for POSTs. Since it only happens via one of our code paths I suspect we are causing this in some way, although the code is extremely simple.

Kind regards,

Silvio


On 11/29/19 12:27 AM, Greg Wilkins wrote:

Silvio,

I believe it is ignorable and you can turn the HttpChannelState logger level down to suppress them.
However, if there a stack trace associated with that warning then it is not what I think it is and you need to provide more information.

What I believe is happening is that while a request is being processed, the associated HTTP/2 stream is being reset (probably by the client?)
This asynchronous error is detected but because the request is not async, it cannot be delivered to the request and instead we warn.  This is probably over verbose as clients can do silly things like close mid request handling.

[hidden email]  what do you think?

cheers











On Fri, 29 Nov 2019 at 09:03, Silvio Bierman <[hidden email]> wrote:
Hello all,

Ever since upgrading to 9.4.24 our stderr-log is filled with these messages:

2019-11-28 22:56:27.469:WARN:oejs.HttpChannelState:qtp1519100796-25:
org.eclipse.jetty.io.EofException: Reset cancel_stream_error

Can anyone tell me what this means? I take it the situation is not
critical because the application has worked flawlessly for years with
earlier Jetty versions without these messages. Can I turn this off?

Kind regards,

Silvio

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users


--

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users


_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users
_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users