[jetty-dev] Invalid Ping Frame Rate with Jetty 9.4.21 v20190926

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

[jetty-dev] Invalid Ping Frame Rate with Jetty 9.4.21 v20190926

Michele Rossi
hi all,
I recently upgraded my system to the latest Jetty build, 9.4.21.v20190926/, and since then we started seeing loads of error messages like the one reported below.

I am now reverting to 9.4.20, will report to the mailing list if that fixes the issue.

thanks,
Michele




org.eclipse.jetty.io.RuntimeIOException: java.io.IOException: 11/invalid_ping_frame_rate
    at org.eclipse.jetty.server.Request.extractFormParameters(Request.java:573)
    at org.eclipse.jetty.server.Request.extractContentParameters(Request.java:519)
    at org.eclipse.jetty.server.Request.getParameters(Request.java:430)
    at org.eclipse.jetty.server.Request.getParameterMap(Request.java:1068)
    at javax.servlet.ServletRequestWrapper.getParameterMap(ServletRequestWrapper.java:203)
    at org.jboss.resteasy.plugins.server.servlet.HttpServletInputMessage.getDecodedFormParameters(HttpServletInputMessage.java:156)
    at org.jboss.resteasy.core.FormParamInjector.inject(FormParamInjector.java:37)
    at org.jboss.resteasy.core.MethodInjectorImpl.lambda$injectArguments$1(MethodInjectorImpl.java:96)
    at java.base/java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1106)
    at java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2235)
    at java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:143)
    at org.jboss.resteasy.core.MethodInjectorImpl.injectArguments(MethodInjectorImpl.java:96)
    at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:121)
    at org.jboss.resteasy.core.ResourceMethodInvoker.internalInvokeOnTarget(ResourceMethodInvoker.java:594)
    at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTargetAfterFilter(ResourceMethodInvoker.java:468)
    at org.jboss.resteasy.core.ResourceMethodInvoker.lambda$invokeOnTarget$2(ResourceMethodInvoker.java:421)
    at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:363)
    at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:423)
    at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:391)
    at org.jboss.resteasy.core.ResourceMethodInvoker.lambda$invoke$1(ResourceMethodInvoker.java:365)
    at java.base/java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1106)
    at java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2235)
    at java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:143)
    at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:365)
    at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:477)
    at org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:252)
    at org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:153)
    at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:363)
    at org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:156)
    at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:238)
    at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:249)
    at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:60)
    at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:760)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1617)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:89)
    at com.iontrading.iod.guice.RequestIpFilter.doFilter(RequestIpFilter.java:17)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
    at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:121)
    at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:536)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1589)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1296)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1559)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1211)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
    at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221)
    at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
    at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
    at org.eclipse.jetty.webapp.logging.ContextLogHandler.handle(ContextLogHandler.java:62)
    at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:724)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
    at org.eclipse.jetty.server.Server.handle(Server.java:500)
    at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:386)
    at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:560)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:378)
    at org.eclipse.jetty.http2.server.HttpChannelOverHTTP2.handleWithContext(HttpChannelOverHTTP2.java:339)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:782)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:914)
    at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.io.IOException: 11/invalid_ping_frame_rate
    at org.eclipse.jetty.http2.HTTP2Session.onConnectionFailure(HTTP2Session.java:513)
    at org.eclipse.jetty.http2.HTTP2Session.onConnectionFailure(HTTP2Session.java:508)
    at org.eclipse.jetty.http2.parser.Parser$Listener$Wrapper.onConnectionFailure(Parser.java:414)
    at org.eclipse.jetty.http2.HTTP2Connection$ParserListener.onConnectionFailure(HTTP2Connection.java:384)
    at org.eclipse.jetty.http2.parser.BodyParser.notifyConnectionFailure(BodyParser.java:223)
    at org.eclipse.jetty.http2.parser.BodyParser.connectionFailure(BodyParser.java:215)
    at org.eclipse.jetty.http2.parser.PingBodyParser.onPing(PingBodyParser.java:99)
    at org.eclipse.jetty.http2.parser.PingBodyParser.parse(PingBodyParser.java:69)
    at org.eclipse.jetty.http2.parser.Parser.parseBody(Parser.java:198)
    at org.eclipse.jetty.http2.parser.Parser.parse(Parser.java:127)
    at org.eclipse.jetty.http2.parser.ServerParser.parse(ServerParser.java:115)
    at org.eclipse.jetty.http2.HTTP2Connection$HTTP2Producer.produce(HTTP2Connection.java:248)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:360)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:184)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:135)
    at org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:170)
    at org.eclipse.jetty.http2.HTTP2Connection.onFillable(HTTP2Connection.java:125)
    at org.eclipse.jetty.http2.HTTP2Connection$FillableCallback.succeeded(HTTP2Connection.java:348)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
    at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:426)
    at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:320)
    at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:158)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
    at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
    at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:367)
     ... 3 common frames omitted 

_______________________________________________
jetty-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-dev
Reply | Threaded
Open this post in threaded view
|

Re: [jetty-dev] Invalid Ping Frame Rate with Jetty 9.4.21 v20190926

Simone Bordet-3
Hi.

On Fri, Oct 18, 2019 at 3:32 PM Michele Rossi <[hidden email]> wrote:
>
> hi all,
> I recently upgraded my system to the latest Jetty build, 9.4.21.v20190926/, and since then we started seeing loads of error messages like the one reported below.

This addressed a potential security vulnerability with HTTP/2.
The clients are collectively sending more than 20 pings/s.
How many active connections do you have?
What clients are connected?

> I am now reverting to 9.4.20, will report to the mailing list if that fixes the issue.

I guess it will.
9.4.22 will have this parameter easily configurable in *.ini files.

--
Simone Bordet
----
http://cometd.org
http://webtide.com
Developer advice, training, services and support
from the Jetty & CometD experts.
_______________________________________________
jetty-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-dev
Reply | Threaded
Open this post in threaded view
|

Re: [jetty-dev] Invalid Ping Frame Rate with Jetty 9.4.21 v20190926

Michele Rossi
hi,
I am not sure how many clients we had connected but probably more than 20.

Each client keeps a cometd long poll connection open that probably meant loads of http2 connections.

Yes reverting to the previous build fixed it.
If this becomes configurable how do I know what number to set it to in advance?

What will the configuration parameter be called?


thanks a lot,
Michele





On Fri, 18 Oct 2019 at 19:45, Simone Bordet <[hidden email]> wrote:
Hi.

On Fri, Oct 18, 2019 at 3:32 PM Michele Rossi <[hidden email]> wrote:
>
> hi all,
> I recently upgraded my system to the latest Jetty build, 9.4.21.v20190926/, and since then we started seeing loads of error messages like the one reported below.

This addressed a potential security vulnerability with HTTP/2.
The clients are collectively sending more than 20 pings/s.
How many active connections do you have?
What clients are connected?

> I am now reverting to 9.4.20, will report to the mailing list if that fixes the issue.

I guess it will.
9.4.22 will have this parameter easily configurable in *.ini files.

--
Simone Bordet
----
http://cometd.org
http://webtide.com
Developer advice, training, services and support
from the Jetty & CometD experts.
_______________________________________________
jetty-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-dev
--
Sent from Gmail Mobile

_______________________________________________
jetty-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-dev
Reply | Threaded
Open this post in threaded view
|

Re: [jetty-dev] Invalid Ping Frame Rate with Jetty 9.4.21 v20190926

Simone Bordet-3
Hi,

On Fri, Oct 18, 2019 at 7:53 PM Michele Rossi <[hidden email]> wrote:
>
> hi,
> I am not sure how many clients we had connected but probably more than 20.

Still, having each one sending a PING every second or so looks like a
DDoS attack.

> Each client keeps a cometd long poll connection open that probably meant loads of http2 connections.

Well, not "loads", just 1 per client.

> Yes reverting to the previous build fixed it.
> If this becomes configurable how do I know what number to set it to in advance?

You should first understand who's sending all those PINGs as it really
looks like an attack.
I would imagine a client needing to send a PING every, say, 1 minute.
To exceed the default limit in this case you'd need 1200 clients.
Do you have such numbers?

--
Simone Bordet
----
http://cometd.org
http://webtide.com
Developer advice, training, services and support
from the Jetty & CometD experts.
_______________________________________________
jetty-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-dev
Reply | Threaded
Open this post in threaded view
|

Re: [jetty-dev] Invalid Ping Frame Rate with Jetty 9.4.21 v20190926

Michele Rossi
I would seriously doubt that there were 1200 clients connected.
Is it possible that some older browser sends too many pings?

Is there an easy way to find out how many clients are connected?

We count the number of sessions but that is a different thing in a way.

thanks,
Michele

On Fri, 18 Oct 2019 at 20:18, Simone Bordet <[hidden email]> wrote:
Hi,

On Fri, Oct 18, 2019 at 7:53 PM Michele Rossi <[hidden email]> wrote:
>
> hi,
> I am not sure how many clients we had connected but probably more than 20.

Still, having each one sending a PING every second or so looks like a
DDoS attack.

> Each client keeps a cometd long poll connection open that probably meant loads of http2 connections.

Well, not "loads", just 1 per client.

> Yes reverting to the previous build fixed it.
> If this becomes configurable how do I know what number to set it to in advance?

You should first understand who's sending all those PINGs as it really
looks like an attack.
I would imagine a client needing to send a PING every, say, 1 minute.
To exceed the default limit in this case you'd need 1200 clients.
Do you have such numbers?

--
Simone Bordet
----
http://cometd.org
http://webtide.com
Developer advice, training, services and support
from the Jetty & CometD experts.
_______________________________________________
jetty-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-dev
--
Sent from Gmail Mobile

_______________________________________________
jetty-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-dev
Reply | Threaded
Open this post in threaded view
|

Re: [jetty-dev] Invalid Ping Frame Rate with Jetty 9.4.21 v20190926

Simone Bordet-3
Hi,

On Fri, Oct 18, 2019 at 8:31 PM Michele Rossi <[hidden email]> wrote:
>
> I would seriously doubt that there were 1200 clients connected.
> Is it possible that some older browser sends too many pings?

I doubt this many per seconds.

> Is there an easy way to find out how many clients are connected?

Jetty has JMX statistics for that.

> We count the number of sessions but that is a different thing in a way.

That would be a good approximation.

--
Simone Bordet
----
http://cometd.org
http://webtide.com
Developer advice, training, services and support
from the Jetty & CometD experts.
_______________________________________________
jetty-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-dev