[jetty-dev] Question regarding Websocket and MappedByteBufferPool behavior

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

[jetty-dev] Question regarding Websocket and MappedByteBufferPool behavior

Ofir Prizat
Hi,

I wanted to ask about how MappedByteBufferPool behaves when held by WebSocketClient. 

From what I can see, Jetty is instantiating a MappedByteBufferPool which serves as a buffer that is used by all WebSocketClient threads. 

Using heapdump/threaddump analysis I can see that websocket messages that are sent from clients (inbound messages) are stored in that buffer. I can also see that there are multiple threads in RUNNABLE state, which point to that buffer. However, when those clients disconnect the websocket, I can see those WebScocket threads change their state to TIMED_WAITING (All but onem which stays RUNNABLE) but the buffer does not clear itself up nor gets cleared by GC, because there is still a single thread pointing to it, and eventually causing our application to run out of memory.

Is this the expected behavior when working with websocket? Are we doing something wrong? Is there some way to configure that BufferPool or the buffers inside, so they are cleared once a connection is closed?

Our app is running on Jetty 9.4.18 via Spring-Boot 1.5.21.

Attached some screenshots.

Many thanks in advance,

Ofir Prizat
Senior Soft. Developer
Liveperson Inc.

_______________________________________________
jetty-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-dev

Screen Shot 2019-07-30 at 12.11.17.png (203K) Download Attachment
Screen Shot 2019-07-30 at 12.19.44.png (445K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [jetty-dev] Question regarding Websocket and MappedByteBufferPool behavior

Simone Bordet-3
Hi,

On Tue, Jul 30, 2019 at 11:21 AM Ofir Prizat <[hidden email]> wrote:
>
> Hi,
>
> I wanted to ask about how MappedByteBufferPool behaves when held by WebSocketClient.
>
> From what I can see, Jetty is instantiating a MappedByteBufferPool which serves as a buffer that is used by all WebSocketClient threads.

It does not serve as a "buffer" but as a "pool".

> Using heapdump/threaddump analysis I can see that websocket messages that are sent from clients (inbound messages) are stored in that buffer. I can also see that there are multiple threads in RUNNABLE state, which point to that buffer. However, when those clients disconnect the websocket, I can see those WebScocket threads change their state to TIMED_WAITING (All but onem which stays RUNNABLE) but the buffer does not clear itself up nor gets cleared by GC, because there is still a single thread pointing to it, and eventually causing our application to run out of memory.
>

It is normal that WebSocket client references MappedByteBufferPool so
that even when your system is idle, they stay alive.
They are long lived components,  and until you stop the WebSocket
client (or shut down the JVM), you will have them around.

I have doubts that is the MappedByteBufferPool that is causing your
application to run out of memory.
It's a pool and we reuse the ByteBuffers it contains and don't
allocate more than it's needed.

Your screenshots do not show any evidence of your memory problems, so
it's difficult to say, but I would first double check your application
for memory leaks.

--
Simone Bordet
----
http://cometd.org
http://webtide.com
Developer advice, training, services and support
from the Jetty & CometD experts.
_______________________________________________
jetty-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-dev
Reply | Threaded
Open this post in threaded view
|

Re: [jetty-dev] Question regarding Websocket and MappedByteBufferPool behavior

Ofir Prizat
Hi Simone,

Thanks for the reply.

It is totally understandable that the pool components are long lived. What we do not understand is how come the ByteBufferPool size only keeps on increasing over time, even though Websocket connections come and go.

What could be the trigger for the allocation of more and more buffers over time? Is there any scenario/timing they should be cleared / garbage collected?

Our app handles hundreds of websocket connection which come and go every day, and looking at the heapdump (taken every morning) we can see that the ByteBuffer only grows, until eventually reaches out of memory.

How can we avoid this outcome?

Thanks again,
Ofir

On Tue, Jul 30, 2019 at 12:33 PM Simone Bordet <[hidden email]> wrote:
Hi,

On Tue, Jul 30, 2019 at 11:21 AM Ofir Prizat <[hidden email]> wrote:
>
> Hi,
>
> I wanted to ask about how MappedByteBufferPool behaves when held by WebSocketClient.
>
> From what I can see, Jetty is instantiating a MappedByteBufferPool which serves as a buffer that is used by all WebSocketClient threads.

It does not serve as a "buffer" but as a "pool".

> Using heapdump/threaddump analysis I can see that websocket messages that are sent from clients (inbound messages) are stored in that buffer. I can also see that there are multiple threads in RUNNABLE state, which point to that buffer. However, when those clients disconnect the websocket, I can see those WebScocket threads change their state to TIMED_WAITING (All but onem which stays RUNNABLE) but the buffer does not clear itself up nor gets cleared by GC, because there is still a single thread pointing to it, and eventually causing our application to run out of memory.
>

It is normal that WebSocket client references MappedByteBufferPool so
that even when your system is idle, they stay alive.
They are long lived components,  and until you stop the WebSocket
client (or shut down the JVM), you will have them around.

I have doubts that is the MappedByteBufferPool that is causing your
application to run out of memory.
It's a pool and we reuse the ByteBuffers it contains and don't
allocate more than it's needed.

Your screenshots do not show any evidence of your memory problems, so
it's difficult to say, but I would first double check your application
for memory leaks.

--
Simone Bordet
----
http://cometd.org
http://webtide.com
Developer advice, training, services and support
from the Jetty & CometD experts.
_______________________________________________
jetty-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-dev

_______________________________________________
jetty-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-dev
Reply | Threaded
Open this post in threaded view
|

Re: [jetty-dev] Question regarding Websocket and MappedByteBufferPool behavior

Simone Bordet-3
Hi,

On Tue, Jul 30, 2019 at 1:05 PM Ofir Prizat <[hidden email]> wrote:

>
> Hi Simone,
>
> Thanks for the reply.
>
> It is totally understandable that the pool components are long lived. What we do not understand is how come the ByteBufferPool size only keeps on increasing over time, even though Websocket connections come and go.
>
> What could be the trigger for the allocation of more and more buffers over time? Is there any scenario/timing they should be cleared / garbage collected?
>
> Our app handles hundreds of websocket connection which come and go every day, and looking at the heapdump (taken every morning) we can see that the ByteBuffer only grows, until eventually reaches out of memory.

So you may be hitting this other bug:
https://github.com/eclipse/jetty.project/issues/3871

But AFAIK, MappedByteBufferPool does not leak, so again it's something
that we need proof of - just you saying that MappedByteBufferPool
retains memory until OOME is not enough.

MappedByteBufferPool retained memory can be constrained via
MappedByteBufferPool.maxHeapMemory and
MappedByteBufferPool.maxDirectMemory.

--
Simone Bordet
----
http://cometd.org
http://webtide.com
Developer advice, training, services and support
from the Jetty & CometD experts.
_______________________________________________
jetty-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-dev