[jetty-users] DirectByteBuffer native memory problems (OOM) with Jetty 9.0/9.1

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

[jetty-users] DirectByteBuffer native memory problems (OOM) with Jetty 9.0/9.1

Emmeran Seehuber
Hello everybody,

I´m using jetty as standalone embedded web server behind a nginx and it is working great so far. Recently I upgraded from 7.6.4 to 9.0 and two days ago to 9.1.0.v20131115.

Since my upgrade to 9.0 i experience native memory leaks. Also with 9.1 i still have the same leaks. Native memory leaks mean, that the process memory size grows very large and then I get OOM Exceptions for DirectByteBuffers. But to get there the server process has to run for about two weeks.

Since the process has a 7 GB heap, the MaxDirectMemory Limit also seems to be 7 GB.

First i thought that the many threads started/stopped in the thread pool are the problem. I specified a idle time of -1, and still many threads are restarted. Also some answers on the web about DirectByteBuffer OOM Exceptions pointed into this direction. But i don’t think thats the problem.

I think the problem may be in ArrayByteBufferPool.  It allocates an unbound number of ByteBuffers and caches them in a unbound ConcurrentLinkedQueue. So at some peak times many buffers are used and never freed.  On the other side this doesn’t really fit with the allocated memory. At the moment the server has this direct memory allocations (according to the MBeans info):

java.nio
BufferPool
name=direct
-  Count 384
-  MemoryUsed 2728821147
-  Name direct
-  ObjectName java.nio:type=BufferPool,name=direct
-  TotalCapacity 2728821147

That would mean an medium buffer size of about 7 MB. But the ArrayByteBufferPool does, as far as i understand, only allocate a max of 64 kb sized buffers by default ?!

Current threads are: Count = 230, Maximum = 233, Started overall = 599

This numbers are after the process is running for about 32 hours. I´ve got about 320 request/minute at peak times. All this numbers are according to the embedded JavaMelody monitoring.  

The JVM arguments are:
-server -XX:+UseCompressedOops -XX:MaxPermSize=512m -Xms7000M -Xmx7000M -XX:+UseParallelOldGC -XX:+DoEscapeAnalysis -XX:+OptimizeStringConcat
JDK 1.7.0_45

The thread pool is configured this way:

tp = new QueuedThreadPool();
tp.setMaxThreads(256);
tp.setMinThreads(5);
tp.setIdleTimeout(-1);
tp.setName("Http Server Thread Pool");
tp.setDaemon(true);

I had no such problems with jetty 7.x. Regularly calling System.gc() every hour does not help. Also the i´ve got no other GC problems.

Any ideas what could cause this problems?
Why are threads restarted in the pool even when i specify a idle timeout of -1? I also regulary schedule some own runnables on the thread pool - but they should not cause this problems, should they? They do in 99.99% of all cases not throw any exceptions.

Thanks.

cu,
 Emmy
--
Mit freundlichen Grüßen aus Augsburg,

Emmeran Seehuber
Dipl. Inf. (FH)
Schrannenstraße 8
86150 Augsburg
USt-IdNr.: DE266070804

Mit freundlichen Grüßen aus Augsburg,

Emmeran Seehuber
Dipl. Inf. (FH)
Schrannenstraße 8
86150 Augsburg
USt-IdNr.: DE266070804


_______________________________________________
jetty-users mailing list
[hidden email]
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: [jetty-users] DirectByteBuffer native memory problems (OOM) with Jetty 9.0/9.1

Simone Bordet-2
Hi,

On Thu, Nov 21, 2013 at 2:47 PM, Emmeran Seehuber <[hidden email]> wrote:

> Hello everybody,
>
> I´m using jetty as standalone embedded web server behind a nginx and it is
> working great so far. Recently I upgraded from 7.6.4 to 9.0 and two days ago
> to 9.1.0.v20131115.
>
> Since my upgrade to 9.0 i experience native memory leaks. Also with 9.1 i
> still have the same leaks. Native memory leaks mean, that the process memory
> size grows very large and then I get OOM Exceptions for DirectByteBuffers.
> But to get there the server process has to run for about two weeks.
>
> Since the process has a 7 GB heap, the MaxDirectMemory Limit also seems to
> be 7 GB.
>
> First i thought that the many threads started/stopped in the thread pool are
> the problem. I specified a idle time of -1, and still many threads are
> restarted. Also some answers on the web about DirectByteBuffer OOM
> Exceptions pointed into this direction. But i don’t think thats the problem.
>
> I think the problem may be in ArrayByteBufferPool.  It allocates an unbound
> number of ByteBuffers and caches them in a unbound ConcurrentLinkedQueue. So
> at some peak times many buffers are used and never freed.  On the other side
> this doesn’t really fit with the allocated memory. At the moment the server
> has this direct memory allocations (according to the MBeans info):
>
> java.nio
> BufferPool
> name=direct
> -  Count 384
> -  MemoryUsed 2728821147
> -  Name direct
> -  ObjectName java.nio:type=BufferPool,name=direct
> -  TotalCapacity 2728821147
>
> That would mean an medium buffer size of about 7 MB. But the
> ArrayByteBufferPool does, as far as i understand, only allocate a max of 64
> kb sized buffers by default ?!
>
> Current threads are: Count = 230, Maximum = 233, Started overall = 599
>
> This numbers are after the process is running for about 32 hours. I´ve got
> about 320 request/minute at peak times. All this numbers are according to
> the embedded JavaMelody monitoring.
>
> The JVM arguments are:
> -server -XX:+UseCompressedOops -XX:MaxPermSize=512m -Xms7000M -Xmx7000M
> -XX:+UseParallelOldGC -XX:+DoEscapeAnalysis -XX:+OptimizeStringConcat
> JDK 1.7.0_45
>
> The thread pool is configured this way:
>
> tp = new QueuedThreadPool();
> tp.setMaxThreads(256);
> tp.setMinThreads(5);
> tp.setIdleTimeout(-1);
> tp.setName("Http Server Thread Pool");
> tp.setDaemon(true);
>
> I had no such problems with jetty 7.x. Regularly calling System.gc() every
> hour does not help. Also the i´ve got no other GC problems.
>
> Any ideas what could cause this problems?
> Why are threads restarted in the pool even when i specify a idle timeout of
> -1? I also regulary schedule some own runnables on the thread pool - but
> they should not cause this problems, should they? They do in 99.99% of all
> cases not throw any exceptions.

Seems like a bug.
I would keep the ThreadPool at default configuration for now, to have
less variables in the system.

Would you be able to replace usage of ArrayByteBufferPool with
MappedByteBufferPool and see if the problem persist ?

Also, please file an issue about this, it really looks from JMX that
there is a problem.

Thanks !

--
Simone Bordet
----
http://cometd.org
http://webtide.com
http://intalio.com
Developer advice, training, services and support
from the Jetty & CometD experts.
Intalio, the modern way to build business applications.
_______________________________________________
jetty-users mailing list
[hidden email]
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: [jetty-users] DirectByteBuffer native memory problems (OOM) with Jetty 9.0/9.1

Jan Bartel-3
And when you open the bug, please make sure to state which operating
system and version you are using.

thanks
Jan

On 24 November 2013 09:55, Simone Bordet <[hidden email]> wrote:

> Hi,
>
> On Thu, Nov 21, 2013 at 2:47 PM, Emmeran Seehuber <[hidden email]> wrote:
>> Hello everybody,
>>
>> I´m using jetty as standalone embedded web server behind a nginx and it is
>> working great so far. Recently I upgraded from 7.6.4 to 9.0 and two days ago
>> to 9.1.0.v20131115.
>>
>> Since my upgrade to 9.0 i experience native memory leaks. Also with 9.1 i
>> still have the same leaks. Native memory leaks mean, that the process memory
>> size grows very large and then I get OOM Exceptions for DirectByteBuffers.
>> But to get there the server process has to run for about two weeks.
>>
>> Since the process has a 7 GB heap, the MaxDirectMemory Limit also seems to
>> be 7 GB.
>>
>> First i thought that the many threads started/stopped in the thread pool are
>> the problem. I specified a idle time of -1, and still many threads are
>> restarted. Also some answers on the web about DirectByteBuffer OOM
>> Exceptions pointed into this direction. But i don’t think thats the problem.
>>
>> I think the problem may be in ArrayByteBufferPool.  It allocates an unbound
>> number of ByteBuffers and caches them in a unbound ConcurrentLinkedQueue. So
>> at some peak times many buffers are used and never freed.  On the other side
>> this doesn’t really fit with the allocated memory. At the moment the server
>> has this direct memory allocations (according to the MBeans info):
>>
>> java.nio
>> BufferPool
>> name=direct
>> -  Count 384
>> -  MemoryUsed 2728821147
>> -  Name direct
>> -  ObjectName java.nio:type=BufferPool,name=direct
>> -  TotalCapacity 2728821147
>>
>> That would mean an medium buffer size of about 7 MB. But the
>> ArrayByteBufferPool does, as far as i understand, only allocate a max of 64
>> kb sized buffers by default ?!
>>
>> Current threads are: Count = 230, Maximum = 233, Started overall = 599
>>
>> This numbers are after the process is running for about 32 hours. I´ve got
>> about 320 request/minute at peak times. All this numbers are according to
>> the embedded JavaMelody monitoring.
>>
>> The JVM arguments are:
>> -server -XX:+UseCompressedOops -XX:MaxPermSize=512m -Xms7000M -Xmx7000M
>> -XX:+UseParallelOldGC -XX:+DoEscapeAnalysis -XX:+OptimizeStringConcat
>> JDK 1.7.0_45
>>
>> The thread pool is configured this way:
>>
>> tp = new QueuedThreadPool();
>> tp.setMaxThreads(256);
>> tp.setMinThreads(5);
>> tp.setIdleTimeout(-1);
>> tp.setName("Http Server Thread Pool");
>> tp.setDaemon(true);
>>
>> I had no such problems with jetty 7.x. Regularly calling System.gc() every
>> hour does not help. Also the i´ve got no other GC problems.
>>
>> Any ideas what could cause this problems?
>> Why are threads restarted in the pool even when i specify a idle timeout of
>> -1? I also regulary schedule some own runnables on the thread pool - but
>> they should not cause this problems, should they? They do in 99.99% of all
>> cases not throw any exceptions.
>
> Seems like a bug.
> I would keep the ThreadPool at default configuration for now, to have
> less variables in the system.
>
> Would you be able to replace usage of ArrayByteBufferPool with
> MappedByteBufferPool and see if the problem persist ?
>
> Also, please file an issue about this, it really looks from JMX that
> there is a problem.
>
> Thanks !
>
> --
> Simone Bordet
> ----
> http://cometd.org
> http://webtide.com
> http://intalio.com
> Developer advice, training, services and support
> from the Jetty & CometD experts.
> Intalio, the modern way to build business applications.
> _______________________________________________
> jetty-users mailing list
> [hidden email]
> https://dev.eclipse.org/mailman/listinfo/jetty-users



--
Jan Bartel <[hidden email]>
www.webtide.com
'Expert Jetty/CometD developer,production,operations advice'
_______________________________________________
jetty-users mailing list
[hidden email]
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: [jetty-users] DirectByteBuffer native memory problems (OOM) with Jetty 9.0/9.1

Emmeran Seehuber
Hi,

I already found the cause for this problem: I do big writes.

The user upload wedding photos (the software is a web2print wedding card shop), and I echo them back to the user. To do so I read them fully into a byte[] and then do one big

request.getOutputStream().write(data);

As the images vary in size from 3 - 20+ MB, so do the byte[]…

Jetty internally just wraps the byte[] into a ByteBuffer and passes it to NIO. There it finally lands in

IOUtil.write()
http://www.docjar.com/html/api/sun/nio/ch/IOUtil.java.html

If the ByteBuffer is a DirectBuffer, everything is fine, and the whole thing is passed to the native write. But if it is not, it allocates a temporary DirectByteBuffer of the full size of the ByteBuffer using

Util.getTemporaryDirectBuffer
http://www.docjar.com/html/api/sun/nio/ch/Util.java.html

The problem is, this DirectByteBuffer are then after the write cached in a thread local pool of 8 such DirectByteBuffer.  The pool keeps the biggest 8 requested buffers.

If you now have about 200+ live threads and any of them can do a 8, 10, … MB write, then you end up using much direct memory. Over time this pool just grows and grows ….

After trying to avoid reading the image data into a byte[] as much as possible and if i have to read the image fully into a byte[] just doing chunked writes of the byte buffer, the problem is gone. I.e. i do now a

                ServletOutputStream outputStream = res.getOutputStream();
                int i = 0;
                final int MAX_BYTES_TO_SEND = 64 * 1024;
                while (i < data.length) {
                        final int maxSendSize = Math.min(MAX_BYTES_TO_SEND, data.length - i);
                        outputStream.write(data, i, maxSendSize);
                        i += maxSendSize;
                }

and everything is fine.

Of course, this was no problem with Jetty 7, because there I did not use the NIO connector there.

I am not sure if you can consider this a bug of Jetty, or if this is rather a bug of the JDK. At least it is very unexpected.

Maybe Jetty should workaround this buggy behavior and just always use its internal DirectByteBuffers and do chunked writes with DirectByteBuffers itself?

I´m not sure if the JDK behavior gives any speed advantages if you cache DirectByteBuffers bigger then a certain size (e.g. 64KB) - or just simple  trashes the CPU cache instead ...

Should I still open a bug for this?

Thanks.

cu,
  Emmy

(BTW: The OS is Linux)

Am 25.11.2013 um 02:37 schrieb Jan Bartel <[hidden email]>:

> And when you open the bug, please make sure to state which operating
> system and version you are using.
>
> thanks
> Jan
>
> On 24 November 2013 09:55, Simone Bordet <[hidden email]> wrote:
>> Hi,
>>
>> On Thu, Nov 21, 2013 at 2:47 PM, Emmeran Seehuber <[hidden email]> wrote:
>>> Hello everybody,
>>>
>>> I´m using jetty as standalone embedded web server behind a nginx and it is
>>> working great so far. Recently I upgraded from 7.6.4 to 9.0 and two days ago
>>> to 9.1.0.v20131115.
>>>
>>> Since my upgrade to 9.0 i experience native memory leaks. Also with 9.1 i
>>> still have the same leaks. Native memory leaks mean, that the process memory
>>> size grows very large and then I get OOM Exceptions for DirectByteBuffers.
>>> But to get there the server process has to run for about two weeks.
>>>
>>> Since the process has a 7 GB heap, the MaxDirectMemory Limit also seems to
>>> be 7 GB.
>>>
>>> First i thought that the many threads started/stopped in the thread pool are
>>> the problem. I specified a idle time of -1, and still many threads are
>>> restarted. Also some answers on the web about DirectByteBuffer OOM
>>> Exceptions pointed into this direction. But i don’t think thats the problem.
>>>
>>> I think the problem may be in ArrayByteBufferPool.  It allocates an unbound
>>> number of ByteBuffers and caches them in a unbound ConcurrentLinkedQueue. So
>>> at some peak times many buffers are used and never freed.  On the other side
>>> this doesn’t really fit with the allocated memory. At the moment the server
>>> has this direct memory allocations (according to the MBeans info):
>>>
>>> java.nio
>>> BufferPool
>>> name=direct
>>> -  Count 384
>>> -  MemoryUsed 2728821147
>>> -  Name direct
>>> -  ObjectName java.nio:type=BufferPool,name=direct
>>> -  TotalCapacity 2728821147
>>>
>>> That would mean an medium buffer size of about 7 MB. But the
>>> ArrayByteBufferPool does, as far as i understand, only allocate a max of 64
>>> kb sized buffers by default ?!
>>>
>>> Current threads are: Count = 230, Maximum = 233, Started overall = 599
>>>
>>> This numbers are after the process is running for about 32 hours. I´ve got
>>> about 320 request/minute at peak times. All this numbers are according to
>>> the embedded JavaMelody monitoring.
>>>
>>> The JVM arguments are:
>>> -server -XX:+UseCompressedOops -XX:MaxPermSize=512m -Xms7000M -Xmx7000M
>>> -XX:+UseParallelOldGC -XX:+DoEscapeAnalysis -XX:+OptimizeStringConcat
>>> JDK 1.7.0_45
>>>
>>> The thread pool is configured this way:
>>>
>>> tp = new QueuedThreadPool();
>>> tp.setMaxThreads(256);
>>> tp.setMinThreads(5);
>>> tp.setIdleTimeout(-1);
>>> tp.setName("Http Server Thread Pool");
>>> tp.setDaemon(true);
>>>
>>> I had no such problems with jetty 7.x. Regularly calling System.gc() every
>>> hour does not help. Also the i´ve got no other GC problems.
>>>
>>> Any ideas what could cause this problems?
>>> Why are threads restarted in the pool even when i specify a idle timeout of
>>> -1? I also regulary schedule some own runnables on the thread pool - but
>>> they should not cause this problems, should they? They do in 99.99% of all
>>> cases not throw any exceptions.
>>
>> Seems like a bug.
>> I would keep the ThreadPool at default configuration for now, to have
>> less variables in the system.
>>
>> Would you be able to replace usage of ArrayByteBufferPool with
>> MappedByteBufferPool and see if the problem persist ?
>>
>> Also, please file an issue about this, it really looks from JMX that
>> there is a problem.
>>
>> Thanks !
>>
>> --
>> Simone Bordet
>> ----
>> http://cometd.org
>> http://webtide.com
>> http://intalio.com
>> Developer advice, training, services and support
>> from the Jetty & CometD experts.
>> Intalio, the modern way to build business applications.
>> _______________________________________________
>> jetty-users mailing list
>> [hidden email]
>> https://dev.eclipse.org/mailman/listinfo/jetty-users
>
>
>
> --
> Jan Bartel <[hidden email]>
> www.webtide.com
> 'Expert Jetty/CometD developer,production,operations advice'
> _______________________________________________
> jetty-users mailing list
> [hidden email]
> https://dev.eclipse.org/mailman/listinfo/jetty-users
>

Mit freundlichen Grüßen aus Augsburg,

Emmeran Seehuber
Dipl. Inf. (FH)
Schrannenstraße 8
86150 Augsburg
USt-IdNr.: DE266070804

_______________________________________________
jetty-users mailing list
[hidden email]
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: [jetty-users] DirectByteBuffer native memory problems (OOM) with Jetty 9.0/9.1

Jan Bartel-3
Hi Emmeran,

This is a known bug in the jdk that has been around for quite a while.
The jetty doc on it is here (see subsection "Direct ByteBuffers"):
http://www.eclipse.org/jetty/documentation/current/preventing-memory-leaks.html#jvm-bugs

Sadly, there is still no fix in the jvm for it. If you've got a good
workaround, I'd go with that.

cheers
Jan

On 26 November 2013 21:10, Emmeran Seehuber <[hidden email]> wrote:

> Hi,
>
> I already found the cause for this problem: I do big writes.
>
> The user upload wedding photos (the software is a web2print wedding card shop), and I echo them back to the user. To do so I read them fully into a byte[] and then do one big
>
> request.getOutputStream().write(data);
>
> As the images vary in size from 3 - 20+ MB, so do the byte[]…
>
> Jetty internally just wraps the byte[] into a ByteBuffer and passes it to NIO. There it finally lands in
>
> IOUtil.write()
> http://www.docjar.com/html/api/sun/nio/ch/IOUtil.java.html
>
> If the ByteBuffer is a DirectBuffer, everything is fine, and the whole thing is passed to the native write. But if it is not, it allocates a temporary DirectByteBuffer of the full size of the ByteBuffer using
>
> Util.getTemporaryDirectBuffer
> http://www.docjar.com/html/api/sun/nio/ch/Util.java.html
>
> The problem is, this DirectByteBuffer are then after the write cached in a thread local pool of 8 such DirectByteBuffer.  The pool keeps the biggest 8 requested buffers.
>
> If you now have about 200+ live threads and any of them can do a 8, 10, … MB write, then you end up using much direct memory. Over time this pool just grows and grows ….
>
> After trying to avoid reading the image data into a byte[] as much as possible and if i have to read the image fully into a byte[] just doing chunked writes of the byte buffer, the problem is gone. I.e. i do now a
>
>                 ServletOutputStream outputStream = res.getOutputStream();
>                 int i = 0;
>                 final int MAX_BYTES_TO_SEND = 64 * 1024;
>                 while (i < data.length) {
>                         final int maxSendSize = Math.min(MAX_BYTES_TO_SEND, data.length - i);
>                         outputStream.write(data, i, maxSendSize);
>                         i += maxSendSize;
>                 }
>
> and everything is fine.
>
> Of course, this was no problem with Jetty 7, because there I did not use the NIO connector there.
>
> I am not sure if you can consider this a bug of Jetty, or if this is rather a bug of the JDK. At least it is very unexpected.
>
> Maybe Jetty should workaround this buggy behavior and just always use its internal DirectByteBuffers and do chunked writes with DirectByteBuffers itself?
>
> I´m not sure if the JDK behavior gives any speed advantages if you cache DirectByteBuffers bigger then a certain size (e.g. 64KB) - or just simple  trashes the CPU cache instead ...
>
> Should I still open a bug for this?
>
> Thanks.
>
> cu,
>   Emmy
>
> (BTW: The OS is Linux)
>
> Am 25.11.2013 um 02:37 schrieb Jan Bartel <[hidden email]>:
>
>> And when you open the bug, please make sure to state which operating
>> system and version you are using.
>>
>> thanks
>> Jan
>>
>> On 24 November 2013 09:55, Simone Bordet <[hidden email]> wrote:
>>> Hi,
>>>
>>> On Thu, Nov 21, 2013 at 2:47 PM, Emmeran Seehuber <[hidden email]> wrote:
>>>> Hello everybody,
>>>>
>>>> I´m using jetty as standalone embedded web server behind a nginx and it is
>>>> working great so far. Recently I upgraded from 7.6.4 to 9.0 and two days ago
>>>> to 9.1.0.v20131115.
>>>>
>>>> Since my upgrade to 9.0 i experience native memory leaks. Also with 9.1 i
>>>> still have the same leaks. Native memory leaks mean, that the process memory
>>>> size grows very large and then I get OOM Exceptions for DirectByteBuffers.
>>>> But to get there the server process has to run for about two weeks.
>>>>
>>>> Since the process has a 7 GB heap, the MaxDirectMemory Limit also seems to
>>>> be 7 GB.
>>>>
>>>> First i thought that the many threads started/stopped in the thread pool are
>>>> the problem. I specified a idle time of -1, and still many threads are
>>>> restarted. Also some answers on the web about DirectByteBuffer OOM
>>>> Exceptions pointed into this direction. But i don’t think thats the problem.
>>>>
>>>> I think the problem may be in ArrayByteBufferPool.  It allocates an unbound
>>>> number of ByteBuffers and caches them in a unbound ConcurrentLinkedQueue. So
>>>> at some peak times many buffers are used and never freed.  On the other side
>>>> this doesn’t really fit with the allocated memory. At the moment the server
>>>> has this direct memory allocations (according to the MBeans info):
>>>>
>>>> java.nio
>>>> BufferPool
>>>> name=direct
>>>> -  Count 384
>>>> -  MemoryUsed 2728821147
>>>> -  Name direct
>>>> -  ObjectName java.nio:type=BufferPool,name=direct
>>>> -  TotalCapacity 2728821147
>>>>
>>>> That would mean an medium buffer size of about 7 MB. But the
>>>> ArrayByteBufferPool does, as far as i understand, only allocate a max of 64
>>>> kb sized buffers by default ?!
>>>>
>>>> Current threads are: Count = 230, Maximum = 233, Started overall = 599
>>>>
>>>> This numbers are after the process is running for about 32 hours. I´ve got
>>>> about 320 request/minute at peak times. All this numbers are according to
>>>> the embedded JavaMelody monitoring.
>>>>
>>>> The JVM arguments are:
>>>> -server -XX:+UseCompressedOops -XX:MaxPermSize=512m -Xms7000M -Xmx7000M
>>>> -XX:+UseParallelOldGC -XX:+DoEscapeAnalysis -XX:+OptimizeStringConcat
>>>> JDK 1.7.0_45
>>>>
>>>> The thread pool is configured this way:
>>>>
>>>> tp = new QueuedThreadPool();
>>>> tp.setMaxThreads(256);
>>>> tp.setMinThreads(5);
>>>> tp.setIdleTimeout(-1);
>>>> tp.setName("Http Server Thread Pool");
>>>> tp.setDaemon(true);
>>>>
>>>> I had no such problems with jetty 7.x. Regularly calling System.gc() every
>>>> hour does not help. Also the i´ve got no other GC problems.
>>>>
>>>> Any ideas what could cause this problems?
>>>> Why are threads restarted in the pool even when i specify a idle timeout of
>>>> -1? I also regulary schedule some own runnables on the thread pool - but
>>>> they should not cause this problems, should they? They do in 99.99% of all
>>>> cases not throw any exceptions.
>>>
>>> Seems like a bug.
>>> I would keep the ThreadPool at default configuration for now, to have
>>> less variables in the system.
>>>
>>> Would you be able to replace usage of ArrayByteBufferPool with
>>> MappedByteBufferPool and see if the problem persist ?
>>>
>>> Also, please file an issue about this, it really looks from JMX that
>>> there is a problem.
>>>
>>> Thanks !
>>>
>>> --
>>> Simone Bordet
>>> ----
>>> http://cometd.org
>>> http://webtide.com
>>> http://intalio.com
>>> Developer advice, training, services and support
>>> from the Jetty & CometD experts.
>>> Intalio, the modern way to build business applications.
>>> _______________________________________________
>>> jetty-users mailing list
>>> [hidden email]
>>> https://dev.eclipse.org/mailman/listinfo/jetty-users
>>
>>
>>
>> --
>> Jan Bartel <[hidden email]>
>> www.webtide.com
>> 'Expert Jetty/CometD developer,production,operations advice'
>> _______________________________________________
>> jetty-users mailing list
>> [hidden email]
>> https://dev.eclipse.org/mailman/listinfo/jetty-users
>>
>
> Mit freundlichen Grüßen aus Augsburg,
>
> Emmeran Seehuber
> Dipl. Inf. (FH)
> Schrannenstraße 8
> 86150 Augsburg
> USt-IdNr.: DE266070804
>
> _______________________________________________
> jetty-users mailing list
> [hidden email]
> https://dev.eclipse.org/mailman/listinfo/jetty-users



--
Jan Bartel <[hidden email]>
www.webtide.com
'Expert Jetty/CometD developer,production,operations advice'
_______________________________________________
jetty-users mailing list
[hidden email]
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: [jetty-users] DirectByteBuffer native memory problems (OOM) with Jetty 9.0/9.1

Greg Wilkins-3
Emmeran,

on reflection, I think this is something we should deal with in jetty.  I have opened:
https://bugs.eclipse.org/bugs/show_bug.cgi?id=422807

and am working on a fix that will break up writes into getBufferSize() blocks, so that large direct buffers are not create/cached.

If you want to write a large direct buffer, you can always create it yourself and use the jetty APIs to do the write (that will be really efficient!).

cheers





On 28 November 2013 10:49, Jan Bartel <[hidden email]> wrote:
Hi Emmeran,

This is a known bug in the jdk that has been around for quite a while.
The jetty doc on it is here (see subsection "Direct ByteBuffers"):
http://www.eclipse.org/jetty/documentation/current/preventing-memory-leaks.html#jvm-bugs

Sadly, there is still no fix in the jvm for it. If you've got a good
workaround, I'd go with that.

cheers
Jan

On 26 November 2013 21:10, Emmeran Seehuber <[hidden email]> wrote:
> Hi,
>
> I already found the cause for this problem: I do big writes.
>
> The user upload wedding photos (the software is a web2print wedding card shop), and I echo them back to the user. To do so I read them fully into a byte[] and then do one big
>
> request.getOutputStream().write(data);
>
> As the images vary in size from 3 - 20+ MB, so do the byte[]…
>
> Jetty internally just wraps the byte[] into a ByteBuffer and passes it to NIO. There it finally lands in
>
> IOUtil.write()
> http://www.docjar.com/html/api/sun/nio/ch/IOUtil.java.html
>
> If the ByteBuffer is a DirectBuffer, everything is fine, and the whole thing is passed to the native write. But if it is not, it allocates a temporary DirectByteBuffer of the full size of the ByteBuffer using
>
> Util.getTemporaryDirectBuffer
> http://www.docjar.com/html/api/sun/nio/ch/Util.java.html
>
> The problem is, this DirectByteBuffer are then after the write cached in a thread local pool of 8 such DirectByteBuffer.  The pool keeps the biggest 8 requested buffers.
>
> If you now have about 200+ live threads and any of them can do a 8, 10, … MB write, then you end up using much direct memory. Over time this pool just grows and grows ….
>
> After trying to avoid reading the image data into a byte[] as much as possible and if i have to read the image fully into a byte[] just doing chunked writes of the byte buffer, the problem is gone. I.e. i do now a
>
>                 ServletOutputStream outputStream = res.getOutputStream();
>                 int i = 0;
>                 final int MAX_BYTES_TO_SEND = 64 * 1024;
>                 while (i < data.length) {
>                         final int maxSendSize = Math.min(MAX_BYTES_TO_SEND, data.length - i);
>                         outputStream.write(data, i, maxSendSize);
>                         i += maxSendSize;
>                 }
>
> and everything is fine.
>
> Of course, this was no problem with Jetty 7, because there I did not use the NIO connector there.
>
> I am not sure if you can consider this a bug of Jetty, or if this is rather a bug of the JDK. At least it is very unexpected.
>
> Maybe Jetty should workaround this buggy behavior and just always use its internal DirectByteBuffers and do chunked writes with DirectByteBuffers itself?
>
> I´m not sure if the JDK behavior gives any speed advantages if you cache DirectByteBuffers bigger then a certain size (e.g. 64KB) - or just simple  trashes the CPU cache instead ...
>
> Should I still open a bug for this?
>
> Thanks.
>
> cu,
>   Emmy
>
> (BTW: The OS is Linux)
>
> Am 25.11.2013 um 02:37 schrieb Jan Bartel <[hidden email]>:
>
>> And when you open the bug, please make sure to state which operating
>> system and version you are using.
>>
>> thanks
>> Jan
>>
>> On 24 November 2013 09:55, Simone Bordet <[hidden email]> wrote:
>>> Hi,
>>>
>>> On Thu, Nov 21, 2013 at 2:47 PM, Emmeran Seehuber <[hidden email]> wrote:
>>>> Hello everybody,
>>>>
>>>> I´m using jetty as standalone embedded web server behind a nginx and it is
>>>> working great so far. Recently I upgraded from 7.6.4 to 9.0 and two days ago
>>>> to 9.1.0.v20131115.
>>>>
>>>> Since my upgrade to 9.0 i experience native memory leaks. Also with 9.1 i
>>>> still have the same leaks. Native memory leaks mean, that the process memory
>>>> size grows very large and then I get OOM Exceptions for DirectByteBuffers.
>>>> But to get there the server process has to run for about two weeks.
>>>>
>>>> Since the process has a 7 GB heap, the MaxDirectMemory Limit also seems to
>>>> be 7 GB.
>>>>
>>>> First i thought that the many threads started/stopped in the thread pool are
>>>> the problem. I specified a idle time of -1, and still many threads are
>>>> restarted. Also some answers on the web about DirectByteBuffer OOM
>>>> Exceptions pointed into this direction. But i don’t think thats the problem.
>>>>
>>>> I think the problem may be in ArrayByteBufferPool.  It allocates an unbound
>>>> number of ByteBuffers and caches them in a unbound ConcurrentLinkedQueue. So
>>>> at some peak times many buffers are used and never freed.  On the other side
>>>> this doesn’t really fit with the allocated memory. At the moment the server
>>>> has this direct memory allocations (according to the MBeans info):
>>>>
>>>> java.nio
>>>> BufferPool
>>>> name=direct
>>>> -  Count 384
>>>> -  MemoryUsed 2728821147
>>>> -  Name direct
>>>> -  ObjectName java.nio:type=BufferPool,name=direct
>>>> -  TotalCapacity 2728821147
>>>>
>>>> That would mean an medium buffer size of about 7 MB. But the
>>>> ArrayByteBufferPool does, as far as i understand, only allocate a max of 64
>>>> kb sized buffers by default ?!
>>>>
>>>> Current threads are: Count = 230, Maximum = 233, Started overall = 599
>>>>
>>>> This numbers are after the process is running for about 32 hours. I´ve got
>>>> about 320 request/minute at peak times. All this numbers are according to
>>>> the embedded JavaMelody monitoring.
>>>>
>>>> The JVM arguments are:
>>>> -server -XX:+UseCompressedOops -XX:MaxPermSize=512m -Xms7000M -Xmx7000M
>>>> -XX:+UseParallelOldGC -XX:+DoEscapeAnalysis -XX:+OptimizeStringConcat
>>>> JDK 1.7.0_45
>>>>
>>>> The thread pool is configured this way:
>>>>
>>>> tp = new QueuedThreadPool();
>>>> tp.setMaxThreads(256);
>>>> tp.setMinThreads(5);
>>>> tp.setIdleTimeout(-1);
>>>> tp.setName("Http Server Thread Pool");
>>>> tp.setDaemon(true);
>>>>
>>>> I had no such problems with jetty 7.x. Regularly calling System.gc() every
>>>> hour does not help. Also the i´ve got no other GC problems.
>>>>
>>>> Any ideas what could cause this problems?
>>>> Why are threads restarted in the pool even when i specify a idle timeout of
>>>> -1? I also regulary schedule some own runnables on the thread pool - but
>>>> they should not cause this problems, should they? They do in 99.99% of all
>>>> cases not throw any exceptions.
>>>
>>> Seems like a bug.
>>> I would keep the ThreadPool at default configuration for now, to have
>>> less variables in the system.
>>>
>>> Would you be able to replace usage of ArrayByteBufferPool with
>>> MappedByteBufferPool and see if the problem persist ?
>>>
>>> Also, please file an issue about this, it really looks from JMX that
>>> there is a problem.
>>>
>>> Thanks !
>>>
>>> --
>>> Simone Bordet
>>> ----
>>> http://cometd.org
>>> http://webtide.com
>>> http://intalio.com
>>> Developer advice, training, services and support
>>> from the Jetty & CometD experts.
>>> Intalio, the modern way to build business applications.
>>> _______________________________________________
>>> jetty-users mailing list
>>> [hidden email]
>>> https://dev.eclipse.org/mailman/listinfo/jetty-users
>>
>>
>>
>> --
>> Jan Bartel <[hidden email]>
>> www.webtide.com
>> 'Expert Jetty/CometD developer,production,operations advice'
>> _______________________________________________
>> jetty-users mailing list
>> [hidden email]
>> https://dev.eclipse.org/mailman/listinfo/jetty-users
>>
>
> Mit freundlichen Grüßen aus Augsburg,
>
> Emmeran Seehuber
> Dipl. Inf. (FH)
> Schrannenstraße 8
> 86150 Augsburg
> USt-IdNr.: DE266070804
>
> _______________________________________________
> jetty-users mailing list
> [hidden email]
> https://dev.eclipse.org/mailman/listinfo/jetty-users



--
Jan Bartel <[hidden email]>
www.webtide.com
'Expert Jetty/CometD developer,production,operations advice'
_______________________________________________
jetty-users mailing list
[hidden email]
https://dev.eclipse.org/mailman/listinfo/jetty-users



--
Greg Wilkins <[hidden email]>
http://www.webtide.com
Developer advice and support from the Jetty & CometD experts.
Intalio, the modern way to build business applications.

_______________________________________________
jetty-users mailing list
[hidden email]
https://dev.eclipse.org/mailman/listinfo/jetty-users