[jetty-users] jetty native memory leak

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

[jetty-users] jetty native memory leak

dhirajpraj
I am using Jetty 9.1.2 in my application. The application uses huge native memory space. In the thread dump, I see a lot of instances like: 

"qtp2042324703-524-selector-ServerConnectorManager@5c95306c/7" prio=10 tid=0x00007f4c1985a000 nid=0xb7bb runnable [0x00007f4c0ef6e000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
        - locked <0x00000000bff941a8> (a sun.nio.ch.Util$2)
        - locked <0x00000000bff94198> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000bff941b8> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
        at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:531)
        at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:484)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
        at java.lang.Thread.run(Thread.java:744)

I need some help to sort this out. Is this a known bug which is fixed in a later release?

Regards,
Dhiraj

_______________________________________________
jetty-users mailing list
[hidden email]
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: [jetty-users] jetty native memory leak

Jan Bartel-3
Dhiraj,

That thread dump just shows an idle thread. In past versions of jetty,
there were issues with jdk epoll bugs, but they manifested themselves
as cpu spin. So if you're not seeing undue cpu activity then this is
not a problem.

Jan

On 23 June 2014 14:57, dhiraj prajapati <[hidden email]> wrote:

> I am using Jetty 9.1.2 in my application. The application uses huge native
> memory space. In the thread dump, I see a lot of instances like:
>
> "qtp2042324703-524-selector-ServerConnectorManager@5c95306c/7" prio=10
> tid=0x00007f4c1985a000 nid=0xb7bb runnable [0x00007f4c0ef6e000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>         - locked <0x00000000bff941a8> (a sun.nio.ch.Util$2)
>         - locked <0x00000000bff94198> (a
> java.util.Collections$UnmodifiableSet)
>         - locked <0x00000000bff941b8> (a sun.nio.ch.EPollSelectorImpl)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>         at
> org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:531)
>         at
> org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:484)
>         at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
>         at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
>         at java.lang.Thread.run(Thread.java:744)
>
> I need some help to sort this out. Is this a known bug which is fixed in a
> later release?
>
> Regards,
>
> Dhiraj
>
>
> _______________________________________________
> jetty-users mailing list
> [hidden email]
> https://dev.eclipse.org/mailman/listinfo/jetty-users
>



--
Jan Bartel <[hidden email]>
www.webtide.com
'Expert Jetty/CometD developer,production,operations advice'
_______________________________________________
jetty-users mailing list
[hidden email]
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: [jetty-users] jetty native memory leak

dhirajpraj
Hi,
But the application is using a lot of native memory and it ends up using a lot of swap space after which I am forced to restart the application.

Regards,
Dhiraj


On Tue, Jun 24, 2014 at 1:20 PM, Jan Bartel <[hidden email]> wrote:
Dhiraj,

That thread dump just shows an idle thread. In past versions of jetty,
there were issues with jdk epoll bugs, but they manifested themselves
as cpu spin. So if you're not seeing undue cpu activity then this is
not a problem.

Jan

On 23 June 2014 14:57, dhiraj prajapati <[hidden email]> wrote:
> I am using Jetty 9.1.2 in my application. The application uses huge native
> memory space. In the thread dump, I see a lot of instances like:
>
> "qtp2042324703-524-selector-ServerConnectorManager@5c95306c/7" prio=10
> tid=0x00007f4c1985a000 nid=0xb7bb runnable [0x00007f4c0ef6e000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>         - locked <0x00000000bff941a8> (a sun.nio.ch.Util$2)
>         - locked <0x00000000bff94198> (a
> java.util.Collections$UnmodifiableSet)
>         - locked <0x00000000bff941b8> (a sun.nio.ch.EPollSelectorImpl)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>         at
> org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:531)
>         at
> org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:484)
>         at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
>         at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
>         at java.lang.Thread.run(Thread.java:744)
>
> I need some help to sort this out. Is this a known bug which is fixed in a
> later release?
>
> Regards,
>
> Dhiraj
>
>
> _______________________________________________
> jetty-users mailing list
> [hidden email]
> https://dev.eclipse.org/mailman/listinfo/jetty-users
>



--
Jan Bartel <[hidden email]>
www.webtide.com
'Expert Jetty/CometD developer,production,operations advice'
_______________________________________________
jetty-users mailing list
[hidden email]
https://dev.eclipse.org/mailman/listinfo/jetty-users


_______________________________________________
jetty-users mailing list
[hidden email]
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: [jetty-users] jetty native memory leak

dhirajpraj
Also, the thread dump shows many such idle threads in RUNNABLE state (more than 500). Isn't that an issue?

Regards,
Dhiraj


On Tue, Jun 24, 2014 at 2:33 PM, dhiraj prajapati <[hidden email]> wrote:
Hi,
But the application is using a lot of native memory and it ends up using a lot of swap space after which I am forced to restart the application.

Regards,
Dhiraj


On Tue, Jun 24, 2014 at 1:20 PM, Jan Bartel <[hidden email]> wrote:
Dhiraj,

That thread dump just shows an idle thread. In past versions of jetty,
there were issues with jdk epoll bugs, but they manifested themselves
as cpu spin. So if you're not seeing undue cpu activity then this is
not a problem.

Jan

On 23 June 2014 14:57, dhiraj prajapati <[hidden email]> wrote:
> I am using Jetty 9.1.2 in my application. The application uses huge native
> memory space. In the thread dump, I see a lot of instances like:
>
> "qtp2042324703-524-selector-ServerConnectorManager@5c95306c/7" prio=10
> tid=0x00007f4c1985a000 nid=0xb7bb runnable [0x00007f4c0ef6e000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>         - locked <0x00000000bff941a8> (a sun.nio.ch.Util$2)
>         - locked <0x00000000bff94198> (a
> java.util.Collections$UnmodifiableSet)
>         - locked <0x00000000bff941b8> (a sun.nio.ch.EPollSelectorImpl)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>         at
> org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:531)
>         at
> org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:484)
>         at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
>         at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
>         at java.lang.Thread.run(Thread.java:744)
>
> I need some help to sort this out. Is this a known bug which is fixed in a
> later release?
>
> Regards,
>
> Dhiraj
>
>
> _______________________________________________
> jetty-users mailing list
> [hidden email]
> https://dev.eclipse.org/mailman/listinfo/jetty-users
>



--
Jan Bartel <[hidden email]>
www.webtide.com
'Expert Jetty/CometD developer,production,operations advice'
_______________________________________________
jetty-users mailing list
[hidden email]
https://dev.eclipse.org/mailman/listinfo/jetty-users



_______________________________________________
jetty-users mailing list
[hidden email]
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: [jetty-users] jetty native memory leak

Thomas Becker-2
In reply to this post by dhirajpraj
Hi Dhiraj,

then use jmap to create a heap dump and browse that dump by using a tool like Eclipse Memory Analyzer, jhat or a Java Profiler that can read heap dumps. There you'll find what's occupying your memory. The idle thread you postet below is no harm as Jan already stated.

Cheers,
Thomas
On 24/06/14 11:03, dhiraj prajapati wrote:
Hi,
But the application is using a lot of native memory and it ends up using a lot of swap space after which I am forced to restart the application.

Regards,
Dhiraj


On Tue, Jun 24, 2014 at 1:20 PM, Jan Bartel <[hidden email]> wrote:
Dhiraj,

That thread dump just shows an idle thread. In past versions of jetty,
there were issues with jdk epoll bugs, but they manifested themselves
as cpu spin. So if you're not seeing undue cpu activity then this is
not a problem.

Jan

On 23 June 2014 14:57, dhiraj prajapati <[hidden email]> wrote:
> I am using Jetty 9.1.2 in my application. The application uses huge native
> memory space. In the thread dump, I see a lot of instances like:
>
> "qtp2042324703-524-selector-ServerConnectorManager@5c95306c/7" prio=10
> tid=0x00007f4c1985a000 nid=0xb7bb runnable [0x00007f4c0ef6e000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>         - locked <0x00000000bff941a8> (a sun.nio.ch.Util$2)
>         - locked <0x00000000bff94198> (a
> java.util.Collections$UnmodifiableSet)
>         - locked <0x00000000bff941b8> (a sun.nio.ch.EPollSelectorImpl)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>         at
> org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:531)
>         at
> org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:484)
>         at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
>         at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
>         at java.lang.Thread.run(Thread.java:744)
>
> I need some help to sort this out. Is this a known bug which is fixed in a
> later release?
>
> Regards,
>
> Dhiraj
>
>
> _______________________________________________
> jetty-users mailing list
> [hidden email]
> https://dev.eclipse.org/mailman/listinfo/jetty-users
>



--
Jan Bartel <[hidden email]>
www.webtide.com
'Expert Jetty/CometD developer,production,operations advice'
_______________________________________________
jetty-users mailing list
[hidden email]
https://dev.eclipse.org/mailman/listinfo/jetty-users



_______________________________________________
jetty-users mailing list
[hidden email]
https://dev.eclipse.org/mailman/listinfo/jetty-users


_______________________________________________
jetty-users mailing list
[hidden email]
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: [jetty-users] jetty native memory leak

dhirajpraj
Thanks Thomas.
But It is the native memory that is being eaten up and not the heap memory. So I don't think Memory Analyzer will be of any help here. I am just concerned about the sheer number of RUNNABLE threads. It was more than 500. Can we be sure that they are just lying idle, because the thread dump shows them in RUNNABLE state?

Regards,
Dhiraj


On Tue, Jun 24, 2014 at 2:38 PM, Thomas Becker <[hidden email]> wrote:
Hi Dhiraj,

then use jmap to create a heap dump and browse that dump by using a tool like Eclipse Memory Analyzer, jhat or a Java Profiler that can read heap dumps. There you'll find what's occupying your memory. The idle thread you postet below is no harm as Jan already stated.

Cheers,
Thomas

On 24/06/14 11:03, dhiraj prajapati wrote:
Hi,
But the application is using a lot of native memory and it ends up using a lot of swap space after which I am forced to restart the application.

Regards,
Dhiraj


On Tue, Jun 24, 2014 at 1:20 PM, Jan Bartel <[hidden email]> wrote:
Dhiraj,

That thread dump just shows an idle thread. In past versions of jetty,
there were issues with jdk epoll bugs, but they manifested themselves
as cpu spin. So if you're not seeing undue cpu activity then this is
not a problem.

Jan

On 23 June 2014 14:57, dhiraj prajapati <[hidden email]> wrote:
> I am using Jetty 9.1.2 in my application. The application uses huge native
> memory space. In the thread dump, I see a lot of instances like:
>
> "qtp2042324703-524-selector-ServerConnectorManager@5c95306c/7" prio=10
> tid=0x00007f4c1985a000 nid=0xb7bb runnable [0x00007f4c0ef6e000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>         - locked <0x00000000bff941a8> (a sun.nio.ch.Util$2)
>         - locked <0x00000000bff94198> (a
> java.util.Collections$UnmodifiableSet)
>         - locked <0x00000000bff941b8> (a sun.nio.ch.EPollSelectorImpl)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>         at
> org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:531)
>         at
> org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:484)
>         at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
>         at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
>         at java.lang.Thread.run(Thread.java:744)
>
> I need some help to sort this out. Is this a known bug which is fixed in a
> later release?
>
> Regards,
>
> Dhiraj
>
>
> _______________________________________________
> jetty-users mailing list
> [hidden email]
> https://dev.eclipse.org/mailman/listinfo/jetty-users
>



--
Jan Bartel <[hidden email]>
www.webtide.com
'Expert Jetty/CometD developer,production,operations advice'
_______________________________________________
jetty-users mailing list
[hidden email]
https://dev.eclipse.org/mailman/listinfo/jetty-users



_______________________________________________
jetty-users mailing list
[hidden email]
https://dev.eclipse.org/mailman/listinfo/jetty-users


_______________________________________________
jetty-users mailing list
[hidden email]
https://dev.eclipse.org/mailman/listinfo/jetty-users



_______________________________________________
jetty-users mailing list
[hidden email]
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: [jetty-users] jetty native memory leak

Jan Bartel-3
Dhiraj,

500 threads in the pool lying idle is not necessarily a problem - you
should check your connector configuration. Also do several thread
dumps a minute or so apart to check on the thread activity. If there's
no traffic or very little traffic, then your threads may all indeed
mostly be idle, particularly if you had a traffic spike and jetty
needed to spin up more threads (up to the max you have configured) to
deal with it.

In any case, have a read of our page on memory leak issues:
http://www.eclipse.org/jetty/documentation/current/preventing-memory-leaks.html

Jan

On 24 June 2014 11:15, dhiraj prajapati <[hidden email]> wrote:

> Thanks Thomas.
> But It is the native memory that is being eaten up and not the heap memory.
> So I don't think Memory Analyzer will be of any help here. I am just
> concerned about the sheer number of RUNNABLE threads. It was more than 500.
> Can we be sure that they are just lying idle, because the thread dump shows
> them in RUNNABLE state?
>
> Regards,
> Dhiraj
>
>
> On Tue, Jun 24, 2014 at 2:38 PM, Thomas Becker <[hidden email]>
> wrote:
>>
>> Hi Dhiraj,
>>
>> then use jmap to create a heap dump and browse that dump by using a tool
>> like Eclipse Memory Analyzer, jhat or a Java Profiler that can read heap
>> dumps. There you'll find what's occupying your memory. The idle thread you
>> postet below is no harm as Jan already stated.
>>
>> Cheers,
>> Thomas
>>
>> On 24/06/14 11:03, dhiraj prajapati wrote:
>>
>> Hi,
>> But the application is using a lot of native memory and it ends up using a
>> lot of swap space after which I am forced to restart the application.
>>
>> Regards,
>> Dhiraj
>>
>>
>> On Tue, Jun 24, 2014 at 1:20 PM, Jan Bartel <[hidden email]> wrote:
>>>
>>> Dhiraj,
>>>
>>> That thread dump just shows an idle thread. In past versions of jetty,
>>> there were issues with jdk epoll bugs, but they manifested themselves
>>> as cpu spin. So if you're not seeing undue cpu activity then this is
>>> not a problem.
>>>
>>> Jan
>>>
>>> On 23 June 2014 14:57, dhiraj prajapati <[hidden email]> wrote:
>>> > I am using Jetty 9.1.2 in my application. The application uses huge
>>> > native
>>> > memory space. In the thread dump, I see a lot of instances like:
>>> >
>>> > "qtp2042324703-524-selector-ServerConnectorManager@5c95306c/7" prio=10
>>> > tid=0x00007f4c1985a000 nid=0xb7bb runnable [0x00007f4c0ef6e000]
>>> >    java.lang.Thread.State: RUNNABLE
>>> >         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> >         at
>>> > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> >         at
>>> > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> >         at
>>> > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>> >         - locked <0x00000000bff941a8> (a sun.nio.ch.Util$2)
>>> >         - locked <0x00000000bff94198> (a
>>> > java.util.Collections$UnmodifiableSet)
>>> >         - locked <0x00000000bff941b8> (a sun.nio.ch.EPollSelectorImpl)
>>> >         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>> >         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>>> >         at
>>> >
>>> > org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:531)
>>> >         at
>>> >
>>> > org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:484)
>>> >         at
>>> >
>>> > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
>>> >         at
>>> >
>>> > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
>>> >         at java.lang.Thread.run(Thread.java:744)
>>> >
>>> > I need some help to sort this out. Is this a known bug which is fixed
>>> > in a
>>> > later release?
>>> >
>>> > Regards,
>>> >
>>> > Dhiraj
>>> >
>>> >
>>> > _______________________________________________
>>> > jetty-users mailing list
>>> > [hidden email]
>>> > https://dev.eclipse.org/mailman/listinfo/jetty-users
>>> >
>>>
>>>
>>>
>>> --
>>> Jan Bartel <[hidden email]>
>>> www.webtide.com
>>> 'Expert Jetty/CometD developer,production,operations advice'
>>> _______________________________________________
>>> jetty-users mailing list
>>> [hidden email]
>>> https://dev.eclipse.org/mailman/listinfo/jetty-users
>>
>>
>>
>>
>> _______________________________________________
>> jetty-users mailing list
>> [hidden email]
>> https://dev.eclipse.org/mailman/listinfo/jetty-users
>>
>>
>>
>> _______________________________________________
>> jetty-users mailing list
>> [hidden email]
>> https://dev.eclipse.org/mailman/listinfo/jetty-users
>>
>
>
> _______________________________________________
> jetty-users mailing list
> [hidden email]
> https://dev.eclipse.org/mailman/listinfo/jetty-users
>



--
Jan Bartel <[hidden email]>
www.webtide.com
'Expert Jetty/CometD developer,production,operations advice'
_______________________________________________
jetty-users mailing list
[hidden email]
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: [jetty-users] jetty native memory leak

dhirajpraj
I took 3 thread dumps after 5 minute gaps. The number of those threads was constant. And all were in RUNNABLE state. Please note that there was very very low traffic on the application.

Regards,
Dhiraj


On Tue, Jun 24, 2014 at 2:53 PM, Jan Bartel <[hidden email]> wrote:
Dhiraj,

500 threads in the pool lying idle is not necessarily a problem - you
should check your connector configuration. Also do several thread
dumps a minute or so apart to check on the thread activity. If there's
no traffic or very little traffic, then your threads may all indeed
mostly be idle, particularly if you had a traffic spike and jetty
needed to spin up more threads (up to the max you have configured) to
deal with it.

In any case, have a read of our page on memory leak issues:
http://www.eclipse.org/jetty/documentation/current/preventing-memory-leaks.html

Jan

On 24 June 2014 11:15, dhiraj prajapati <[hidden email]> wrote:
> Thanks Thomas.
> But It is the native memory that is being eaten up and not the heap memory.
> So I don't think Memory Analyzer will be of any help here. I am just
> concerned about the sheer number of RUNNABLE threads. It was more than 500.
> Can we be sure that they are just lying idle, because the thread dump shows
> them in RUNNABLE state?
>
> Regards,
> Dhiraj
>
>
> On Tue, Jun 24, 2014 at 2:38 PM, Thomas Becker <[hidden email]>
> wrote:
>>
>> Hi Dhiraj,
>>
>> then use jmap to create a heap dump and browse that dump by using a tool
>> like Eclipse Memory Analyzer, jhat or a Java Profiler that can read heap
>> dumps. There you'll find what's occupying your memory. The idle thread you
>> postet below is no harm as Jan already stated.
>>
>> Cheers,
>> Thomas
>>
>> On 24/06/14 11:03, dhiraj prajapati wrote:
>>
>> Hi,
>> But the application is using a lot of native memory and it ends up using a
>> lot of swap space after which I am forced to restart the application.
>>
>> Regards,
>> Dhiraj
>>
>>
>> On Tue, Jun 24, 2014 at 1:20 PM, Jan Bartel <[hidden email]> wrote:
>>>
>>> Dhiraj,
>>>
>>> That thread dump just shows an idle thread. In past versions of jetty,
>>> there were issues with jdk epoll bugs, but they manifested themselves
>>> as cpu spin. So if you're not seeing undue cpu activity then this is
>>> not a problem.
>>>
>>> Jan
>>>
>>> On 23 June 2014 14:57, dhiraj prajapati <[hidden email]> wrote:
>>> > I am using Jetty 9.1.2 in my application. The application uses huge
>>> > native
>>> > memory space. In the thread dump, I see a lot of instances like:
>>> >
>>> > "qtp2042324703-524-selector-ServerConnectorManager@5c95306c/7" prio=10
>>> > tid=0x00007f4c1985a000 nid=0xb7bb runnable [0x00007f4c0ef6e000]
>>> >    java.lang.Thread.State: RUNNABLE
>>> >         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> >         at
>>> > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>>> >         at
>>> > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>>> >         at
>>> > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>> >         - locked <0x00000000bff941a8> (a sun.nio.ch.Util$2)
>>> >         - locked <0x00000000bff94198> (a
>>> > java.util.Collections$UnmodifiableSet)
>>> >         - locked <0x00000000bff941b8> (a sun.nio.ch.EPollSelectorImpl)
>>> >         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>> >         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>>> >         at
>>> >
>>> > org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:531)
>>> >         at
>>> >
>>> > org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:484)
>>> >         at
>>> >
>>> > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
>>> >         at
>>> >
>>> > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
>>> >         at java.lang.Thread.run(Thread.java:744)
>>> >
>>> > I need some help to sort this out. Is this a known bug which is fixed
>>> > in a
>>> > later release?
>>> >
>>> > Regards,
>>> >
>>> > Dhiraj
>>> >
>>> >
>>> > _______________________________________________
>>> > jetty-users mailing list
>>> > [hidden email]
>>> > https://dev.eclipse.org/mailman/listinfo/jetty-users
>>> >
>>>
>>>
>>>
>>> --
>>> Jan Bartel <[hidden email]>
>>> www.webtide.com
>>> 'Expert Jetty/CometD developer,production,operations advice'
>>> _______________________________________________
>>> jetty-users mailing list
>>> [hidden email]
>>> https://dev.eclipse.org/mailman/listinfo/jetty-users
>>
>>
>>
>>
>> _______________________________________________
>> jetty-users mailing list
>> [hidden email]
>> https://dev.eclipse.org/mailman/listinfo/jetty-users
>>
>>
>>
>> _______________________________________________
>> jetty-users mailing list
>> [hidden email]
>> https://dev.eclipse.org/mailman/listinfo/jetty-users
>>
>
>
> _______________________________________________
> jetty-users mailing list
> [hidden email]
> https://dev.eclipse.org/mailman/listinfo/jetty-users
>



--
Jan Bartel <[hidden email]>
www.webtide.com
'Expert Jetty/CometD developer,production,operations advice'
_______________________________________________
jetty-users mailing list
[hidden email]
https://dev.eclipse.org/mailman/listinfo/jetty-users


_______________________________________________
jetty-users mailing list
[hidden email]
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: [jetty-users] jetty native memory leak

Simone Bordet-2
Hi,

On Tue, Jun 24, 2014 at 11:29 AM, dhiraj prajapati <[hidden email]> wrote:
> I took 3 thread dumps after 5 minute gaps. The number of those threads was
> constant. And all were in RUNNABLE state. Please note that there was very
> very low traffic on the application.

You don't say what tool you used to take the thread dumps, you don't
show the stack traces of those dumps, you say your application uses
*huge* native memory, but you never said actually how much, you did
not say what JDK you are using.

We have basically zero information to help you, apart from
"aaaaahhhhh" which is not very descriptive.

Read this: http://www.catb.org/esr/faqs/smart-questions.html

When you have completed that read, try again posting here, and I am
sure we will be able to help you out.

--
Simone Bordet
----
http://cometd.org
http://webtide.com
http://intalio.com
Developer advice, training, services and support
from the Jetty & CometD experts.
Intalio, the modern way to build business applications.
_______________________________________________
jetty-users mailing list
[hidden email]
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: [jetty-users] jetty native memory leak

dhirajpraj
Hi,
I used jstack to take the thread dumps.
PFA the thread dumps taken at intervals of 5 minutes.(order is td2014062301, then td2014062302 and then td2014062303 )
JDK version is jdk1.7.0_45
Jetty version is jetty-all-9.1.2.v20140210.jar.
I allocated 1 GB of heap space to the application, but the application was using 6.6 GB memory (2.7 GB swap space) before I restarted the application. RAM on the machine is 8GB.

Regards,
Dhiraj



On Tue, Jun 24, 2014 at 3:51 PM, Simone Bordet <[hidden email]> wrote:
Hi,

On Tue, Jun 24, 2014 at 11:29 AM, dhiraj prajapati <[hidden email]> wrote:
> I took 3 thread dumps after 5 minute gaps. The number of those threads was
> constant. And all were in RUNNABLE state. Please note that there was very
> very low traffic on the application.

You don't say what tool you used to take the thread dumps, you don't
show the stack traces of those dumps, you say your application uses
*huge* native memory, but you never said actually how much, you did
not say what JDK you are using.

We have basically zero information to help you, apart from
"aaaaahhhhh" which is not very descriptive.

Read this: http://www.catb.org/esr/faqs/smart-questions.html

When you have completed that read, try again posting here, and I am
sure we will be able to help you out.

--
Simone Bordet
----
http://cometd.org
http://webtide.com
http://intalio.com
Developer advice, training, services and support
from the Jetty & CometD experts.
Intalio, the modern way to build business applications.
_______________________________________________
jetty-users mailing list
[hidden email]
https://dev.eclipse.org/mailman/listinfo/jetty-users


_______________________________________________
jetty-users mailing list
[hidden email]
https://dev.eclipse.org/mailman/listinfo/jetty-users

td2014062303.txt (996K) Download Attachment
td2014062302.txt (996K) Download Attachment
td2014062301.txt (996K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [jetty-users] jetty native memory leak

Simone Bordet-2
Hi,

On Tue, Jun 24, 2014 at 12:52 PM, dhiraj prajapati <[hidden email]> wrote:
> Hi,
> I used jstack to take the thread dumps.
> PFA the thread dumps taken at intervals of 5 minutes.(order is td2014062301,
> then td2014062302 and then td2014062303 )
> JDK version is jdk1.7.0_45
> Jetty version is jetty-all-9.1.2.v20140210.jar.
> I allocated 1 GB of heap space to the application, but the application was
> using 6.6 GB memory (2.7 GB swap space) before I restarted the application.
> RAM on the machine is 8GB.

From your own stack traces, Jetty allocates 8 selector threads.

You application also uses Netty, and it's Netty that allocates the
other (~500) selector threads.
I suspect you have misconfigured Netty.

--
Simone Bordet
----
http://cometd.org
http://webtide.com
http://intalio.com
Developer advice, training, services and support
from the Jetty & CometD experts.
Intalio, the modern way to build business applications.
_______________________________________________
jetty-users mailing list
[hidden email]
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: jetty native memory leak

dhirajpraj

Hi,
The problem seems to be solved after upgrading jetty to version 9.2

Thanks,
Dhiraj Prajapati

On 24 Jun 2014 16:37, "Simone Bordet" <[hidden email]> wrote:
Hi,

On Tue, Jun 24, 2014 at 12:52 PM, dhiraj prajapati <[hidden email]> wrote:
> Hi,
> I used jstack to take the thread dumps.
> PFA the thread dumps taken at intervals of 5 minutes.(order is td2014062301,
> then td2014062302 and then td2014062303 )
> JDK version is jdk1.7.0_45
> Jetty version is jetty-all-9.1.2.v20140210.jar.
> I allocated 1 GB of heap space to the application, but the application was
> using 6.6 GB memory (2.7 GB swap space) before I restarted the application.
> RAM on the machine is 8GB.

From your own stack traces, Jetty allocates 8 selector threads.

You application also uses Netty, and it's Netty that allocates the
other (~500) selector threads.
I suspect you have misconfigured Netty.

--
Simone Bordet
----
http://cometd.org
http://webtide.com
http://intalio.com
Developer advice, training, services and support
from the Jetty & CometD experts.
Intalio, the modern way to build business applications.
_______________________________________________
jetty-users mailing list
[hidden email]
https://dev.eclipse.org/mailman/listinfo/jetty-users

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users