[jetty-users] Simplest way to detect: Hey, where'd my thread go?

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

[jetty-users] Simplest way to detect: Hey, where'd my thread go?

Tamás Cservenák
Hi Jetty list!

I have a situation where I confirmed that Jetty's pool is "loosing" threads. I bet it's my code (runtime or maybe even Error is thrown somewhere I assume?), but...

What is the easiest thing with Jetty (8.1.x), to instrument the threads, or install some "custom" UncaughtExceptionHandler or alike (to detect and see where any why is this happening)?

Also, what is the default Jetty behavior with Threads that happens to throw some Runtime Exception or Error? The Jetty log contains no entry about anything, so I am just speculating the reasons why threads are disappearing....

What UncaughtExceptionHandler is by default used?


Any help appreciated.

Reference to:


Thanks,
~t~

_______________________________________________
jetty-users mailing list
[hidden email]
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: [jetty-users] Simplest way to detect: Hey, where'd my thread go?

Tamás Cservenák
New info:

It's a nice OutOfMemoryError in Nexus code.

I remember I saw some thread (or Jira issue) about why Jetty intentionally does not handle this... am I right?


Thanks,
~t~

On Tue, Jun 26, 2012 at 4:14 PM, Tamás Cservenák <[hidden email]> wrote:
Hi Jetty list!

I have a situation where I confirmed that Jetty's pool is "loosing" threads. I bet it's my code (runtime or maybe even Error is thrown somewhere I assume?), but...

What is the easiest thing with Jetty (8.1.x), to instrument the threads, or install some "custom" UncaughtExceptionHandler or alike (to detect and see where any why is this happening)?

Also, what is the default Jetty behavior with Threads that happens to throw some Runtime Exception or Error? The Jetty log contains no entry about anything, so I am just speculating the reasons why threads are disappearing....

What UncaughtExceptionHandler is by default used?


Any help appreciated.

Reference to:


Thanks,
~t~


_______________________________________________
jetty-users mailing list
[hidden email]
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: [jetty-users] Simplest way to detect: Hey, where'd my thread go?

Tamás Cservenák
To conclude, after I debugged a bit:

Jetty is nice and dandy, logs the problem, as long you have some runtime exception (anything that descends from Exception)...

But is grumpy and silent when you have an Error (like OutOfMemoryError).

And in both cases, thread gets removed from the pool, but in 2nd case you have no notification about it at all in logs...


Thanks,
~t~

On Tue, Jun 26, 2012 at 4:45 PM, Tamás Cservenák <[hidden email]> wrote:
New info:

It's a nice OutOfMemoryError in Nexus code.

I remember I saw some thread (or Jira issue) about why Jetty intentionally does not handle this... am I right?


Thanks,
~t~


On Tue, Jun 26, 2012 at 4:14 PM, Tamás Cservenák <[hidden email]> wrote:
Hi Jetty list!

I have a situation where I confirmed that Jetty's pool is "loosing" threads. I bet it's my code (runtime or maybe even Error is thrown somewhere I assume?), but...

What is the easiest thing with Jetty (8.1.x), to instrument the threads, or install some "custom" UncaughtExceptionHandler or alike (to detect and see where any why is this happening)?

Also, what is the default Jetty behavior with Threads that happens to throw some Runtime Exception or Error? The Jetty log contains no entry about anything, so I am just speculating the reasons why threads are disappearing....

What UncaughtExceptionHandler is by default used?


Any help appreciated.

Reference to:


Thanks,
~t~



_______________________________________________
jetty-users mailing list
[hidden email]
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: [jetty-users] Simplest way to detect: Hey, where'd my thread go?

Jesse McConnell
well, after an OOME all bets are generally off and your jvm is going
to have to go down to be predictable again...I suspect that is part of
the rational

jesse

--
jesse mcconnell
[hidden email]


On Tue, Jun 26, 2012 at 10:58 AM, Tamás Cservenák <[hidden email]> wrote:

> To conclude, after I debugged a bit:
>
> Jetty is nice and dandy, logs the problem, as long you have some runtime
> exception (anything that descends from Exception)...
>
> But is grumpy and silent when you have an Error (like OutOfMemoryError).
>
> And in both cases, thread gets removed from the pool, but in 2nd case you
> have no notification about it at all in logs...
>
>
> Thanks,
> ~t~
>
> On Tue, Jun 26, 2012 at 4:45 PM, Tamás Cservenák <[hidden email]>
> wrote:
>>
>> New info:
>>
>> It's a nice OutOfMemoryError in Nexus code.
>>
>> I remember I saw some thread (or Jira issue) about why Jetty intentionally
>> does not handle this... am I right?
>>
>>
>> Thanks,
>> ~t~
>>
>>
>> On Tue, Jun 26, 2012 at 4:14 PM, Tamás Cservenák <[hidden email]>
>> wrote:
>>>
>>> Hi Jetty list!
>>>
>>> I have a situation where I confirmed that Jetty's pool is "loosing"
>>> threads. I bet it's my code (runtime or maybe even Error is thrown somewhere
>>> I assume?), but...
>>>
>>> What is the easiest thing with Jetty (8.1.x), to instrument the threads,
>>> or install some "custom" UncaughtExceptionHandler or alike (to detect and
>>> see where any why is this happening)?
>>>
>>> Also, what is the default Jetty behavior with Threads that happens to
>>> throw some Runtime Exception or Error? The Jetty log contains no entry about
>>> anything, so I am just speculating the reasons why threads
>>> are disappearing....
>>>
>>> What UncaughtExceptionHandler is by default used?
>>>
>>>
>>> Any help appreciated.
>>>
>>> Reference to:
>>> http://www.ibm.com/developerworks/java/library/j-jtp0924/index.html
>>>
>>>
>>> Thanks,
>>> ~t~
>>
>>
>
>
> _______________________________________________
> jetty-users mailing list
> [hidden email]
> https://dev.eclipse.org/mailman/listinfo/jetty-users
>
_______________________________________________
jetty-users mailing list
[hidden email]
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: [jetty-users] Simplest way to detect: Hey, where'd my thread go?

Tamás Cservenák
I totally agree with you and rational.

But still, my question stands: what is the simplest or intended way (asking, since you guys might have something similar in Jetty already), to have some UncaughtExceptionHandler (or maybe some Jetty specific class) installed in Jetty threads (when running embedded)?

Point is, that _nothing_ got logged after OOM, nothing is visible "from outside", and my Nexus continued to live and happily served (!) things after the incident. I have to point out that OOM is not due to a leak, but one request in buggy code causes it, and in a moment it died and thread removed, JVM got plenty of memory...

So, to be able to emit at least a big bold warning, to try to shut down Jetty, _something_ ... , as leaving OOM unnoticed -- even if I agree with rationale -- is not okay for me in this case...

Do I have access to ThreadGroup? How can I instrument threads and/or thread creation in Jetty?

Thanks,
~t~


On Tue, Jun 26, 2012 at 6:09 PM, Jesse McConnell <[hidden email]> wrote:
well, after an OOME all bets are generally off and your jvm is going
to have to go down to be predictable again...I suspect that is part of
the rational

jesse

--
jesse mcconnell
[hidden email]


_______________________________________________
jetty-users mailing list
[hidden email]
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: [jetty-users] Simplest way to detect: Hey, where'd my thread go?

Tamás Cservenák
Ok, I spent day experimenting with our P2 bug, that is very specific of OOM: it has a "peak" of memory consumption that is immediately reclaimed too.

In short, I tried following:

* UncaughtExceptionHandler installed with my own "instrumented QTP". Did not have quite much effect, to me, seems like it was not even invoked in case of OOM.
* wrapping Jetty's _runnable (from QTP) with my own and logging from there... same thing, like my code was not invoked at all.

But, as I realized, _other threads_ in system kept living happily during OOM, it was only the failing thread that was heavily disrupted. I assume this is due to "type" or "case" of the OOM I have. Again, this was not a leak, this was one processing step allocating way too much and immediately releasing the same stuff. I assume this would not help for real leaks, as whole JVM would be disrupted (just to emphasize: my app was around 150MB of heap while "working/under load", one HTTP call causes to have heap jump to 850-900MB, but the very next GC also takes it back to 150-200MB). So, I went will 3rd approach that proved working for me:

* a separated "watchdog" daemon thread, that watches for QTP thread deaths, and start you nagging, and does it endlessly once death detected.



I think this was my goal, to have _any_ feedback in case of OOMs, as threads were just disappearing, and users had NO whatsoever feedback, that they might running the system "on the edge".


I'd really like to see some of these (if only the "instrumented QTP" and ability to pass in any ThreadFactory!) in Jetty!




Thanks,
~t~


On Tue, Jun 26, 2012 at 7:14 PM, Tamás Cservenák <[hidden email]> wrote:
I totally agree with you and rational.

But still, my question stands: what is the simplest or intended way (asking, since you guys might have something similar in Jetty already), to have some UncaughtExceptionHandler (or maybe some Jetty specific class) installed in Jetty threads (when running embedded)?

Point is, that _nothing_ got logged after OOM, nothing is visible "from outside", and my Nexus continued to live and happily served (!) things after the incident. I have to point out that OOM is not due to a leak, but one request in buggy code causes it, and in a moment it died and thread removed, JVM got plenty of memory...

So, to be able to emit at least a big bold warning, to try to shut down Jetty, _something_ ... , as leaving OOM unnoticed -- even if I agree with rationale -- is not okay for me in this case...

Do I have access to ThreadGroup? How can I instrument threads and/or thread creation in Jetty?

Thanks,
~t~


On Tue, Jun 26, 2012 at 6:09 PM, Jesse McConnell <[hidden email]> wrote:
well, after an OOME all bets are generally off and your jvm is going
to have to go down to be predictable again...I suspect that is part of
the rational

jesse

--
jesse mcconnell
[hidden email]



_______________________________________________
jetty-users mailing list
[hidden email]
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: [jetty-users] Simplest way to detect: Hey, where'd my thread go?

Jesse McConnell
> But, as I realized, _other threads_ in system kept living happily during
> OOM, it was only the failing thread that was heavily disrupted. I assume
> this is due to "type" or "case" of the OOM I have. Again, this was not a
> leak, this was one processing step allocating way too much
> and immediately releasing the same stuff. I assume this would not help for
> real leaks, as whole JVM would be disrupted (just to emphasize: my app was
> around 150MB of heap while "working/under load", one HTTP call causes to
> have heap jump to 850-900MB, but the very next GC also takes it back to
> 150-200MB). So, I went will 3rd approach that proved working for me:

this just sounds like you got 'lucky' more then anything :)

> * a separated "watchdog" daemon thread, that watches for QTP thread deaths,
> and start you nagging, and does it endlessly once death detected.
>
> https://github.com/sonatype/sisu-jetty8/commit/0cf9572896d4f0cf77bbaa57c41359db86cf9586

This is interesting, I'll remind greg about this next week and see
what he thinks, QTP is his baby but this sort of thing could be an
optional addition to the normally running qtp I think.  Generally OOME
are just bad juju, but that you had no other notification of it
indicates there might be something more we ought to do here..

thanks!
jesse
_______________________________________________
jetty-users mailing list
[hidden email]
https://dev.eclipse.org/mailman/listinfo/jetty-users