Jetty server crashes

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Jetty server crashes

Kapil
We are using jetty server and when it is on load then the server is crashed without showing much of information.
We have also put it on JMX. The CPU usage and memory looks fine, but still Jetty server crashes. Please let me know what parameters we should look for and how can we isolate issue which is causing jetty server crash.

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Jetty server crashes

Greg Wilkins

Kapil,

I'm sorry but you've not given us enough information to say anything.  What do you mean by "crashed"?  If it is the JVM that stops, then that is not a Jetty problem.  Is there a stack trace? Is there any logging?  Is the process still running? Can any requests be served? Is the connector still listening?  Can you run with debug?  Why do you think it is jetty and not some other component?

regards



On 10 January 2018 at 05:37, kapil gupta <[hidden email]> wrote:
We are using jetty server and when it is on load then the server is crashed without showing much of information.
We have also put it on JMX. The CPU usage and memory looks fine, but still Jetty server crashes. Please let me know what parameters we should look for and how can we isolate issue which is causing jetty server crash.

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users



--

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Jetty server crashes

Kapil
Greg,

Actually below things are observed:
  1. No process of jetty server running
  2. MySQL still runs
  3. CPU usage shows data fine
  4. RAM is also fine
Not sure how can we isolate if issue could be because of JVM, can you please guide on setting some parameters or moniotring them.

Thanks

On Wed, Jan 10, 2018 at 1:40 PM, Greg Wilkins <[hidden email]> wrote:

Kapil,

I'm sorry but you've not given us enough information to say anything.  What do you mean by "crashed"?  If it is the JVM that stops, then that is not a Jetty problem.  Is there a stack trace? Is there any logging?  Is the process still running? Can any requests be served? Is the connector still listening?  Can you run with debug?  Why do you think it is jetty and not some other component?

regards



On 10 January 2018 at 05:37, kapil gupta <[hidden email]> wrote:
We are using jetty server and when it is on load then the server is crashed without showing much of information.
We have also put it on JMX. The CPU usage and memory looks fine, but still Jetty server crashes. Please let me know what parameters we should look for and how can we isolate issue which is causing jetty server crash.

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users



--

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users


_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Jetty server crashes

Greg Wilkins
Kapil,

java is a very safe language and there should be no way for a java program to "crash" and end up with no process running... unless the program calls System.exit(), which Jetty does not.     Thus it is possible that you have hit a JVM problem, as sometimes the JVM will segv due to a bug and crash.

I suggest making sure your operating system is up to date with latest patches and checking your JVM is a recent stable release.

If it still happens, you need to capture how it failed.   Perhaps the start script can log the exit code? Are there any crash files in the directory where it ran?

regards




On 10 January 2018 at 12:50, kapil gupta <[hidden email]> wrote:
Greg,

Actually below things are observed:
  1. No process of jetty server running
  2. MySQL still runs
  3. CPU usage shows data fine
  4. RAM is also fine
Not sure how can we isolate if issue could be because of JVM, can you please guide on setting some parameters or moniotring them.

Thanks

On Wed, Jan 10, 2018 at 1:40 PM, Greg Wilkins <[hidden email]> wrote:

Kapil,

I'm sorry but you've not given us enough information to say anything.  What do you mean by "crashed"?  If it is the JVM that stops, then that is not a Jetty problem.  Is there a stack trace? Is there any logging?  Is the process still running? Can any requests be served? Is the connector still listening?  Can you run with debug?  Why do you think it is jetty and not some other component?

regards



On 10 January 2018 at 05:37, kapil gupta <[hidden email]> wrote:
We are using jetty server and when it is on load then the server is crashed without showing much of information.
We have also put it on JMX. The CPU usage and memory looks fine, but still Jetty server crashes. Please let me know what parameters we should look for and how can we isolate issue which is causing jetty server crash.

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users



--

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users


_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users



--

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Jetty server crashes

Brian Reichert
On Wed, Jan 10, 2018 at 04:07:09PM +0100, Greg Wilkins wrote:
> Kapil,
>
> java is a very safe language and there should be no way for a java program
> to "crash" and end up with no process running... unless the program calls
> System.exit(), which Jetty does not.     Thus it is possible that you have
> hit a JVM problem, as sometimes the JVM will segv due to a bug and crash.

See if you're using any of these directives with your JVM:

        -XX:+ExitOnOutOfMemoryError
        -XX:+CrashOnOutOfMemoryError

That's one way the JVM could exit.

The JVM used to run jetty will, by default, generate output to
STDOUT and STDERR.  Are you collecting that output into a log?
Unless the JVM was terminated with a KILL signal, it likely would
have recorded that it was shutting down for a specific reason.

It's possible your server has an Out-of-Memory killer, to help
manage a system running very low on resources, e.g.:

  https://www.memset.com/docs/additional-information/oom-killer/

I have no direct experience with that feature of the OS, though.



--
Brian Reichert <[hidden email]>
BSD admin/developer at large
_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Jetty server crashes

Joakim Erdfelt-8
In reply to this post by Greg Wilkins
If the Java JVM did crash, I would expect a hs_err_pid<pid>.log file to exist telling you why the java JVM crashed.

You can find that file in the working directory from where you started the JVM.

Or you can configure the location on the JVM.


Joakim Erdfelt / [hidden email]

On Wed, Jan 10, 2018 at 9:07 AM, Greg Wilkins <[hidden email]> wrote:
Kapil,

java is a very safe language and there should be no way for a java program to "crash" and end up with no process running... unless the program calls System.exit(), which Jetty does not.     Thus it is possible that you have hit a JVM problem, as sometimes the JVM will segv due to a bug and crash.

I suggest making sure your operating system is up to date with latest patches and checking your JVM is a recent stable release.

If it still happens, you need to capture how it failed.   Perhaps the start script can log the exit code? Are there any crash files in the directory where it ran?

regards




On 10 January 2018 at 12:50, kapil gupta <[hidden email]> wrote:
Greg,

Actually below things are observed:
  1. No process of jetty server running
  2. MySQL still runs
  3. CPU usage shows data fine
  4. RAM is also fine
Not sure how can we isolate if issue could be because of JVM, can you please guide on setting some parameters or moniotring them.

Thanks

On Wed, Jan 10, 2018 at 1:40 PM, Greg Wilkins <[hidden email]> wrote:

Kapil,

I'm sorry but you've not given us enough information to say anything.  What do you mean by "crashed"?  If it is the JVM that stops, then that is not a Jetty problem.  Is there a stack trace? Is there any logging?  Is the process still running? Can any requests be served? Is the connector still listening?  Can you run with debug?  Why do you think it is jetty and not some other component?

regards



On 10 January 2018 at 05:37, kapil gupta <[hidden email]> wrote:
We are using jetty server and when it is on load then the server is crashed without showing much of information.
We have also put it on JMX. The CPU usage and memory looks fine, but still Jetty server crashes. Please let me know what parameters we should look for and how can we isolate issue which is causing jetty server crash.

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users



--

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users


_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users



--

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users


_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Jetty server crashes

Silvio Bierman
In reply to this post by Kapil
Kapil,

If you are running on Linux check the system log for kernel panics due to lack of memory. Linux will ruthlessly kill the process with highest memory load in such a situation. That has happened to me plenty of times and also left me wondering where my JVM process had gone.

And to add to what Greg said: the System.exit thing can also happen in a library. I used a library once that would do a System.exit in some error situations killing my server now and then. That was a real puzzler.

Cheers,

Silvio


On 10-01-18 12:50, kapil gupta wrote:
Greg,

Actually below things are observed:
  1. No process of jetty server running
  2. MySQL still runs
  3. CPU usage shows data fine
  4. RAM is also fine
Not sure how can we isolate if issue could be because of JVM, can you please guide on setting some parameters or moniotring them.

Thanks

On Wed, Jan 10, 2018 at 1:40 PM, Greg Wilkins <[hidden email]> wrote:

Kapil,

I'm sorry but you've not given us enough information to say anything.  What do you mean by "crashed"?  If it is the JVM that stops, then that is not a Jetty problem.  Is there a stack trace? Is there any logging?  Is the process still running? Can any requests be served? Is the connector still listening?  Can you run with debug?  Why do you think it is jetty and not some other component?

regards



On 10 January 2018 at 05:37, kapil gupta <[hidden email]> wrote:
We are using jetty server and when it is on load then the server is crashed without showing much of information.
We have also put it on JMX. The CPU usage and memory looks fine, but still Jetty server crashes. Please let me know what parameters we should look for and how can we isolate issue which is causing jetty server crash.

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users



--

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users



_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users


_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Jetty server crashes

Kapil
Thanks everyone for your replies. 

Some more points into it:
  1. @Silvio - As you mentioned, yes sometimes OOM killer invokes jetty process to stop, but many of times it doesnt.
  2. @Greg - We are using cloud, and running on CentOS 7. We have stopped updates due to some reasons.
  3. @Joakim - Yes, checked the thread dump and some times OOM killer invokes jetty crash, sometimes I see below properties in dump file:
    1. Current thread (0x00007f18e01b9800):  VMThread [stack: 0x00007f18d33f4000,0x00007f18d34f5000] [id=11584]

    2. Stack: [0x00007f18d33f4000,0x00007f18d34f5000],  sp=0x00007f18d34f3140,  free space=1020k
    3. Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
    4. V  [libjvm.so+0x9a320a]  VMError::report_and_die()+0x2ea
    5. V  [libjvm.so+0x498d3b]  report_vm_out_of_memory(char const*, int, unsigned long, char const*)+0x9b
    6. V  [libjvm.so+0x82191e]  os::Linux::commit_memory_impl(char*, unsigned long, bool)+0xfe
    7. V  [libjvm.so+0x821e69]  os::pd_commit_memory(char*, unsigned long, unsigned long, bool)+0x29
    8. V  [libjvm.so+0x81bb6a]  os::commit_memory(char*, unsigned long, unsigned long, bool)+0x2a
    9. V  [libjvm.so+0x88d623]  PSVirtualSpace::expand_by(unsigned long)+0x53
    10. V  [libjvm.so+0x88e9f8]  PSYoungGen::resize_generation(unsigned long, unsigned long)+0xf8
    11. V  [libjvm.so+0x88db62]  PSYoungGen::resize(unsigned long, unsigned long)+0x22
    12. V  [libjvm.so+0x88af1d]  PSScavenge::invoke_no_policy()+0xf3d
    13. V  [libjvm.so+0x88b761]  PSScavenge::invoke()+0x41
    14. V  [libjvm.so+0x843f40]  ParallelScavengeHeap::failed_mem_allocate(unsigned long)+0x70
    15. V  [libjvm.so+0x9a4a97]  VM_ParallelGCFailedAllocation::doit()+0x97
    16. V  [libjvm.so+0x9abf35]  VM_Operation::evaluate()+0x55
    17. V  [libjvm.so+0x9aa2fa]  VMThread::evaluate_operation(VM_Operation*)+0xba
    18. V  [libjvm.so+0x9aa67e]  VMThread::loop()+0x1ce
    19. V  [libjvm.so+0x9aaaf0]  VMThread::run()+0x70
    20. V  [libjvm.so+0x8238c8]  java_start(Thread*)+0x108
I do not have much of knowledge in to these configuration, need you guys help.
Is this because of my server goes OutOfMemory, if yes then why it doesnt show the memory occupied more in VisualVM when we check heap size


On Thu, Jan 11, 2018 at 3:00 PM, Silvio Bierman <[hidden email]> wrote:
Kapil,

If you are running on Linux check the system log for kernel panics due to lack of memory. Linux will ruthlessly kill the process with highest memory load in such a situation. That has happened to me plenty of times and also left me wondering where my JVM process had gone.

And to add to what Greg said: the System.exit thing can also happen in a library. I used a library once that would do a System.exit in some error situations killing my server now and then. That was a real puzzler.

Cheers,

Silvio



On 10-01-18 12:50, kapil gupta wrote:
Greg,

Actually below things are observed:
  1. No process of jetty server running
  2. MySQL still runs
  3. CPU usage shows data fine
  4. RAM is also fine
Not sure how can we isolate if issue could be because of JVM, can you please guide on setting some parameters or moniotring them.

Thanks

On Wed, Jan 10, 2018 at 1:40 PM, Greg Wilkins <[hidden email]> wrote:

Kapil,

I'm sorry but you've not given us enough information to say anything.  What do you mean by "crashed"?  If it is the JVM that stops, then that is not a Jetty problem.  Is there a stack trace? Is there any logging?  Is the process still running? Can any requests be served? Is the connector still listening?  Can you run with debug?  Why do you think it is jetty and not some other component?

regards



On 10 January 2018 at 05:37, kapil gupta <[hidden email]> wrote:
We are using jetty server and when it is on load then the server is crashed without showing much of information.
We have also put it on JMX. The CPU usage and memory looks fine, but still Jetty server crashes. Please let me know what parameters we should look for and how can we isolate issue which is causing jetty server crash.

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users



--

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users



_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users


_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users


_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Jetty server crashes

Greg Wilkins
Kapil,

if your server is running out of memory, then it is most likely the application rather than jetty itself.
I would advise:
 + increasing the memory allocated to your production JVMs
 + running a test system with realistic load and profiling the applications memory usage both over time (for leaks) or just for memory hungry operations.

regards


On 12 January 2018 at 05:38, kapil gupta <[hidden email]> wrote:
Thanks everyone for your replies. 

Some more points into it:
  1. @Silvio - As you mentioned, yes sometimes OOM killer invokes jetty process to stop, but many of times it doesnt.
  2. @Greg - We are using cloud, and running on CentOS 7. We have stopped updates due to some reasons.
  3. @Joakim - Yes, checked the thread dump and some times OOM killer invokes jetty crash, sometimes I see below properties in dump file:
    1. Current thread (0x00007f18e01b9800):  VMThread [stack: 0x00007f18d33f4000,0x00007f18d34f5000] [id=11584]

    2. Stack: [0x00007f18d33f4000,0x00007f18d34f5000],  sp=0x00007f18d34f3140,  free space=1020k
    3. Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
    4. V  [libjvm.so+0x9a320a]  VMError::report_and_die()+0x2ea
    5. V  [libjvm.so+0x498d3b]  report_vm_out_of_memory(char const*, int, unsigned long, char const*)+0x9b
    6. V  [libjvm.so+0x82191e]  os::Linux::commit_memory_impl(char*, unsigned long, bool)+0xfe
    7. V  [libjvm.so+0x821e69]  os::pd_commit_memory(char*, unsigned long, unsigned long, bool)+0x29
    8. V  [libjvm.so+0x81bb6a]  os::commit_memory(char*, unsigned long, unsigned long, bool)+0x2a
    9. V  [libjvm.so+0x88d623]  PSVirtualSpace::expand_by(unsigned long)+0x53
    10. V  [libjvm.so+0x88e9f8]  PSYoungGen::resize_generation(unsigned long, unsigned long)+0xf8
    11. V  [libjvm.so+0x88db62]  PSYoungGen::resize(unsigned long, unsigned long)+0x22
    12. V  [libjvm.so+0x88af1d]  PSScavenge::invoke_no_policy()+0xf3d
    13. V  [libjvm.so+0x88b761]  PSScavenge::invoke()+0x41
    14. V  [libjvm.so+0x843f40]  ParallelScavengeHeap::failed_mem_allocate(unsigned long)+0x70
    15. V  [libjvm.so+0x9a4a97]  VM_ParallelGCFailedAllocation::doit()+0x97
    16. V  [libjvm.so+0x9abf35]  VM_Operation::evaluate()+0x55
    17. V  [libjvm.so+0x9aa2fa]  VMThread::evaluate_operation(VM_Operation*)+0xba
    18. V  [libjvm.so+0x9aa67e]  VMThread::loop()+0x1ce
    19. V  [libjvm.so+0x9aaaf0]  VMThread::run()+0x70
    20. V  [libjvm.so+0x8238c8]  java_start(Thread*)+0x108
I do not have much of knowledge in to these configuration, need you guys help.
Is this because of my server goes OutOfMemory, if yes then why it doesnt show the memory occupied more in VisualVM when we check heap size


On Thu, Jan 11, 2018 at 3:00 PM, Silvio Bierman <[hidden email]> wrote:
Kapil,

If you are running on Linux check the system log for kernel panics due to lack of memory. Linux will ruthlessly kill the process with highest memory load in such a situation. That has happened to me plenty of times and also left me wondering where my JVM process had gone.

And to add to what Greg said: the System.exit thing can also happen in a library. I used a library once that would do a System.exit in some error situations killing my server now and then. That was a real puzzler.

Cheers,

Silvio



On 10-01-18 12:50, kapil gupta wrote:
Greg,

Actually below things are observed:
  1. No process of jetty server running
  2. MySQL still runs
  3. CPU usage shows data fine
  4. RAM is also fine
Not sure how can we isolate if issue could be because of JVM, can you please guide on setting some parameters or moniotring them.

Thanks

On Wed, Jan 10, 2018 at 1:40 PM, Greg Wilkins <[hidden email]> wrote:

Kapil,

I'm sorry but you've not given us enough information to say anything.  What do you mean by "crashed"?  If it is the JVM that stops, then that is not a Jetty problem.  Is there a stack trace? Is there any logging?  Is the process still running? Can any requests be served? Is the connector still listening?  Can you run with debug?  Why do you think it is jetty and not some other component?

regards



On 10 January 2018 at 05:37, kapil gupta <[hidden email]> wrote:
We are using jetty server and when it is on load then the server is crashed without showing much of information.
We have also put it on JMX. The CPU usage and memory looks fine, but still Jetty server crashes. Please let me know what parameters we should look for and how can we isolate issue which is causing jetty server crash.

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users



--

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users



_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users


_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users


_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users



--

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Jetty server crashes

Kapil
Sure Greg, 

I will check out for the points suggested. But one thing, why my VisualVM doesn show heap size hike, it hardly consumes 50-60% of allocated memory.

On Fri, Jan 12, 2018 at 3:34 PM, Greg Wilkins <[hidden email]> wrote:
Kapil,

if your server is running out of memory, then it is most likely the application rather than jetty itself.
I would advise:
 + increasing the memory allocated to your production JVMs
 + running a test system with realistic load and profiling the applications memory usage both over time (for leaks) or just for memory hungry operations.

regards


On 12 January 2018 at 05:38, kapil gupta <[hidden email]> wrote:
Thanks everyone for your replies. 

Some more points into it:
  1. @Silvio - As you mentioned, yes sometimes OOM killer invokes jetty process to stop, but many of times it doesnt.
  2. @Greg - We are using cloud, and running on CentOS 7. We have stopped updates due to some reasons.
  3. @Joakim - Yes, checked the thread dump and some times OOM killer invokes jetty crash, sometimes I see below properties in dump file:
    1. Current thread (0x00007f18e01b9800):  VMThread [stack: 0x00007f18d33f4000,0x00007f18d34f5000] [id=11584]

    2. Stack: [0x00007f18d33f4000,0x00007f18d34f5000],  sp=0x00007f18d34f3140,  free space=1020k
    3. Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
    4. V  [libjvm.so+0x9a320a]  VMError::report_and_die()+0x2ea
    5. V  [libjvm.so+0x498d3b]  report_vm_out_of_memory(char const*, int, unsigned long, char const*)+0x9b
    6. V  [libjvm.so+0x82191e]  os::Linux::commit_memory_impl(char*, unsigned long, bool)+0xfe
    7. V  [libjvm.so+0x821e69]  os::pd_commit_memory(char*, unsigned long, unsigned long, bool)+0x29
    8. V  [libjvm.so+0x81bb6a]  os::commit_memory(char*, unsigned long, unsigned long, bool)+0x2a
    9. V  [libjvm.so+0x88d623]  PSVirtualSpace::expand_by(unsigned long)+0x53
    10. V  [libjvm.so+0x88e9f8]  PSYoungGen::resize_generation(unsigned long, unsigned long)+0xf8
    11. V  [libjvm.so+0x88db62]  PSYoungGen::resize(unsigned long, unsigned long)+0x22
    12. V  [libjvm.so+0x88af1d]  PSScavenge::invoke_no_policy()+0xf3d
    13. V  [libjvm.so+0x88b761]  PSScavenge::invoke()+0x41
    14. V  [libjvm.so+0x843f40]  ParallelScavengeHeap::failed_mem_allocate(unsigned long)+0x70
    15. V  [libjvm.so+0x9a4a97]  VM_ParallelGCFailedAllocation::doit()+0x97
    16. V  [libjvm.so+0x9abf35]  VM_Operation::evaluate()+0x55
    17. V  [libjvm.so+0x9aa2fa]  VMThread::evaluate_operation(VM_Operation*)+0xba
    18. V  [libjvm.so+0x9aa67e]  VMThread::loop()+0x1ce
    19. V  [libjvm.so+0x9aaaf0]  VMThread::run()+0x70
    20. V  [libjvm.so+0x8238c8]  java_start(Thread*)+0x108
I do not have much of knowledge in to these configuration, need you guys help.
Is this because of my server goes OutOfMemory, if yes then why it doesnt show the memory occupied more in VisualVM when we check heap size


On Thu, Jan 11, 2018 at 3:00 PM, Silvio Bierman <[hidden email]> wrote:
Kapil,

If you are running on Linux check the system log for kernel panics due to lack of memory. Linux will ruthlessly kill the process with highest memory load in such a situation. That has happened to me plenty of times and also left me wondering where my JVM process had gone.

And to add to what Greg said: the System.exit thing can also happen in a library. I used a library once that would do a System.exit in some error situations killing my server now and then. That was a real puzzler.

Cheers,

Silvio



On 10-01-18 12:50, kapil gupta wrote:
Greg,

Actually below things are observed:
  1. No process of jetty server running
  2. MySQL still runs
  3. CPU usage shows data fine
  4. RAM is also fine
Not sure how can we isolate if issue could be because of JVM, can you please guide on setting some parameters or moniotring them.

Thanks

On Wed, Jan 10, 2018 at 1:40 PM, Greg Wilkins <[hidden email]> wrote:

Kapil,

I'm sorry but you've not given us enough information to say anything.  What do you mean by "crashed"?  If it is the JVM that stops, then that is not a Jetty problem.  Is there a stack trace? Is there any logging?  Is the process still running? Can any requests be served? Is the connector still listening?  Can you run with debug?  Why do you think it is jetty and not some other component?

regards



On 10 January 2018 at 05:37, kapil gupta <[hidden email]> wrote:
We are using jetty server and when it is on load then the server is crashed without showing much of information.
We have also put it on JMX. The CPU usage and memory looks fine, but still Jetty server crashes. Please let me know what parameters we should look for and how can we isolate issue which is causing jetty server crash.

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users



--

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users



_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users


_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users


_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users



--

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users


_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Jetty server crashes

Kapil
Also one more thing we are facing issues when the request is not going to jetty server, there is no entry in access-log and application keeps loading.

Any pointers would be helpful

On Fri, Jan 12, 2018 at 3:45 PM, kapil gupta <[hidden email]> wrote:
Sure Greg, 

I will check out for the points suggested. But one thing, why my VisualVM doesn show heap size hike, it hardly consumes 50-60% of allocated memory.

On Fri, Jan 12, 2018 at 3:34 PM, Greg Wilkins <[hidden email]> wrote:
Kapil,

if your server is running out of memory, then it is most likely the application rather than jetty itself.
I would advise:
 + increasing the memory allocated to your production JVMs
 + running a test system with realistic load and profiling the applications memory usage both over time (for leaks) or just for memory hungry operations.

regards


On 12 January 2018 at 05:38, kapil gupta <[hidden email]> wrote:
Thanks everyone for your replies. 

Some more points into it:
  1. @Silvio - As you mentioned, yes sometimes OOM killer invokes jetty process to stop, but many of times it doesnt.
  2. @Greg - We are using cloud, and running on CentOS 7. We have stopped updates due to some reasons.
  3. @Joakim - Yes, checked the thread dump and some times OOM killer invokes jetty crash, sometimes I see below properties in dump file:
    1. Current thread (0x00007f18e01b9800):  VMThread [stack: 0x00007f18d33f4000,0x00007f18d34f5000] [id=11584]

    2. Stack: [0x00007f18d33f4000,0x00007f18d34f5000],  sp=0x00007f18d34f3140,  free space=1020k
    3. Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
    4. V  [libjvm.so+0x9a320a]  VMError::report_and_die()+0x2ea
    5. V  [libjvm.so+0x498d3b]  report_vm_out_of_memory(char const*, int, unsigned long, char const*)+0x9b
    6. V  [libjvm.so+0x82191e]  os::Linux::commit_memory_impl(char*, unsigned long, bool)+0xfe
    7. V  [libjvm.so+0x821e69]  os::pd_commit_memory(char*, unsigned long, unsigned long, bool)+0x29
    8. V  [libjvm.so+0x81bb6a]  os::commit_memory(char*, unsigned long, unsigned long, bool)+0x2a
    9. V  [libjvm.so+0x88d623]  PSVirtualSpace::expand_by(unsigned long)+0x53
    10. V  [libjvm.so+0x88e9f8]  PSYoungGen::resize_generation(unsigned long, unsigned long)+0xf8
    11. V  [libjvm.so+0x88db62]  PSYoungGen::resize(unsigned long, unsigned long)+0x22
    12. V  [libjvm.so+0x88af1d]  PSScavenge::invoke_no_policy()+0xf3d
    13. V  [libjvm.so+0x88b761]  PSScavenge::invoke()+0x41
    14. V  [libjvm.so+0x843f40]  ParallelScavengeHeap::failed_mem_allocate(unsigned long)+0x70
    15. V  [libjvm.so+0x9a4a97]  VM_ParallelGCFailedAllocation::doit()+0x97
    16. V  [libjvm.so+0x9abf35]  VM_Operation::evaluate()+0x55
    17. V  [libjvm.so+0x9aa2fa]  VMThread::evaluate_operation(VM_Operation*)+0xba
    18. V  [libjvm.so+0x9aa67e]  VMThread::loop()+0x1ce
    19. V  [libjvm.so+0x9aaaf0]  VMThread::run()+0x70
    20. V  [libjvm.so+0x8238c8]  java_start(Thread*)+0x108
I do not have much of knowledge in to these configuration, need you guys help.
Is this because of my server goes OutOfMemory, if yes then why it doesnt show the memory occupied more in VisualVM when we check heap size


On Thu, Jan 11, 2018 at 3:00 PM, Silvio Bierman <[hidden email]> wrote:
Kapil,

If you are running on Linux check the system log for kernel panics due to lack of memory. Linux will ruthlessly kill the process with highest memory load in such a situation. That has happened to me plenty of times and also left me wondering where my JVM process had gone.

And to add to what Greg said: the System.exit thing can also happen in a library. I used a library once that would do a System.exit in some error situations killing my server now and then. That was a real puzzler.

Cheers,

Silvio



On 10-01-18 12:50, kapil gupta wrote:
Greg,

Actually below things are observed:
  1. No process of jetty server running
  2. MySQL still runs
  3. CPU usage shows data fine
  4. RAM is also fine
Not sure how can we isolate if issue could be because of JVM, can you please guide on setting some parameters or moniotring them.

Thanks

On Wed, Jan 10, 2018 at 1:40 PM, Greg Wilkins <[hidden email]> wrote:

Kapil,

I'm sorry but you've not given us enough information to say anything.  What do you mean by "crashed"?  If it is the JVM that stops, then that is not a Jetty problem.  Is there a stack trace? Is there any logging?  Is the process still running? Can any requests be served? Is the connector still listening?  Can you run with debug?  Why do you think it is jetty and not some other component?

regards



On 10 January 2018 at 05:37, kapil gupta <[hidden email]> wrote:
We are using jetty server and when it is on load then the server is crashed without showing much of information.
We have also put it on JMX. The CPU usage and memory looks fine, but still Jetty server crashes. Please let me know what parameters we should look for and how can we isolate issue which is causing jetty server crash.

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users



--

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users



_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users


_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users


_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users



--

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users



_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users