[jetty-dev] jetty and 'hot-deploy', not freeing diskspace?

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

[jetty-dev] jetty and 'hot-deploy', not freeing diskspace?

Brian Reichert
(I posted this recently to jetty-users, but hadn't seen a response
there.  Reposting here, and hoping for the best. :)

We're employing the 'hot deploy' feature in jetty to redeploy an
updated webapp.

What we're observing is that the redeployment doesn't discard the
disk resources of the prior deployment.

Using 'lsof' as such:

  lsof -nP +L1 /var | grep /var | grep '(deleted)' | less

We can see dozens of old deployments (old extracted war files?), like this:

  java      26734        root   82r   REG    8,6   575389     0  32470
  /var/tmp/webapp_tmp/jetty-0.0.0.0-80-webapp-_idp-any-4794158382650079166.dir/webinf/WEB-INF/lib/commons-collections-3.2.1.jar
  (deleted)
  java      26734        root  110r   REG    8,6   575389     0  32561
  /var/tmp/webapp_tmp/jetty-0.0.0.0-80-webapp-_idp-any-2567933012053298590.dir/webinf/WEB-INF/lib/commons-collections-3.2.1.jar
  (deleted)
  java      26734        root  318r   REG    8,6   575389     0  40227
  /var/tmp/webapp_tmp/jetty-0.0.0.0-80-webapp-_idp-any-4949209254558295977.dir/webinf/WEB-INF/lib/commons-collections-3.2.1.jar
  (deleted)
  ...

As the JVM is still holding a reference, the disk space never frees,
and our /var/ partition eventually fills. :/

Do other people see this?

Are we doing this wrong?  All we did was modify the scanInterval
setting to a nonzero value.

Is there some configuration switch we can throw to coerce the JVM
to really release these resources, or are we stuck waiting for the
GC to eventually cleanup?

Details of our environment:

  jetty-distribution-9.3.8.v20160314

  OpenJDK Runtime Environment (build 1.8.0_71-b15)

  CentOS release 6.7 (Final)

  # uname -r
  2.6.32-573.18.1.el6.x86_64

--
Brian Reichert <[hidden email]>
BSD admin/developer at large
_______________________________________________
jetty-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-dev
Reply | Threaded
Open this post in threaded view
|

Re: [jetty-dev] jetty and 'hot-deploy', not freeing diskspace?

Jesse McConnell
Hi Brian,

Did you take a peek through these? It is typically the reason we see this sort of behavior.


cheers,
Jesse

--
jesse mcconnell
[hidden email]

On Tue, Aug 23, 2016 at 5:07 PM, Brian Reichert <[hidden email]> wrote:
(I posted this recently to jetty-users, but hadn't seen a response
there.  Reposting here, and hoping for the best. :)

We're employing the 'hot deploy' feature in jetty to redeploy an
updated webapp.

What we're observing is that the redeployment doesn't discard the
disk resources of the prior deployment.

Using 'lsof' as such:

  lsof -nP +L1 /var | grep /var | grep '(deleted)' | less

We can see dozens of old deployments (old extracted war files?), like this:

  java      26734        root   82r   REG    8,6   575389     0  32470
  /var/tmp/webapp_tmp/jetty-0.0.0.0-80-webapp-_idp-any-4794158382650079166.dir/webinf/WEB-INF/lib/commons-collections-3.2.1.jar
  (deleted)
  java      26734        root  110r   REG    8,6   575389     0  32561
  /var/tmp/webapp_tmp/jetty-0.0.0.0-80-webapp-_idp-any-2567933012053298590.dir/webinf/WEB-INF/lib/commons-collections-3.2.1.jar
  (deleted)
  java      26734        root  318r   REG    8,6   575389     0  40227
  /var/tmp/webapp_tmp/jetty-0.0.0.0-80-webapp-_idp-any-4949209254558295977.dir/webinf/WEB-INF/lib/commons-collections-3.2.1.jar
  (deleted)
  ...

As the JVM is still holding a reference, the disk space never frees,
and our /var/ partition eventually fills. :/

Do other people see this?

Are we doing this wrong?  All we did was modify the scanInterval
setting to a nonzero value.

Is there some configuration switch we can throw to coerce the JVM
to really release these resources, or are we stuck waiting for the
GC to eventually cleanup?

Details of our environment:

  jetty-distribution-9.3.8.v20160314

  OpenJDK Runtime Environment (build 1.8.0_71-b15)

  CentOS release 6.7 (Final)

  # uname -r
  2.6.32-573.18.1.el6.x86_64

--
Brian Reichert                          <[hidden email]>
BSD admin/developer at large
_______________________________________________
jetty-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-dev


_______________________________________________
jetty-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-dev
Reply | Threaded
Open this post in threaded view
|

Re: [jetty-dev] jetty and 'hot-deploy', not freeing diskspace?

Brian Reichert
On Tue, Aug 23, 2016 at 05:11:24PM -0500, Jesse McConnell wrote:
> Hi Brian,
>
> Did you take a peek through these? It is typically the reason we see this
> sort of behavior.
>
> https://www.eclipse.org/jetty/documentation/current/preventing-memory-leaks.html

Will do; thanks for the pointer!

>
> cheers,
> Jesse
>
> --
> jesse mcconnell
> [hidden email]

--
Brian Reichert <[hidden email]>
BSD admin/developer at large
_______________________________________________
jetty-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-dev
Reply | Threaded
Open this post in threaded view
|

Re: [jetty-dev] jetty and 'hot-deploy', not freeing diskspace?

Brian Reichert
On Wed, Aug 24, 2016 at 01:17:01PM -0400, Brian Reichert wrote:
> On Tue, Aug 23, 2016 at 05:11:24PM -0500, Jesse McConnell wrote:
> > Hi Brian,
> >
> > Did you take a peek through these? It is typically the reason we see this
> > sort of behavior.
> >
> > https://www.eclipse.org/jetty/documentation/current/preventing-memory-leaks.html
>
> Will do; thanks for the pointer!

That was a quick read. :)

The pattern we see seems specific to the behavior of hot-deploy.

This is easily reproducible with stock jetty code:

In one window, spin up the demo-base app, with hot-deploy set:

  # cd /usr/jetty-distribution-9.3.8.v20160314/demo-base/
  # java -jar ../start.jar jetty.deploy.scanInterval=1

In another window, isolate the process ID, and show the current
count of deleted files, and the usage of the /tmp partition.

  # lsof -i :8080
  COMMAND  PID USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
  java    1463 root   89u  IPv6 5047191      0t0  TCP *:webcache (LISTEN)
 
  # lsof -nP +L1 | grep '(deleted)' | grep 1463 | wc -l
  0
  # df /tmp
  Filesystem     1K-blocks   Used Available Use% Mounted on
  /dev/sda2        2519792 470052   1921740  20% /
  # lsof -nP +L1 | grep '(deleted)' | grep 1463 | wc -l
  0

And periodically trigger hot-deploy, and observe the presence of
deleted files that the PID is still holding a reference to, and the
increase of utilization of the /tmp partition:

  # touch webapps/test.xml
  # lsof -nP +L1 | grep '(deleted)' | grep 1463 | wc -l
  7
  # df /tmp
  Filesystem     1K-blocks   Used Available Use% Mounted on
  /dev/sda2        2519792 470908   1920884  20% /

  # touch webapps/test.xml
  # lsof -nP +L1 | grep '(deleted)' | grep 1463 | wc -l
  14
  # df /tmp
  Filesystem     1K-blocks   Used Available Use% Mounted on
  /dev/sda2        2519792 471764   1920028  20% /

  # touch webapps/test.xml
  # lsof -nP +L1 | grep '(deleted)' | grep 1463 | wc -l
  21
  # df /tmp
  Filesystem     1K-blocks   Used Available Use% Mounted on
  /dev/sda2        2519792 472620   1919172  20% /
 
These count of deleted-but-referenced file keeps increasing, and
more of /tmp keeps getting consumed.

All of the deleted files are of the pattern:

  /tmp/jetty-0.0.0.0-8080-test.war-_test-any-XXXXXXXXXXXXXXXX.dir/...

Does anyone else see this behavior?

Again, details of our environment:

Details of our environment:

  jetty-distribution-9.3.8.v20160314

  OpenJDK Runtime Environment (build 1.8.0_71-b15)

  CentOS release 6.7 (Final)

  # uname -r
  2.6.32-573.18.1.el6.x86_64


--
Brian Reichert <[hidden email]>
BSD admin/developer at large
_______________________________________________
jetty-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-dev
Reply | Threaded
Open this post in threaded view
|

Re: [jetty-dev] jetty and 'hot-deploy', not freeing diskspace?

Jesse McConnell
well that doesn't look right, could you open a bug at https://github.com/eclipse/jetty.project/issues and we'll take a look

--
jesse mcconnell
[hidden email]

On Wed, Aug 24, 2016 at 12:59 PM, Brian Reichert <[hidden email]> wrote:
On Wed, Aug 24, 2016 at 01:17:01PM -0400, Brian Reichert wrote:
> On Tue, Aug 23, 2016 at 05:11:24PM -0500, Jesse McConnell wrote:
> > Hi Brian,
> >
> > Did you take a peek through these? It is typically the reason we see this
> > sort of behavior.
> >
> > https://www.eclipse.org/jetty/documentation/current/preventing-memory-leaks.html
>
> Will do; thanks for the pointer!

That was a quick read. :)

The pattern we see seems specific to the behavior of hot-deploy.

This is easily reproducible with stock jetty code:

In one window, spin up the demo-base app, with hot-deploy set:

  # cd /usr/jetty-distribution-9.3.8.v20160314/demo-base/
  # java -jar ../start.jar jetty.deploy.scanInterval=1

In another window, isolate the process ID, and show the current
count of deleted files, and the usage of the /tmp partition.

  # lsof -i :8080
  COMMAND  PID USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
  java    1463 root   89u  IPv6 5047191      0t0  TCP *:webcache (LISTEN)

  # lsof -nP +L1 | grep '(deleted)' | grep 1463 | wc -l
  0
  # df /tmp
  Filesystem     1K-blocks   Used Available Use% Mounted on
  /dev/sda2        2519792 470052   1921740  20% /
  # lsof -nP +L1 | grep '(deleted)' | grep 1463 | wc -l
  0

And periodically trigger hot-deploy, and observe the presence of
deleted files that the PID is still holding a reference to, and the
increase of utilization of the /tmp partition:

  # touch webapps/test.xml
  # lsof -nP +L1 | grep '(deleted)' | grep 1463 | wc -l
  7
  # df /tmp
  Filesystem     1K-blocks   Used Available Use% Mounted on
  /dev/sda2        2519792 470908   1920884  20% /

  # touch webapps/test.xml
  # lsof -nP +L1 | grep '(deleted)' | grep 1463 | wc -l
  14
  # df /tmp
  Filesystem     1K-blocks   Used Available Use% Mounted on
  /dev/sda2        2519792 471764   1920028  20% /

  # touch webapps/test.xml
  # lsof -nP +L1 | grep '(deleted)' | grep 1463 | wc -l
  21
  # df /tmp
  Filesystem     1K-blocks   Used Available Use% Mounted on
  /dev/sda2        2519792 472620   1919172  20% /

These count of deleted-but-referenced file keeps increasing, and
more of /tmp keeps getting consumed.

All of the deleted files are of the pattern:

  /tmp/jetty-0.0.0.0-8080-test.war-_test-any-XXXXXXXXXXXXXXXX.dir/...

Does anyone else see this behavior?

Again, details of our environment:

Details of our environment:

  jetty-distribution-9.3.8.v20160314

  OpenJDK Runtime Environment (build 1.8.0_71-b15)

  CentOS release 6.7 (Final)

  # uname -r
  2.6.32-573.18.1.el6.x86_64


--
Brian Reichert                          <[hidden email]>
BSD admin/developer at large
_______________________________________________
jetty-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-dev


_______________________________________________
jetty-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-dev
Reply | Threaded
Open this post in threaded view
|

Re: [jetty-dev] jetty and 'hot-deploy', not freeing diskspace?

Brian Reichert
On Wed, Aug 24, 2016 at 03:25:34PM -0500, Jesse McConnell wrote:
> well that doesn't look right, could you open a bug at
> https://github.com/eclipse/jetty.project/issues and we'll take a look

Done:

  https://github.com/eclipse/jetty.project/issues/867


--
Brian Reichert <[hidden email]>
BSD admin/developer at large
_______________________________________________
jetty-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-dev