Memory leak when server is loaded?

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Memory leak when server is loaded?

nate510
Hi all, hopefully someone can help me with this.

I'm seeing my Jetty stand-alone servers chew up available memory under load (-Xmx4g eventually all gets used).  Based on Java Object histograms, it seems like the problem has to do with Jetty, or how I have Jetty configured.  Here's the info I have:

Server: Jetty Hightide 7.0.0 (problem occurred before upgrading, tho, previously 6.1.11).
Setup: Stand-alone, running one production Java webapp: Spring 2.5, JSP
Load: ~20-250 req./second over the course of a day, distributed across three server instances on separate machines

Histogram shows large number of Jetty objects being retained at the request level:
# cat /tmp/histo.out | grep jetty
   5:       1683568      121216896  org.eclipse.jetty.http.HttpFields$Field
   6:       1370718       98691696  org.eclipse.jetty.io.View
  14:        550942       39667824  org.eclipse.jetty.io.ByteArrayBuffer
  15:         98505       29945520  org.eclipse.jetty.server.Request
  18:         98505       18124920  org.eclipse.jetty.server.nio.SelectChannelConnector$2
  22:        197325       14207400  org.eclipse.jetty.io.View$CaseInsensitive
  23:         98505       14184720  org.eclipse.jetty.http.HttpParser
  28:         98505       12608640  org.eclipse.jetty.http.HttpGenerator
  29:        116775       12144600  org.eclipse.jetty.server.session.HashSessionManager$Session
  30:         98505       11032560  org.eclipse.jetty.io.nio.SelectChannelEndPoint
  31:         98505        9456480  org.eclipse.jetty.server.Response
  32:         98505        8668440  org.eclipse.jetty.http.HttpURI
  33:         98068        8629984  org.eclipse.jetty.server.HttpConnection$Output
  38:        197010        7880400  org.eclipse.jetty.http.HttpFields
  41:         98505        7092360  org.eclipse.jetty.io.nio.SelectChannelEndPoint$IdleTask
  45:         98505        6304320  org.eclipse.jetty.server.AsyncContinuation
  46:         70473        6201624  org.eclipse.jetty.io.BufferCache$CachedBuffer
  58:         98557        3153824  org.eclipse.jetty.util.Utf8StringBuilder
  59:         98515        3152480  org.eclipse.jetty.util.MultiMap
  60:         98505        3152160  org.eclipse.jetty.server.HttpConnection$RequestHandler
  72:         98502        2364048  org.eclipse.jetty.util.AttributesMap
  87:         24008        1152384  org.eclipse.jetty.servlet.ServletHandler$CachedChain
 101:         11734         563232  org.eclipse.jetty.server.CookieCutter
(--truncated at 10K instance count--)

There are no comparable/suspicious counts for my-domain-level objects.  Based on a session-scoped object, there appear to be about 29K active sessions when this snapshot was taken, which seems a bit high, but not inconceivable, given our traffic type (Facebook apps).  There are <100 active requests present at the time of the snapshot.  

Seems like maybe old, fulfilled requests are not getting GCed and are hanging around in the idle pool?  Any ideas?

Thanks in advance!