[jetty-dev] Troubles with embedded Jetty Session scavenging

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

[jetty-dev] Troubles with embedded Jetty Session scavenging

Paul Palaszewski

Hi everybody,

 

I’ve been using Jetty for quite a while and a few months ago we decided to switch from the standard start.jar-launcher to our own. Main reasons are greatly improved deployments (startup) time by using a own largly simplified hot-WebApp deployer and automatic SSL certificate updates through our own letsencrypt-client.

 

Having troubles with the Session HouseKeeper:

Using simple, standard in-memory sessions. Scavenging starts and initially works fine. I can debug it and see that it collects expired sessions.

But as soon as I let it run on productive systems for a few days at some point in time it stops and all expired sessions remain in memory which leads to OutOfMemory-errors after a few weeks.

Works fine with the original start.jar-launcher, experienced this problem only with our custom server.

Using jetty 9.4.7 and 9.4.8, running on Oracle JDK 9.0.1 (was the same with 9.0) on Windows 10, various Linux distros and macOS 10.13.2 .

 

I’m initializing Server / Session Manager this way …

    server = new org.eclipse.jetty.server.Server();

    DefaultSessionIdManager session = new DefaultSessionIdManager(server);

    HouseKeeper houseKeeper = new HouseKeeper();

    houseKeeper.setIntervalSec(30);

    session.setSessionHouseKeeper(houseKeeper);

    server.setSessionIdManager(session);

 

Have not found a "Stopped scavenging" or any other clue in the logs.

 

Any ideas, what I’m doing wrong or how to narrow down the issue?

 

Best regards,

Paul

 

And the server dump …

org.eclipse.jetty.server.Server@3aefe5e5[9.4.z-SNAPSHOT] - STARTED

+= QueuedThreadPool@qtp674483268{STARTED,8<=10<=200,i=2,q=0} - STARTED

|   +- 401 qtp674483268-401 IDLE TIMED_WAITING @ java.base@9.0.1/jdk.internal.misc.Unsafe.park(Native Method)

|   +- 402 qtp674483268-402 IDLE TIMED_WAITING @ java.base@9.0.1/jdk.internal.misc.Unsafe.park(Native Method)

|   +- 221 qtp674483268-221 SELECTING RUNNABLE @ java.base@9.0.1/sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

|   +- 18 qtp674483268-18-acceptor-0@ee4204f-ServerConnector@3fa247d1{SSL,[ssl, alpn, h2, http/1.1]}{0.0.0.0:443} ACCEPTING RUNNABLE @ java.base@9.0.1/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) prio=3

|   +- 20 qtp674483268-20 SELECTING RUNNABLE @ java.base@9.0.1/sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

|   +- 400 qtp674483268-400 RESERVED TIMED_WAITING @ java.base@9.0.1/jdk.internal.misc.Unsafe.park(Native Method)

|   +- 21 qtp674483268-21 SELECTING RUNNABLE @ java.base@9.0.1/sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

|   +- 22 qtp674483268-22-acceptor-0@2c9c6f53-ServerConnector@5c86a017{HTTP/1.1,[http/1.1, h2c]}{0.0.0.0:80} ACCEPTING RUNNABLE @ java.base@9.0.1/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) prio=3

|   +- 23 0:0:0:0:0:0:0:1 2018-01-12 15:36:35.055 GET /About RUNNABLE @ java.base/java.lang.Thread.getStackTrace(Thread.java:1654)

|   +- 78 qtp674483268-78 SELECTING RUNNABLE @ java.base@9.0.1/sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

|   +- jobs

+= org.eclipse.jetty.server.session.DefaultSessionIdManager@1de47212[worker=node0] - STARTED

|   +? org.eclipse.jetty.server.session.HouseKeeper@501792b1[interval=33000, ownscheduler=true] - STARTED

+= ServerConnector@5c86a017{HTTP/1.1,[http/1.1, h2c]}{0.0.0.0:80} - STARTED

|   +~ org.eclipse.jetty.server.Server@3aefe5e5[9.4.z-SNAPSHOT] - STARTED

|   +~ QueuedThreadPool@qtp674483268{STARTED,8<=10<=200,i=2,q=0} - STARTED

|   += org.eclipse.jetty.util.thread.ScheduledExecutorScheduler@b7dd107 - STARTED

|   |   +- java.base@9.0.1/jdk.internal.misc.Unsafe.park(Native Method)

|   |   +- java.base@9.0.1/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)

|   |   +- java.base@9.0.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2104)

|   |   +- java.base@9.0.1/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1131)

|   |   +- java.base@9.0.1/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:848)

|   |   +- java.base@9.0.1/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1092)

|   |   +- java.base@9.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152)

|   |   +- java.base@9.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)

|   |   +- java.base@9.0.1/java.lang.Thread.run(Thread.java:844)

|   +- org.eclipse.jetty.io.ArrayByteBufferPool@7caa80e4

|   += HttpConnectionFactory@cb0ed20[HTTP/1.1] - STARTED

|   |   +- HttpConfiguration@18c5cd7d{32768/8192,8192/8192,https://:443,[]}

|   += SelectorManager@ServerConnector@5c86a017{HTTP/1.1,[http/1.1, h2c]}{0.0.0.0:80} - STARTED

|   |   += ReservedThreadExecutor@268f5895{s=1/4,p=0}@SelectorManager@ServerConnector@5c86a017{HTTP/1.1,[http/1.1, h2c]}{0.0.0.0:80} - STARTED

|   |   += org.eclipse.jetty.io.ManagedSelector@4a61c4ef id=0 keys=0 selected=0 actions=0 - STARTED

|   |   |   += EatWhatYouKill@62e4c0e7/SelectorProducer@1d4ff7ab/PRODUCING/ReservedThreadExecutor@268f5895{s=1/4,p=0}@SelectorManager@ServerConnector@5c86a017{HTTP/1.1,[http/1.1, h2c]}{0.0.0.0:80} - STARTED

|   |   |       +- SelectorProducer@1d4ff7ab

|   |   |   +- keys

|   |   |   |   +- sun.nio.ch.WindowsSelectorImpl@3f46ce65 keys=0

|   |   |   +- actions

|   |   += org.eclipse.jetty.io.ManagedSelector@5a02a8a4 id=1 keys=1 selected=0 actions=0 - STARTED

|   |       += EatWhatYouKill@1e02ae9b/SelectorProducer@7c92e05f/PRODUCING/ReservedThreadExecutor@268f5895{s=1/4,p=0}@SelectorManager@ServerConnector@5c86a017{HTTP/1.1,[http/1.1, h2c]}{0.0.0.0:80} - STARTED

|   |           +- SelectorProducer@7c92e05f

|   |       +- keys

|   |       |   +- sun.nio.ch.WindowsSelectorImpl@2bdff7b9 keys=1

|   |       |   +- SelectionKey@67dc299b{i=0}->SocketChannelEndPoint@168b474c{/0:0:0:0:0:0:0:1:52677<->/0:0:0:0:0:0:0:1:80,OPEN,fill=-,flush=-,to=119/30000}{io=0/0,kio=0,kro=1}->HttpConnection@5d013a2c[p=HttpParser{s=END,0 of -1},g=HttpGenerator@593eb554{s=START}]=>HttpChannelOverHttp@1c3c3017{r=1,c=false,a=DISPATCHED,uri=//localhost/4p/About?b9=}

|   |       +- actions

|   += HTTP2CServerConnectionFactory@2fd66ad3[h2c] - STARTED

|   |   +- HttpConfiguration@18c5cd7d{32768/8192,8192/8192,https://:443,[]}

|   +- sun.nio.ch.ServerSocketChannelImpl[/0:0:0:0:0:0:0:0:80]

|   +- qtp674483268-22-acceptor-0@2c9c6f53-ServerConnector@5c86a017{HTTP/1.1,[http/1.1, h2c]}{0.0.0.0:80}

+= ServerConnector@3fa247d1{SSL,[ssl, alpn, h2, http/1.1]}{0.0.0.0:443} - STARTED

|   +~ org.eclipse.jetty.server.Server@3aefe5e5[9.4.z-SNAPSHOT] - STARTED

|   +~ QueuedThreadPool@qtp674483268{STARTED,8<=10<=200,i=2,q=0} - STARTED

|   += org.eclipse.jetty.util.thread.ScheduledExecutorScheduler@4b9e255 - STARTED

|   +- org.eclipse.jetty.io.ArrayByteBufferPool@2cc73a0f

|   += SelectorManager@ServerConnector@3fa247d1{SSL,[ssl, alpn, h2, http/1.1]}{0.0.0.0:443} - STARTED

|   |   += ReservedThreadExecutor@63cff0bb{s=0/4,p=0}@SelectorManager@ServerConnector@3fa247d1{SSL,[ssl, alpn, h2, http/1.1]}{0.0.0.0:443} - STARTED

|   |   += org.eclipse.jetty.io.ManagedSelector@6832fc2f id=0 keys=0 selected=0 actions=0 - STARTED

|   |   |   += EatWhatYouKill@3c9696e/SelectorProducer@16cb2c5a/PRODUCING/ReservedThreadExecutor@63cff0bb{s=0/4,p=0}@SelectorManager@ServerConnector@3fa247d1{SSL,[ssl, alpn, h2, http/1.1]}{0.0.0.0:443} - STARTED

|   |   |       +- SelectorProducer@16cb2c5a

|   |   |   +- keys

|   |   |   |   +- sun.nio.ch.WindowsSelectorImpl@5f633211 keys=0

|   |   |   +- actions

|   |   += org.eclipse.jetty.io.ManagedSelector@a9d5c10 id=1 keys=0 selected=0 actions=0 - STARTED

|   |       += EatWhatYouKill@75311ae8/SelectorProducer@3d7b9f39/PRODUCING/ReservedThreadExecutor@63cff0bb{s=0/4,p=0}@SelectorManager@ServerConnector@3fa247d1{SSL,[ssl, alpn, h2, http/1.1]}{0.0.0.0:443} - STARTED

|   |           +- SelectorProducer@3d7b9f39

|   |       +- keys

|   |       |   +- sun.nio.ch.WindowsSelectorImpl@5c443958 keys=0

|   |       +- actions

|   += SslConnectionFactory@d6da883{SSL->alpn} - STARTED

|   |   += SslContextFactory@642a7222[provider=null,keyStore=file:///C:/Entw/btkrt/etc/keystore,trustStore=null] trustAll=false

|   |       +- Protocol Selections

|   |       |   +- Enabled (size=3)

|   |       |   |   +- TLSv1

|   |       |   |   +- TLSv1.1

|   |       |   |   +- TLSv1.2

|   |       |   +- Disabled (size=2)

|   |       |       +- SSLv2Hello - ConfigExcluded:'SSLv2Hello'

|   |       |       +- SSLv3 - JreDisabled:java.security, ConfigExcluded:'SSLv3'

|   |       +- Cipher Suite Selections

|   |           +- Enabled (size=13)

|   |           |   +- SSL_RSA_WITH_3DES_EDE_CBC_SHA

|   |           |   +- TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA

|   |           |   +- TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256

|   |           |   +- TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256

|   |           |   +- TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA

|   |           |   +- TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384

|   |           |   +- TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384

|   |           |   +- TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA

|   |           |   +- TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256

|   |           |   +- TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256

|   |           |   +- TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA

|   |           |   +- TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384

|   |           |   +- TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384

|   |           +- Disabled (size=69)

|   |               +- SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified

|   |               +- SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA - ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified

|   |               +- SSL_DHE_DSS_WITH_DES_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified

|   |               +- SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified

|   |               +- SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- SSL_DHE_RSA_WITH_DES_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified

|   |               +- SSL_DH_anon_EXPORT_WITH_DES40_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified

|   |               +- SSL_DH_anon_WITH_3DES_EDE_CBC_SHA - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- SSL_DH_anon_WITH_DES_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified

|   |               +- SSL_RSA_EXPORT_WITH_DES40_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified

|   |               +- SSL_RSA_WITH_DES_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified

|   |               +- SSL_RSA_WITH_NULL_MD5 - JreDisabled:java.security, ConfigExcluded:'.*NULL.*', ConfigExcluded:'.*MD5.*', ConfigIncluded:NotSpecified

|   |               +- SSL_RSA_WITH_NULL_SHA - JreDisabled:java.security, ConfigExcluded:'.*NULL.*', ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_DSS_WITH_AES_128_CBC_SHA - ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_DSS_WITH_AES_128_CBC_SHA256 - ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_DSS_WITH_AES_128_GCM_SHA256 - ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_DSS_WITH_AES_256_CBC_SHA - ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_DSS_WITH_AES_256_CBC_SHA256 - ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_DSS_WITH_AES_256_GCM_SHA384 - ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_RSA_WITH_AES_128_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_RSA_WITH_AES_128_CBC_SHA256 - ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 - ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_RSA_WITH_AES_256_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 - ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 - ConfigIncluded:NotSpecified

|   |               +- TLS_DH_anon_WITH_AES_128_CBC_SHA - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- TLS_DH_anon_WITH_AES_128_CBC_SHA256 - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- TLS_DH_anon_WITH_AES_128_GCM_SHA256 - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- TLS_DH_anon_WITH_AES_256_CBC_SHA - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- TLS_DH_anon_WITH_AES_256_CBC_SHA256 - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- TLS_DH_anon_WITH_AES_256_GCM_SHA384 - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDHE_ECDSA_WITH_NULL_SHA - JreDisabled:java.security, ConfigExcluded:'.*NULL.*', ConfigIncluded:NotSpecified

|   |               +- TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDHE_RSA_WITH_NULL_SHA - JreDisabled:java.security, ConfigExcluded:'.*NULL.*', ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256 - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256 - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384 - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384 - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_ECDSA_WITH_NULL_SHA - JreDisabled:java.security, ConfigExcluded:'.*NULL.*', ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_RSA_WITH_AES_128_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256 - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256 - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_RSA_WITH_AES_256_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384 - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384 - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_RSA_WITH_NULL_SHA - JreDisabled:java.security, ConfigExcluded:'.*NULL.*', ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_anon_WITH_3DES_EDE_CBC_SHA - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_anon_WITH_AES_128_CBC_SHA - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_anon_WITH_AES_256_CBC_SHA - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_anon_WITH_NULL_SHA - JreDisabled:java.security, ConfigExcluded:'.*NULL.*', ConfigIncluded:NotSpecified

|   |               +- TLS_EMPTY_RENEGOTIATION_INFO_SCSV - ConfigIncluded:NotSpecified

|   |               +- TLS_KRB5_EXPORT_WITH_DES_CBC_40_MD5 - JreDisabled:java.security, ConfigExcluded:'.*MD5.*', ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified

|   |               +- TLS_KRB5_EXPORT_WITH_DES_CBC_40_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified

|   |               +- TLS_KRB5_WITH_3DES_EDE_CBC_MD5 - JreDisabled:java.security, ConfigExcluded:'.*MD5.*', ConfigIncluded:NotSpecified

|   |               +- TLS_KRB5_WITH_3DES_EDE_CBC_SHA - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- TLS_KRB5_WITH_DES_CBC_MD5 - JreDisabled:java.security, ConfigExcluded:'.*MD5.*', ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified

|   |               +- TLS_KRB5_WITH_DES_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified

|   |               +- TLS_RSA_WITH_AES_128_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_RSA_WITH_AES_128_CBC_SHA256 - ConfigIncluded:NotSpecified

|   |               +- TLS_RSA_WITH_AES_128_GCM_SHA256 - ConfigIncluded:NotSpecified

|   |               +- TLS_RSA_WITH_AES_256_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_RSA_WITH_AES_256_CBC_SHA256 - ConfigIncluded:NotSpecified

|   |               +- TLS_RSA_WITH_AES_256_GCM_SHA384 - ConfigIncluded:NotSpecified

|   |               +- TLS_RSA_WITH_NULL_SHA256 - JreDisabled:java.security, ConfigExcluded:'.*NULL.*', ConfigIncluded:NotSpecified

|   += ALPNServerConnectionFactory@45afc369{[alpn],HTTP/1.1,[]} - STARTED

|   += HTTP2ServerConnectionFactory@70beb599[h2] - STARTED

|   |   +- HttpConfiguration@7dd64052{32768/8192,8192/8192,https://:443,[SecureRequestCustomizer@35528b51]}

|   += HttpConnectionFactory@4e41089d[HTTP/1.1] - STARTED

|   |   +- HttpConfiguration@7dd64052{32768/8192,8192/8192,https://:443,[SecureRequestCustomizer@35528b51]}

|   +- sun.nio.ch.ServerSocketChannelImpl[/0:0:0:0:0:0:0:0:443]

|   +- qtp674483268-18-acceptor-0@ee4204f-ServerConnector@3fa247d1{SSL,[ssl, alpn, h2, http/1.1]}{0.0.0.0:443}

+= com.bdm.bwebserver.Server$2@190bae5a - STARTED

|   +> webapps

|       +- /bc - STARTED

|       +- /bdm - STARTED

+= org.eclipse.jetty.server.handler.ErrorHandler@67eb0042 - STARTED

+> jdk.internal.loader.ClassLoaders$AppClassLoader@504bae78

     +- jdk.internal.loader.ClassLoaders$PlatformClassLoader@17ed40e0


_______________________________________________
jetty-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-dev
Reply | Threaded
Open this post in threaded view
|

Re: [jetty-dev] Troubles with embedded Jetty Session scavenging

Jan Bartel
Are you sure that the OOM happens after the scavenger dies and not that the scavenger dies because of OOM?

I would connect to it with jconsole or similar tool and keep looking at the running threads and the memory and see when things start to go wrong.

Also turn up session logging to debug level to capture more info.

Also try with a profiler and see if you can grab a heap dump to analyse.

Can you reproduce in a test environment?

Can you tell me what all of the session related configuration is, including in your web.xml, and anything you do in code, and I'll see if I can reproduce.

Jan

On 12 Jan. 2018 16:16, "Paul Palaszewski" <[hidden email]> wrote:

Hi everybody,

 

I’ve been using Jetty for quite a while and a few months ago we decided to switch from the standard start.jar-launcher to our own. Main reasons are greatly improved deployments (startup) time by using a own largly simplified hot-WebApp deployer and automatic SSL certificate updates through our own letsencrypt-client.

 

Having troubles with the Session HouseKeeper:

Using simple, standard in-memory sessions. Scavenging starts and initially works fine. I can debug it and see that it collects expired sessions.

But as soon as I let it run on productive systems for a few days at some point in time it stops and all expired sessions remain in memory which leads to OutOfMemory-errors after a few weeks.

Works fine with the original start.jar-launcher, experienced this problem only with our custom server.

Using jetty 9.4.7 and 9.4.8, running on Oracle JDK 9.0.1 (was the same with 9.0) on Windows 10, various Linux distros and macOS 10.13.2 .

 

I’m initializing Server / Session Manager this way …

    server = new org.eclipse.jetty.server.Server();

    DefaultSessionIdManager session = new DefaultSessionIdManager(server);

    HouseKeeper houseKeeper = new HouseKeeper();

    houseKeeper.setIntervalSec(30);

    session.setSessionHouseKeeper(houseKeeper);

    server.setSessionIdManager(session);

 

Have not found a "Stopped scavenging" or any other clue in the logs.

 

Any ideas, what I’m doing wrong or how to narrow down the issue?

 

Best regards,

Paul

 

And the server dump …

org.eclipse.jetty.server.Server@3aefe5e5[9.4.z-SNAPSHOT] - STARTED

+= QueuedThreadPool@qtp674483268{STARTED,8<=10<=200,i=2,q=0} - STARTED

|   +- 401 qtp674483268-401 IDLE TIMED_WAITING @ java.base@9.0.1/jdk.internal.misc.Unsafe.park(Native Method)

|   +- 402 qtp674483268-402 IDLE TIMED_WAITING @ java.base@9.0.1/jdk.internal.misc.Unsafe.park(Native Method)

|   +- 221 qtp674483268-221 SELECTING RUNNABLE @ java.base@9.0.1/sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

|   +- 18 qtp674483268-18-acceptor-0@ee4204f-ServerConnector@3fa247d1{SSL,[ssl, alpn, h2, http/1.1]}{0.0.0.0:443} ACCEPTING RUNNABLE @ java.base@9.0.1/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) prio=3

|   +- 20 qtp674483268-20 SELECTING RUNNABLE @ java.base@9.0.1/sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

|   +- 400 qtp674483268-400 RESERVED TIMED_WAITING @ java.base@9.0.1/jdk.internal.misc.Unsafe.park(Native Method)

|   +- 21 qtp674483268-21 SELECTING RUNNABLE @ java.base@9.0.1/sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

|   +- 22 qtp674483268-22-acceptor-0@2c9c6f53-ServerConnector@5c86a017{HTTP/1.1,[http/1.1, h2c]}{0.0.0.0:80} ACCEPTING RUNNABLE @ java.base@9.0.1/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) prio=3

|   +- 23 0:0:0:0:0:0:0:1 2018-01-12 15:36:35.055 GET /About RUNNABLE @ java.base/java.lang.Thread.getStackTrace(Thread.java:1654)

|   +- 78 qtp674483268-78 SELECTING RUNNABLE @ java.base@9.0.1/sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

|   +- jobs

+= org.eclipse.jetty.server.session.DefaultSessionIdManager@1de47212[worker=node0] - STARTED

|   +? org.eclipse.jetty.server.session.HouseKeeper@501792b1[interval=33000, ownscheduler=true] - STARTED

+= ServerConnector@5c86a017{HTTP/1.1,[http/1.1, h2c]}{0.0.0.0:80} - STARTED

|   +~ org.eclipse.jetty.server.Server@3aefe5e5[9.4.z-SNAPSHOT] - STARTED

|   +~ QueuedThreadPool@qtp674483268{STARTED,8<=10<=200,i=2,q=0} - STARTED

|   += org.eclipse.jetty.util.thread.ScheduledExecutorScheduler@b7dd107 - STARTED

|   |   +- java.base@9.0.1/jdk.internal.misc.Unsafe.park(Native Method)

|   |   +- java.base@9.0.1/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)

|   |   +- java.base@9.0.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2104)

|   |   +- java.base@9.0.1/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1131)

|   |   +- java.base@9.0.1/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:848)

|   |   +- java.base@9.0.1/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1092)

|   |   +- java.base@9.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152)

|   |   +- java.base@9.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)

|   |   +- java.base@9.0.1/java.lang.Thread.run(Thread.java:844)

|   +- org.eclipse.jetty.io.ArrayByteBufferPool@7caa80e4

|   += HttpConnectionFactory@cb0ed20[HTTP/1.1] - STARTED

|   |   +- HttpConfiguration@18c5cd7d{32768/8192,8192/8192,https://:443,[]}

|   += SelectorManager@ServerConnector@5c86a017{HTTP/1.1,[http/1.1, h2c]}{0.0.0.0:80} - STARTED

|   |   += ReservedThreadExecutor@268f5895{s=1/4,p=0}@SelectorManager@ServerConnector@5c86a017{HTTP/1.1,[http/1.1, h2c]}{0.0.0.0:80} - STARTED

|   |   += org.eclipse.jetty.io.ManagedSelector@4a61c4ef id=0 keys=0 selected=0 actions=0 - STARTED

|   |   |   += EatWhatYouKill@62e4c0e7/SelectorProducer@1d4ff7ab/PRODUCING/ReservedThreadExecutor@268f5895{s=1/4,p=0}@SelectorManager@ServerConnector@5c86a017{HTTP/1.1,[http/1.1, h2c]}{0.0.0.0:80} - STARTED

|   |   |       +- SelectorProducer@1d4ff7ab

|   |   |   +- keys

|   |   |   |   +- sun.nio.ch.WindowsSelectorImpl@3f46ce65 keys=0

|   |   |   +- actions

|   |   += org.eclipse.jetty.io.ManagedSelector@5a02a8a4 id=1 keys=1 selected=0 actions=0 - STARTED

|   |       += EatWhatYouKill@1e02ae9b/SelectorProducer@7c92e05f/PRODUCING/ReservedThreadExecutor@268f5895{s=1/4,p=0}@SelectorManager@ServerConnector@5c86a017{HTTP/1.1,[http/1.1, h2c]}{0.0.0.0:80} - STARTED

|   |           +- SelectorProducer@7c92e05f

|   |       +- keys

|   |       |   +- sun.nio.ch.WindowsSelectorImpl@2bdff7b9 keys=1

|   |       |   +- SelectionKey@67dc299b{i=0}->SocketChannelEndPoint@168b474c{/0:0:0:0:0:0:0:1:52677<->/0:0:0:0:0:0:0:1:80,OPEN,fill=-,flush=-,to=119/30000}{io=0/0,kio=0,kro=1}->HttpConnection@5d013a2c[p=HttpParser{s=END,0 of -1},g=HttpGenerator@593eb554{s=START}]=>HttpChannelOverHttp@1c3c3017{r=1,c=false,a=DISPATCHED,uri=//localhost/4p/About?b9=}

|   |       +- actions

|   += HTTP2CServerConnectionFactory@2fd66ad3[h2c] - STARTED

|   |   +- HttpConfiguration@18c5cd7d{32768/8192,8192/8192,https://:443,[]}

|   +- sun.nio.ch.ServerSocketChannelImpl[/0:0:0:0:0:0:0:0:80]

|   +- qtp674483268-22-acceptor-0@2c9c6f53-ServerConnector@5c86a017{HTTP/1.1,[http/1.1, h2c]}{0.0.0.0:80}

+= ServerConnector@3fa247d1{SSL,[ssl, alpn, h2, http/1.1]}{0.0.0.0:443} - STARTED

|   +~ org.eclipse.jetty.server.Server@3aefe5e5[9.4.z-SNAPSHOT] - STARTED

|   +~ QueuedThreadPool@qtp674483268{STARTED,8<=10<=200,i=2,q=0} - STARTED

|   += org.eclipse.jetty.util.thread.ScheduledExecutorScheduler@4b9e255 - STARTED

|   +- org.eclipse.jetty.io.ArrayByteBufferPool@2cc73a0f

|   += SelectorManager@ServerConnector@3fa247d1{SSL,[ssl, alpn, h2, http/1.1]}{0.0.0.0:443} - STARTED

|   |   += ReservedThreadExecutor@63cff0bb{s=0/4,p=0}@SelectorManager@ServerConnector@3fa247d1{SSL,[ssl, alpn, h2, http/1.1]}{0.0.0.0:443} - STARTED

|   |   += org.eclipse.jetty.io.ManagedSelector@6832fc2f id=0 keys=0 selected=0 actions=0 - STARTED

|   |   |   += EatWhatYouKill@3c9696e/SelectorProducer@16cb2c5a/PRODUCING/ReservedThreadExecutor@63cff0bb{s=0/4,p=0}@SelectorManager@ServerConnector@3fa247d1{SSL,[ssl, alpn, h2, http/1.1]}{0.0.0.0:443} - STARTED

|   |   |       +- SelectorProducer@16cb2c5a

|   |   |   +- keys

|   |   |   |   +- sun.nio.ch.WindowsSelectorImpl@5f633211 keys=0

|   |   |   +- actions

|   |   += org.eclipse.jetty.io.ManagedSelector@a9d5c10 id=1 keys=0 selected=0 actions=0 - STARTED

|   |       += EatWhatYouKill@75311ae8/SelectorProducer@3d7b9f39/PRODUCING/ReservedThreadExecutor@63cff0bb{s=0/4,p=0}@SelectorManager@ServerConnector@3fa247d1{SSL,[ssl, alpn, h2, http/1.1]}{0.0.0.0:443} - STARTED

|   |           +- SelectorProducer@3d7b9f39

|   |       +- keys

|   |       |   +- sun.nio.ch.WindowsSelectorImpl@5c443958 keys=0

|   |       +- actions

|   += SslConnectionFactory@d6da883{SSL->alpn} - STARTED

|   |   += SslContextFactory@642a7222[provider=null,keyStore=file:///C:/Entw/btkrt/etc/keystore,trustStore=null] trustAll=false

|   |       +- Protocol Selections

|   |       |   +- Enabled (size=3)

|   |       |   |   +- TLSv1

|   |       |   |   +- TLSv1.1

|   |       |   |   +- TLSv1.2

|   |       |   +- Disabled (size=2)

|   |       |       +- SSLv2Hello - ConfigExcluded:'SSLv2Hello'

|   |       |       +- SSLv3 - JreDisabled:java.security, ConfigExcluded:'SSLv3'

|   |       +- Cipher Suite Selections

|   |           +- Enabled (size=13)

|   |           |   +- SSL_RSA_WITH_3DES_EDE_CBC_SHA

|   |           |   +- TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA

|   |           |   +- TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256

|   |           |   +- TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256

|   |           |   +- TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA

|   |           |   +- TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384

|   |           |   +- TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384

|   |           |   +- TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA

|   |           |   +- TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256

|   |           |   +- TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256

|   |           |   +- TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA

|   |           |   +- TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384

|   |           |   +- TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384

|   |           +- Disabled (size=69)

|   |               +- SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified

|   |               +- SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA - ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified

|   |               +- SSL_DHE_DSS_WITH_DES_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified

|   |               +- SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified

|   |               +- SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- SSL_DHE_RSA_WITH_DES_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified

|   |               +- SSL_DH_anon_EXPORT_WITH_DES40_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified

|   |               +- SSL_DH_anon_WITH_3DES_EDE_CBC_SHA - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- SSL_DH_anon_WITH_DES_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified

|   |               +- SSL_RSA_EXPORT_WITH_DES40_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified

|   |               +- SSL_RSA_WITH_DES_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified

|   |               +- SSL_RSA_WITH_NULL_MD5 - JreDisabled:java.security, ConfigExcluded:'.*NULL.*', ConfigExcluded:'.*MD5.*', ConfigIncluded:NotSpecified

|   |               +- SSL_RSA_WITH_NULL_SHA - JreDisabled:java.security, ConfigExcluded:'.*NULL.*', ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_DSS_WITH_AES_128_CBC_SHA - ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_DSS_WITH_AES_128_CBC_SHA256 - ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_DSS_WITH_AES_128_GCM_SHA256 - ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_DSS_WITH_AES_256_CBC_SHA - ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_DSS_WITH_AES_256_CBC_SHA256 - ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_DSS_WITH_AES_256_GCM_SHA384 - ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_RSA_WITH_AES_128_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_RSA_WITH_AES_128_CBC_SHA256 - ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 - ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_RSA_WITH_AES_256_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 - ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 - ConfigIncluded:NotSpecified

|   |               +- TLS_DH_anon_WITH_AES_128_CBC_SHA - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- TLS_DH_anon_WITH_AES_128_CBC_SHA256 - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- TLS_DH_anon_WITH_AES_128_GCM_SHA256 - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- TLS_DH_anon_WITH_AES_256_CBC_SHA - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- TLS_DH_anon_WITH_AES_256_CBC_SHA256 - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- TLS_DH_anon_WITH_AES_256_GCM_SHA384 - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDHE_ECDSA_WITH_NULL_SHA - JreDisabled:java.security, ConfigExcluded:'.*NULL.*', ConfigIncluded:NotSpecified

|   |               +- TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDHE_RSA_WITH_NULL_SHA - JreDisabled:java.security, ConfigExcluded:'.*NULL.*', ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256 - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256 - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384 - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384 - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_ECDSA_WITH_NULL_SHA - JreDisabled:java.security, ConfigExcluded:'.*NULL.*', ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_RSA_WITH_AES_128_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256 - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256 - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_RSA_WITH_AES_256_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384 - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384 - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_RSA_WITH_NULL_SHA - JreDisabled:java.security, ConfigExcluded:'.*NULL.*', ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_anon_WITH_3DES_EDE_CBC_SHA - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_anon_WITH_AES_128_CBC_SHA - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_anon_WITH_AES_256_CBC_SHA - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_anon_WITH_NULL_SHA - JreDisabled:java.security, ConfigExcluded:'.*NULL.*', ConfigIncluded:NotSpecified

|   |               +- TLS_EMPTY_RENEGOTIATION_INFO_SCSV - ConfigIncluded:NotSpecified

|   |               +- TLS_KRB5_EXPORT_WITH_DES_CBC_40_MD5 - JreDisabled:java.security, ConfigExcluded:'.*MD5.*', ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified

|   |               +- TLS_KRB5_EXPORT_WITH_DES_CBC_40_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified

|   |               +- TLS_KRB5_WITH_3DES_EDE_CBC_MD5 - JreDisabled:java.security, ConfigExcluded:'.*MD5.*', ConfigIncluded:NotSpecified

|   |               +- TLS_KRB5_WITH_3DES_EDE_CBC_SHA - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- TLS_KRB5_WITH_DES_CBC_MD5 - JreDisabled:java.security, ConfigExcluded:'.*MD5.*', ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified

|   |               +- TLS_KRB5_WITH_DES_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified

|   |               +- TLS_RSA_WITH_AES_128_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_RSA_WITH_AES_128_CBC_SHA256 - ConfigIncluded:NotSpecified

|   |               +- TLS_RSA_WITH_AES_128_GCM_SHA256 - ConfigIncluded:NotSpecified

|   |               +- TLS_RSA_WITH_AES_256_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_RSA_WITH_AES_256_CBC_SHA256 - ConfigIncluded:NotSpecified

|   |               +- TLS_RSA_WITH_AES_256_GCM_SHA384 - ConfigIncluded:NotSpecified

|   |               +- TLS_RSA_WITH_NULL_SHA256 - JreDisabled:java.security, ConfigExcluded:'.*NULL.*', ConfigIncluded:NotSpecified

|   += ALPNServerConnectionFactory@45afc369{[alpn],HTTP/1.1,[]} - STARTED

|   += HTTP2ServerConnectionFactory@70beb599[h2] - STARTED

|   |   +- HttpConfiguration@7dd64052{32768/8192,8192/8192,https://:443,[SecureRequestCustomizer@35528b51]}

|   += HttpConnectionFactory@4e41089d[HTTP/1.1] - STARTED

|   |   +- HttpConfiguration@7dd64052{32768/8192,8192/8192,https://:443,[SecureRequestCustomizer@35528b51]}

|   +- sun.nio.ch.ServerSocketChannelImpl[/0:0:0:0:0:0:0:0:443]

|   +- qtp674483268-18-acceptor-0@ee4204f-ServerConnector@3fa247d1{SSL,[ssl, alpn, h2, http/1.1]}{0.0.0.0:443}

+= com.bdm.bwebserver.Server$2@190bae5a - STARTED

|   +> webapps

|       +- /bc - STARTED

|       +- /bdm - STARTED

+= org.eclipse.jetty.server.handler.ErrorHandler@67eb0042 - STARTED

+> jdk.internal.loader.ClassLoaders$AppClassLoader@504bae78

     +- jdk.internal.loader.ClassLoaders$PlatformClassLoader@17ed40e0


_______________________________________________
jetty-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-dev

_______________________________________________
jetty-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-dev
Reply | Threaded
Open this post in threaded view
|

Re: [jetty-dev] Troubles with embedded Jetty Session scavenging

Paul Palaszewski
Hi Jan,

thanks for the first feedback!

yes, I’m sure that OutOfMemory occurs because of the ungarbaged sessions. Loosing a few kb with every open session. Our web apps have an end-user-friendly jmx-like monitoring. there I can see it pretty clear. Our apps are usually running for months and years without interrupt, we have no leaking memory issues there.

No, up to now I could not reproduce it - otherwise I would have tackled it already myself. 

One more thing that should not, but might interfere: The app keeps track of active sessions. It implements HttpSessionListener interface and our WebApp (extending ServletContextHandler) uses addEventListener() for itself and for the session manager to capture sessionCreated() and sessionDestroyed() to update it’s own session list. Up to some time it get the events, later on it does not anymore.

I usually use that in our monitoring view to verify if any new changes solve the issue - again, for some time I see sessions disappearing and then they remain that while it shows that the maxInactiveInterval was set to something reasonable but it sticks there for days without any further usage/access.

WebApp / Session setup:
bdm.Server creates DefaultSessionIdManager and HouseKeeper as described in my previous mail
bdm.Server scans for .war-files and creates a bdm.WebApp extending ServletContextHandler
bdm.WepApp reads WEB-INF/web.xml which contains 1 servlet with load-on-startup + 1 servlet filter + 1 listener-class + following session config
<session-config>
<cookie-config>
<name>SESSIONID</name>
<http-only>true</http-only>
<max-age>30000</max-age>
</cookie-config>
</session-config>
which is applied to the WebApps getSessionHandler().getCookieConfig()

on war-file change the app is stopped, removed from the contexts and a new one is added. Happens every few weeks. The hanging scavenger occurs regardless also if we don’t update / redeploy any apps.

The app has a config option to change the session expiration which is by default set to 3 hours. It’s applied when the session is created with httpSession.setMaxInactiveInterval(inSeconds). As mentioned, in production we usually have up to 6 hours, for development / testing I usually change it to 1 or 2 min.

I can easily obtain Heap Dumps of machines after startup where scavenging works and after some time where it does not. I do see, that sessions are referenced by the session manager and our session list, not by anyone else. How does that help me?

I am familiar with profilers and use them on a regular basis. I think debugging or temporarily adding more debug infos to the dump-output is more suitable in the given case. What / where should I look at to gain any insights?

Paul

Am 12.01.2018 um 17:46 schrieb Jan Bartel <[hidden email]>:

Are you sure that the OOM happens after the scavenger dies and not that the scavenger dies because of OOM?

I would connect to it with jconsole or similar tool and keep looking at the running threads and the memory and see when things start to go wrong.

Also turn up session logging to debug level to capture more info.

Also try with a profiler and see if you can grab a heap dump to analyse.

Can you reproduce in a test environment?

Can you tell me what all of the session related configuration is, including in your web.xml, and anything you do in code, and I'll see if I can reproduce.

Jan

On 12 Jan. 2018 16:16, "Paul Palaszewski" <[hidden email]> wrote:

Hi everybody,

 

I’ve been using Jetty for quite a while and a few months ago we decided to switch from the standard start.jar-launcher to our own. Main reasons are greatly improved deployments (startup) time by using a own largly simplified hot-WebApp deployer and automatic SSL certificate updates through our own letsencrypt-client.

 

Having troubles with the Session HouseKeeper:

Using simple, standard in-memory sessions. Scavenging starts and initially works fine. I can debug it and see that it collects expired sessions.

But as soon as I let it run on productive systems for a few days at some point in time it stops and all expired sessions remain in memory which leads to OutOfMemory-errors after a few weeks.

Works fine with the original start.jar-launcher, experienced this problem only with our custom server.

Using jetty 9.4.7 and 9.4.8, running on Oracle JDK 9.0.1 (was the same with 9.0) on Windows 10, various Linux distros and macOS 10.13.2 .

 

I’m initializing Server / Session Manager this way …

    server = new org.eclipse.jetty.server.Server();

    DefaultSessionIdManager session = new DefaultSessionIdManager(server);

    HouseKeeper houseKeeper = new HouseKeeper();

    houseKeeper.setIntervalSec(30);

    session.setSessionHouseKeeper(houseKeeper);

    server.setSessionIdManager(session);

 

Have not found a "Stopped scavenging" or any other clue in the logs.

 

Any ideas, what I’m doing wrong or how to narrow down the issue?

 

Best regards,

Paul

 

And the server dump …

org.eclipse.jetty.server.Server@3aefe5e5[9.4.z-SNAPSHOT] - STARTED

+= QueuedThreadPool@qtp674483268{STARTED,8<=10<=200,i=2,q=0} - STARTED

|   +- 401 qtp674483268-401 IDLE TIMED_WAITING @ java.base@9.0.1/jdk.internal.misc.Unsafe.park(Native Method)

|   +- 402 qtp674483268-402 IDLE TIMED_WAITING @ java.base@9.0.1/jdk.internal.misc.Unsafe.park(Native Method)

|   +- 221 qtp674483268-221 SELECTING RUNNABLE @ java.base@9.0.1/sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

|   +- 18 qtp674483268-18-acceptor-0@ee4204f-ServerConnector@3fa247d1{SSL,[ssl, alpn, h2, http/1.1]}{0.0.0.0:443} ACCEPTING RUNNABLE @ java.base@9.0.1/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) prio=3

|   +- 20 qtp674483268-20 SELECTING RUNNABLE @ java.base@9.0.1/sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

|   +- 400 qtp674483268-400 RESERVED TIMED_WAITING @ java.base@9.0.1/jdk.internal.misc.Unsafe.park(Native Method)

|   +- 21 qtp674483268-21 SELECTING RUNNABLE @ java.base@9.0.1/sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

|   +- 22 qtp674483268-22-acceptor-0@2c9c6f53-ServerConnector@5c86a017{HTTP/1.1,[http/1.1, h2c]}{0.0.0.0:80} ACCEPTING RUNNABLE @ java.base@9.0.1/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) prio=3

|   +- 23 0:0:0:0:0:0:0:1 2018-01-12 15:36:35.055 GET /About RUNNABLE @ java.base/java.lang.Thread.getStackTrace(Thread.java:1654)

|   +- 78 qtp674483268-78 SELECTING RUNNABLE @ java.base@9.0.1/sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

|   +- jobs

+= org.eclipse.jetty.server.session.DefaultSessionIdManager@1de47212[worker=node0] - STARTED

|   +? org.eclipse.jetty.server.session.HouseKeeper@501792b1[interval=33000, ownscheduler=true] - STARTED

+= ServerConnector@5c86a017{HTTP/1.1,[http/1.1, h2c]}{0.0.0.0:80} - STARTED

|   +~ org.eclipse.jetty.server.Server@3aefe5e5[9.4.z-SNAPSHOT] - STARTED

|   +~ QueuedThreadPool@qtp674483268{STARTED,8<=10<=200,i=2,q=0} - STARTED

|   += org.eclipse.jetty.util.thread.ScheduledExecutorScheduler@b7dd107 - STARTED

|   |   +- java.base@9.0.1/jdk.internal.misc.Unsafe.park(Native Method)

|   |   +- java.base@9.0.1/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)

|   |   +- java.base@9.0.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2104)

|   |   +- java.base@9.0.1/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1131)

|   |   +- java.base@9.0.1/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:848)

|   |   +- java.base@9.0.1/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1092)

|   |   +- java.base@9.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152)

|   |   +- java.base@9.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)

|   |   +- java.base@9.0.1/java.lang.Thread.run(Thread.java:844)

|   +- org.eclipse.jetty.io.ArrayByteBufferPool@7caa80e4

|   += HttpConnectionFactory@cb0ed20[HTTP/1.1] - STARTED

|   |   +- HttpConfiguration@18c5cd7d{32768/8192,8192/8192,https://:443,[]}

|   += SelectorManager@ServerConnector@5c86a017{HTTP/1.1,[http/1.1, h2c]}{0.0.0.0:80} - STARTED

|   |   += ReservedThreadExecutor@268f5895{s=1/4,p=0}@SelectorManager@ServerConnector@5c86a017{HTTP/1.1,[http/1.1, h2c]}{0.0.0.0:80} - STARTED

|   |   += org.eclipse.jetty.io.ManagedSelector@4a61c4ef id=0 keys=0 selected=0 actions=0 - STARTED

|   |   |   += EatWhatYouKill@62e4c0e7/SelectorProducer@1d4ff7ab/PRODUCING/ReservedThreadExecutor@268f5895{s=1/4,p=0}@SelectorManager@ServerConnector@5c86a017{HTTP/1.1,[http/1.1, h2c]}{0.0.0.0:80} - STARTED

|   |   |       +- SelectorProducer@1d4ff7ab

|   |   |   +- keys

|   |   |   |   +- sun.nio.ch.WindowsSelectorImpl@3f46ce65 keys=0

|   |   |   +- actions

|   |   += org.eclipse.jetty.io.ManagedSelector@5a02a8a4 id=1 keys=1 selected=0 actions=0 - STARTED

|   |       += EatWhatYouKill@1e02ae9b/SelectorProducer@7c92e05f/PRODUCING/ReservedThreadExecutor@268f5895{s=1/4,p=0}@SelectorManager@ServerConnector@5c86a017{HTTP/1.1,[http/1.1, h2c]}{0.0.0.0:80} - STARTED

|   |           +- SelectorProducer@7c92e05f

|   |       +- keys

|   |       |   +- sun.nio.ch.WindowsSelectorImpl@2bdff7b9 keys=1

|   |       |   +- SelectionKey@67dc299b{i=0}->SocketChannelEndPoint@168b474c{/0:0:0:0:0:0:0:1:52677<->/0:0:0:0:0:0:0:1:80,OPEN,fill=-,flush=-,to=119/30000}{io=0/0,kio=0,kro=1}->HttpConnection@5d013a2c[p=HttpParser{s=END,0 of -1},g=HttpGenerator@593eb554{s=START}]=>HttpChannelOverHttp@1c3c3017{r=1,c=false,a=DISPATCHED,uri=//localhost/4p/About?b9=}

|   |       +- actions

|   += HTTP2CServerConnectionFactory@2fd66ad3[h2c] - STARTED

|   |   +- HttpConfiguration@18c5cd7d{32768/8192,8192/8192,https://:443,[]}

|   +- sun.nio.ch.ServerSocketChannelImpl[/0:0:0:0:0:0:0:0:80]

|   +- qtp674483268-22-acceptor-0@2c9c6f53-ServerConnector@5c86a017{HTTP/1.1,[http/1.1, h2c]}{0.0.0.0:80}

+= ServerConnector@3fa247d1{SSL,[ssl, alpn, h2, http/1.1]}{0.0.0.0:443} - STARTED

|   +~ org.eclipse.jetty.server.Server@3aefe5e5[9.4.z-SNAPSHOT] - STARTED

|   +~ QueuedThreadPool@qtp674483268{STARTED,8<=10<=200,i=2,q=0} - STARTED

|   += org.eclipse.jetty.util.thread.ScheduledExecutorScheduler@4b9e255 - STARTED

|   +- org.eclipse.jetty.io.ArrayByteBufferPool@2cc73a0f

|   += SelectorManager@ServerConnector@3fa247d1{SSL,[ssl, alpn, h2, http/1.1]}{0.0.0.0:443} - STARTED

|   |   += ReservedThreadExecutor@63cff0bb{s=0/4,p=0}@SelectorManager@ServerConnector@3fa247d1{SSL,[ssl, alpn, h2, http/1.1]}{0.0.0.0:443} - STARTED

|   |   += org.eclipse.jetty.io.ManagedSelector@6832fc2f id=0 keys=0 selected=0 actions=0 - STARTED

|   |   |   += EatWhatYouKill@3c9696e/SelectorProducer@16cb2c5a/PRODUCING/ReservedThreadExecutor@63cff0bb{s=0/4,p=0}@SelectorManager@ServerConnector@3fa247d1{SSL,[ssl, alpn, h2, http/1.1]}{0.0.0.0:443} - STARTED

|   |   |       +- SelectorProducer@16cb2c5a

|   |   |   +- keys

|   |   |   |   +- sun.nio.ch.WindowsSelectorImpl@5f633211 keys=0

|   |   |   +- actions

|   |   += org.eclipse.jetty.io.ManagedSelector@a9d5c10 id=1 keys=0 selected=0 actions=0 - STARTED

|   |       += EatWhatYouKill@75311ae8/SelectorProducer@3d7b9f39/PRODUCING/ReservedThreadExecutor@63cff0bb{s=0/4,p=0}@SelectorManager@ServerConnector@3fa247d1{SSL,[ssl, alpn, h2, http/1.1]}{0.0.0.0:443} - STARTED

|   |           +- SelectorProducer@3d7b9f39

|   |       +- keys

|   |       |   +- sun.nio.ch.WindowsSelectorImpl@5c443958 keys=0

|   |       +- actions

|   += SslConnectionFactory@d6da883{SSL->alpn} - STARTED

|   |   += SslContextFactory@642a7222[provider=null,keyStore=file:///C:/Entw/btkrt/etc/keystore,trustStore=null] trustAll=false

|   |       +- Protocol Selections

|   |       |   +- Enabled (size=3)

|   |       |   |   +- TLSv1

|   |       |   |   +- TLSv1.1

|   |       |   |   +- TLSv1.2

|   |       |   +- Disabled (size=2)

|   |       |       +- SSLv2Hello - ConfigExcluded:'SSLv2Hello'

|   |       |       +- SSLv3 - JreDisabled:java.security, ConfigExcluded:'SSLv3'

|   |       +- Cipher Suite Selections

|   |           +- Enabled (size=13)

|   |           |   +- SSL_RSA_WITH_3DES_EDE_CBC_SHA

|   |           |   +- TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA

|   |           |   +- TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256

|   |           |   +- TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256

|   |           |   +- TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA

|   |           |   +- TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384

|   |           |   +- TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384

|   |           |   +- TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA

|   |           |   +- TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256

|   |           |   +- TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256

|   |           |   +- TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA

|   |           |   +- TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384

|   |           |   +- TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384

|   |           +- Disabled (size=69)

|   |               +- SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified

|   |               +- SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA - ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified

|   |               +- SSL_DHE_DSS_WITH_DES_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified

|   |               +- SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified

|   |               +- SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- SSL_DHE_RSA_WITH_DES_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified

|   |               +- SSL_DH_anon_EXPORT_WITH_DES40_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified

|   |               +- SSL_DH_anon_WITH_3DES_EDE_CBC_SHA - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- SSL_DH_anon_WITH_DES_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified

|   |               +- SSL_RSA_EXPORT_WITH_DES40_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified

|   |               +- SSL_RSA_WITH_DES_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified

|   |               +- SSL_RSA_WITH_NULL_MD5 - JreDisabled:java.security, ConfigExcluded:'.*NULL.*', ConfigExcluded:'.*MD5.*', ConfigIncluded:NotSpecified

|   |               +- SSL_RSA_WITH_NULL_SHA - JreDisabled:java.security, ConfigExcluded:'.*NULL.*', ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_DSS_WITH_AES_128_CBC_SHA - ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_DSS_WITH_AES_128_CBC_SHA256 - ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_DSS_WITH_AES_128_GCM_SHA256 - ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_DSS_WITH_AES_256_CBC_SHA - ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_DSS_WITH_AES_256_CBC_SHA256 - ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_DSS_WITH_AES_256_GCM_SHA384 - ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_RSA_WITH_AES_128_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_RSA_WITH_AES_128_CBC_SHA256 - ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 - ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_RSA_WITH_AES_256_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 - ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 - ConfigIncluded:NotSpecified

|   |               +- TLS_DH_anon_WITH_AES_128_CBC_SHA - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- TLS_DH_anon_WITH_AES_128_CBC_SHA256 - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- TLS_DH_anon_WITH_AES_128_GCM_SHA256 - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- TLS_DH_anon_WITH_AES_256_CBC_SHA - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- TLS_DH_anon_WITH_AES_256_CBC_SHA256 - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- TLS_DH_anon_WITH_AES_256_GCM_SHA384 - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDHE_ECDSA_WITH_NULL_SHA - JreDisabled:java.security, ConfigExcluded:'.*NULL.*', ConfigIncluded:NotSpecified

|   |               +- TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDHE_RSA_WITH_NULL_SHA - JreDisabled:java.security, ConfigExcluded:'.*NULL.*', ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256 - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256 - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384 - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384 - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_ECDSA_WITH_NULL_SHA - JreDisabled:java.security, ConfigExcluded:'.*NULL.*', ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_RSA_WITH_AES_128_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256 - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256 - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_RSA_WITH_AES_256_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384 - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384 - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_RSA_WITH_NULL_SHA - JreDisabled:java.security, ConfigExcluded:'.*NULL.*', ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_anon_WITH_3DES_EDE_CBC_SHA - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_anon_WITH_AES_128_CBC_SHA - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_anon_WITH_AES_256_CBC_SHA - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_anon_WITH_NULL_SHA - JreDisabled:java.security, ConfigExcluded:'.*NULL.*', ConfigIncluded:NotSpecified

|   |               +- TLS_EMPTY_RENEGOTIATION_INFO_SCSV - ConfigIncluded:NotSpecified

|   |               +- TLS_KRB5_EXPORT_WITH_DES_CBC_40_MD5 - JreDisabled:java.security, ConfigExcluded:'.*MD5.*', ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified

|   |               +- TLS_KRB5_EXPORT_WITH_DES_CBC_40_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified

|   |               +- TLS_KRB5_WITH_3DES_EDE_CBC_MD5 - JreDisabled:java.security, ConfigExcluded:'.*MD5.*', ConfigIncluded:NotSpecified

|   |               +- TLS_KRB5_WITH_3DES_EDE_CBC_SHA - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- TLS_KRB5_WITH_DES_CBC_MD5 - JreDisabled:java.security, ConfigExcluded:'.*MD5.*', ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified

|   |               +- TLS_KRB5_WITH_DES_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified

|   |               +- TLS_RSA_WITH_AES_128_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_RSA_WITH_AES_128_CBC_SHA256 - ConfigIncluded:NotSpecified

|   |               +- TLS_RSA_WITH_AES_128_GCM_SHA256 - ConfigIncluded:NotSpecified

|   |               +- TLS_RSA_WITH_AES_256_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_RSA_WITH_AES_256_CBC_SHA256 - ConfigIncluded:NotSpecified

|   |               +- TLS_RSA_WITH_AES_256_GCM_SHA384 - ConfigIncluded:NotSpecified

|   |               +- TLS_RSA_WITH_NULL_SHA256 - JreDisabled:java.security, ConfigExcluded:'.*NULL.*', ConfigIncluded:NotSpecified

|   += ALPNServerConnectionFactory@45afc369{[alpn],HTTP/1.1,[]} - STARTED

|   += HTTP2ServerConnectionFactory@70beb599[h2] - STARTED

|   |   +- HttpConfiguration@7dd64052{32768/8192,8192/8192,https://:443,[SecureRequestCustomizer@35528b51]}

|   += HttpConnectionFactory@4e41089d[HTTP/1.1] - STARTED

|   |   +- HttpConfiguration@7dd64052{32768/8192,8192/8192,https://:443,[SecureRequestCustomizer@35528b51]}

|   +- sun.nio.ch.ServerSocketChannelImpl[/0:0:0:0:0:0:0:0:443]

|   +- qtp674483268-18-acceptor-0@ee4204f-ServerConnector@3fa247d1{SSL,[ssl, alpn, h2, http/1.1]}{0.0.0.0:443}

+= com.bdm.bwebserver.Server$2@190bae5a - STARTED

|   +> webapps

|       +- /bc - STARTED

|       +- /bdm - STARTED

+= org.eclipse.jetty.server.handler.ErrorHandler@67eb0042 - STARTED

+> jdk.internal.loader.ClassLoaders$AppClassLoader@504bae78

     +- jdk.internal.loader.ClassLoaders$PlatformClassLoader@17ed40e0


_______________________________________________
jetty-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-dev
_______________________________________________
jetty-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-dev


_______________________________________________
jetty-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-dev
Reply | Threaded
Open this post in threaded view
|

Re: [jetty-dev] Troubles with embedded Jetty Session scavenging

Jan Bartel
Paul,

One thing that I noticed is that your HouseKeeper is using its own scheduler, rather than using one that it obtains from the server. Any chance you can change that and run again and see if the problem disappears?

As you can see from the HouseKeeper code, every time the scavenge cycle completes, it schedules another cycle iff the scheduler is still running. See line 72. It does that in a finally block, so even if the scavenge threw some kind of exception/error, the next scavenge should be scheduled.  You may have to checkout and build the jetty 9.4.x branch locally and try putting in a catch and/or some printlns to see what is going on, although I would have expected some log output if there was an error.

You also say the app is keeping its own list of sessions - is that a full reference to the session or just the session id? If the former, I wouldn't recommend you do that, as the management of the lifecycle of that object is not under the control of the app, the container needs to manage it.

The server dump from your original email looks a little odd - I don't see the normal dump for the 2 webapps /bc and /bdm -  I would expect to see a ContextHandlerCollection and a normal dump of the 2 contexts. That would also show the configuration setup for the SessionHandler with the default session maxIdleTime.

Another thing to look into is the Session objects themselves. After this problem happens, do you see log lines like "Timer expired for session for session xyz"? Or ""Inspecting session xyz, valid=true"? Or "Session xyz is candidate for expiry"?  What about " Expiring xyz" ?  Can you look at the heap dump for the SessionHandler._candidateSessionIdsForExpiry and see if it has any entries for expired sessions?  What are the values for the fields of the sessions that you say are still in the DefaultSessionCache?

regards
Jan

On 14 January 2018 at 14:01, Paul Palaszewski <[hidden email]> wrote:
Hi Jan,

thanks for the first feedback!

yes, I’m sure that OutOfMemory occurs because of the ungarbaged sessions. Loosing a few kb with every open session. Our web apps have an end-user-friendly jmx-like monitoring. there I can see it pretty clear. Our apps are usually running for months and years without interrupt, we have no leaking memory issues there.

No, up to now I could not reproduce it - otherwise I would have tackled it already myself. 

One more thing that should not, but might interfere: The app keeps track of active sessions. It implements HttpSessionListener interface and our WebApp (extending ServletContextHandler) uses addEventListener() for itself and for the session manager to capture sessionCreated() and sessionDestroyed() to update it’s own session list. Up to some time it get the events, later on it does not anymore.

I usually use that in our monitoring view to verify if any new changes solve the issue - again, for some time I see sessions disappearing and then they remain that while it shows that the maxInactiveInterval was set to something reasonable but it sticks there for days without any further usage/access.

WebApp / Session setup:
bdm.Server creates DefaultSessionIdManager and HouseKeeper as described in my previous mail
bdm.Server scans for .war-files and creates a bdm.WebApp extending ServletContextHandler
bdm.WepApp reads WEB-INF/web.xml which contains 1 servlet with load-on-startup + 1 servlet filter + 1 listener-class + following session config
<session-config>
<cookie-config>
<name>SESSIONID</name>
<http-only>true</http-only>
<max-age>30000</max-age>
</cookie-config>
</session-config>
which is applied to the WebApps getSessionHandler().getCookieConfig()

on war-file change the app is stopped, removed from the contexts and a new one is added. Happens every few weeks. The hanging scavenger occurs regardless also if we don’t update / redeploy any apps.

The app has a config option to change the session expiration which is by default set to 3 hours. It’s applied when the session is created with httpSession.setMaxInactiveInterval(inSeconds). As mentioned, in production we usually have up to 6 hours, for development / testing I usually change it to 1 or 2 min.

I can easily obtain Heap Dumps of machines after startup where scavenging works and after some time where it does not. I do see, that sessions are referenced by the session manager and our session list, not by anyone else. How does that help me?

I am familiar with profilers and use them on a regular basis. I think debugging or temporarily adding more debug infos to the dump-output is more suitable in the given case. What / where should I look at to gain any insights?

Paul

Am 12.01.2018 um 17:46 schrieb Jan Bartel <[hidden email]>:

Are you sure that the OOM happens after the scavenger dies and not that the scavenger dies because of OOM?

I would connect to it with jconsole or similar tool and keep looking at the running threads and the memory and see when things start to go wrong.

Also turn up session logging to debug level to capture more info.

Also try with a profiler and see if you can grab a heap dump to analyse.

Can you reproduce in a test environment?

Can you tell me what all of the session related configuration is, including in your web.xml, and anything you do in code, and I'll see if I can reproduce.

Jan

On 12 Jan. 2018 16:16, "Paul Palaszewski" <[hidden email]> wrote:

Hi everybody,

 

I’ve been using Jetty for quite a while and a few months ago we decided to switch from the standard start.jar-launcher to our own. Main reasons are greatly improved deployments (startup) time by using a own largly simplified hot-WebApp deployer and automatic SSL certificate updates through our own letsencrypt-client.

 

Having troubles with the Session HouseKeeper:

Using simple, standard in-memory sessions. Scavenging starts and initially works fine. I can debug it and see that it collects expired sessions.

But as soon as I let it run on productive systems for a few days at some point in time it stops and all expired sessions remain in memory which leads to OutOfMemory-errors after a few weeks.

Works fine with the original start.jar-launcher, experienced this problem only with our custom server.

Using jetty 9.4.7 and 9.4.8, running on Oracle JDK 9.0.1 (was the same with 9.0) on Windows 10, various Linux distros and macOS 10.13.2 .

 

I’m initializing Server / Session Manager this way …

    server = new org.eclipse.jetty.server.Server();

    DefaultSessionIdManager session = new DefaultSessionIdManager(server);

    HouseKeeper houseKeeper = new HouseKeeper();

    houseKeeper.setIntervalSec(30);

    session.setSessionHouseKeeper(houseKeeper);

    server.setSessionIdManager(session);

 

Have not found a "Stopped scavenging" or any other clue in the logs.

 

Any ideas, what I’m doing wrong or how to narrow down the issue?

 

Best regards,

Paul

 

And the server dump …

org.eclipse.jetty.server.Server@3aefe5e5[9.4.z-SNAPSHOT] - STARTED

+= QueuedThreadPool@qtp674483268{STARTED,8<=10<=200,i=2,q=0} - STARTED

|   +- 401 qtp674483268-401 IDLE TIMED_WAITING @ java.base@9.0.1/jdk.internal.misc.Unsafe.park(Native Method)

|   +- 402 qtp674483268-402 IDLE TIMED_WAITING @ java.base@9.0.1/jdk.internal.misc.Unsafe.park(Native Method)

|   +- 221 qtp674483268-221 SELECTING RUNNABLE @ java.base@9.0.1/sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

|   +- 18 qtp674483268-18-acceptor-0@ee4204f-ServerConnector@3fa247d1{SSL,[ssl, alpn, h2, http/1.1]}{0.0.0.0:443} ACCEPTING RUNNABLE @ java.base@9.0.1/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) prio=3

|   +- 20 qtp674483268-20 SELECTING RUNNABLE @ java.base@9.0.1/sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

|   +- 400 qtp674483268-400 RESERVED TIMED_WAITING @ java.base@9.0.1/jdk.internal.misc.Unsafe.park(Native Method)

|   +- 21 qtp674483268-21 SELECTING RUNNABLE @ java.base@9.0.1/sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

|   +- 22 qtp674483268-22-acceptor-0@2c9c6f53-ServerConnector@5c86a017{HTTP/1.1,[http/1.1, h2c]}{0.0.0.0:80} ACCEPTING RUNNABLE @ java.base@9.0.1/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) prio=3

|   +- 23 0:0:0:0:0:0:0:1 2018-01-12 15:36:35.055 GET /About RUNNABLE @ java.base/java.lang.Thread.getStackTrace(Thread.java:1654)

|   +- 78 qtp674483268-78 SELECTING RUNNABLE @ java.base@9.0.1/sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

|   +- jobs

+= org.eclipse.jetty.server.session.DefaultSessionIdManager@1de47212[worker=node0] - STARTED

|   +? org.eclipse.jetty.server.session.HouseKeeper@501792b1[interval=33000, ownscheduler=true] - STARTED

+= ServerConnector@5c86a017{HTTP/1.1,[http/1.1, h2c]}{0.0.0.0:80} - STARTED

|   +~ org.eclipse.jetty.server.Server@3aefe5e5[9.4.z-SNAPSHOT] - STARTED

|   +~ QueuedThreadPool@qtp674483268{STARTED,8<=10<=200,i=2,q=0} - STARTED

|   += org.eclipse.jetty.util.thread.ScheduledExecutorScheduler@b7dd107 - STARTED

|   |   +- java.base@9.0.1/jdk.internal.misc.Unsafe.park(Native Method)

|   |   +- java.base@9.0.1/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)

|   |   +- java.base@9.0.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2104)

|   |   +- java.base@9.0.1/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1131)

|   |   +- java.base@9.0.1/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:848)

|   |   +- java.base@9.0.1/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1092)

|   |   +- java.base@9.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152)

|   |   +- java.base@9.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)

|   |   +- java.base@9.0.1/java.lang.Thread.run(Thread.java:844)

|   +- org.eclipse.jetty.io.ArrayByteBufferPool@7caa80e4

|   += HttpConnectionFactory@cb0ed20[HTTP/1.1] - STARTED

|   |   +- HttpConfiguration@18c5cd7d{32768/8192,8192/8192,https://:443,[]}

|   += SelectorManager@ServerConnector@5c86a017{HTTP/1.1,[http/1.1, h2c]}{0.0.0.0:80} - STARTED

|   |   += ReservedThreadExecutor@268f5895{s=1/4,p=0}@SelectorManager@ServerConnector@5c86a017{HTTP/1.1,[http/1.1, h2c]}{0.0.0.0:80} - STARTED

|   |   += org.eclipse.jetty.io.ManagedSelector@4a61c4ef id=0 keys=0 selected=0 actions=0 - STARTED

|   |   |   += EatWhatYouKill@62e4c0e7/SelectorProducer@1d4ff7ab/PRODUCING/ReservedThreadExecutor@268f5895{s=1/4,p=0}@SelectorManager@ServerConnector@5c86a017{HTTP/1.1,[http/1.1, h2c]}{0.0.0.0:80} - STARTED

|   |   |       +- SelectorProducer@1d4ff7ab

|   |   |   +- keys

|   |   |   |   +- sun.nio.ch.WindowsSelectorImpl@3f46ce65 keys=0

|   |   |   +- actions

|   |   += org.eclipse.jetty.io.ManagedSelector@5a02a8a4 id=1 keys=1 selected=0 actions=0 - STARTED

|   |       += EatWhatYouKill@1e02ae9b/SelectorProducer@7c92e05f/PRODUCING/ReservedThreadExecutor@268f5895{s=1/4,p=0}@SelectorManager@ServerConnector@5c86a017{HTTP/1.1,[http/1.1, h2c]}{0.0.0.0:80} - STARTED

|   |           +- SelectorProducer@7c92e05f

|   |       +- keys

|   |       |   +- sun.nio.ch.WindowsSelectorImpl@2bdff7b9 keys=1

|   |       |   +- SelectionKey@67dc299b{i=0}->SocketChannelEndPoint@168b474c{/0:0:0:0:0:0:0:1:52677<->/0:0:0:0:0:0:0:1:80,OPEN,fill=-,flush=-,to=119/30000}{io=0/0,kio=0,kro=1}->HttpConnection@5d013a2c[p=HttpParser{s=END,0 of -1},g=HttpGenerator@593eb554{s=START}]=>HttpChannelOverHttp@1c3c3017{r=1,c=false,a=DISPATCHED,uri=//localhost/4p/About?b9=}

|   |       +- actions

|   += HTTP2CServerConnectionFactory@2fd66ad3[h2c] - STARTED

|   |   +- HttpConfiguration@18c5cd7d{32768/8192,8192/8192,https://:443,[]}

|   +- sun.nio.ch.ServerSocketChannelImpl[/0:0:0:0:0:0:0:0:80]

|   +- qtp674483268-22-acceptor-0@2c9c6f53-ServerConnector@5c86a017{HTTP/1.1,[http/1.1, h2c]}{0.0.0.0:80}

+= ServerConnector@3fa247d1{SSL,[ssl, alpn, h2, http/1.1]}{0.0.0.0:443} - STARTED

|   +~ org.eclipse.jetty.server.Server@3aefe5e5[9.4.z-SNAPSHOT] - STARTED

|   +~ QueuedThreadPool@qtp674483268{STARTED,8<=10<=200,i=2,q=0} - STARTED

|   += org.eclipse.jetty.util.thread.ScheduledExecutorScheduler@4b9e255 - STARTED

|   +- org.eclipse.jetty.io.ArrayByteBufferPool@2cc73a0f

|   += SelectorManager@ServerConnector@3fa247d1{SSL,[ssl, alpn, h2, http/1.1]}{0.0.0.0:443} - STARTED

|   |   += ReservedThreadExecutor@63cff0bb{s=0/4,p=0}@SelectorManager@ServerConnector@3fa247d1{SSL,[ssl, alpn, h2, http/1.1]}{0.0.0.0:443} - STARTED

|   |   += org.eclipse.jetty.io.ManagedSelector@6832fc2f id=0 keys=0 selected=0 actions=0 - STARTED

|   |   |   += EatWhatYouKill@3c9696e/SelectorProducer@16cb2c5a/PRODUCING/ReservedThreadExecutor@63cff0bb{s=0/4,p=0}@SelectorManager@ServerConnector@3fa247d1{SSL,[ssl, alpn, h2, http/1.1]}{0.0.0.0:443} - STARTED

|   |   |       +- SelectorProducer@16cb2c5a

|   |   |   +- keys

|   |   |   |   +- sun.nio.ch.WindowsSelectorImpl@5f633211 keys=0

|   |   |   +- actions

|   |   += org.eclipse.jetty.io.ManagedSelector@a9d5c10 id=1 keys=0 selected=0 actions=0 - STARTED

|   |       += EatWhatYouKill@75311ae8/SelectorProducer@3d7b9f39/PRODUCING/ReservedThreadExecutor@63cff0bb{s=0/4,p=0}@SelectorManager@ServerConnector@3fa247d1{SSL,[ssl, alpn, h2, http/1.1]}{0.0.0.0:443} - STARTED

|   |           +- SelectorProducer@3d7b9f39

|   |       +- keys

|   |       |   +- sun.nio.ch.WindowsSelectorImpl@5c443958 keys=0

|   |       +- actions

|   += SslConnectionFactory@d6da883{SSL->alpn} - STARTED

|   |   += SslContextFactory@642a7222[provider=null,keyStore=file:///C:/Entw/btkrt/etc/keystore,trustStore=null] trustAll=false

|   |       +- Protocol Selections

|   |       |   +- Enabled (size=3)

|   |       |   |   +- TLSv1

|   |       |   |   +- TLSv1.1

|   |       |   |   +- TLSv1.2

|   |       |   +- Disabled (size=2)

|   |       |       +- SSLv2Hello - ConfigExcluded:'SSLv2Hello'

|   |       |       +- SSLv3 - JreDisabled:java.security, ConfigExcluded:'SSLv3'

|   |       +- Cipher Suite Selections

|   |           +- Enabled (size=13)

|   |           |   +- SSL_RSA_WITH_3DES_EDE_CBC_SHA

|   |           |   +- TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA

|   |           |   +- TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256

|   |           |   +- TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256

|   |           |   +- TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA

|   |           |   +- TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384

|   |           |   +- TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384

|   |           |   +- TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA

|   |           |   +- TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256

|   |           |   +- TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256

|   |           |   +- TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA

|   |           |   +- TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384

|   |           |   +- TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384

|   |           +- Disabled (size=69)

|   |               +- SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified

|   |               +- SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA - ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified

|   |               +- SSL_DHE_DSS_WITH_DES_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified

|   |               +- SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified

|   |               +- SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- SSL_DHE_RSA_WITH_DES_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified

|   |               +- SSL_DH_anon_EXPORT_WITH_DES40_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified

|   |               +- SSL_DH_anon_WITH_3DES_EDE_CBC_SHA - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- SSL_DH_anon_WITH_DES_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified

|   |               +- SSL_RSA_EXPORT_WITH_DES40_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified

|   |               +- SSL_RSA_WITH_DES_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified

|   |               +- SSL_RSA_WITH_NULL_MD5 - JreDisabled:java.security, ConfigExcluded:'.*NULL.*', ConfigExcluded:'.*MD5.*', ConfigIncluded:NotSpecified

|   |               +- SSL_RSA_WITH_NULL_SHA - JreDisabled:java.security, ConfigExcluded:'.*NULL.*', ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_DSS_WITH_AES_128_CBC_SHA - ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_DSS_WITH_AES_128_CBC_SHA256 - ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_DSS_WITH_AES_128_GCM_SHA256 - ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_DSS_WITH_AES_256_CBC_SHA - ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_DSS_WITH_AES_256_CBC_SHA256 - ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_DSS_WITH_AES_256_GCM_SHA384 - ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_RSA_WITH_AES_128_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_RSA_WITH_AES_128_CBC_SHA256 - ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 - ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_RSA_WITH_AES_256_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 - ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 - ConfigIncluded:NotSpecified

|   |               +- TLS_DH_anon_WITH_AES_128_CBC_SHA - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- TLS_DH_anon_WITH_AES_128_CBC_SHA256 - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- TLS_DH_anon_WITH_AES_128_GCM_SHA256 - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- TLS_DH_anon_WITH_AES_256_CBC_SHA - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- TLS_DH_anon_WITH_AES_256_CBC_SHA256 - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- TLS_DH_anon_WITH_AES_256_GCM_SHA384 - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDHE_ECDSA_WITH_NULL_SHA - JreDisabled:java.security, ConfigExcluded:'.*NULL.*', ConfigIncluded:NotSpecified

|   |               +- TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDHE_RSA_WITH_NULL_SHA - JreDisabled:java.security, ConfigExcluded:'.*NULL.*', ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256 - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256 - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384 - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384 - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_ECDSA_WITH_NULL_SHA - JreDisabled:java.security, ConfigExcluded:'.*NULL.*', ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_RSA_WITH_AES_128_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256 - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256 - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_RSA_WITH_AES_256_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384 - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384 - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_RSA_WITH_NULL_SHA - JreDisabled:java.security, ConfigExcluded:'.*NULL.*', ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_anon_WITH_3DES_EDE_CBC_SHA - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_anon_WITH_AES_128_CBC_SHA - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_anon_WITH_AES_256_CBC_SHA - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_anon_WITH_NULL_SHA - JreDisabled:java.security, ConfigExcluded:'.*NULL.*', ConfigIncluded:NotSpecified

|   |               +- TLS_EMPTY_RENEGOTIATION_INFO_SCSV - ConfigIncluded:NotSpecified

|   |               +- TLS_KRB5_EXPORT_WITH_DES_CBC_40_MD5 - JreDisabled:java.security, ConfigExcluded:'.*MD5.*', ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified

|   |               +- TLS_KRB5_EXPORT_WITH_DES_CBC_40_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified

|   |               +- TLS_KRB5_WITH_3DES_EDE_CBC_MD5 - JreDisabled:java.security, ConfigExcluded:'.*MD5.*', ConfigIncluded:NotSpecified

|   |               +- TLS_KRB5_WITH_3DES_EDE_CBC_SHA - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- TLS_KRB5_WITH_DES_CBC_MD5 - JreDisabled:java.security, ConfigExcluded:'.*MD5.*', ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified

|   |               +- TLS_KRB5_WITH_DES_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified

|   |               +- TLS_RSA_WITH_AES_128_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_RSA_WITH_AES_128_CBC_SHA256 - ConfigIncluded:NotSpecified

|   |               +- TLS_RSA_WITH_AES_128_GCM_SHA256 - ConfigIncluded:NotSpecified

|   |               +- TLS_RSA_WITH_AES_256_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_RSA_WITH_AES_256_CBC_SHA256 - ConfigIncluded:NotSpecified

|   |               +- TLS_RSA_WITH_AES_256_GCM_SHA384 - ConfigIncluded:NotSpecified

|   |               +- TLS_RSA_WITH_NULL_SHA256 - JreDisabled:java.security, ConfigExcluded:'.*NULL.*', ConfigIncluded:NotSpecified

|   += ALPNServerConnectionFactory@45afc369{[alpn],HTTP/1.1,[]} - STARTED

|   += HTTP2ServerConnectionFactory@70beb599[h2] - STARTED

|   |   +- HttpConfiguration@7dd64052{32768/8192,8192/8192,https://:443,[SecureRequestCustomizer@35528b51]}

|   += HttpConnectionFactory@4e41089d[HTTP/1.1] - STARTED

|   |   +- HttpConfiguration@7dd64052{32768/8192,8192/8192,https://:443,[SecureRequestCustomizer@35528b51]}

|   +- sun.nio.ch.ServerSocketChannelImpl[/0:0:0:0:0:0:0:0:443]

|   +- qtp674483268-18-acceptor-0@ee4204f-ServerConnector@3fa247d1{SSL,[ssl, alpn, h2, http/1.1]}{0.0.0.0:443}

+= com.bdm.bwebserver.Server$2@190bae5a - STARTED

|   +> webapps

|       +- /bc - STARTED

|       +- /bdm - STARTED

+= org.eclipse.jetty.server.handler.ErrorHandler@67eb0042 - STARTED

+> jdk.internal.loader.ClassLoaders$AppClassLoader@504bae78

     +- jdk.internal.loader.ClassLoaders$PlatformClassLoader@17ed40e0


_______________________________________________
jetty-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-dev
_______________________________________________
jetty-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-dev


_______________________________________________
jetty-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-dev



--
Jan Bartel <[hidden email]>
www.webtide.com
Expert assistance from the creators of Jetty and CometD


_______________________________________________
jetty-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-dev
Reply | Threaded
Open this post in threaded view
|

Re: [jetty-dev] Troubles with embedded Jetty Session scavenging

Jan Bartel
Hi Paul,

Another thing to check is the state of the scheduledExecutor inside the HouseKeeper:   if that thread is blocked, say during a callback to one of the application listeners, then subsequent scavenges will never be scheduled. So check the thread dump and the profiler for the state of that and let me know.

If there's anything to be done in jetty then I want to get it in before release 9.4.9, which is coming up very shortly, so it would be great if you could report back as soon as you are able.

regards 
Jan

On 15 January 2018 at 11:01, Jan Bartel <[hidden email]> wrote:
Paul,

One thing that I noticed is that your HouseKeeper is using its own scheduler, rather than using one that it obtains from the server. Any chance you can change that and run again and see if the problem disappears?

As you can see from the HouseKeeper code, every time the scavenge cycle completes, it schedules another cycle iff the scheduler is still running. See line 72. It does that in a finally block, so even if the scavenge threw some kind of exception/error, the next scavenge should be scheduled.  You may have to checkout and build the jetty 9.4.x branch locally and try putting in a catch and/or some printlns to see what is going on, although I would have expected some log output if there was an error.

You also say the app is keeping its own list of sessions - is that a full reference to the session or just the session id? If the former, I wouldn't recommend you do that, as the management of the lifecycle of that object is not under the control of the app, the container needs to manage it.

The server dump from your original email looks a little odd - I don't see the normal dump for the 2 webapps /bc and /bdm -  I would expect to see a ContextHandlerCollection and a normal dump of the 2 contexts. That would also show the configuration setup for the SessionHandler with the default session maxIdleTime.

Another thing to look into is the Session objects themselves. After this problem happens, do you see log lines like "Timer expired for session for session xyz"? Or ""Inspecting session xyz, valid=true"? Or "Session xyz is candidate for expiry"?  What about " Expiring xyz" ?  Can you look at the heap dump for the SessionHandler._candidateSessionIdsForExpiry and see if it has any entries for expired sessions?  What are the values for the fields of the sessions that you say are still in the DefaultSessionCache?

regards
Jan

On 14 January 2018 at 14:01, Paul Palaszewski <[hidden email]> wrote:
Hi Jan,

thanks for the first feedback!

yes, I’m sure that OutOfMemory occurs because of the ungarbaged sessions. Loosing a few kb with every open session. Our web apps have an end-user-friendly jmx-like monitoring. there I can see it pretty clear. Our apps are usually running for months and years without interrupt, we have no leaking memory issues there.

No, up to now I could not reproduce it - otherwise I would have tackled it already myself. 

One more thing that should not, but might interfere: The app keeps track of active sessions. It implements HttpSessionListener interface and our WebApp (extending ServletContextHandler) uses addEventListener() for itself and for the session manager to capture sessionCreated() and sessionDestroyed() to update it’s own session list. Up to some time it get the events, later on it does not anymore.

I usually use that in our monitoring view to verify if any new changes solve the issue - again, for some time I see sessions disappearing and then they remain that while it shows that the maxInactiveInterval was set to something reasonable but it sticks there for days without any further usage/access.

WebApp / Session setup:
bdm.Server creates DefaultSessionIdManager and HouseKeeper as described in my previous mail
bdm.Server scans for .war-files and creates a bdm.WebApp extending ServletContextHandler
bdm.WepApp reads WEB-INF/web.xml which contains 1 servlet with load-on-startup + 1 servlet filter + 1 listener-class + following session config
<session-config>
<cookie-config>
<name>SESSIONID</name>
<http-only>true</http-only>
<max-age>30000</max-age>
</cookie-config>
</session-config>
which is applied to the WebApps getSessionHandler().getCookieConfig()

on war-file change the app is stopped, removed from the contexts and a new one is added. Happens every few weeks. The hanging scavenger occurs regardless also if we don’t update / redeploy any apps.

The app has a config option to change the session expiration which is by default set to 3 hours. It’s applied when the session is created with httpSession.setMaxInactiveInterval(inSeconds). As mentioned, in production we usually have up to 6 hours, for development / testing I usually change it to 1 or 2 min.

I can easily obtain Heap Dumps of machines after startup where scavenging works and after some time where it does not. I do see, that sessions are referenced by the session manager and our session list, not by anyone else. How does that help me?

I am familiar with profilers and use them on a regular basis. I think debugging or temporarily adding more debug infos to the dump-output is more suitable in the given case. What / where should I look at to gain any insights?

Paul

Am 12.01.2018 um 17:46 schrieb Jan Bartel <[hidden email]>:

Are you sure that the OOM happens after the scavenger dies and not that the scavenger dies because of OOM?

I would connect to it with jconsole or similar tool and keep looking at the running threads and the memory and see when things start to go wrong.

Also turn up session logging to debug level to capture more info.

Also try with a profiler and see if you can grab a heap dump to analyse.

Can you reproduce in a test environment?

Can you tell me what all of the session related configuration is, including in your web.xml, and anything you do in code, and I'll see if I can reproduce.

Jan

On 12 Jan. 2018 16:16, "Paul Palaszewski" <[hidden email]> wrote:

Hi everybody,

 

I’ve been using Jetty for quite a while and a few months ago we decided to switch from the standard start.jar-launcher to our own. Main reasons are greatly improved deployments (startup) time by using a own largly simplified hot-WebApp deployer and automatic SSL certificate updates through our own letsencrypt-client.

 

Having troubles with the Session HouseKeeper:

Using simple, standard in-memory sessions. Scavenging starts and initially works fine. I can debug it and see that it collects expired sessions.

But as soon as I let it run on productive systems for a few days at some point in time it stops and all expired sessions remain in memory which leads to OutOfMemory-errors after a few weeks.

Works fine with the original start.jar-launcher, experienced this problem only with our custom server.

Using jetty 9.4.7 and 9.4.8, running on Oracle JDK 9.0.1 (was the same with 9.0) on Windows 10, various Linux distros and macOS 10.13.2 .

 

I’m initializing Server / Session Manager this way …

    server = new org.eclipse.jetty.server.Server();

    DefaultSessionIdManager session = new DefaultSessionIdManager(server);

    HouseKeeper houseKeeper = new HouseKeeper();

    houseKeeper.setIntervalSec(30);

    session.setSessionHouseKeeper(houseKeeper);

    server.setSessionIdManager(session);

 

Have not found a "Stopped scavenging" or any other clue in the logs.

 

Any ideas, what I’m doing wrong or how to narrow down the issue?

 

Best regards,

Paul

 

And the server dump …

org.eclipse.jetty.server.Server@3aefe5e5[9.4.z-SNAPSHOT] - STARTED

+= QueuedThreadPool@qtp674483268{STARTED,8<=10<=200,i=2,q=0} - STARTED

|   +- 401 qtp674483268-401 IDLE TIMED_WAITING @ java.base@9.0.1/jdk.internal.misc.Unsafe.park(Native Method)

|   +- 402 qtp674483268-402 IDLE TIMED_WAITING @ java.base@9.0.1/jdk.internal.misc.Unsafe.park(Native Method)

|   +- 221 qtp674483268-221 SELECTING RUNNABLE @ java.base@9.0.1/sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

|   +- 18 qtp674483268-18-acceptor-0@ee4204f-ServerConnector@3fa247d1{SSL,[ssl, alpn, h2, http/1.1]}{0.0.0.0:443} ACCEPTING RUNNABLE @ java.base@9.0.1/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) prio=3

|   +- 20 qtp674483268-20 SELECTING RUNNABLE @ java.base@9.0.1/sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

|   +- 400 qtp674483268-400 RESERVED TIMED_WAITING @ java.base@9.0.1/jdk.internal.misc.Unsafe.park(Native Method)

|   +- 21 qtp674483268-21 SELECTING RUNNABLE @ java.base@9.0.1/sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

|   +- 22 qtp674483268-22-acceptor-0@2c9c6f53-ServerConnector@5c86a017{HTTP/1.1,[http/1.1, h2c]}{0.0.0.0:80} ACCEPTING RUNNABLE @ java.base@9.0.1/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) prio=3

|   +- 23 0:0:0:0:0:0:0:1 2018-01-12 15:36:35.055 GET /About RUNNABLE @ java.base/java.lang.Thread.getStackTrace(Thread.java:1654)

|   +- 78 qtp674483268-78 SELECTING RUNNABLE @ java.base@9.0.1/sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

|   +- jobs

+= org.eclipse.jetty.server.session.DefaultSessionIdManager@1de47212[worker=node0] - STARTED

|   +? org.eclipse.jetty.server.session.HouseKeeper@501792b1[interval=33000, ownscheduler=true] - STARTED

+= ServerConnector@5c86a017{HTTP/1.1,[http/1.1, h2c]}{0.0.0.0:80} - STARTED

|   +~ org.eclipse.jetty.server.Server@3aefe5e5[9.4.z-SNAPSHOT] - STARTED

|   +~ QueuedThreadPool@qtp674483268{STARTED,8<=10<=200,i=2,q=0} - STARTED

|   += org.eclipse.jetty.util.thread.ScheduledExecutorScheduler@b7dd107 - STARTED

|   |   +- java.base@9.0.1/jdk.internal.misc.Unsafe.park(Native Method)

|   |   +- java.base@9.0.1/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)

|   |   +- java.base@9.0.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2104)

|   |   +- java.base@9.0.1/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1131)

|   |   +- java.base@9.0.1/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:848)

|   |   +- java.base@9.0.1/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1092)

|   |   +- java.base@9.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152)

|   |   +- java.base@9.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)

|   |   +- java.base@9.0.1/java.lang.Thread.run(Thread.java:844)

|   +- org.eclipse.jetty.io.ArrayByteBufferPool@7caa80e4

|   += HttpConnectionFactory@cb0ed20[HTTP/1.1] - STARTED

|   |   +- HttpConfiguration@18c5cd7d{32768/8192,8192/8192,https://:443,[]}

|   += SelectorManager@ServerConnector@5c86a017{HTTP/1.1,[http/1.1, h2c]}{0.0.0.0:80} - STARTED

|   |   += ReservedThreadExecutor@268f5895{s=1/4,p=0}@SelectorManager@ServerConnector@5c86a017{HTTP/1.1,[http/1.1, h2c]}{0.0.0.0:80} - STARTED

|   |   += org.eclipse.jetty.io.ManagedSelector@4a61c4ef id=0 keys=0 selected=0 actions=0 - STARTED

|   |   |   += EatWhatYouKill@62e4c0e7/SelectorProducer@1d4ff7ab/PRODUCING/ReservedThreadExecutor@268f5895{s=1/4,p=0}@SelectorManager@ServerConnector@5c86a017{HTTP/1.1,[http/1.1, h2c]}{0.0.0.0:80} - STARTED

|   |   |       +- SelectorProducer@1d4ff7ab

|   |   |   +- keys

|   |   |   |   +- sun.nio.ch.WindowsSelectorImpl@3f46ce65 keys=0

|   |   |   +- actions

|   |   += org.eclipse.jetty.io.ManagedSelector@5a02a8a4 id=1 keys=1 selected=0 actions=0 - STARTED

|   |       += EatWhatYouKill@1e02ae9b/SelectorProducer@7c92e05f/PRODUCING/ReservedThreadExecutor@268f5895{s=1/4,p=0}@SelectorManager@ServerConnector@5c86a017{HTTP/1.1,[http/1.1, h2c]}{0.0.0.0:80} - STARTED

|   |           +- SelectorProducer@7c92e05f

|   |       +- keys

|   |       |   +- sun.nio.ch.WindowsSelectorImpl@2bdff7b9 keys=1

|   |       |   +- SelectionKey@67dc299b{i=0}->SocketChannelEndPoint@168b474c{/0:0:0:0:0:0:0:1:52677<->/0:0:0:0:0:0:0:1:80,OPEN,fill=-,flush=-,to=119/30000}{io=0/0,kio=0,kro=1}->HttpConnection@5d013a2c[p=HttpParser{s=END,0 of -1},g=HttpGenerator@593eb554{s=START}]=>HttpChannelOverHttp@1c3c3017{r=1,c=false,a=DISPATCHED,uri=//localhost/4p/About?b9=}

|   |       +- actions

|   += HTTP2CServerConnectionFactory@2fd66ad3[h2c] - STARTED

|   |   +- HttpConfiguration@18c5cd7d{32768/8192,8192/8192,https://:443,[]}

|   +- sun.nio.ch.ServerSocketChannelImpl[/0:0:0:0:0:0:0:0:80]

|   +- qtp674483268-22-acceptor-0@2c9c6f53-ServerConnector@5c86a017{HTTP/1.1,[http/1.1, h2c]}{0.0.0.0:80}

+= ServerConnector@3fa247d1{SSL,[ssl, alpn, h2, http/1.1]}{0.0.0.0:443} - STARTED

|   +~ org.eclipse.jetty.server.Server@3aefe5e5[9.4.z-SNAPSHOT] - STARTED

|   +~ QueuedThreadPool@qtp674483268{STARTED,8<=10<=200,i=2,q=0} - STARTED

|   += org.eclipse.jetty.util.thread.ScheduledExecutorScheduler@4b9e255 - STARTED

|   +- org.eclipse.jetty.io.ArrayByteBufferPool@2cc73a0f

|   += SelectorManager@ServerConnector@3fa247d1{SSL,[ssl, alpn, h2, http/1.1]}{0.0.0.0:443} - STARTED

|   |   += ReservedThreadExecutor@63cff0bb{s=0/4,p=0}@SelectorManager@ServerConnector@3fa247d1{SSL,[ssl, alpn, h2, http/1.1]}{0.0.0.0:443} - STARTED

|   |   += org.eclipse.jetty.io.ManagedSelector@6832fc2f id=0 keys=0 selected=0 actions=0 - STARTED

|   |   |   += EatWhatYouKill@3c9696e/SelectorProducer@16cb2c5a/PRODUCING/ReservedThreadExecutor@63cff0bb{s=0/4,p=0}@SelectorManager@ServerConnector@3fa247d1{SSL,[ssl, alpn, h2, http/1.1]}{0.0.0.0:443} - STARTED

|   |   |       +- SelectorProducer@16cb2c5a

|   |   |   +- keys

|   |   |   |   +- sun.nio.ch.WindowsSelectorImpl@5f633211 keys=0

|   |   |   +- actions

|   |   += org.eclipse.jetty.io.ManagedSelector@a9d5c10 id=1 keys=0 selected=0 actions=0 - STARTED

|   |       += EatWhatYouKill@75311ae8/SelectorProducer@3d7b9f39/PRODUCING/ReservedThreadExecutor@63cff0bb{s=0/4,p=0}@SelectorManager@ServerConnector@3fa247d1{SSL,[ssl, alpn, h2, http/1.1]}{0.0.0.0:443} - STARTED

|   |           +- SelectorProducer@3d7b9f39

|   |       +- keys

|   |       |   +- sun.nio.ch.WindowsSelectorImpl@5c443958 keys=0

|   |       +- actions

|   += SslConnectionFactory@d6da883{SSL->alpn} - STARTED

|   |   += SslContextFactory@642a7222[provider=null,keyStore=file:///C:/Entw/btkrt/etc/keystore,trustStore=null] trustAll=false

|   |       +- Protocol Selections

|   |       |   +- Enabled (size=3)

|   |       |   |   +- TLSv1

|   |       |   |   +- TLSv1.1

|   |       |   |   +- TLSv1.2

|   |       |   +- Disabled (size=2)

|   |       |       +- SSLv2Hello - ConfigExcluded:'SSLv2Hello'

|   |       |       +- SSLv3 - JreDisabled:java.security, ConfigExcluded:'SSLv3'

|   |       +- Cipher Suite Selections

|   |           +- Enabled (size=13)

|   |           |   +- SSL_RSA_WITH_3DES_EDE_CBC_SHA

|   |           |   +- TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA

|   |           |   +- TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256

|   |           |   +- TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256

|   |           |   +- TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA

|   |           |   +- TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384

|   |           |   +- TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384

|   |           |   +- TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA

|   |           |   +- TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256

|   |           |   +- TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256

|   |           |   +- TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA

|   |           |   +- TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384

|   |           |   +- TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384

|   |           +- Disabled (size=69)

|   |               +- SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified

|   |               +- SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA - ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified

|   |               +- SSL_DHE_DSS_WITH_DES_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified

|   |               +- SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified

|   |               +- SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- SSL_DHE_RSA_WITH_DES_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified

|   |               +- SSL_DH_anon_EXPORT_WITH_DES40_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified

|   |               +- SSL_DH_anon_WITH_3DES_EDE_CBC_SHA - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- SSL_DH_anon_WITH_DES_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified

|   |               +- SSL_RSA_EXPORT_WITH_DES40_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified

|   |               +- SSL_RSA_WITH_DES_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified

|   |               +- SSL_RSA_WITH_NULL_MD5 - JreDisabled:java.security, ConfigExcluded:'.*NULL.*', ConfigExcluded:'.*MD5.*', ConfigIncluded:NotSpecified

|   |               +- SSL_RSA_WITH_NULL_SHA - JreDisabled:java.security, ConfigExcluded:'.*NULL.*', ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_DSS_WITH_AES_128_CBC_SHA - ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_DSS_WITH_AES_128_CBC_SHA256 - ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_DSS_WITH_AES_128_GCM_SHA256 - ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_DSS_WITH_AES_256_CBC_SHA - ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_DSS_WITH_AES_256_CBC_SHA256 - ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_DSS_WITH_AES_256_GCM_SHA384 - ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_RSA_WITH_AES_128_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_RSA_WITH_AES_128_CBC_SHA256 - ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 - ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_RSA_WITH_AES_256_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 - ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 - ConfigIncluded:NotSpecified

|   |               +- TLS_DH_anon_WITH_AES_128_CBC_SHA - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- TLS_DH_anon_WITH_AES_128_CBC_SHA256 - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- TLS_DH_anon_WITH_AES_128_GCM_SHA256 - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- TLS_DH_anon_WITH_AES_256_CBC_SHA - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- TLS_DH_anon_WITH_AES_256_CBC_SHA256 - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- TLS_DH_anon_WITH_AES_256_GCM_SHA384 - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDHE_ECDSA_WITH_NULL_SHA - JreDisabled:java.security, ConfigExcluded:'.*NULL.*', ConfigIncluded:NotSpecified

|   |               +- TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDHE_RSA_WITH_NULL_SHA - JreDisabled:java.security, ConfigExcluded:'.*NULL.*', ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256 - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256 - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384 - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384 - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_ECDSA_WITH_NULL_SHA - JreDisabled:java.security, ConfigExcluded:'.*NULL.*', ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_RSA_WITH_AES_128_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256 - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256 - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_RSA_WITH_AES_256_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384 - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384 - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_RSA_WITH_NULL_SHA - JreDisabled:java.security, ConfigExcluded:'.*NULL.*', ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_anon_WITH_3DES_EDE_CBC_SHA - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_anon_WITH_AES_128_CBC_SHA - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_anon_WITH_AES_256_CBC_SHA - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_anon_WITH_NULL_SHA - JreDisabled:java.security, ConfigExcluded:'.*NULL.*', ConfigIncluded:NotSpecified

|   |               +- TLS_EMPTY_RENEGOTIATION_INFO_SCSV - ConfigIncluded:NotSpecified

|   |               +- TLS_KRB5_EXPORT_WITH_DES_CBC_40_MD5 - JreDisabled:java.security, ConfigExcluded:'.*MD5.*', ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified

|   |               +- TLS_KRB5_EXPORT_WITH_DES_CBC_40_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified

|   |               +- TLS_KRB5_WITH_3DES_EDE_CBC_MD5 - JreDisabled:java.security, ConfigExcluded:'.*MD5.*', ConfigIncluded:NotSpecified

|   |               +- TLS_KRB5_WITH_3DES_EDE_CBC_SHA - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- TLS_KRB5_WITH_DES_CBC_MD5 - JreDisabled:java.security, ConfigExcluded:'.*MD5.*', ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified

|   |               +- TLS_KRB5_WITH_DES_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified

|   |               +- TLS_RSA_WITH_AES_128_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_RSA_WITH_AES_128_CBC_SHA256 - ConfigIncluded:NotSpecified

|   |               +- TLS_RSA_WITH_AES_128_GCM_SHA256 - ConfigIncluded:NotSpecified

|   |               +- TLS_RSA_WITH_AES_256_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_RSA_WITH_AES_256_CBC_SHA256 - ConfigIncluded:NotSpecified

|   |               +- TLS_RSA_WITH_AES_256_GCM_SHA384 - ConfigIncluded:NotSpecified

|   |               +- TLS_RSA_WITH_NULL_SHA256 - JreDisabled:java.security, ConfigExcluded:'.*NULL.*', ConfigIncluded:NotSpecified

|   += ALPNServerConnectionFactory@45afc369{[alpn],HTTP/1.1,[]} - STARTED

|   += HTTP2ServerConnectionFactory@70beb599[h2] - STARTED

|   |   +- HttpConfiguration@7dd64052{32768/8192,8192/8192,https://:443,[SecureRequestCustomizer@35528b51]}

|   += HttpConnectionFactory@4e41089d[HTTP/1.1] - STARTED

|   |   +- HttpConfiguration@7dd64052{32768/8192,8192/8192,https://:443,[SecureRequestCustomizer@35528b51]}

|   +- sun.nio.ch.ServerSocketChannelImpl[/0:0:0:0:0:0:0:0:443]

|   +- qtp674483268-18-acceptor-0@ee4204f-ServerConnector@3fa247d1{SSL,[ssl, alpn, h2, http/1.1]}{0.0.0.0:443}

+= com.bdm.bwebserver.Server$2@190bae5a - STARTED

|   +> webapps

|       +- /bc - STARTED

|       +- /bdm - STARTED

+= org.eclipse.jetty.server.handler.ErrorHandler@67eb0042 - STARTED

+> jdk.internal.loader.ClassLoaders$AppClassLoader@504bae78

     +- jdk.internal.loader.ClassLoaders$PlatformClassLoader@17ed40e0


_______________________________________________
jetty-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-dev
_______________________________________________
jetty-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-dev


_______________________________________________
jetty-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-dev



--
Jan Bartel <[hidden email]>
www.webtide.com
Expert assistance from the creators of Jetty and CometD




--
Jan Bartel <[hidden email]>
www.webtide.com
Expert assistance from the creators of Jetty and CometD


_______________________________________________
jetty-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-dev
Reply | Threaded
Open this post in threaded view
|

Re: [jetty-dev] Troubles with embedded Jetty Session scavenging

Paul Palaszewski

Hi Jan,

 

Big thanks for your suggestions! Sorry for the delay, testing took some time to get reliable feedback.

 

Yes, the server dump looks odd because of our own simplified WebApp – it is really not based on your WebAppContext, just reads web.xml and nothing else so there is not much to configure / to dump.

 

Keeping hard references to the sessions was in my opinion the only J2EE conform way of having actual session info in a way that runs on any servlet engine = without having access to the session manager. Since we are now fix on jetty, I agree – it makes sense to change that.

 

No, had no log entries about sessions and did not suppress any.

 

In the meantime I also tested with jdk 9.0.4 – but it did not change anything.

 

I cross-checked with dumps and right after start … the Scheduler threads are not blocked. Right after the server start I only have one Scheduler. Remains 1 even if I add more web apps and also when I redeploy. But in the dump where the server stopped garbaging I had 4 waiting Schedulers … don’t know how to Interpret that.

 

I’ve tried to find the attached session listners using YourKit inspecting the memory dump … there are so many listener collections – hard to tell if adding + removing is right.

My WebApps overriden ServletContextHandler’s addEventListener() and removeEventListener() add/remove session Event listeners to/from the session handler. add is called during start(), remove() is never called.

 

The HouseKeeper has it’s own Scheduler because I created the Server simply with

    server = new org.eclipse.jetty.server.Server();

// set DefaultSessionIdManager to get rid of No SessionScavenger set log message, using defaults

    DefaultSessionIdManager session = new DefaultSessionIdManager(server);

   HouseKeeper houseKeeper = new HouseKeeper();

    houseKeeper.setIntervalSec(30);

    session.setSessionHouseKeeper(houseKeeper);

    server.setSessionIdManager(session);

 

cross-checked with HouseKeeper-code and jetty startup-files, and tried if it works better if I add

server.addBean(new ScheduledExecutorScheduler());

 

The sessions are not garbaged. The only change I could notice is that I now have only one scavenger thread, typically [WAITING] when I look at it.

Checked the sessions in the memory dump hours after they should have expired – only references to sessions are from DefaultSessionCache and my static session list. Session has correct _created, _expiry, _lastAccessed and _maxInactiveMs but _dirty = false and _state = VALID. I also kept the browser-window open, luckily when I click after a few hours the server acts as if the session was expired – can’t 100% say if it’s cookie expiration or really the server.

 

Any further ideas? Any specific part of the code that I should post to help analysis?

Paul

 

Von: [hidden email] [mailto:[hidden email]] Im Auftrag von Jan Bartel
Gesendet: Dienstag, 16. Januar 2018 11:41
An: Jetty @ Eclipse developer discussion list
Betreff: Re: [jetty-dev] Troubles with embedded Jetty Session scavenging

 

Hi Paul,

 

Another thing to check is the state of the scheduledExecutor inside the HouseKeeper:   if that thread is blocked, say during a callback to one of the application listeners, then subsequent scavenges will never be scheduled. So check the thread dump and the profiler for the state of that and let me know.

 

If there's anything to be done in jetty then I want to get it in before release 9.4.9, which is coming up very shortly, so it would be great if you could report back as soon as you are able.

 

regards 

Jan

 

On 15 January 2018 at 11:01, Jan Bartel <[hidden email]> wrote:

Paul,

 

One thing that I noticed is that your HouseKeeper is using its own scheduler, rather than using one that it obtains from the server. Any chance you can change that and run again and see if the problem disappears?

 

As you can see from the HouseKeeper code, every time the scavenge cycle completes, it schedules another cycle iff the scheduler is still running. See line 72. It does that in a finally block, so even if the scavenge threw some kind of exception/error, the next scavenge should be scheduled.  You may have to checkout and build the jetty 9.4.x branch locally and try putting in a catch and/or some printlns to see what is going on, although I would have expected some log output if there was an error.

 

You also say the app is keeping its own list of sessions - is that a full reference to the session or just the session id? If the former, I wouldn't recommend you do that, as the management of the lifecycle of that object is not under the control of the app, the container needs to manage it.

 

The server dump from your original email looks a little odd - I don't see the normal dump for the 2 webapps /bc and /bdm -  I would expect to see a ContextHandlerCollection and a normal dump of the 2 contexts. That would also show the configuration setup for the SessionHandler with the default session maxIdleTime.

 

Another thing to look into is the Session objects themselves. After this problem happens, do you see log lines like "Timer expired for session for session xyz"? Or ""Inspecting session xyz, valid=true"? Or "Session xyz is candidate for expiry"?  What about " Expiring xyz" ?  Can you look at the heap dump for the SessionHandler._candidateSessionIdsForExpiry and see if it has any entries for expired sessions?  What are the values for the fields of the sessions that you say are still in the DefaultSessionCache?

 

regards

Jan

 

On 14 January 2018 at 14:01, Paul Palaszewski <[hidden email]> wrote:

Hi Jan,

 

thanks for the first feedback!

 

yes, I’m sure that OutOfMemory occurs because of the ungarbaged sessions. Loosing a few kb with every open session. Our web apps have an end-user-friendly jmx-like monitoring. there I can see it pretty clear. Our apps are usually running for months and years without interrupt, we have no leaking memory issues there.

 

No, up to now I could not reproduce it - otherwise I would have tackled it already myself. 

 

One more thing that should not, but might interfere: The app keeps track of active sessions. It implements HttpSessionListener interface and our WebApp (extending ServletContextHandler) uses addEventListener() for itself and for the session manager to capture sessionCreated() and sessionDestroyed() to update it’s own session list. Up to some time it get the events, later on it does not anymore.

 

I usually use that in our monitoring view to verify if any new changes solve the issue - again, for some time I see sessions disappearing and then they remain that while it shows that the maxInactiveInterval was set to something reasonable but it sticks there for days without any further usage/access.

 

WebApp / Session setup:

bdm.Server creates DefaultSessionIdManager and HouseKeeper as described in my previous mail

bdm.Server scans for .war-files and creates a bdm.WebApp extending ServletContextHandler

bdm.WepApp reads WEB-INF/web.xml which contains 1 servlet with load-on-startup + 1 servlet filter + 1 listener-class + following session config

<session-config>

<cookie-config>

<name>SESSIONID</name>

<http-only>true</http-only>

<max-age>30000</max-age>

</cookie-config>

</session-config>

which is applied to the WebApps getSessionHandler().getCookieConfig()

 

on war-file change the app is stopped, removed from the contexts and a new one is added. Happens every few weeks. The hanging scavenger occurs regardless also if we don’t update / redeploy any apps.

 

The app has a config option to change the session expiration which is by default set to 3 hours. It’s applied when the session is created with httpSession.setMaxInactiveInterval(inSeconds). As mentioned, in production we usually have up to 6 hours, for development / testing I usually change it to 1 or 2 min.

 

I can easily obtain Heap Dumps of machines after startup where scavenging works and after some time where it does not. I do see, that sessions are referenced by the session manager and our session list, not by anyone else. How does that help me?

 

I am familiar with profilers and use them on a regular basis. I think debugging or temporarily adding more debug infos to the dump-output is more suitable in the given case. What / where should I look at to gain any insights?

 

Paul



Am 12.01.2018 um 17:46 schrieb Jan Bartel <[hidden email]>:

 

Are you sure that the OOM happens after the scavenger dies and not that the scavenger dies because of OOM?

 

I would connect to it with jconsole or similar tool and keep looking at the running threads and the memory and see when things start to go wrong.

 

Also turn up session logging to debug level to capture more info.

 

Also try with a profiler and see if you can grab a heap dump to analyse.

 

Can you reproduce in a test environment?

 

Can you tell me what all of the session related configuration is, including in your web.xml, and anything you do in code, and I'll see if I can reproduce.

 

Jan

 

On 12 Jan. 2018 16:16, "Paul Palaszewski" <[hidden email]> wrote:

Hi everybody,

 

I’ve been using Jetty for quite a while and a few months ago we decided to switch from the standard start.jar-launcher to our own. Main reasons are greatly improved deployments (startup) time by using a own largly simplified hot-WebApp deployer and automatic SSL certificate updates through our own letsencrypt-client.

 

Having troubles with the Session HouseKeeper:

Using simple, standard in-memory sessions. Scavenging starts and initially works fine. I can debug it and see that it collects expired sessions.

But as soon as I let it run on productive systems for a few days at some point in time it stops and all expired sessions remain in memory which leads to OutOfMemory-errors after a few weeks.

Works fine with the original start.jar-launcher, experienced this problem only with our custom server.

Using jetty 9.4.7 and 9.4.8, running on Oracle JDK 9.0.1 (was the same with 9.0) on Windows 10, various Linux distros and macOS 10.13.2 .

 

I’m initializing Server / Session Manager this way …

    server = new org.eclipse.jetty.server.Server();

    DefaultSessionIdManager session = new DefaultSessionIdManager(server);

    HouseKeeper houseKeeper = new HouseKeeper();

    houseKeeper.setIntervalSec(30);

    session.setSessionHouseKeeper(houseKeeper);

    server.setSessionIdManager(session);

 

Have not found a "Stopped scavenging" or any other clue in the logs.

 

Any ideas, what I’m doing wrong or how to narrow down the issue?

 

Best regards,

Paul

 

And the server dump …

[hidden email]] - STARTED

+= QueuedThreadPool@qtp674483268{STARTED,8<=10<=200,i=2,q=0} - STARTED

|   +- 401 qtp674483268-401 IDLE TIMED_WAITING @ [hidden email] Method)

|   +- 402 qtp674483268-402 IDLE TIMED_WAITING @ [hidden email] Method)

|   +- 221 qtp674483268-221 SELECTING RUNNABLE @ java.base@9.0.1/sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

|   +- 18 qtp674483268-18-acceptor-0@ee4204f-ServerConnector@3fa247d1{SSL,[ssl, alpn, h2, http/1.1]}{0.0.0.0:443} ACCEPTING RUNNABLE @ java.base@9.0.1/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) prio=3

|   +- 20 qtp674483268-20 SELECTING RUNNABLE @ java.base@9.0.1/sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

|   +- 400 qtp674483268-400 RESERVED TIMED_WAITING @ [hidden email] Method)

|   +- 21 qtp674483268-21 SELECTING RUNNABLE @ java.base@9.0.1/sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

|   +- 22 [hidden email], h2c]}{0.0.0.0:80} ACCEPTING RUNNABLE @ java.base@9.0.1/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) prio=3

|   +- 23 0:0:0:0:0:0:0:1 2018-01-12 15:36:35.055 GET /About RUNNABLE @ java.base/java.lang.Thread.getStackTrace(Thread.java:1654)

|   +- 78 qtp674483268-78 SELECTING RUNNABLE @ java.base@9.0.1/sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

|   +- jobs

+= [hidden email]] - STARTED

|   +? [hidden email], ownscheduler=true] - STARTED

+= [hidden email], h2c]}{0.0.0.0:80} - STARTED

|   +~ [hidden email]] - STARTED

|   +~ QueuedThreadPool@qtp674483268{STARTED,8<=10<=200,i=2,q=0} - STARTED

|   += [hidden email] - STARTED

|   |   +- [hidden email] Method)

|   |   +- [hidden email]

|   |   +- [hidden email]

|   |   +- [hidden email]

|   |   +- [hidden email]

|   |   +- [hidden email]

|   |   +- [hidden email]

|   |   +- [hidden email]

|   |   +- [hidden email]

|   +- org.eclipse.jetty.io.ArrayByteBufferPool@7caa80e4

|   += [hidden email]] - STARTED

|   |   +- HttpConfiguration@18c5cd7d{32768/8192,8192/8192,https://:443,[]}

|   += [hidden email], h2c]}{0.0.0.0:80} - STARTED

|   |   += [hidden email], h2c]}{0.0.0.0:80} - STARTED

|   |   += org.eclipse.jetty.io.ManagedSelector@4a61c4ef id=0 keys=0 selected=0 actions=0 - STARTED

|   |   |   += [hidden email], h2c]}{0.0.0.0:80} - STARTED

|   |   |       +- SelectorProducer@1d4ff7ab

|   |   |   +- keys

|   |   |   |   +- sun.nio.ch.WindowsSelectorImpl@3f46ce65 keys=0

|   |   |   +- actions

|   |   += org.eclipse.jetty.io.ManagedSelector@5a02a8a4 id=1 keys=1 selected=0 actions=0 - STARTED

|   |       += [hidden email], h2c]}{0.0.0.0:80} - STARTED

|   |           +- SelectorProducer@7c92e05f

|   |       +- keys

|   |       |   +- sun.nio.ch.WindowsSelectorImpl@2bdff7b9 keys=1

|   |       |   +- SelectionKey@67dc299b{i=0}->SocketChannelEndPoint@168b474c{/0:0:0:0:0:0:0:1:52677<->/0:0:0:0:0:0:0:1:80,OPEN,fill=-,flush=-,to=119/30000}{io=0/0,kio=0,kro=1}->HttpConnection@5d013a2c[p=HttpParser{s=END,0 of -1},g=HttpGenerator@593eb554{s=START}]=>HttpChannelOverHttp@1c3c3017{r=1,c=false,a=DISPATCHED,uri=//localhost/4p/About?b9=}

|   |       +- actions

|   += HTTP2CServerConnectionFactory@2fd66ad3[h2c] - STARTED

|   |   +- HttpConfiguration@18c5cd7d{32768/8192,8192/8192,https://:443,[]}

|   +- sun.nio.ch.ServerSocketChannelImpl[/0:0:0:0:0:0:0:0:80]

|   +- [hidden email], h2c]}{0.0.0.0:80}

+= ServerConnector@3fa247d1{SSL,[ssl, alpn, h2, http/1.1]}{0.0.0.0:443} - STARTED

|   +~ [hidden email]] - STARTED

|   +~ QueuedThreadPool@qtp674483268{STARTED,8<=10<=200,i=2,q=0} - STARTED

|   += [hidden email] - STARTED

|   +- org.eclipse.jetty.io.ArrayByteBufferPool@2cc73a0f

|   += SelectorManager@ServerConnector@3fa247d1{SSL,[ssl, alpn, h2, http/1.1]}{0.0.0.0:443} - STARTED

|   |   += ReservedThreadExecutor@63cff0bb{s=0/4,p=0}@SelectorManager@ServerConnector@3fa247d1{SSL,[ssl, alpn, h2, http/1.1]}{0.0.0.0:443} - STARTED

|   |   += org.eclipse.jetty.io.ManagedSelector@6832fc2f id=0 keys=0 selected=0 actions=0 - STARTED

|   |   |   += EatWhatYouKill@3c9696e/SelectorProducer@16cb2c5a/PRODUCING/ReservedThreadExecutor@63cff0bb{s=0/4,p=0}@SelectorManager@ServerConnector@3fa247d1{SSL,[ssl, alpn, h2, http/1.1]}{0.0.0.0:443} - STARTED

|   |   |       +- SelectorProducer@16cb2c5a

|   |   |   +- keys

|   |   |   |   +- sun.nio.ch.WindowsSelectorImpl@5f633211 keys=0

|   |   |   +- actions

|   |   += org.eclipse.jetty.io.ManagedSelector@a9d5c10 id=1 keys=0 selected=0 actions=0 - STARTED

|   |       += EatWhatYouKill@75311ae8/SelectorProducer@3d7b9f39/PRODUCING/ReservedThreadExecutor@63cff0bb{s=0/4,p=0}@SelectorManager@ServerConnector@3fa247d1{SSL,[ssl, alpn, h2, http/1.1]}{0.0.0.0:443} - STARTED

|   |           +- SelectorProducer@3d7b9f39

|   |       +- keys

|   |       |   +- sun.nio.ch.WindowsSelectorImpl@5c443958 keys=0

|   |       +- actions

|   += SslConnectionFactory@d6da883{SSL->alpn} - STARTED

|   |   += SslContextFactory@642a7222[provider=null,keyStore=file:///C:/Entw/btkrt/etc/keystore,trustStore=null] trustAll=false

|   |       +- Protocol Selections

|   |       |   +- Enabled (size=3)

|   |       |   |   +- TLSv1

|   |       |   |   +- TLSv1.1

|   |       |   |   +- TLSv1.2

|   |       |   +- Disabled (size=2)

|   |       |       +- SSLv2Hello - ConfigExcluded:'SSLv2Hello'

|   |       |       +- SSLv3 - JreDisabled:java.security, ConfigExcluded:'SSLv3'

|   |       +- Cipher Suite Selections

|   |           +- Enabled (size=13)

|   |           |   +- SSL_RSA_WITH_3DES_EDE_CBC_SHA

|   |           |   +- TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA

|   |           |   +- TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256

|   |           |   +- TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256

|   |           |   +- TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA

|   |           |   +- TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384

|   |           |   +- TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384

|   |           |   +- TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA

|   |           |   +- TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256

|   |           |   +- TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256

|   |           |   +- TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA

|   |           |   +- TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384

|   |           |   +- TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384

|   |           +- Disabled (size=69)

|   |               +- SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified

|   |               +- SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA - ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified

|   |               +- SSL_DHE_DSS_WITH_DES_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified

|   |               +- SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified

|   |               +- SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- SSL_DHE_RSA_WITH_DES_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified

|   |               +- SSL_DH_anon_EXPORT_WITH_DES40_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified

|   |               +- SSL_DH_anon_WITH_3DES_EDE_CBC_SHA - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- SSL_DH_anon_WITH_DES_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified

|   |               +- SSL_RSA_EXPORT_WITH_DES40_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified

|   |               +- SSL_RSA_WITH_DES_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified

|   |               +- SSL_RSA_WITH_NULL_MD5 - JreDisabled:java.security, ConfigExcluded:'.*NULL.*', ConfigExcluded:'.*MD5.*', ConfigIncluded:NotSpecified

|   |               +- SSL_RSA_WITH_NULL_SHA - JreDisabled:java.security, ConfigExcluded:'.*NULL.*', ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_DSS_WITH_AES_128_CBC_SHA - ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_DSS_WITH_AES_128_CBC_SHA256 - ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_DSS_WITH_AES_128_GCM_SHA256 - ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_DSS_WITH_AES_256_CBC_SHA - ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_DSS_WITH_AES_256_CBC_SHA256 - ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_DSS_WITH_AES_256_GCM_SHA384 - ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_RSA_WITH_AES_128_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_RSA_WITH_AES_128_CBC_SHA256 - ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 - ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_RSA_WITH_AES_256_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 - ConfigIncluded:NotSpecified

|   |               +- TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 - ConfigIncluded:NotSpecified

|   |               +- TLS_DH_anon_WITH_AES_128_CBC_SHA - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- TLS_DH_anon_WITH_AES_128_CBC_SHA256 - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- TLS_DH_anon_WITH_AES_128_GCM_SHA256 - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- TLS_DH_anon_WITH_AES_256_CBC_SHA - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- TLS_DH_anon_WITH_AES_256_CBC_SHA256 - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- TLS_DH_anon_WITH_AES_256_GCM_SHA384 - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDHE_ECDSA_WITH_NULL_SHA - JreDisabled:java.security, ConfigExcluded:'.*NULL.*', ConfigIncluded:NotSpecified

|   |               +- TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDHE_RSA_WITH_NULL_SHA - JreDisabled:java.security, ConfigExcluded:'.*NULL.*', ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256 - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256 - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384 - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384 - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_ECDSA_WITH_NULL_SHA - JreDisabled:java.security, ConfigExcluded:'.*NULL.*', ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_RSA_WITH_AES_128_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256 - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256 - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_RSA_WITH_AES_256_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384 - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384 - ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_RSA_WITH_NULL_SHA - JreDisabled:java.security, ConfigExcluded:'.*NULL.*', ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_anon_WITH_3DES_EDE_CBC_SHA - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_anon_WITH_AES_128_CBC_SHA - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_anon_WITH_AES_256_CBC_SHA - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- TLS_ECDH_anon_WITH_NULL_SHA - JreDisabled:java.security, ConfigExcluded:'.*NULL.*', ConfigIncluded:NotSpecified

|   |               +- TLS_EMPTY_RENEGOTIATION_INFO_SCSV - ConfigIncluded:NotSpecified

|   |               +- TLS_KRB5_EXPORT_WITH_DES_CBC_40_MD5 - JreDisabled:java.security, ConfigExcluded:'.*MD5.*', ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified

|   |               +- TLS_KRB5_EXPORT_WITH_DES_CBC_40_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified

|   |               +- TLS_KRB5_WITH_3DES_EDE_CBC_MD5 - JreDisabled:java.security, ConfigExcluded:'.*MD5.*', ConfigIncluded:NotSpecified

|   |               +- TLS_KRB5_WITH_3DES_EDE_CBC_SHA - JreDisabled:java.security, ConfigIncluded:NotSpecified

|   |               +- TLS_KRB5_WITH_DES_CBC_MD5 - JreDisabled:java.security, ConfigExcluded:'.*MD5.*', ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified

|   |               +- TLS_KRB5_WITH_DES_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified

|   |               +- TLS_RSA_WITH_AES_128_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_RSA_WITH_AES_128_CBC_SHA256 - ConfigIncluded:NotSpecified

|   |               +- TLS_RSA_WITH_AES_128_GCM_SHA256 - ConfigIncluded:NotSpecified

|   |               +- TLS_RSA_WITH_AES_256_CBC_SHA - ConfigIncluded:NotSpecified

|   |               +- TLS_RSA_WITH_AES_256_CBC_SHA256 - ConfigIncluded:NotSpecified

|   |               +- TLS_RSA_WITH_AES_256_GCM_SHA384 - ConfigIncluded:NotSpecified

|   |               +- TLS_RSA_WITH_NULL_SHA256 - JreDisabled:java.security, ConfigExcluded:'.*NULL.*', ConfigIncluded:NotSpecified

|   += [hidden email] - STARTED

|   += HTTP2ServerConnectionFactory@70beb599[h2] - STARTED

|   |   +- HttpConfiguration@7dd64052{32768/8192,8192/8192,https://:443,[SecureRequestCustomizer@35528b51]}

|   += [hidden email]] - STARTED

|   |   +- HttpConfiguration@7dd64052{32768/8192,8192/8192,https://:443,[SecureRequestCustomizer@35528b51]}

|   +- sun.nio.ch.ServerSocketChannelImpl[/0:0:0:0:0:0:0:0:443]

|   +- qtp674483268-18-acceptor-0@ee4204f-ServerConnector@3fa247d1{SSL,[ssl, alpn, h2, http/1.1]}{0.0.0.0:443}

+= [hidden email] - STARTED

|   +> webapps

|       +- /bc - STARTED

|       +- /bdm - STARTED

+= [hidden email] - STARTED

+> [hidden email]

     +- [hidden email]


_______________________________________________
jetty-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-dev

_______________________________________________
jetty-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-dev

 


_______________________________________________
jetty-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-dev



 

--

Jan Bartel <[hidden email]>

www.webtide.com
Expert assistance from the creators of Jetty and CometD

 



 

--

Jan Bartel <[hidden email]>

www.webtide.com
Expert assistance from the creators of Jetty and CometD

 


_______________________________________________
jetty-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-dev