Jetty sessions leaking

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Jetty sessions leaking

Carey Evans-2
Hi.

We're trying to use Jetty 9.4.14 with an application that uses a lot of sessions, and we're having trouble with the DefaultSessionCache not deleting some old sessions, until we ended up with 6.7 GB consumed by 393216 Session objects in the heap after our load test. Looking at the sessions that hadn't expired after a few hours, I see that each of them has _requests set to 1, when it looks like they should all be 0.

We're using a JDBC session store, and a lot of sessions didn't seem to make it to the database table, although the ones that did were expired and removed as expected.

This is a fairly large application with a lot of proprietary code, so unfortunately I don't have a way to reproduce this that I can share at the moment. We do a fair bit of asynchronous servlet processing, though, which makes me suspicious. Where can I start to look for a solution to this?



--
Laissez lire, et laissez danser; ces deux amusements ne feront jamais de mal au monde.

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Jetty sessions leaking

Jan Bartel
Hi Carey,

So sessions will only be written to the store (jdbc) when the request count drops to zero. If the count remains above 0, then that is why they are not written nor expired.

I will take a particularly close look at the Async handling myself, but if you came up with a repro test case that does some simple async handling that would be great .

Jan

On Thu., 13 Dec. 2018, 13:24 Carey Evans <[hidden email] wrote:
Hi.

We're trying to use Jetty 9.4.14 with an application that uses a lot of sessions, and we're having trouble with the DefaultSessionCache not deleting some old sessions, until we ended up with 6.7 GB consumed by 393216 Session objects in the heap after our load test. Looking at the sessions that hadn't expired after a few hours, I see that each of them has _requests set to 1, when it looks like they should all be 0.

We're using a JDBC session store, and a lot of sessions didn't seem to make it to the database table, although the ones that did were expired and removed as expected.

This is a fairly large application with a lot of proprietary code, so unfortunately I don't have a way to reproduce this that I can share at the moment. We do a fair bit of asynchronous servlet processing, though, which makes me suspicious. Where can I start to look for a solution to this?



--
Laissez lire, et laissez danser; ces deux amusements ne feront jamais de mal au monde.
_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Jetty sessions leaking

Jan Bartel
Carey,

Some follow up on this.

I wrote a test with async servlet handling accessing sessions, and it passes:  the request count is as it should be.

I need more details from you:
+ do you wrap your requests
+ do you have multiple simultaneous requests accessing the same session
+ what SessionCache implementation do you use, and what configuration do you have on it
+ what is the max inactive interval on your sessions
+ how does your async handling terminate, is it with dispatch() or complete()
+ do you do any forwarding between contexts
+ does your application call Request.changeSessionId()
+ does your application use form authentication

Jan

On Thu, 13 Dec 2018 at 13:38, Jan Bartel <[hidden email]> wrote:
Hi Carey,

So sessions will only be written to the store (jdbc) when the request count drops to zero. If the count remains above 0, then that is why they are not written nor expired.

I will take a particularly close look at the Async handling myself, but if you came up with a repro test case that does some simple async handling that would be great .

Jan

On Thu., 13 Dec. 2018, 13:24 Carey Evans <[hidden email] wrote:
Hi.

We're trying to use Jetty 9.4.14 with an application that uses a lot of sessions, and we're having trouble with the DefaultSessionCache not deleting some old sessions, until we ended up with 6.7 GB consumed by 393216 Session objects in the heap after our load test. Looking at the sessions that hadn't expired after a few hours, I see that each of them has _requests set to 1, when it looks like they should all be 0.

We're using a JDBC session store, and a lot of sessions didn't seem to make it to the database table, although the ones that did were expired and removed as expected.

This is a fairly large application with a lot of proprietary code, so unfortunately I don't have a way to reproduce this that I can share at the moment. We do a fair bit of asynchronous servlet processing, though, which makes me suspicious. Where can I start to look for a solution to this?



--
Laissez lire, et laissez danser; ces deux amusements ne feront jamais de mal au monde.
_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users


--
Jan Bartel <[hidden email]>
www.webtide.com
Expert assistance from the creators of Jetty and CometD


_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Jetty sessions leaking

Carey Evans-2
On Thu, 13 Dec 2018 at 19:00, Jan Bartel <[hidden email]> wrote:
Carey,

Some follow up on this.

I wrote a test with async servlet handling accessing sessions, and it passes:  the request count is as it should be.

I need more details from you:
+ do you wrap your requests
+ do you have multiple simultaneous requests accessing the same session
+ what SessionCache implementation do you use, and what configuration do you have on it
+ what is the max inactive interval on your sessions
+ how does your async handling terminate, is it with dispatch() or complete()
+ do you do any forwarding between contexts
+ does your application call Request.changeSessionId()
+ does your application use form authentication

I believe I have a test case at https://github.com/carey/jetty-sessions-bug that reproduces the important parts of our application, though I don't know which part is actually the problem. For better or worse, we're using QoSFilter so that the JVM doesn't end up thrashing itself to a standstill under high load, which I'm guessing is involved.

Once this application is running, in this case using the Jetty Maven plugin, http://localhost:8080/info shows a single line for the current request, which is using a session itself. After this, something like:

for i in 1 2 3 4 5 6 7 8 ; do curl -d 'delay=10' http://localhost:8080/test & done

will increase the number of sessions listed with _requests = 1 by six each time.

We are wrapping our requests further down the filter chain, but that doesn't seem to be necessary to reproduce the issue. We have some simultaneous requests, but not as many as the leaked sessions that we see.






 

Jan

On Thu, 13 Dec 2018 at 13:38, Jan Bartel <[hidden email]> wrote:
Hi Carey,

So sessions will only be written to the store (jdbc) when the request count drops to zero. If the count remains above 0, then that is why they are not written nor expired.

I will take a particularly close look at the Async handling myself, but if you came up with a repro test case that does some simple async handling that would be great .

Jan

On Thu., 13 Dec. 2018, 13:24 Carey Evans <[hidden email] wrote:
Hi.

We're trying to use Jetty 9.4.14 with an application that uses a lot of sessions, and we're having trouble with the DefaultSessionCache not deleting some old sessions, until we ended up with 6.7 GB consumed by 393216 Session objects in the heap after our load test. Looking at the sessions that hadn't expired after a few hours, I see that each of them has _requests set to 1, when it looks like they should all be 0.

We're using a JDBC session store, and a lot of sessions didn't seem to make it to the database table, although the ones that did were expired and removed as expected.

This is a fairly large application with a lot of proprietary code, so unfortunately I don't have a way to reproduce this that I can share at the moment. We do a fair bit of asynchronous servlet processing, though, which makes me suspicious. Where can I start to look for a solution to this?



--
Laissez lire, et laissez danser; ces deux amusements ne feront jamais de mal au monde.
_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users


--
Jan Bartel <[hidden email]>
www.webtide.com
Expert assistance from the creators of Jetty and CometD

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users


--
Laissez lire, et laissez danser; ces deux amusements ne feront jamais de mal au monde.

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Jetty sessions leaking

Carey Evans-2
On Thu, 13 Dec 2018 at 20:20, Carey Evans <[hidden email]> wrote:
We are wrapping our requests further down the filter chain, but that doesn't seem to be necessary to reproduce the issue. We have some simultaneous requests, but not as many as the leaked sessions that we see.

An update: I added
<%@ page language="java" contentType="text/html; charset=UTF-8" pageEncoding="UTF-8" session="false" isErrorPage="true" %>
to the top of 503.jsp, and I no longer see the issue using this code.

--
Laissez lire, et laissez danser; ces deux amusements ne feront jamais de mal au monde.

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Jetty sessions leaking

Carey Evans-2

On Thu, 13 Dec 2018 at 20:31, Carey Evans <[hidden email]> wrote:
An update: I added
<%@ page language="java" contentType="text/html; charset=UTF-8" pageEncoding="UTF-8" session="false" isErrorPage="true" %>
to the top of 503.jsp, and I no longer see the issue using this code.

I've simplified the code at https://github.com/carey/jetty-sessions-bug. Now all that's needed to create sessions that never expire is an error handling servlet that starts a new session from an async dispatch.


--
Laissez lire, et laissez danser; ces deux amusements ne feront jamais de mal au monde.

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Jetty sessions leaking

Jan Bartel
By all means raise it as an issue. And thank you for the repro test case, I'm on it.

Jan

On Sat, 15 Dec 2018 at 08:51, Carey Evans <[hidden email]> wrote:

On Thu, 13 Dec 2018 at 20:31, Carey Evans <[hidden email]> wrote:
An update: I added
<%@ page language="java" contentType="text/html; charset=UTF-8" pageEncoding="UTF-8" session="false" isErrorPage="true" %>
to the top of 503.jsp, and I no longer see the issue using this code.

I've simplified the code at https://github.com/carey/jetty-sessions-bug. Now all that's needed to create sessions that never expire is an error handling servlet that starts a new session from an async dispatch.


--
Laissez lire, et laissez danser; ces deux amusements ne feront jamais de mal au monde.
_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users


--
Jan Bartel <[hidden email]>
www.webtide.com
Expert assistance from the creators of Jetty and CometD


_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Jetty sessions leaking

Jan Bartel
Carey,

if you open the issue, I've got a fix ready to go.

Jan

On Sat, 15 Dec 2018 at 09:22, Jan Bartel <[hidden email]> wrote:
By all means raise it as an issue. And thank you for the repro test case, I'm on it.

Jan

On Sat, 15 Dec 2018 at 08:51, Carey Evans <[hidden email]> wrote:

On Thu, 13 Dec 2018 at 20:31, Carey Evans <[hidden email]> wrote:
An update: I added
<%@ page language="java" contentType="text/html; charset=UTF-8" pageEncoding="UTF-8" session="false" isErrorPage="true" %>
to the top of 503.jsp, and I no longer see the issue using this code.

I've simplified the code at https://github.com/carey/jetty-sessions-bug. Now all that's needed to create sessions that never expire is an error handling servlet that starts a new session from an async dispatch.


--
Laissez lire, et laissez danser; ces deux amusements ne feront jamais de mal au monde.
_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users


--
Jan Bartel <[hidden email]>
www.webtide.com
Expert assistance from the creators of Jetty and CometD



--
Jan Bartel <[hidden email]>
www.webtide.com
Expert assistance from the creators of Jetty and CometD


_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Jetty sessions leaking

Carey Evans-2
On Sat, 15 Dec 2018 at 15:40, Jan Bartel <[hidden email]> wrote:
Carey,

if you open the issue, I've got a fix ready to go.


--
Laissez lire, et laissez danser; ces deux amusements ne feront jamais de mal au monde.

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users