[jetty-dev] Concurrency issue causing EnvConfiguration.configure to throw a NameNotFoundException; remaining name 'env'

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

[jetty-dev] Concurrency issue causing EnvConfiguration.configure to throw a NameNotFoundException; remaining name 'env'

Matthew Sheppard-2
Hi Folks,

I'm currently debugging an issue we've seen running our system, which contains two embedded jetty servers, where rarely (about one in 6000 restarts is the best estimate I have at the moment) we see one of our web apps fail to deploy with jetty throwing a NameNotFoundException with the message "remaining name 'env'" (Full stack trace included for reference below).

If I understand the design of the code correctly, that shouldn't be possible because the EnvConfiguration should be ensuring that the 'env' subcontext is always created in preConfigure before its configure starts trying to bind Env Entries. (See https://github.com/eclipse/jetty.project/blob/jetty-9.4.x/jetty-plus/src/main/java/org/eclipse/jetty/plus/webapp/EnvConfiguration.java#L63)

I went looking through some of the relevant code and came to wondering whether https://github.com/eclipse/jetty.project/blob/jetty-9.4.x/jetty-plus/src/main/java/org/eclipse/jetty/plus/webapp/EnvConfiguration.java#L249 was possibly the cause. I think in writing this I've convinced myself, however, that since we start each jetty server in a separate thread there should not be a problem with messing with the current thread's context class loader as I initially though, but maybe there's something else there I'm overlooking.

I'm working at the moment to try to reduce the problem down into a form I can demonstrate it outside our system, but figured I'd post this in the meanwhile in the hope someone might have bumped into this before or have some pointers about where I ought to be looking.

Many thanks,
Matt Sheppard

—-
The full stack trace we see in jetty's log is as follows. I believe it's using jetty version 9.4.6.v20170531. I'm working on setting up a version with the latest jetty to reproduce the problem there, but it'll take some time given the infrequency with which it occurs.

javax.naming.NameNotFoundException; remaining name 'env'
        at org.eclipse.jetty.jndi.NamingContext.lookup(NamingContext.java:476)
        at org.eclipse.jetty.jndi.NamingContext.lookup(NamingContext.java:563)
        at org.eclipse.jetty.jndi.NamingContext.lookup(NamingContext.java:578)
        at org.eclipse.jetty.jndi.java.javaRootURLContext.lookup(javaRootURLContext.java:106)
        at javax.naming.InitialContext.lookup(InitialContext.java:417)
        at org.eclipse.jetty.plus.webapp.EnvConfiguration.bindEnvEntries(EnvConfiguration.java:209)
        at org.eclipse.jetty.plus.webapp.EnvConfiguration.configure(EnvConfiguration.java:128)
        at org.eclipse.jetty.webapp.WebAppContext.configure(WebAppContext.java:517)
        at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1458)
        at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:785)
        at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:261)
        at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:545)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.eclipse.jetty.deploy.bindings.StandardStarter.processBinding(StandardStarter.java:41)
        at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:188)
        at org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:502)
        at org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:150)
        at org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:180)
        at org.eclipse.jetty.deploy.providers.WebAppProvider.fileAdded(WebAppProvider.java:453)
        at org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:64)
        at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:610)
        at org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:529)
        at org.eclipse.jetty.util.Scanner.scan(Scanner.java:392)
        at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:313)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:150)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:564)
        at org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:239)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
        at org.eclipse.jetty.server.Server.start(Server.java:452)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113)
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
        at org.eclipse.jetty.server.Server.doStart(Server.java:419)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at (our code which calls start)

_______________________________________________
jetty-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-dev
Reply | Threaded
Open this post in threaded view
|

Re: [jetty-dev] Concurrency issue causing EnvConfiguration.configure to throw a NameNotFoundException; remaining name 'env'

Jan Bartel
This kind of sounds familiar, let me see if I can find the relevant bug ....  found the one I was thinking of:  https://github.com/eclipse/jetty.project/issues/1602

Although that problem was caused when the EnvConfiguration has already created the jndi context and the deployer failed to deploy it, whereas you're stacktrace shows that the context doesn't exist, so probably not the same cause.

You'd have to show us how you're starting your servers and deploying your webapps - do you use the deployer, or do you deploy the webapps directly to the ContextHandlerCollection? Do you set the classloader of the threads starting the jetty servers at all? Is there any chance more than 1 thread is used to deploy a single webapp?

A repro would be good.

Jan

On Tue, 28 May 2019 at 03:39, Matthew Sheppard <[hidden email]> wrote:
Hi Folks,

I'm currently debugging an issue we've seen running our system, which contains two embedded jetty servers, where rarely (about one in 6000 restarts is the best estimate I have at the moment) we see one of our web apps fail to deploy with jetty throwing a NameNotFoundException with the message "remaining name 'env'" (Full stack trace included for reference below).

If I understand the design of the code correctly, that shouldn't be possible because the EnvConfiguration should be ensuring that the 'env' subcontext is always created in preConfigure before its configure starts trying to bind Env Entries. (See https://github.com/eclipse/jetty.project/blob/jetty-9.4.x/jetty-plus/src/main/java/org/eclipse/jetty/plus/webapp/EnvConfiguration.java#L63)

I went looking through some of the relevant code and came to wondering whether https://github.com/eclipse/jetty.project/blob/jetty-9.4.x/jetty-plus/src/main/java/org/eclipse/jetty/plus/webapp/EnvConfiguration.java#L249 was possibly the cause. I think in writing this I've convinced myself, however, that since we start each jetty server in a separate thread there should not be a problem with messing with the current thread's context class loader as I initially though, but maybe there's something else there I'm overlooking.

I'm working at the moment to try to reduce the problem down into a form I can demonstrate it outside our system, but figured I'd post this in the meanwhile in the hope someone might have bumped into this before or have some pointers about where I ought to be looking.

Many thanks,
Matt Sheppard

—-
The full stack trace we see in jetty's log is as follows. I believe it's using jetty version 9.4.6.v20170531. I'm working on setting up a version with the latest jetty to reproduce the problem there, but it'll take some time given the infrequency with which it occurs.

javax.naming.NameNotFoundException; remaining name 'env'
        at org.eclipse.jetty.jndi.NamingContext.lookup(NamingContext.java:476)
        at org.eclipse.jetty.jndi.NamingContext.lookup(NamingContext.java:563)
        at org.eclipse.jetty.jndi.NamingContext.lookup(NamingContext.java:578)
        at org.eclipse.jetty.jndi.java.javaRootURLContext.lookup(javaRootURLContext.java:106)
        at javax.naming.InitialContext.lookup(InitialContext.java:417)
        at org.eclipse.jetty.plus.webapp.EnvConfiguration.bindEnvEntries(EnvConfiguration.java:209)
        at org.eclipse.jetty.plus.webapp.EnvConfiguration.configure(EnvConfiguration.java:128)
        at org.eclipse.jetty.webapp.WebAppContext.configure(WebAppContext.java:517)
        at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1458)
        at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:785)
        at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:261)
        at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:545)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.eclipse.jetty.deploy.bindings.StandardStarter.processBinding(StandardStarter.java:41)
        at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:188)
        at org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:502)
        at org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:150)
        at org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:180)
        at org.eclipse.jetty.deploy.providers.WebAppProvider.fileAdded(WebAppProvider.java:453)
        at org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:64)
        at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:610)
        at org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:529)
        at org.eclipse.jetty.util.Scanner.scan(Scanner.java:392)
        at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:313)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:150)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:564)
        at org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:239)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
        at org.eclipse.jetty.server.Server.start(Server.java:452)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113)
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
        at org.eclipse.jetty.server.Server.doStart(Server.java:419)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at (our code which calls start)
_______________________________________________
jetty-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-dev


--
Jan Bartel <[hidden email]>
www.webtide.com
Expert assistance from the creators of Jetty and CometD


_______________________________________________
jetty-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-dev
Reply | Threaded
Open this post in threaded view
|

Re: [jetty-dev] Concurrency issue causing EnvConfiguration.configure to throw a NameNotFoundException; remaining name 'env'

Matthew Sheppard-2
In reply to this post by Matthew Sheppard-2
Hi Jan et al,

After a bit of messing around I've managed to get a reasonably simple reproduction written up for the "NameNotFoundException; remaining name 'env'" issue I saw.

I've bundled it up into a github repo at https://github.com/mattsheppard/jetty-NameNotFoundException-remaining-name-env-reproduction with some basic instructions on how to run it and what you can expect to see when it fails.

If there's any more information I can usefully provide to help track the issue down please let me know.

Many thanks,
Matt Sheppard


P.S. if you'd prefer me to raise it in https://github.com/eclipse/jetty.project/issues now that I can at least reliably reproduce it I'm very happy to do that.
 
This kind of sounds familiar, let me see if I can find the relevant bug
....  found the one I was thinking of:
https://github.com/eclipse/jetty.project/issues/1602

Although that problem was caused when the EnvConfiguration has already
created the jndi context and the deployer failed to deploy it, whereas
you're stacktrace shows that the context doesn't exist, so probably not the
same cause.

You'd have to show us how you're starting your servers and deploying your
webapps - do you use the deployer, or do you deploy the webapps directly to
the ContextHandlerCollection? Do you set the classloader of the threads
starting the jetty servers at all? Is there any chance more than 1 thread
is used to deploy a single webapp?

A repro would be good.

Jan

On Tue, 28 May 2019 at 03:39, Matthew Sheppard <[hidden email]>
wrote:

> Hi Folks,
>
> I'm currently debugging an issue we've seen running our system, which
> contains two embedded jetty servers, where rarely (about one in 6000
> restarts is the best estimate I have at the moment) we see one of our web
> apps fail to deploy with jetty throwing a NameNotFoundException with the
> message "remaining name 'env'" (Full stack trace included for reference
> below).
>
> If I understand the design of the code correctly, that shouldn't be
> possible because the EnvConfiguration should be ensuring that the 'env'
> subcontext is always created in preConfigure before its configure starts
> trying to bind Env Entries. (See
> https://github.com/eclipse/jetty.project/blob/jetty-9.4.x/jetty-plus/src/main/java/org/eclipse/jetty/plus/webapp/EnvConfiguration.java#L63
> )
>
> I went looking through some of the relevant code and came to wondering
> whether
> https://github.com/eclipse/jetty.project/blob/jetty-9.4.x/jetty-plus/src/main/java/org/eclipse/jetty/plus/webapp/EnvConfiguration.java#L249
> was possibly the cause. I think in writing this I've convinced myself,
> however, that since we start each jetty server in a separate thread there
> should not be a problem with messing with the current thread's context
> class loader as I initially though, but maybe there's something else there
> I'm overlooking.
>
> I'm working at the moment to try to reduce the problem down into a form I
> can demonstrate it outside our system, but figured I'd post this in the
> meanwhile in the hope someone might have bumped into this before or have
> some pointers about where I ought to be looking.
>
> Many thanks,
> Matt Sheppard
>
> ?-
> The full stack trace we see in jetty's log is as follows. I believe it's
> using jetty version 9.4.6.v20170531. I'm working on setting up a version
> with the latest jetty to reproduce the problem there, but it'll take some
> time given the infrequency with which it occurs.
>
> javax.naming.NameNotFoundException; remaining name 'env'
>         at
> org.eclipse.jetty.jndi.NamingContext.lookup(NamingContext.java:476)
>         at
> org.eclipse.jetty.jndi.NamingContext.lookup(NamingContext.java:563)
>         at
> org.eclipse.jetty.jndi.NamingContext.lookup(NamingContext.java:578)
>         at
> org.eclipse.jetty.jndi.java.javaRootURLContext.lookup(javaRootURLContext.java:106)
>         at javax.naming.InitialContext.lookup(InitialContext.java:417)
>         at
> org.eclipse.jetty.plus.webapp.EnvConfiguration.bindEnvEntries(EnvConfiguration.java:209)
>         at
> org.eclipse.jetty.plus.webapp.EnvConfiguration.configure(EnvConfiguration.java:128)
>         at
> org.eclipse.jetty.webapp.WebAppContext.configure(WebAppContext.java:517)
>         at
> org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1458)
>         at
> org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:785)
>         at
> org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:261)
>         at
> org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:545)
>         at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>         at
> org.eclipse.jetty.deploy.bindings.StandardStarter.processBinding(StandardStarter.java:41)
>         at
> org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:188)
>         at
> org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:502)
>         at
> org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:150)
>         at
> org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:180)
>         at
> org.eclipse.jetty.deploy.providers.WebAppProvider.fileAdded(WebAppProvider.java:453)
>         at
> org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:64)
>         at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:610)
>         at
> org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:529)
>         at org.eclipse.jetty.util.Scanner.scan(Scanner.java:392)
>         at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:313)
>         at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>         at
> org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:150)
>         at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>         at
> org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:564)
>         at
> org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:239)
>         at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>         at
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
>         at org.eclipse.jetty.server.Server.start(Server.java:452)
>         at
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113)
>         at
> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
>         at org.eclipse.jetty.server.Server.doStart(Server.java:419)
>         at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>         at (our code which calls start)
> _______________________________________________
> jetty-dev mailing list
> [hidden email]
> To change your delivery options, retrieve your password, or unsubscribe
> from this list, visit
> https://www.eclipse.org/mailman/listinfo/jetty-dev



--
Jan Bartel <[hidden email]>
www.webtide.com
*Expert assistance from the creators of Jetty and CometD*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.eclipse.org/mailman/private/jetty-dev/attachments/20190528/86c27234/attachment.html>

------------------------------

_______________________________________________
jetty-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-dev

End of jetty-dev Digest, Vol 123, Issue 11
******************************************

_______________________________________________
jetty-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-dev
Reply | Threaded
Open this post in threaded view
|

Re: [jetty-dev] Concurrency issue causing EnvConfiguration.configure to throw a NameNotFoundException; remaining name 'env'

Jan Bartel
Matt,

Thanks for the repro test case!

I've let it run for nearly 3mins and didn't get the problem, however I did get an OutOfMemoryException, so there's something not quite right with the test:

Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
    at java.util.concurrent.FutureTask.report (FutureTask.java:122)
    at java.util.concurrent.FutureTask.get (FutureTask.java:191)
    at Main.main (Main.java:61)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:566)
    at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:297)
    at java.lang.Thread.run (Thread.java:834)

I think you should open an issue in the jetty issue tracker, and link to your repro on github and we'll take it from there. Be good if you took another look at the repro and fixed the OOM issue so we aren't muddying the waters.

cheers
Jan

On Tue, 4 Jun 2019 at 08:01, Matthew Sheppard <[hidden email]> wrote:
Hi Jan et al,

After a bit of messing around I've managed to get a reasonably simple reproduction written up for the "NameNotFoundException; remaining name 'env'" issue I saw.

I've bundled it up into a github repo at https://github.com/mattsheppard/jetty-NameNotFoundException-remaining-name-env-reproduction with some basic instructions on how to run it and what you can expect to see when it fails.

If there's any more information I can usefully provide to help track the issue down please let me know.

Many thanks,
Matt Sheppard


P.S. if you'd prefer me to raise it in https://github.com/eclipse/jetty.project/issues now that I can at least reliably reproduce it I'm very happy to do that.
 
This kind of sounds familiar, let me see if I can find the relevant bug
....  found the one I was thinking of:
https://github.com/eclipse/jetty.project/issues/1602

Although that problem was caused when the EnvConfiguration has already
created the jndi context and the deployer failed to deploy it, whereas
you're stacktrace shows that the context doesn't exist, so probably not the
same cause.

You'd have to show us how you're starting your servers and deploying your
webapps - do you use the deployer, or do you deploy the webapps directly to
the ContextHandlerCollection? Do you set the classloader of the threads
starting the jetty servers at all? Is there any chance more than 1 thread
is used to deploy a single webapp?

A repro would be good.

Jan

On Tue, 28 May 2019 at 03:39, Matthew Sheppard <[hidden email]>
wrote:

> Hi Folks,
>
> I'm currently debugging an issue we've seen running our system, which
> contains two embedded jetty servers, where rarely (about one in 6000
> restarts is the best estimate I have at the moment) we see one of our web
> apps fail to deploy with jetty throwing a NameNotFoundException with the
> message "remaining name 'env'" (Full stack trace included for reference
> below).
>
> If I understand the design of the code correctly, that shouldn't be
> possible because the EnvConfiguration should be ensuring that the 'env'
> subcontext is always created in preConfigure before its configure starts
> trying to bind Env Entries. (See
> https://github.com/eclipse/jetty.project/blob/jetty-9.4.x/jetty-plus/src/main/java/org/eclipse/jetty/plus/webapp/EnvConfiguration.java#L63
> )
>
> I went looking through some of the relevant code and came to wondering
> whether
> https://github.com/eclipse/jetty.project/blob/jetty-9.4.x/jetty-plus/src/main/java/org/eclipse/jetty/plus/webapp/EnvConfiguration.java#L249
> was possibly the cause. I think in writing this I've convinced myself,
> however, that since we start each jetty server in a separate thread there
> should not be a problem with messing with the current thread's context
> class loader as I initially though, but maybe there's something else there
> I'm overlooking.
>
> I'm working at the moment to try to reduce the problem down into a form I
> can demonstrate it outside our system, but figured I'd post this in the
> meanwhile in the hope someone might have bumped into this before or have
> some pointers about where I ought to be looking.
>
> Many thanks,
> Matt Sheppard
>
> ?-
> The full stack trace we see in jetty's log is as follows. I believe it's
> using jetty version 9.4.6.v20170531. I'm working on setting up a version
> with the latest jetty to reproduce the problem there, but it'll take some
> time given the infrequency with which it occurs.
>
> javax.naming.NameNotFoundException; remaining name 'env'
>         at
> org.eclipse.jetty.jndi.NamingContext.lookup(NamingContext.java:476)
>         at
> org.eclipse.jetty.jndi.NamingContext.lookup(NamingContext.java:563)
>         at
> org.eclipse.jetty.jndi.NamingContext.lookup(NamingContext.java:578)
>         at
> org.eclipse.jetty.jndi.java.javaRootURLContext.lookup(javaRootURLContext.java:106)
>         at javax.naming.InitialContext.lookup(InitialContext.java:417)
>         at
> org.eclipse.jetty.plus.webapp.EnvConfiguration.bindEnvEntries(EnvConfiguration.java:209)
>         at
> org.eclipse.jetty.plus.webapp.EnvConfiguration.configure(EnvConfiguration.java:128)
>         at
> org.eclipse.jetty.webapp.WebAppContext.configure(WebAppContext.java:517)
>         at
> org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1458)
>         at
> org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:785)
>         at
> org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:261)
>         at
> org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:545)
>         at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>         at
> org.eclipse.jetty.deploy.bindings.StandardStarter.processBinding(StandardStarter.java:41)
>         at
> org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:188)
>         at
> org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:502)
>         at
> org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:150)
>         at
> org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:180)
>         at
> org.eclipse.jetty.deploy.providers.WebAppProvider.fileAdded(WebAppProvider.java:453)
>         at
> org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:64)
>         at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:610)
>         at
> org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:529)
>         at org.eclipse.jetty.util.Scanner.scan(Scanner.java:392)
>         at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:313)
>         at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>         at
> org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:150)
>         at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>         at
> org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:564)
>         at
> org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:239)
>         at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>         at
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
>         at org.eclipse.jetty.server.Server.start(Server.java:452)
>         at
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113)
>         at
> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
>         at org.eclipse.jetty.server.Server.doStart(Server.java:419)
>         at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>         at (our code which calls start)
> _______________________________________________
> jetty-dev mailing list
> [hidden email]
> To change your delivery options, retrieve your password, or unsubscribe
> from this list, visit
> https://www.eclipse.org/mailman/listinfo/jetty-dev



--
Jan Bartel <[hidden email]>
www.webtide.com
*Expert assistance from the creators of Jetty and CometD*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.eclipse.org/mailman/private/jetty-dev/attachments/20190528/86c27234/attachment.html>

------------------------------

_______________________________________________
jetty-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-dev

End of jetty-dev Digest, Vol 123, Issue 11
******************************************
_______________________________________________
jetty-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-dev


--
Jan Bartel <[hidden email]>
www.webtide.com
Expert assistance from the creators of Jetty and CometD


_______________________________________________
jetty-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-dev
Reply | Threaded
Open this post in threaded view
|

Re: [jetty-dev] Concurrency issue causing EnvConfiguration.configure to throw a NameNotFoundException; remaining name 'env'

Jan Bartel
So broadly, I think the issue is that jndi is not thread safe, as per the javadoc on javax.naming.Context:

 "A Context instance is not guaranteed to be synchronized against
 * concurrent access by multiple threads.  Threads that need to access
 * a single Context instance concurrently should synchronize amongst
 * themselves and provide the necessary locking. ..."

So there is no doubt some race in the creation/access of java:comp/env between the 2 threads controlling the servers' startups.

I'll see if there's something we can do to make the java:comp/env stuff threadsafe - at least at startup.

Please make the issue and then I'll comment on it.

cheers
Jan

On Tue, 4 Jun 2019 at 11:33, Jan Bartel <[hidden email]> wrote:
Matt,

Thanks for the repro test case!

I've let it run for nearly 3mins and didn't get the problem, however I did get an OutOfMemoryException, so there's something not quite right with the test:

Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
    at java.util.concurrent.FutureTask.report (FutureTask.java:122)
    at java.util.concurrent.FutureTask.get (FutureTask.java:191)
    at Main.main (Main.java:61)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:566)
    at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:297)
    at java.lang.Thread.run (Thread.java:834)

I think you should open an issue in the jetty issue tracker, and link to your repro on github and we'll take it from there. Be good if you took another look at the repro and fixed the OOM issue so we aren't muddying the waters.

cheers
Jan

On Tue, 4 Jun 2019 at 08:01, Matthew Sheppard <[hidden email]> wrote:
Hi Jan et al,

After a bit of messing around I've managed to get a reasonably simple reproduction written up for the "NameNotFoundException; remaining name 'env'" issue I saw.

I've bundled it up into a github repo at https://github.com/mattsheppard/jetty-NameNotFoundException-remaining-name-env-reproduction with some basic instructions on how to run it and what you can expect to see when it fails.

If there's any more information I can usefully provide to help track the issue down please let me know.

Many thanks,
Matt Sheppard


P.S. if you'd prefer me to raise it in https://github.com/eclipse/jetty.project/issues now that I can at least reliably reproduce it I'm very happy to do that.
 
This kind of sounds familiar, let me see if I can find the relevant bug
....  found the one I was thinking of:
https://github.com/eclipse/jetty.project/issues/1602

Although that problem was caused when the EnvConfiguration has already
created the jndi context and the deployer failed to deploy it, whereas
you're stacktrace shows that the context doesn't exist, so probably not the
same cause.

You'd have to show us how you're starting your servers and deploying your
webapps - do you use the deployer, or do you deploy the webapps directly to
the ContextHandlerCollection? Do you set the classloader of the threads
starting the jetty servers at all? Is there any chance more than 1 thread
is used to deploy a single webapp?

A repro would be good.

Jan

On Tue, 28 May 2019 at 03:39, Matthew Sheppard <[hidden email]>
wrote:

> Hi Folks,
>
> I'm currently debugging an issue we've seen running our system, which
> contains two embedded jetty servers, where rarely (about one in 6000
> restarts is the best estimate I have at the moment) we see one of our web
> apps fail to deploy with jetty throwing a NameNotFoundException with the
> message "remaining name 'env'" (Full stack trace included for reference
> below).
>
> If I understand the design of the code correctly, that shouldn't be
> possible because the EnvConfiguration should be ensuring that the 'env'
> subcontext is always created in preConfigure before its configure starts
> trying to bind Env Entries. (See
> https://github.com/eclipse/jetty.project/blob/jetty-9.4.x/jetty-plus/src/main/java/org/eclipse/jetty/plus/webapp/EnvConfiguration.java#L63
> )
>
> I went looking through some of the relevant code and came to wondering
> whether
> https://github.com/eclipse/jetty.project/blob/jetty-9.4.x/jetty-plus/src/main/java/org/eclipse/jetty/plus/webapp/EnvConfiguration.java#L249
> was possibly the cause. I think in writing this I've convinced myself,
> however, that since we start each jetty server in a separate thread there
> should not be a problem with messing with the current thread's context
> class loader as I initially though, but maybe there's something else there
> I'm overlooking.
>
> I'm working at the moment to try to reduce the problem down into a form I
> can demonstrate it outside our system, but figured I'd post this in the
> meanwhile in the hope someone might have bumped into this before or have
> some pointers about where I ought to be looking.
>
> Many thanks,
> Matt Sheppard
>
> ?-
> The full stack trace we see in jetty's log is as follows. I believe it's
> using jetty version 9.4.6.v20170531. I'm working on setting up a version
> with the latest jetty to reproduce the problem there, but it'll take some
> time given the infrequency with which it occurs.
>
> javax.naming.NameNotFoundException; remaining name 'env'
>         at
> org.eclipse.jetty.jndi.NamingContext.lookup(NamingContext.java:476)
>         at
> org.eclipse.jetty.jndi.NamingContext.lookup(NamingContext.java:563)
>         at
> org.eclipse.jetty.jndi.NamingContext.lookup(NamingContext.java:578)
>         at
> org.eclipse.jetty.jndi.java.javaRootURLContext.lookup(javaRootURLContext.java:106)
>         at javax.naming.InitialContext.lookup(InitialContext.java:417)
>         at
> org.eclipse.jetty.plus.webapp.EnvConfiguration.bindEnvEntries(EnvConfiguration.java:209)
>         at
> org.eclipse.jetty.plus.webapp.EnvConfiguration.configure(EnvConfiguration.java:128)
>         at
> org.eclipse.jetty.webapp.WebAppContext.configure(WebAppContext.java:517)
>         at
> org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1458)
>         at
> org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:785)
>         at
> org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:261)
>         at
> org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:545)
>         at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>         at
> org.eclipse.jetty.deploy.bindings.StandardStarter.processBinding(StandardStarter.java:41)
>         at
> org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:188)
>         at
> org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:502)
>         at
> org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:150)
>         at
> org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:180)
>         at
> org.eclipse.jetty.deploy.providers.WebAppProvider.fileAdded(WebAppProvider.java:453)
>         at
> org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:64)
>         at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:610)
>         at
> org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:529)
>         at org.eclipse.jetty.util.Scanner.scan(Scanner.java:392)
>         at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:313)
>         at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>         at
> org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:150)
>         at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>         at
> org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:564)
>         at
> org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:239)
>         at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>         at
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
>         at org.eclipse.jetty.server.Server.start(Server.java:452)
>         at
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113)
>         at
> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
>         at org.eclipse.jetty.server.Server.doStart(Server.java:419)
>         at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>         at (our code which calls start)
> _______________________________________________
> jetty-dev mailing list
> [hidden email]
> To change your delivery options, retrieve your password, or unsubscribe
> from this list, visit
> https://www.eclipse.org/mailman/listinfo/jetty-dev



--
Jan Bartel <[hidden email]>
www.webtide.com
*Expert assistance from the creators of Jetty and CometD*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.eclipse.org/mailman/private/jetty-dev/attachments/20190528/86c27234/attachment.html>

------------------------------

_______________________________________________
jetty-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-dev

End of jetty-dev Digest, Vol 123, Issue 11
******************************************
_______________________________________________
jetty-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-dev


--
Jan Bartel <[hidden email]>
www.webtide.com
Expert assistance from the creators of Jetty and CometD



--
Jan Bartel <[hidden email]>
www.webtide.com
Expert assistance from the creators of Jetty and CometD


_______________________________________________
jetty-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-dev
Reply | Threaded
Open this post in threaded view
|

Re: [jetty-dev] Concurrency issue causing EnvConfiguration.configure to throw a NameNotFoundException; remaining name 'env'

Jan Bartel
OK, I went ahead and made the issue already so I can work on it - see https://github.com/eclipse/jetty.project/issues/3729

Jan

On Tue, 4 Jun 2019 at 13:52, Jan Bartel <[hidden email]> wrote:
So broadly, I think the issue is that jndi is not thread safe, as per the javadoc on javax.naming.Context:

 "A Context instance is not guaranteed to be synchronized against
 * concurrent access by multiple threads.  Threads that need to access
 * a single Context instance concurrently should synchronize amongst
 * themselves and provide the necessary locking. ..."

So there is no doubt some race in the creation/access of java:comp/env between the 2 threads controlling the servers' startups.

I'll see if there's something we can do to make the java:comp/env stuff threadsafe - at least at startup.

Please make the issue and then I'll comment on it.

cheers
Jan

On Tue, 4 Jun 2019 at 11:33, Jan Bartel <[hidden email]> wrote:
Matt,

Thanks for the repro test case!

I've let it run for nearly 3mins and didn't get the problem, however I did get an OutOfMemoryException, so there's something not quite right with the test:

Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
    at java.util.concurrent.FutureTask.report (FutureTask.java:122)
    at java.util.concurrent.FutureTask.get (FutureTask.java:191)
    at Main.main (Main.java:61)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:566)
    at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:297)
    at java.lang.Thread.run (Thread.java:834)

I think you should open an issue in the jetty issue tracker, and link to your repro on github and we'll take it from there. Be good if you took another look at the repro and fixed the OOM issue so we aren't muddying the waters.

cheers
Jan

On Tue, 4 Jun 2019 at 08:01, Matthew Sheppard <[hidden email]> wrote:
Hi Jan et al,

After a bit of messing around I've managed to get a reasonably simple reproduction written up for the "NameNotFoundException; remaining name 'env'" issue I saw.

I've bundled it up into a github repo at https://github.com/mattsheppard/jetty-NameNotFoundException-remaining-name-env-reproduction with some basic instructions on how to run it and what you can expect to see when it fails.

If there's any more information I can usefully provide to help track the issue down please let me know.

Many thanks,
Matt Sheppard


P.S. if you'd prefer me to raise it in https://github.com/eclipse/jetty.project/issues now that I can at least reliably reproduce it I'm very happy to do that.
 
This kind of sounds familiar, let me see if I can find the relevant bug
....  found the one I was thinking of:
https://github.com/eclipse/jetty.project/issues/1602

Although that problem was caused when the EnvConfiguration has already
created the jndi context and the deployer failed to deploy it, whereas
you're stacktrace shows that the context doesn't exist, so probably not the
same cause.

You'd have to show us how you're starting your servers and deploying your
webapps - do you use the deployer, or do you deploy the webapps directly to
the ContextHandlerCollection? Do you set the classloader of the threads
starting the jetty servers at all? Is there any chance more than 1 thread
is used to deploy a single webapp?

A repro would be good.

Jan

On Tue, 28 May 2019 at 03:39, Matthew Sheppard <[hidden email]>
wrote:

> Hi Folks,
>
> I'm currently debugging an issue we've seen running our system, which
> contains two embedded jetty servers, where rarely (about one in 6000
> restarts is the best estimate I have at the moment) we see one of our web
> apps fail to deploy with jetty throwing a NameNotFoundException with the
> message "remaining name 'env'" (Full stack trace included for reference
> below).
>
> If I understand the design of the code correctly, that shouldn't be
> possible because the EnvConfiguration should be ensuring that the 'env'
> subcontext is always created in preConfigure before its configure starts
> trying to bind Env Entries. (See
> https://github.com/eclipse/jetty.project/blob/jetty-9.4.x/jetty-plus/src/main/java/org/eclipse/jetty/plus/webapp/EnvConfiguration.java#L63
> )
>
> I went looking through some of the relevant code and came to wondering
> whether
> https://github.com/eclipse/jetty.project/blob/jetty-9.4.x/jetty-plus/src/main/java/org/eclipse/jetty/plus/webapp/EnvConfiguration.java#L249
> was possibly the cause. I think in writing this I've convinced myself,
> however, that since we start each jetty server in a separate thread there
> should not be a problem with messing with the current thread's context
> class loader as I initially though, but maybe there's something else there
> I'm overlooking.
>
> I'm working at the moment to try to reduce the problem down into a form I
> can demonstrate it outside our system, but figured I'd post this in the
> meanwhile in the hope someone might have bumped into this before or have
> some pointers about where I ought to be looking.
>
> Many thanks,
> Matt Sheppard
>
> ?-
> The full stack trace we see in jetty's log is as follows. I believe it's
> using jetty version 9.4.6.v20170531. I'm working on setting up a version
> with the latest jetty to reproduce the problem there, but it'll take some
> time given the infrequency with which it occurs.
>
> javax.naming.NameNotFoundException; remaining name 'env'
>         at
> org.eclipse.jetty.jndi.NamingContext.lookup(NamingContext.java:476)
>         at
> org.eclipse.jetty.jndi.NamingContext.lookup(NamingContext.java:563)
>         at
> org.eclipse.jetty.jndi.NamingContext.lookup(NamingContext.java:578)
>         at
> org.eclipse.jetty.jndi.java.javaRootURLContext.lookup(javaRootURLContext.java:106)
>         at javax.naming.InitialContext.lookup(InitialContext.java:417)
>         at
> org.eclipse.jetty.plus.webapp.EnvConfiguration.bindEnvEntries(EnvConfiguration.java:209)
>         at
> org.eclipse.jetty.plus.webapp.EnvConfiguration.configure(EnvConfiguration.java:128)
>         at
> org.eclipse.jetty.webapp.WebAppContext.configure(WebAppContext.java:517)
>         at
> org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1458)
>         at
> org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:785)
>         at
> org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:261)
>         at
> org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:545)
>         at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>         at
> org.eclipse.jetty.deploy.bindings.StandardStarter.processBinding(StandardStarter.java:41)
>         at
> org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:188)
>         at
> org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:502)
>         at
> org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:150)
>         at
> org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:180)
>         at
> org.eclipse.jetty.deploy.providers.WebAppProvider.fileAdded(WebAppProvider.java:453)
>         at
> org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:64)
>         at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:610)
>         at
> org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:529)
>         at org.eclipse.jetty.util.Scanner.scan(Scanner.java:392)
>         at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:313)
>         at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>         at
> org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:150)
>         at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>         at
> org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:564)
>         at
> org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:239)
>         at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>         at
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
>         at org.eclipse.jetty.server.Server.start(Server.java:452)
>         at
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113)
>         at
> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
>         at org.eclipse.jetty.server.Server.doStart(Server.java:419)
>         at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>         at (our code which calls start)
> _______________________________________________
> jetty-dev mailing list
> [hidden email]
> To change your delivery options, retrieve your password, or unsubscribe
> from this list, visit
> https://www.eclipse.org/mailman/listinfo/jetty-dev



--
Jan Bartel <[hidden email]>
www.webtide.com
*Expert assistance from the creators of Jetty and CometD*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.eclipse.org/mailman/private/jetty-dev/attachments/20190528/86c27234/attachment.html>

------------------------------

_______________________________________________
jetty-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-dev

End of jetty-dev Digest, Vol 123, Issue 11
******************************************
_______________________________________________
jetty-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-dev


--
Jan Bartel <[hidden email]>
www.webtide.com
Expert assistance from the creators of Jetty and CometD



--
Jan Bartel <[hidden email]>
www.webtide.com
Expert assistance from the creators of Jetty and CometD



--
Jan Bartel <[hidden email]>
www.webtide.com
Expert assistance from the creators of Jetty and CometD


_______________________________________________
jetty-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-dev