Jetty http2 client request hangs indefinitely

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Jetty http2 client request hangs indefinitely

Santhosh Kumar

Hi,

   We are trying to implement a http2 jetty client which makes a multiplexed connection with our remote servers. Our program is as follows,

Security.addProvider(new OpenSSLProvider());//Conscrypt for ALPN support in jdk8
SslContextFactory sslContextFactory = new SslContextFactory.Client(true);
sslContextFactory.setProvider("Conscrypt");

sslContextFactory.setProtocol("TLSv1.3");//Use TLSv1.3
HTTP2Client http2Client = new HTTP2Client();
http2Client.setConnectTimeout(5000);
http2Client.setIdleTimeout(30000);

org.eclipse.jetty.client.HttpClient http2HttpClient = new org.eclipse.jetty.client.HttpClient(new HttpClientTransportOverHTTP2(http2Client), sslContextFactory);
http2HttpClient.setConnectTimeout(5000);
http2HttpClient.setIdleTimeout(30000);
http2HttpClient.start();
http2HttpClient.addBean(sslContextFactory);
http2HttpClient.start();

org.eclipse.jetty.client.api.Request request = http2HttpClient.POST(locationurl+locationposturl);
request.header(HttpHeader.CONTENT_TYPE, "application/json");//No I18N
request.header("Pragma", "no-cache");
request.header("Custom", customheaderName);
request.header("Secure", secureHeader);
request.header("protocol", "h2");
request.content(new StringContentProvider(myXMLRequest,"utf-8"));

request.timeout(420, TimeUnit.SECONDS);
request.idleTimeout(30, TimeUnit.SECONDS);

ContentResponse response = request.send();
if (response != null && (response.getStatus() >= 200 && response.getStatus() < 400)) {
int statuscode=response.getStatus();
byte [] wbresponse = response.getContent();
}

We use synchronous calls and conscrypt(security provider) to support ALPN in jdk8 and use TLSv1.3. we are able to make http2 requests successfully to our remote locations. but we face a problem where most of the threads hangs indefinitely until we restart our servers. we are using tomcat 9.x at both ends and each remote servers are behind dedicated HAproxy(used as h2/ssl terminator).

Client : Security provider used: Conscrypt-2.1.0, JAVA: JDK 8, tomcat: 9.0.21, Jetty: 9.4.19

Server: JAVA: JDK 11, tomcat: 9.0.21, Haproxy: 2.0

I checked jetty docs for possible timeout and we have configured all timeout but still we don't know why threads hung indefinitely. We have thread dump for the same,

34534543535345435(DNS) - Tue Aug 13 04:40:06 PDT 2019" #1081 prio=5 os_prio=0 tid=0x00007f93e40d3000 nid=0xd820 waiting on condition [0x00007f91c7cfa000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000000d9648be0> (a java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
at org.eclipse.jetty.client.util.FutureResponseListener.get(FutureResponseListener.java:100)
at org.eclipse.jetty.client.HttpRequest.send(HttpRequest.java:683)
at com.adventnet.webmon.util.LocationTestUtil.getLocationResponse(LocationTestUtil.java:430)
at com.adventnet.webmon.util.LocationTestUtil.getLocationResponse(LocationTestUtil.java:219)
at com.adventnet.webmon.conf.server.ConfDataCollection.doMultiLocationDC(ConfDataCollection.java:3192)
at com.adventnet.webmon.conf.server.ConfDataCollection.getResponseData(ConfDataCollection.java:2703)
at com.adventnet.webmon.conf.server.ConfDataCollection.invokeDataCollection(ConfDataCollection.java:479)
at com.adventnet.webmon.conf.task.ConfMonitorTask.run(ConfMonitorTask.java:676)
at com.zoho.scheduler.JobWrapper.invokeRunnableJob(JobWrapper.java:116)
at com.zoho.scheduler.instrument.WrappedJobWrapper.invokeRunnableJob(WrappedJobWrapper.java:34)
at com.zoho.scheduler.JobWrapper.executeJob(JobWrapper.java:89)
at com.zoho.scheduler.JobWrapper.run(JobWrapper.java:47)
at com.adventnet.taskengine.inmemory.TaskExecutor$2.call(TaskExecutor.java:113)
at com.adventnet.sas.scheduler.inmemory.WrappedSASExecutorListener.call(WrappedSASExecutorListener.java:33)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
The biggest problem is it does not happen in test grids, its only happening in production servers.

Kindly help to fix this problem

TIA

--
With Regards,
Santhosh Kumar J

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Jetty http2 client request hangs indefinitely

Simone Bordet-3
Hi,

On Wed, Aug 21, 2019 at 11:42 AM Santhosh Kumar <[hidden email]> wrote:
> We are trying to implement a http2 jetty client which makes a multiplexed connection with our remote servers. Our program is as follows,

[snip]

The code is correct, apart perhaps for this:

> request.timeout(420, TimeUnit.SECONDS);

This is typically way too large. Is there any reason you want this
large total timeout?

> We use synchronous calls and conscrypt(security provider) to support ALPN in jdk8 and use TLSv1.3. we are able to make http2 requests successfully to our remote locations. but we face a problem where most of the threads hangs indefinitely until we restart our servers.

You have to explain this a bit better.
From where are you using the client? From a remote client, or from
within a web application deployed in Tomcat?
With "restart our servers" you mean restart the remote servers that
the client calls?
A schematic diagram of client, load balancer, servers, etc will be
great to understand how is the system.

> I checked jetty docs for possible timeout and we have configured all timeout but still we don't know why threads hung indefinitely. We have thread dump for the same,

The thread dump does not show much.
You will have better luck registering HttpClient instances in JMX and
the performing a HttpClient dump via JMX.
This is how you do it:
https://github.com/eclipse/jetty.project/blob/jetty-9.4.20.v20190813/jetty-client/src/test/java/org/eclipse/jetty/client/jmx/HttpClientJMXTest.java
Once you are blocking again, you connect with a JMX console, find the
HttpClient MBean, and invoke its dump() operation to get the
HttpClient internal state.
Then attach it here for analysis.

Do you have the same problem if you don't use OpenSSL, but the
standard JDK TLS implementation?

--
Simone Bordet
----
http://cometd.org
http://webtide.com
Developer advice, training, services and support
from the Jetty & CometD experts.
_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Jetty http2 client request hangs indefinitely

Santhosh Kumar
Hi,

>This is typically way too large. Is there any reason you want this
large total timeout?

We have operations that may 5-10 mins, however threads hung indefinitely even beyond this timeout and even shorter timeouts like 180 seconds are also hanging.

>From where are you using the client? From a remote client, or from
within a web application deployed in Tomcat?

Yes our client runs in an application deployed inside tomcat which makes calls to remotely hosted servers.

>With "restart our servers" you mean restart the remote servers that
the client calls?

Not the remote servers, we restart appserver where jettyclient is running.

>A schematic diagram of client, load balancer, servers, etc will be
great to understand how is the system.

https://imgur.com/bxV3BA9

>The thread dump does not show much.
You will have better luck registering HttpClient instances in JMX and
the performing a HttpClient dump via JMX.
This is how you do it:
https://github.com/eclipse/jetty.project/blob/jetty-9.4.20.v20190813/jetty-client/src/test/java/org/eclipse/jetty/client/jmx/HttpClientJMXTest.java
Once you are blocking again, you connect with a JMX console, find the
HttpClient MBean, and invoke its dump() operation to get the
HttpClient internal state.
Then attach it here for analysis.

I cannot use JMX in live production servers, so I just called httpclient.dump() while it was hanging using a local servlet call,  FYKI we instantiate a static httpclient and it will be used  to establish connection with many remote servers  in a multithreaded environment. PFA links below for client dumps

https://pastebin.com/N017wph4

https://pastebin.com/k21hzLwL

>Do you have the same problem if you don't use OpenSSL, but the
standard JDK TLS implementation?

Our framework does not support JDK 9 or above so we are dependent on conscrypt(OPENSSL) for alpn support in jdk8.

Kidnly let us know if any other details are required 

On Wed, Aug 21, 2019 at 9:29 PM Simone Bordet <[hidden email]> wrote:
Hi,

On Wed, Aug 21, 2019 at 11:42 AM Santhosh Kumar <[hidden email]> wrote:
> We are trying to implement a http2 jetty client which makes a multiplexed connection with our remote servers. Our program is as follows,

[snip]

The code is correct, apart perhaps for this:

> request.timeout(420, TimeUnit.SECONDS);

This is typically way too large. Is there any reason you want this
large total timeout?

> We use synchronous calls and conscrypt(security provider) to support ALPN in jdk8 and use TLSv1.3. we are able to make http2 requests successfully to our remote locations. but we face a problem where most of the threads hangs indefinitely until we restart our servers.

You have to explain this a bit better.
From where are you using the client? From a remote client, or from
within a web application deployed in Tomcat?
With "restart our servers" you mean restart the remote servers that
the client calls?
A schematic diagram of client, load balancer, servers, etc will be
great to understand how is the system.

> I checked jetty docs for possible timeout and we have configured all timeout but still we don't know why threads hung indefinitely. We have thread dump for the same,

The thread dump does not show much.
You will have better luck registering HttpClient instances in JMX and
the performing a HttpClient dump via JMX.
This is how you do it:
https://github.com/eclipse/jetty.project/blob/jetty-9.4.20.v20190813/jetty-client/src/test/java/org/eclipse/jetty/client/jmx/HttpClientJMXTest.java
Once you are blocking again, you connect with a JMX console, find the
HttpClient MBean, and invoke its dump() operation to get the
HttpClient internal state.
Then attach it here for analysis.

Do you have the same problem if you don't use OpenSSL, but the
standard JDK TLS implementation?

--
Simone Bordet
----
http://cometd.org
http://webtide.com
Developer advice, training, services and support
from the Jetty & CometD experts.
_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users


--
With Regards,
Santhosh Kumar J

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Jetty http2 client request hangs indefinitely

Simone Bordet-3
Santhosh,

sorry for the late reply.

On Mon, Aug 26, 2019 at 3:26 PM Santhosh Kumar <[hidden email]> wrote:
> I cannot use JMX in live production servers, so I just called httpclient.dump() while it was hanging using a local servlet call,  FYKI we instantiate a static httpclient and it will be used  to establish connection with many remote servers  in a multithreaded environment. PFA links below for client dumps
>
> https://pastebin.com/N017wph4
>
> https://pastebin.com/k21hzLwL

Good solution, just remember to protect that local Servlet (only you
should be able to call it).

What I see from the dumps are 2 things.
* the client connects to ports 8080 (typically used for clear-text
communication), while the diagram hints there is SSL offload at the
external server.
* the client has a lot of exchanges queued, and the connection pool is
empty of established connections, although 1 was attempted.

This is one client dump that seems to be manufactured as I looked at
it and thought "impossible".

This strikes to me as the client trying to connect to the external
server, but does not receive a TCP reply (or we handle it wrongly).
The client-side connect timeout should fire and fail all the
exchanges, but apparently it does not.

Can you enable DEBUG logs then send the *first* request?
There must be something really weird that happens and we don't cope
well with it.

Thanks for your patience.

--
Simone Bordet
----
http://cometd.org
http://webtide.com
Developer advice, training, services and support
from the Jetty & CometD experts.
_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Jetty http2 client request hangs indefinitely

Santhosh Kumar
Thanks SImone for the analysis,

>Good solution, just remember to protect that local Servlet (only you
should be able to call it).

Yes the servlet will only be accessible by us.

> the client connects to ports 8080 (typically used for clear-text
communication), while the diagram hints there is SSL offload at the
external server.

We have dedicated haproxy before each external servers which runs in 8080 and work as a SSL offloader and pass it to backend server which is a clear-text.

>Can you enable DEBUG logs then send the *first* request?
There must be something really weird that happens and we don't cope
well with it.

Enabling Debug logs in production servers are real headache, anyhow I ll try to get the logs somehow. My doubt is *first* request you mean whether its a very first request fired by the client or the first request which hangs?

On Fri, Sep 20, 2019 at 3:54 PM Simone Bordet <[hidden email]> wrote:
Santhosh,

sorry for the late reply.

On Mon, Aug 26, 2019 at 3:26 PM Santhosh Kumar <[hidden email]> wrote:
> I cannot use JMX in live production servers, so I just called httpclient.dump() while it was hanging using a local servlet call,  FYKI we instantiate a static httpclient and it will be used  to establish connection with many remote servers  in a multithreaded environment. PFA links below for client dumps
>
> https://pastebin.com/N017wph4
>
> https://pastebin.com/k21hzLwL

Good solution, just remember to protect that local Servlet (only you
should be able to call it).

What I see from the dumps are 2 things.
* the client connects to ports 8080 (typically used for clear-text
communication), while the diagram hints there is SSL offload at the
external server.
* the client has a lot of exchanges queued, and the connection pool is
empty of established connections, although 1 was attempted.

This is one client dump that seems to be manufactured as I looked at
it and thought "impossible".

This strikes to me as the client trying to connect to the external
server, but does not receive a TCP reply (or we handle it wrongly).
The client-side connect timeout should fire and fail all the
exchanges, but apparently it does not.

Can you enable DEBUG logs then send the *first* request?
There must be something really weird that happens and we don't cope
well with it.

Thanks for your patience.

--
Simone Bordet
----
http://cometd.org
http://webtide.com
Developer advice, training, services and support
from the Jetty & CometD experts.
_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users


--
With Regards,
Santhosh Kumar J

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Jetty http2 client request hangs indefinitely

Simone Bordet-3
Hi,

On Fri, Sep 20, 2019 at 2:09 PM Santhosh Kumar <[hidden email]> wrote:
> Enabling Debug logs in production servers are real headache, anyhow I ll try to get the logs somehow. My doubt is *first* request you mean whether its a very first request fired by the client or the first request which hangs?

What I see from the dump is that the requests get queued and there are
no established connections.
Since it is the first request that triggers the connection opening, I
want to see the debug logs to understand why the connection
establishment failed, and how it failed (as any failure should have
aborted the queued requests).
You make requests by host or by IP address (i.e. is DNS involved)?

--
Simone Bordet
----
http://cometd.org
http://webtide.com
Developer advice, training, services and support
from the Jetty & CometD experts.
_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Jetty http2 client request hangs indefinitely

Santhosh Kumar
>What I see from the dump is that the requests get queued and there are
no established connections.
Since it is the first request that triggers the connection opening, I
want to see the debug logs to understand why the connection
establishment failed, and how it failed (as any failure should have
aborted the queued requests).

The remote server is always accessible(verified by browser, curl, wget or even another jetty standalone program)  when threads are hung.

>You make requests by host or by IP address (i.e. is DNS involved)?

We make requests by IP address.

I ll try to get the debug logs asap

Thanks :)





On Fri, Sep 20, 2019 at 7:54 PM Simone Bordet <[hidden email]> wrote:
Hi,

On Fri, Sep 20, 2019 at 2:09 PM Santhosh Kumar <[hidden email]> wrote:
> Enabling Debug logs in production servers are real headache, anyhow I ll try to get the logs somehow. My doubt is *first* request you mean whether its a very first request fired by the client or the first request which hangs?

What I see from the dump is that the requests get queued and there are
no established connections.
Since it is the first request that triggers the connection opening, I
want to see the debug logs to understand why the connection
establishment failed, and how it failed (as any failure should have
aborted the queued requests).
You make requests by host or by IP address (i.e. is DNS involved)?

--
Simone Bordet
----
http://cometd.org
http://webtide.com
Developer advice, training, services and support
from the Jetty & CometD experts.
_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users


--
With Regards,
Santhosh Kumar J

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Jetty http2 client request hangs indefinitely

Santhosh Kumar
>Since it is the first request that triggers the connection opening, I
want to see the debug logs to understand why the connection
establishment failed

I m trying to get the debug logs but I face a problem where debug logs are always redirected to RootLogger (SLF4j-log4j). How to redirect the debug logs to logs file other than rootLogger. I have also raised a SO question with other details like how I tried( https://stackoverflow.com/questions/58101857/move-jetty-httpclient-debug-logs-from-rootlogger ). Kindly let us know how to handle this case.

My sample is as follows,

private final static Logger logger =  Logger.getRootLogger();

public static void main(String[] args) {
    DailyRollingFileAppender timeAppender = new DailyRollingFileAppender();
    timeAppender.setDatePattern("'.'yyyy-MM-dd-HH-mm");//write at start of every minute
    timeAppender.setName("FileLogger");
    timeAppender.setFile("/path-to-log/client.log");
    timeAppender.setLayout(new PatternLayout("%t %p %d %-5p [%c{1}] %m%n"));
    timeAppender.setThreshold(Level.ALL);
    timeAppender.setAppend(true);
    timeAppender.activateOptions();
    logger.setAdditivity(false);
    logger.addAppender(timeAppender);

    logger.debug("JDK Used-->"+System.getProperty("java.version"));
    //HttpClient code follows
    ......

and my logger files prints,

main DEBUG 2019-09-25 20:26:07,347 DEBUG [JettyJMX] JDK -->1.8.0_212
main DEBUG 2019-09-25 20:26:07,485 DEBUG [log] Logging to org.slf4j.impl.Log4jLoggerAdapter(org.eclipse.jetty.util.log) via org.eclipse.jetty.util.log.Slf4jLog
main INFO 2019-09-25 20:26:07,488 INFO  [log] Logging initialized @370ms to org.eclipse.jetty.util.log.Slf4jLog
main DEBUG 2019-09-25 20:26:07,591 DEBUG [ContainerLifeCycle] HttpClient@65e579dc{STOPPED} added 
....

But I don't want to use rootLogger as other thirdparty packages may use slf4j in my application so I want to redirect jetty debug logs to a separate file. So, instead of using RootLogger I tried using class logger like

final static Logger logger =  Logger.getLogger(MyClass.class.getName());

which only prints,

main DEBUG 2019-09-25 20:49:03,676 DEBUG [JettyJMX] JDK -->1.8.0_212 

SLF4j Jars Used: log4j-1.2.15.jar, slf4j-api-1.7.25.jar, slf4j-log4j12-1.7.25.jar

Jetty Version Used- 9.4.19

Jdk Used: 1.8

I can get debug logs only if I use rootLogger, is there a way or configuration to redirect debug logs from RootLogger to any other loggers?

Thanks In Advance




On Mon, Sep 23, 2019 at 11:41 AM Santhosh Kumar <[hidden email]> wrote:
>What I see from the dump is that the requests get queued and there are
no established connections.
Since it is the first request that triggers the connection opening, I
want to see the debug logs to understand why the connection
establishment failed, and how it failed (as any failure should have
aborted the queued requests).

The remote server is always accessible(verified by browser, curl, wget or even another jetty standalone program)  when threads are hung.

>You make requests by host or by IP address (i.e. is DNS involved)?

We make requests by IP address.

I ll try to get the debug logs asap

Thanks :)





On Fri, Sep 20, 2019 at 7:54 PM Simone Bordet <[hidden email]> wrote:
Hi,

On Fri, Sep 20, 2019 at 2:09 PM Santhosh Kumar <[hidden email]> wrote:
> Enabling Debug logs in production servers are real headache, anyhow I ll try to get the logs somehow. My doubt is *first* request you mean whether its a very first request fired by the client or the first request which hangs?

What I see from the dump is that the requests get queued and there are
no established connections.
Since it is the first request that triggers the connection opening, I
want to see the debug logs to understand why the connection
establishment failed, and how it failed (as any failure should have
aborted the queued requests).
You make requests by host or by IP address (i.e. is DNS involved)?

--
Simone Bordet
----
http://cometd.org
http://webtide.com
Developer advice, training, services and support
from the Jetty & CometD experts.
_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users


--
With Regards,
Santhosh Kumar J


--
With Regards,
Santhosh Kumar J

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Jetty http2 client request hangs indefinitely

Joakim Erdfelt-8
Loggers are hierarchical.

When you declare a logger for ...

final static Logger logger =  Logger.getLogger(com.acme.project.MyClass.class.getName());

You are asking for a logger on ...
  • / (root)
    • com
      • acme
        • project
          • MyClass <-- this leaf
          • MyOtherClass
          • utils
            • UtilityLib
        • common
          • CoreBehaviors

The logging configuration you give is for that specific logger and all children.

So if you obtain a logger at say "com.acme" via Logger.getLogger("com.acme")
and then setup an appender on that logger then all events that belong within "com.acme.*" get that configuration.
Such as from com.acme.common.CoreBehaviors, and com.acme.project.MyOtherClass, and com.acme.project.utils.UtilityLib, etc...
But not from org.eclipse.jetty.*

You can put the logging configuration anywhere in this hierarchy that best suits you.

log4j is quite capable, but doesn't have some advanced features that other slf4j implementation offer.
Such as SiftingAppenders (which will pick details from the logging event, its NDC, and its MDC to route logging events).
Or Discriminators that allow you to associate Appenders to details in the logging event (or its NDC, or its MDC).

Joakim Erdfelt / [hidden email]


On Thu, Sep 26, 2019 at 1:10 AM Santhosh Kumar <[hidden email]> wrote:
>Since it is the first request that triggers the connection opening, I
want to see the debug logs to understand why the connection
establishment failed

I m trying to get the debug logs but I face a problem where debug logs are always redirected to RootLogger (SLF4j-log4j). How to redirect the debug logs to logs file other than rootLogger. I have also raised a SO question with other details like how I tried( https://stackoverflow.com/questions/58101857/move-jetty-httpclient-debug-logs-from-rootlogger ). Kindly let us know how to handle this case.

My sample is as follows,

private final static Logger logger =  Logger.getRootLogger();

public static void main(String[] args) {
    DailyRollingFileAppender timeAppender = new DailyRollingFileAppender();
    timeAppender.setDatePattern("'.'yyyy-MM-dd-HH-mm");//write at start of every minute
    timeAppender.setName("FileLogger");
    timeAppender.setFile("/path-to-log/client.log");
    timeAppender.setLayout(new PatternLayout("%t %p %d %-5p [%c{1}] %m%n"));
    timeAppender.setThreshold(Level.ALL);
    timeAppender.setAppend(true);
    timeAppender.activateOptions();
    logger.setAdditivity(false);
    logger.addAppender(timeAppender);

    logger.debug("JDK Used-->"+System.getProperty("java.version"));
    //HttpClient code follows
    ......

and my logger files prints,

main DEBUG 2019-09-25 20:26:07,347 DEBUG [JettyJMX] JDK -->1.8.0_212
main DEBUG 2019-09-25 20:26:07,485 DEBUG [log] Logging to org.slf4j.impl.Log4jLoggerAdapter(org.eclipse.jetty.util.log) via org.eclipse.jetty.util.log.Slf4jLog
main INFO 2019-09-25 20:26:07,488 INFO  [log] Logging initialized @370ms to org.eclipse.jetty.util.log.Slf4jLog
main DEBUG 2019-09-25 20:26:07,591 DEBUG [ContainerLifeCycle] HttpClient@65e579dc{STOPPED} added 
....

But I don't want to use rootLogger as other thirdparty packages may use slf4j in my application so I want to redirect jetty debug logs to a separate file. So, instead of using RootLogger I tried using class logger like

final static Logger logger =  Logger.getLogger(MyClass.class.getName());

which only prints,

main DEBUG 2019-09-25 20:49:03,676 DEBUG [JettyJMX] JDK -->1.8.0_212 

SLF4j Jars Used: log4j-1.2.15.jar, slf4j-api-1.7.25.jar, slf4j-log4j12-1.7.25.jar

Jetty Version Used- 9.4.19

Jdk Used: 1.8

I can get debug logs only if I use rootLogger, is there a way or configuration to redirect debug logs from RootLogger to any other loggers?

Thanks In Advance




On Mon, Sep 23, 2019 at 11:41 AM Santhosh Kumar <[hidden email]> wrote:
>What I see from the dump is that the requests get queued and there are
no established connections.
Since it is the first request that triggers the connection opening, I
want to see the debug logs to understand why the connection
establishment failed, and how it failed (as any failure should have
aborted the queued requests).

The remote server is always accessible(verified by browser, curl, wget or even another jetty standalone program)  when threads are hung.

>You make requests by host or by IP address (i.e. is DNS involved)?

We make requests by IP address.

I ll try to get the debug logs asap

Thanks :)





On Fri, Sep 20, 2019 at 7:54 PM Simone Bordet <[hidden email]> wrote:
Hi,

On Fri, Sep 20, 2019 at 2:09 PM Santhosh Kumar <[hidden email]> wrote:
> Enabling Debug logs in production servers are real headache, anyhow I ll try to get the logs somehow. My doubt is *first* request you mean whether its a very first request fired by the client or the first request which hangs?

What I see from the dump is that the requests get queued and there are
no established connections.
Since it is the first request that triggers the connection opening, I
want to see the debug logs to understand why the connection
establishment failed, and how it failed (as any failure should have
aborted the queued requests).
You make requests by host or by IP address (i.e. is DNS involved)?

--
Simone Bordet
----
http://cometd.org
http://webtide.com
Developer advice, training, services and support
from the Jetty & CometD experts.
_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users


--
With Regards,
Santhosh Kumar J


--
With Regards,
Santhosh Kumar J
_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Jetty http2 client request hangs indefinitely

Santhosh Kumar
>oggers are hierarchical.

When you declare a logger for ...

final static Logger logger =  Logger.getLogger(com.acme.project.MyClass.class.getName());

You are asking for a logger on ...
  • / (root)
    • com
      • acme
        • project
          • MyClass <-- this leaf
          • MyOtherClass
          • utils
            • UtilityLib
        • common
          • CoreBehaviors

The logging configuration you give is for that specific logger and all children.

So if you obtain a logger at say "com.acme" via Logger.getLogger("com.acme")
and then setup an appender on that logger then all events that belong within "com.acme.*" get that configuration.
Such as from com.acme.common.CoreBehaviors, and com.acme.project.MyOtherClass, and com.acme.project.utils.UtilityLib, etc...
But not from org.eclipse.jetty.*

You can put the logging configuration anywhere in this hierarchy that best suits you.

log4j is quite capable, but doesn't have some advanced features that other slf4j implementation offer.
Such as SiftingAppenders (which will pick details from the logging event, its NDC, and its MDC to route logging events).
Or Discriminators that allow you to associate Appenders to details in the logging event (or its NDC, or its MDC).

Thanks very much for the explanation Joakim, now I understood how it works :)

On Thu, Sep 26, 2019 at 11:53 PM Joakim Erdfelt <[hidden email]> wrote:
Loggers are hierarchical.

When you declare a logger for ...

final static Logger logger =  Logger.getLogger(com.acme.project.MyClass.class.getName());

You are asking for a logger on ...
  • / (root)
    • com
      • acme
        • project
          • MyClass <-- this leaf
          • MyOtherClass
          • utils
            • UtilityLib
        • common
          • CoreBehaviors

The logging configuration you give is for that specific logger and all children.

So if you obtain a logger at say "com.acme" via Logger.getLogger("com.acme")
and then setup an appender on that logger then all events that belong within "com.acme.*" get that configuration.
Such as from com.acme.common.CoreBehaviors, and com.acme.project.MyOtherClass, and com.acme.project.utils.UtilityLib, etc...
But not from org.eclipse.jetty.*

You can put the logging configuration anywhere in this hierarchy that best suits you.

log4j is quite capable, but doesn't have some advanced features that other slf4j implementation offer.
Such as SiftingAppenders (which will pick details from the logging event, its NDC, and its MDC to route logging events).
Or Discriminators that allow you to associate Appenders to details in the logging event (or its NDC, or its MDC).

Joakim Erdfelt / [hidden email]


On Thu, Sep 26, 2019 at 1:10 AM Santhosh Kumar <[hidden email]> wrote:
>Since it is the first request that triggers the connection opening, I
want to see the debug logs to understand why the connection
establishment failed

I m trying to get the debug logs but I face a problem where debug logs are always redirected to RootLogger (SLF4j-log4j). How to redirect the debug logs to logs file other than rootLogger. I have also raised a SO question with other details like how I tried( https://stackoverflow.com/questions/58101857/move-jetty-httpclient-debug-logs-from-rootlogger ). Kindly let us know how to handle this case.

My sample is as follows,

private final static Logger logger =  Logger.getRootLogger();

public static void main(String[] args) {
    DailyRollingFileAppender timeAppender = new DailyRollingFileAppender();
    timeAppender.setDatePattern("'.'yyyy-MM-dd-HH-mm");//write at start of every minute
    timeAppender.setName("FileLogger");
    timeAppender.setFile("/path-to-log/client.log");
    timeAppender.setLayout(new PatternLayout("%t %p %d %-5p [%c{1}] %m%n"));
    timeAppender.setThreshold(Level.ALL);
    timeAppender.setAppend(true);
    timeAppender.activateOptions();
    logger.setAdditivity(false);
    logger.addAppender(timeAppender);

    logger.debug("JDK Used-->"+System.getProperty("java.version"));
    //HttpClient code follows
    ......

and my logger files prints,

main DEBUG 2019-09-25 20:26:07,347 DEBUG [JettyJMX] JDK -->1.8.0_212
main DEBUG 2019-09-25 20:26:07,485 DEBUG [log] Logging to org.slf4j.impl.Log4jLoggerAdapter(org.eclipse.jetty.util.log) via org.eclipse.jetty.util.log.Slf4jLog
main INFO 2019-09-25 20:26:07,488 INFO  [log] Logging initialized @370ms to org.eclipse.jetty.util.log.Slf4jLog
main DEBUG 2019-09-25 20:26:07,591 DEBUG [ContainerLifeCycle] HttpClient@65e579dc{STOPPED} added 
....

But I don't want to use rootLogger as other thirdparty packages may use slf4j in my application so I want to redirect jetty debug logs to a separate file. So, instead of using RootLogger I tried using class logger like

final static Logger logger =  Logger.getLogger(MyClass.class.getName());

which only prints,

main DEBUG 2019-09-25 20:49:03,676 DEBUG [JettyJMX] JDK -->1.8.0_212 

SLF4j Jars Used: log4j-1.2.15.jar, slf4j-api-1.7.25.jar, slf4j-log4j12-1.7.25.jar

Jetty Version Used- 9.4.19

Jdk Used: 1.8

I can get debug logs only if I use rootLogger, is there a way or configuration to redirect debug logs from RootLogger to any other loggers?

Thanks In Advance




On Mon, Sep 23, 2019 at 11:41 AM Santhosh Kumar <[hidden email]> wrote:
>What I see from the dump is that the requests get queued and there are
no established connections.
Since it is the first request that triggers the connection opening, I
want to see the debug logs to understand why the connection
establishment failed, and how it failed (as any failure should have
aborted the queued requests).

The remote server is always accessible(verified by browser, curl, wget or even another jetty standalone program)  when threads are hung.

>You make requests by host or by IP address (i.e. is DNS involved)?

We make requests by IP address.

I ll try to get the debug logs asap

Thanks :)





On Fri, Sep 20, 2019 at 7:54 PM Simone Bordet <[hidden email]> wrote:
Hi,

On Fri, Sep 20, 2019 at 2:09 PM Santhosh Kumar <[hidden email]> wrote:
> Enabling Debug logs in production servers are real headache, anyhow I ll try to get the logs somehow. My doubt is *first* request you mean whether its a very first request fired by the client or the first request which hangs?

What I see from the dump is that the requests get queued and there are
no established connections.
Since it is the first request that triggers the connection opening, I
want to see the debug logs to understand why the connection
establishment failed, and how it failed (as any failure should have
aborted the queued requests).
You make requests by host or by IP address (i.e. is DNS involved)?

--
Simone Bordet
----
http://cometd.org
http://webtide.com
Developer advice, training, services and support
from the Jetty & CometD experts.
_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users


--
With Regards,
Santhosh Kumar J


--
With Regards,
Santhosh Kumar J
_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users
_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users


--
With Regards,
Santhosh Kumar J

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Jetty http2 client request hangs indefinitely

Santhosh Kumar
Hi, 

   I have reproduced the issue and have the debug logs and client dump

 Is there a separate channel or private window where I can share the logs? (I don't want to share it publicly).

On Fri, Sep 27, 2019 at 11:48 AM Santhosh Kumar <[hidden email]> wrote:
>oggers are hierarchical.

When you declare a logger for ...

final static Logger logger =  Logger.getLogger(com.acme.project.MyClass.class.getName());

You are asking for a logger on ...
  • / (root)
    • com
      • acme
        • project
          • MyClass <-- this leaf
          • MyOtherClass
          • utils
            • UtilityLib
        • common
          • CoreBehaviors

The logging configuration you give is for that specific logger and all children.

So if you obtain a logger at say "com.acme" via Logger.getLogger("com.acme")
and then setup an appender on that logger then all events that belong within "com.acme.*" get that configuration.
Such as from com.acme.common.CoreBehaviors, and com.acme.project.MyOtherClass, and com.acme.project.utils.UtilityLib, etc...
But not from org.eclipse.jetty.*

You can put the logging configuration anywhere in this hierarchy that best suits you.

log4j is quite capable, but doesn't have some advanced features that other slf4j implementation offer.
Such as SiftingAppenders (which will pick details from the logging event, its NDC, and its MDC to route logging events).
Or Discriminators that allow you to associate Appenders to details in the logging event (or its NDC, or its MDC).

Thanks very much for the explanation Joakim, now I understood how it works :)

On Thu, Sep 26, 2019 at 11:53 PM Joakim Erdfelt <[hidden email]> wrote:
Loggers are hierarchical.

When you declare a logger for ...

final static Logger logger =  Logger.getLogger(com.acme.project.MyClass.class.getName());

You are asking for a logger on ...
  • / (root)
    • com
      • acme
        • project
          • MyClass <-- this leaf
          • MyOtherClass
          • utils
            • UtilityLib
        • common
          • CoreBehaviors

The logging configuration you give is for that specific logger and all children.

So if you obtain a logger at say "com.acme" via Logger.getLogger("com.acme")
and then setup an appender on that logger then all events that belong within "com.acme.*" get that configuration.
Such as from com.acme.common.CoreBehaviors, and com.acme.project.MyOtherClass, and com.acme.project.utils.UtilityLib, etc...
But not from org.eclipse.jetty.*

You can put the logging configuration anywhere in this hierarchy that best suits you.

log4j is quite capable, but doesn't have some advanced features that other slf4j implementation offer.
Such as SiftingAppenders (which will pick details from the logging event, its NDC, and its MDC to route logging events).
Or Discriminators that allow you to associate Appenders to details in the logging event (or its NDC, or its MDC).

Joakim Erdfelt / [hidden email]


On Thu, Sep 26, 2019 at 1:10 AM Santhosh Kumar <[hidden email]> wrote:
>Since it is the first request that triggers the connection opening, I
want to see the debug logs to understand why the connection
establishment failed

I m trying to get the debug logs but I face a problem where debug logs are always redirected to RootLogger (SLF4j-log4j). How to redirect the debug logs to logs file other than rootLogger. I have also raised a SO question with other details like how I tried( https://stackoverflow.com/questions/58101857/move-jetty-httpclient-debug-logs-from-rootlogger ). Kindly let us know how to handle this case.

My sample is as follows,

private final static Logger logger =  Logger.getRootLogger();

public static void main(String[] args) {
    DailyRollingFileAppender timeAppender = new DailyRollingFileAppender();
    timeAppender.setDatePattern("'.'yyyy-MM-dd-HH-mm");//write at start of every minute
    timeAppender.setName("FileLogger");
    timeAppender.setFile("/path-to-log/client.log");
    timeAppender.setLayout(new PatternLayout("%t %p %d %-5p [%c{1}] %m%n"));
    timeAppender.setThreshold(Level.ALL);
    timeAppender.setAppend(true);
    timeAppender.activateOptions();
    logger.setAdditivity(false);
    logger.addAppender(timeAppender);

    logger.debug("JDK Used-->"+System.getProperty("java.version"));
    //HttpClient code follows
    ......

and my logger files prints,

main DEBUG 2019-09-25 20:26:07,347 DEBUG [JettyJMX] JDK -->1.8.0_212
main DEBUG 2019-09-25 20:26:07,485 DEBUG [log] Logging to org.slf4j.impl.Log4jLoggerAdapter(org.eclipse.jetty.util.log) via org.eclipse.jetty.util.log.Slf4jLog
main INFO 2019-09-25 20:26:07,488 INFO  [log] Logging initialized @370ms to org.eclipse.jetty.util.log.Slf4jLog
main DEBUG 2019-09-25 20:26:07,591 DEBUG [ContainerLifeCycle] HttpClient@65e579dc{STOPPED} added 
....

But I don't want to use rootLogger as other thirdparty packages may use slf4j in my application so I want to redirect jetty debug logs to a separate file. So, instead of using RootLogger I tried using class logger like

final static Logger logger =  Logger.getLogger(MyClass.class.getName());

which only prints,

main DEBUG 2019-09-25 20:49:03,676 DEBUG [JettyJMX] JDK -->1.8.0_212 

SLF4j Jars Used: log4j-1.2.15.jar, slf4j-api-1.7.25.jar, slf4j-log4j12-1.7.25.jar

Jetty Version Used- 9.4.19

Jdk Used: 1.8

I can get debug logs only if I use rootLogger, is there a way or configuration to redirect debug logs from RootLogger to any other loggers?

Thanks In Advance




On Mon, Sep 23, 2019 at 11:41 AM Santhosh Kumar <[hidden email]> wrote:
>What I see from the dump is that the requests get queued and there are
no established connections.
Since it is the first request that triggers the connection opening, I
want to see the debug logs to understand why the connection
establishment failed, and how it failed (as any failure should have
aborted the queued requests).

The remote server is always accessible(verified by browser, curl, wget or even another jetty standalone program)  when threads are hung.

>You make requests by host or by IP address (i.e. is DNS involved)?

We make requests by IP address.

I ll try to get the debug logs asap

Thanks :)





On Fri, Sep 20, 2019 at 7:54 PM Simone Bordet <[hidden email]> wrote:
Hi,

On Fri, Sep 20, 2019 at 2:09 PM Santhosh Kumar <[hidden email]> wrote:
> Enabling Debug logs in production servers are real headache, anyhow I ll try to get the logs somehow. My doubt is *first* request you mean whether its a very first request fired by the client or the first request which hangs?

What I see from the dump is that the requests get queued and there are
no established connections.
Since it is the first request that triggers the connection opening, I
want to see the debug logs to understand why the connection
establishment failed, and how it failed (as any failure should have
aborted the queued requests).
You make requests by host or by IP address (i.e. is DNS involved)?

--
Simone Bordet
----
http://cometd.org
http://webtide.com
Developer advice, training, services and support
from the Jetty & CometD experts.
_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users


--
With Regards,
Santhosh Kumar J


--
With Regards,
Santhosh Kumar J
_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users
_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users


--
With Regards,
Santhosh Kumar J


--
With Regards,
Santhosh Kumar J

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users