HTTP/2 requirements

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

HTTP/2 requirements

Silvio Bierman
Hello all,

We are running Jetty 9.4.14 embedded in an application that runs on a
number of Ubuntu 18.10 servers. Last night we switched from HTTP1.1 only
to HTTP2. Today we are experiencing serious problems. Users can not
access the application or are experiencing extremely long latencies. At
the same time the server is almost idle.
We have configured the Ubuntu servers as almost default setups.
OpenJDK11 with Jetty using pure Java HTTPS. We have been doing this for
years now and each time we switched from HTTP1.1 to HTTP2 things started
falling apart. In the past we had reports of people not being able to
connect at all receiving spurious messages from their browsers (some of
which are undoubtedly outdated). We expected things to have improved by
now and decided to try again, with the above result.

Are there additional things we need to take care of when using HTTP2
when compared to HTTP1.1? More threads, different queue size, more file
handles etc?

Any pointers would be very much appreciated.

Kind regards,

Silvio

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: HTTP/2 requirements

Joakim Erdfelt-8
How do you enable HTTP/2 in your embedded application?

Also, If you have configured your SslContextFactory, what does this configuration look like?

Lastly, what does your ThreadPool / Executor configuration look like?

Joakim Erdfelt / [hidden email]


On Tue, Nov 20, 2018 at 9:01 AM Silvio Bierman <[hidden email]> wrote:
Hello all,

We are running Jetty 9.4.14 embedded in an application that runs on a
number of Ubuntu 18.10 servers. Last night we switched from HTTP1.1 only
to HTTP2. Today we are experiencing serious problems. Users can not
access the application or are experiencing extremely long latencies. At
the same time the server is almost idle.
We have configured the Ubuntu servers as almost default setups.
OpenJDK11 with Jetty using pure Java HTTPS. We have been doing this for
years now and each time we switched from HTTP1.1 to HTTP2 things started
falling apart. In the past we had reports of people not being able to
connect at all receiving spurious messages from their browsers (some of
which are undoubtedly outdated). We expected things to have improved by
now and decided to try again, with the above result.

Are there additional things we need to take care of when using HTTP2
when compared to HTTP1.1? More threads, different queue size, more file
handles etc?

Any pointers would be very much appreciated.

Kind regards,

Silvio

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: HTTP/2 requirements

Joakim Erdfelt-8
Inline ...

On Tue, Nov 20, 2018 at 9:26 AM Silvio Bierman <[hidden email]> wrote:
Hello Joakim,

Firstly, this is the setup (Scala) code for the SslContextFactory etc.:

/***** start code *****/
val sslContextFactory = new SslContextFactory
       
        val ciphers = Array(
            "TLS_AES_128_GCM_SHA256", /* TLS 1.3 */
            "TLS_AES_256_GCM_SHA384", /* TLS 1.3 */
            "TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384",
            "TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256",
            "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384",
            "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256",
            "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384",
            "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
            "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA", /* TLS 1.0 / 1.1 - IE7/8/9 - Android <= 4.3 */
            "TLS_DHE_DSS_WITH_AES_256_GCM_SHA384",
            "TLS_DHE_DSS_WITH_AES_128_GCM_SHA256",
            "TLS_DHE_DSS_WITH_AES_256_CBC_SHA256")
       
        sslContextFactory.setCipherComparator(HTTP2Cipher.COMPARATOR) 
        sslContextFactory.setUseCipherSuitesOrder(true)

Not relevant, anywhere anymore.
You can get an A+ rating on ssllabs without attempting to control cipher suite order.
(Note: A+ rating is for all HTTP/1.x and HTTP/2 behaviors)

 
        sslContextFactory.setIncludeCipherSuites(ciphers:_*)

Don't include, exclude only.
You eliminate a ton of valid Cipher Suites doing it this way.
 
       
        sslContextFactory.setExcludeCipherSuites("SSL_RSA_WITH_DES_CBC_SHA",
            "SSL_DHE_RSA_WITH_DES_CBC_SHA",
            "SSL_DHE_DSS_WITH_DES_CBC_SHA",
            "SSL_RSA_EXPORT_WITH_RC4_40_MD5",
            "SSL_RSA_EXPORT_WITH_DES40_CBC_SHA",
            "SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA",
            "SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA")

Use defaults, none of those Cipher suites would be selected anyway due to default Cipher Suite exclusions in SslContextFactory AND the JVM disabled cipher suites.

    /** Default Excluded Cipher Suite List */
    private static final String[] DEFAULT_EXCLUDED_CIPHER_SUITES = {
            // Exclude weak / insecure ciphers
            "^.*_(MD5|SHA|SHA1)$",
            // Exclude ciphers that don't support forward secrecy
            "^TLS_RSA_.*$",
            // The following exclusions are present to cleanup known bad cipher
            // suites that may be accidentally included via include patterns.
            // The default enabled cipher list in Java will not include these
            // (but they are available in the supported list).
            "^SSL_.*$",
            "^.*_NULL_.*$",
            "^.*_anon_.*$"
    };

 

        sslContextFactory.addExcludeProtocols("SSLv3")

This is the in the default excluded protocols values, and also excluded/disabled at the JVM level, don't set it.

    /** Default Excluded Protocols List */
    private static final String[] DEFAULT_EXCLUDED_PROTOCOLS = {"SSL", "SSLv2", "SSLv2Hello", "SSLv3"};

 
        sslContextFactory.setRenegotiationAllowed(false)
        sslContextFactory.setEndpointIdentificationAlgorithm("HTTPS")

This is only relevant for a client initiating a connection to a destination machine with TLS.

Comment out all of the above.
It screams old-school behaviors and techniques.
Use defaults for SslContextFactory when it comes to Cipher Suites and Protocols, and don't attempt to support TLS/1.0 or TLS/1.1 Those protocols are disabled in so many places (User-Agents, Browsers, OS level, Java VM level, Server level, various hardware intermediates) now its pointless to attempt to support them.   If this is still important to you, make a different connector for legacy behaviors.

       
        sslContextFactory.setKeyStorePath(keyStore)
        sslContextFactory.setKeyStorePassword(keyPassword)
        sslContextFactory.setKeyManagerPassword(keyPassword)
        sslContextFactory.setKeyStoreType(keyStoreType)
       
        val httpsConfig = new HttpConfiguration
        httpsConfig.setSecureScheme("https")
        httpsConfig.setSecurePort(port)
        httpsConfig.setSendXPoweredBy(false)
    httpsConfig.setSendServerVersion(false)
    httpsConfig.setOutputBufferSize(32768)
        httpsConfig.addCustomizer(new SecureRequestCustomizer)
        val http1 = new HttpConnectionFactory(httpsConfig)
        val connector =
        {
            if (httpVersion != "http/2")
            {
                val ssl = new SslConnectionFactory(sslContextFactory,"http/1.1")
                val http = new HttpConnectionFactory(httpsConfig)
                new ServerConnector(server,ssl,http)
            }
            else
            {
                val http2 = new HTTP2ServerConnectionFactory(httpsConfig)
            val alpn = new ALPNServerConnectionFactory
            alpn.setDefaultProtocol(http1.getProtocol)

Use alpn.setDefaultProtocol(http1.getDefaultProtocol())


                val ssl = new SslConnectionFactory(sslContextFactory,alpn.getProtocol)
                new ServerConnector(server,ssl,alpn,http2,http1)
            }
        }
        if (address != null) connector.setHost(address)
        connector.setPort(port)
        connector.setIdleTimeout(1000L * idleTimeout)
        server.addConnector(connector)
/***** end code *****/

Then for the ThreadPool part:

/***** start code *****/
    val pool = new QueuedThreadPool(maxThreads,minThreads,idleTimeout,new ArrayBlockingQueue(queueSize))
    val server = new Server(pool)
/***** end code *****/

HTTP/2 will seriously stress out your thread pool.
Don't specify the queue, use defaults.
Don't use less then defaults for min/max threads with HTTP/2.
It's good that you are using the QueuedThreadPool, it behaves much better then a raw Executor with regards to HTTP/2.
 

Threads are 10-100, idletimeouts I have tried are from 1000 to 20000, queueSize is now 500 but I have tried both much smaller and larger values.

I have no idea what values would be sensible. The number of concurrent users is relatively low: up to a couple of hundred, rarely more. Users can do requests that take longish to complete (sometimes up to 30 or even 60 seconds) but those are very rare. The majority is relatively quick (< 50ms). Current load on the problematic servers is very low (perhaps 10 users) and as said: low memory and CPU close to zero.
No problems with HTTP/1.1 with this code/settings, problematic with HTTP/2. Strangely enough some servers seem fine (or users just did not report any problems yet).

Start by using a default QueuedThreadPool, no configuration, no min thread, no max threads, all strictly defaults.
Then monitor your usages and behaviors with HTTP/2 enabled over at least a week.
Then you can tune your maxThreads.
As to thread idle timeouts, don't bother setting them yet.
Your connector.setIdleTimeout() is far more useful to you.
 

Any help would be welcome.

Kind regards,

Silvio

Many folks have gotten HTTP/2 working just fine in embedded-jetty.
The two most common issues:
1. Not having alpn support setup properly
2. Too restrictive of an SslContextFactory that is incompatible with HTTP/2

When it comes to the ALPN suppport, you have to worry about what JVM runtime you are on first.

If you are using Java 8 then you have to align your alpn-boot version to the specific Java 8 JVM you are using.
Under Java 8, you MUST use the -Xbootclasspath on JVM startup to have alpn work properly.

However, if you are using Java 11 then you can use the jetty-alpn-java-server artifact instead (no bootclasspath requirement) to get ALPN behaviors required for HTTP/2.

You also have a conscrypt option for running ALPN + HTTP/2 on Jetty, which will work the same regardless of Java 8 or Java 11.

When it comes to too restrictive of an SslContextFactory we have the following advice:

* If you are specifying Include Cipher Suites you are too restrictive.
* Your certificates need to be RSA (not DES) and at a higher bit size (like 2048+)
* If you are attempting to get old Protocols (TLS/1.0 TLS/1.1) working, then you are introducing incompatible Cipher Suites too.
* If you are resetting the Exclude Cipher Suites you are introducing incompatible Cipher Suites.
* Avoid using Include Cipher Suites (this exists for people stuck in the past and not ever wanting to use HTTP/2)
* Avoid using Include Protocols (this exists for people stuck supporting old clients and not wanting to use HTTP/2)
* Add to the existing Exclude Cipher Suites (if needed)
* Add to the existing Exclude Protocols (if needed)
* Understand the OpenJDK JVM Crypto Roadmap and its implications - https://java.com/en/jre-jdk-cryptoroadmap.html
* Always use an Up to date (not-expired) JVM
* Learn how to use the SslContextFactory.dump() methods at runtime to know what the current state of your SslContextFactory is (including all selected cipher suites and protocols)
* Test your running server on https://www.ssllabs.com/ssltest/ to see where you are still having configuration issues with SslContextFactory

We have an example codebase that shows how to start an HTTP/2 server connector.

You can check the Jenkins project for their usage of HTTP/2 on Embedded Jetty as well. (their self executing WAR file does just this)

You could also run h2spec against your server and see if its behaving properly.

- Joakim

 


On 20-11-18 16:06, Joakim Erdfelt wrote:
How do you enable HTTP/2 in your embedded application?

Also, If you have configured your SslContextFactory, what does this configuration look like?

Lastly, what does your ThreadPool / Executor configuration look like?

Joakim Erdfelt / [hidden email]


On Tue, Nov 20, 2018 at 9:01 AM Silvio Bierman <[hidden email]> wrote:
Hello all,

We are running Jetty 9.4.14 embedded in an application that runs on a
number of Ubuntu 18.10 servers. Last night we switched from HTTP1.1 only
to HTTP2. Today we are experiencing serious problems. Users can not
access the application or are experiencing extremely long latencies. At
the same time the server is almost idle.
We have configured the Ubuntu servers as almost default setups.
OpenJDK11 with Jetty using pure Java HTTPS. We have been doing this for
years now and each time we switched from HTTP1.1 to HTTP2 things started
falling apart. In the past we had reports of people not being able to
connect at all receiving spurious messages from their browsers (some of
which are undoubtedly outdated). We expected things to have improved by
now and decided to try again, with the above result.

Are there additional things we need to take care of when using HTTP2
when compared to HTTP1.1? More threads, different queue size, more file
handles etc?

Any pointers would be very much appreciated.

Kind regards,

Silvio

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users


_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: HTTP/2 requirements

Silvio Bierman
Thank you Joachim,

I will try implementing your suggestions and see where that takes me.

The code has been largely in place for some years now using an older Jetty on JDK8 (with bootclasspath etc) up until Jetty 9.4.14 on JDK11 which I am running now. I try to support as up-to-date protocols and ciphers as possible but I am also required (by contract) to support IE9+. The current ssllabs score is A.

I switched to this specific set of included suites (suggested at that time in some post by someone who knows more about the subject than I do) when failing to achieve an acceptable score on  JDK8. The only thing I changed afterwards was a half-hearted attempt to support TLS1.3 on JDK11, which I did not succeed in yet. I put in the setEndpointIdentificationAlgorithm because I got a startup warning about not calling it in 9.4.14.

Thanks again for clearing most of that up. I will follow your suggestions and update the code to match what current Jetty and JDK11 can do for me and try do see what I need to do about IE9+/TLS1.0.

Kind regards,

Silvio


On 20-11-18 19:26, Joakim Erdfelt wrote:
Inline ...

On Tue, Nov 20, 2018 at 9:26 AM Silvio Bierman <[hidden email]> wrote:
Hello Joakim,

Firstly, this is the setup (Scala) code for the SslContextFactory etc.:

/***** start code *****/
val sslContextFactory = new SslContextFactory
       
        val ciphers = Array(
            "TLS_AES_128_GCM_SHA256", /* TLS 1.3 */
            "TLS_AES_256_GCM_SHA384", /* TLS 1.3 */
            "TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384",
            "TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256",
            "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384",
            "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256",
            "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384",
            "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
            "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA", /* TLS 1.0 / 1.1 - IE7/8/9 - Android <= 4.3 */
            "TLS_DHE_DSS_WITH_AES_256_GCM_SHA384",
            "TLS_DHE_DSS_WITH_AES_128_GCM_SHA256",
            "TLS_DHE_DSS_WITH_AES_256_CBC_SHA256")
       
        sslContextFactory.setCipherComparator(HTTP2Cipher.COMPARATOR) 
        sslContextFactory.setUseCipherSuitesOrder(true)

Not relevant, anywhere anymore.
You can get an A+ rating on ssllabs without attempting to control cipher suite order.
(Note: A+ rating is for all HTTP/1.x and HTTP/2 behaviors)

 
        sslContextFactory.setIncludeCipherSuites(ciphers:_*)

Don't include, exclude only.
You eliminate a ton of valid Cipher Suites doing it this way.
 
       
        sslContextFactory.setExcludeCipherSuites("SSL_RSA_WITH_DES_CBC_SHA",
            "SSL_DHE_RSA_WITH_DES_CBC_SHA",
            "SSL_DHE_DSS_WITH_DES_CBC_SHA",
            "SSL_RSA_EXPORT_WITH_RC4_40_MD5",
            "SSL_RSA_EXPORT_WITH_DES40_CBC_SHA",
            "SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA",
            "SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA")

Use defaults, none of those Cipher suites would be selected anyway due to default Cipher Suite exclusions in SslContextFactory AND the JVM disabled cipher suites.

    /** Default Excluded Cipher Suite List */
    private static final String[] DEFAULT_EXCLUDED_CIPHER_SUITES = {
            // Exclude weak / insecure ciphers
            "^.*_(MD5|SHA|SHA1)$",
            // Exclude ciphers that don't support forward secrecy
            "^TLS_RSA_.*$",
            // The following exclusions are present to cleanup known bad cipher
            // suites that may be accidentally included via include patterns.
            // The default enabled cipher list in Java will not include these
            // (but they are available in the supported list).
            "^SSL_.*$",
            "^.*_NULL_.*$",
            "^.*_anon_.*$"
    };

 

        sslContextFactory.addExcludeProtocols("SSLv3")

This is the in the default excluded protocols values, and also excluded/disabled at the JVM level, don't set it.

    /** Default Excluded Protocols List */
    private static final String[] DEFAULT_EXCLUDED_PROTOCOLS = {"SSL", "SSLv2", "SSLv2Hello", "SSLv3"};

 
        sslContextFactory.setRenegotiationAllowed(false)
        sslContextFactory.setEndpointIdentificationAlgorithm("HTTPS")

This is only relevant for a client initiating a connection to a destination machine with TLS.

Comment out all of the above.
It screams old-school behaviors and techniques.
Use defaults for SslContextFactory when it comes to Cipher Suites and Protocols, and don't attempt to support TLS/1.0 or TLS/1.1 Those protocols are disabled in so many places (User-Agents, Browsers, OS level, Java VM level, Server level, various hardware intermediates) now its pointless to attempt to support them.   If this is still important to you, make a different connector for legacy behaviors.

       
        sslContextFactory.setKeyStorePath(keyStore)
        sslContextFactory.setKeyStorePassword(keyPassword)
        sslContextFactory.setKeyManagerPassword(keyPassword)
        sslContextFactory.setKeyStoreType(keyStoreType)
       
        val httpsConfig = new HttpConfiguration
        httpsConfig.setSecureScheme("https")
        httpsConfig.setSecurePort(port)
        httpsConfig.setSendXPoweredBy(false)
    httpsConfig.setSendServerVersion(false)
    httpsConfig.setOutputBufferSize(32768)
        httpsConfig.addCustomizer(new SecureRequestCustomizer)
        val http1 = new HttpConnectionFactory(httpsConfig)
        val connector =
        {
            if (httpVersion != "http/2")
            {
                val ssl = new SslConnectionFactory(sslContextFactory,"http/1.1")
                val http = new HttpConnectionFactory(httpsConfig)
                new ServerConnector(server,ssl,http)
            }
            else
            {
                val http2 = new HTTP2ServerConnectionFactory(httpsConfig)
            val alpn = new ALPNServerConnectionFactory
            alpn.setDefaultProtocol(http1.getProtocol)

Use alpn.setDefaultProtocol(http1.getDefaultProtocol())


                val ssl = new SslConnectionFactory(sslContextFactory,alpn.getProtocol)
                new ServerConnector(server,ssl,alpn,http2,http1)
            }
        }
        if (address != null) connector.setHost(address)
        connector.setPort(port)
        connector.setIdleTimeout(1000L * idleTimeout)
        server.addConnector(connector)
/***** end code *****/

Then for the ThreadPool part:

/***** start code *****/
    val pool = new QueuedThreadPool(maxThreads,minThreads,idleTimeout,new ArrayBlockingQueue(queueSize))
    val server = new Server(pool)
/***** end code *****/

HTTP/2 will seriously stress out your thread pool.
Don't specify the queue, use defaults.
Don't use less then defaults for min/max threads with HTTP/2.
It's good that you are using the QueuedThreadPool, it behaves much better then a raw Executor with regards to HTTP/2.
 

Threads are 10-100, idletimeouts I have tried are from 1000 to 20000, queueSize is now 500 but I have tried both much smaller and larger values.

I have no idea what values would be sensible. The number of concurrent users is relatively low: up to a couple of hundred, rarely more. Users can do requests that take longish to complete (sometimes up to 30 or even 60 seconds) but those are very rare. The majority is relatively quick (< 50ms). Current load on the problematic servers is very low (perhaps 10 users) and as said: low memory and CPU close to zero.
No problems with HTTP/1.1 with this code/settings, problematic with HTTP/2. Strangely enough some servers seem fine (or users just did not report any problems yet).

Start by using a default QueuedThreadPool, no configuration, no min thread, no max threads, all strictly defaults.
Then monitor your usages and behaviors with HTTP/2 enabled over at least a week.
Then you can tune your maxThreads.
As to thread idle timeouts, don't bother setting them yet.
Your connector.setIdleTimeout() is far more useful to you.
 

Any help would be welcome.

Kind regards,

Silvio

Many folks have gotten HTTP/2 working just fine in embedded-jetty.
The two most common issues:
1. Not having alpn support setup properly
2. Too restrictive of an SslContextFactory that is incompatible with HTTP/2

When it comes to the ALPN suppport, you have to worry about what JVM runtime you are on first.

If you are using Java 8 then you have to align your alpn-boot version to the specific Java 8 JVM you are using.
Under Java 8, you MUST use the -Xbootclasspath on JVM startup to have alpn work properly.

However, if you are using Java 11 then you can use the jetty-alpn-java-server artifact instead (no bootclasspath requirement) to get ALPN behaviors required for HTTP/2.

You also have a conscrypt option for running ALPN + HTTP/2 on Jetty, which will work the same regardless of Java 8 or Java 11.

When it comes to too restrictive of an SslContextFactory we have the following advice:

* If you are specifying Include Cipher Suites you are too restrictive.
* Your certificates need to be RSA (not DES) and at a higher bit size (like 2048+)
* If you are attempting to get old Protocols (TLS/1.0 TLS/1.1) working, then you are introducing incompatible Cipher Suites too.
* If you are resetting the Exclude Cipher Suites you are introducing incompatible Cipher Suites.
* Avoid using Include Cipher Suites (this exists for people stuck in the past and not ever wanting to use HTTP/2)
* Avoid using Include Protocols (this exists for people stuck supporting old clients and not wanting to use HTTP/2)
* Add to the existing Exclude Cipher Suites (if needed)
* Add to the existing Exclude Protocols (if needed)
* Understand the OpenJDK JVM Crypto Roadmap and its implications - https://java.com/en/jre-jdk-cryptoroadmap.html
* Always use an Up to date (not-expired) JVM
* Learn how to use the SslContextFactory.dump() methods at runtime to know what the current state of your SslContextFactory is (including all selected cipher suites and protocols)
* Test your running server on https://www.ssllabs.com/ssltest/ to see where you are still having configuration issues with SslContextFactory

We have an example codebase that shows how to start an HTTP/2 server connector.

You can check the Jenkins project for their usage of HTTP/2 on Embedded Jetty as well. (their self executing WAR file does just this)

You could also run h2spec against your server and see if its behaving properly.

- Joakim

 


On 20-11-18 16:06, Joakim Erdfelt wrote:
How do you enable HTTP/2 in your embedded application?

Also, If you have configured your SslContextFactory, what does this configuration look like?

Lastly, what does your ThreadPool / Executor configuration look like?

Joakim Erdfelt / [hidden email]


On Tue, Nov 20, 2018 at 9:01 AM Silvio Bierman <[hidden email]> wrote:
Hello all,

We are running Jetty 9.4.14 embedded in an application that runs on a
number of Ubuntu 18.10 servers. Last night we switched from HTTP1.1 only
to HTTP2. Today we are experiencing serious problems. Users can not
access the application or are experiencing extremely long latencies. At
the same time the server is almost idle.
We have configured the Ubuntu servers as almost default setups.
OpenJDK11 with Jetty using pure Java HTTPS. We have been doing this for
years now and each time we switched from HTTP1.1 to HTTP2 things started
falling apart. In the past we had reports of people not being able to
connect at all receiving spurious messages from their browsers (some of
which are undoubtedly outdated). We expected things to have improved by
now and decided to try again, with the above result.

Are there additional things we need to take care of when using HTTP2
when compared to HTTP1.1? More threads, different queue size, more file
handles etc?

Any pointers would be very much appreciated.

Kind regards,

Silvio

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users


_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users


_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: HTTP/2 requirements

Silvio Bierman
In reply to this post by Joakim Erdfelt-8
One additional question: do you think it is worthwhile to support HTTP/2 for newer browsers (which most of our users have) while at the same time trying to uphold support for IE9?

Cheers,

Silvio


On 20-11-18 19:26, Joakim Erdfelt wrote:
Inline ...

On Tue, Nov 20, 2018 at 9:26 AM Silvio Bierman <[hidden email]> wrote:
Hello Joakim,

Firstly, this is the setup (Scala) code for the SslContextFactory etc.:

/***** start code *****/
val sslContextFactory = new SslContextFactory
       
        val ciphers = Array(
            "TLS_AES_128_GCM_SHA256", /* TLS 1.3 */
            "TLS_AES_256_GCM_SHA384", /* TLS 1.3 */
            "TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384",
            "TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256",
            "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384",
            "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256",
            "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384",
            "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
            "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA", /* TLS 1.0 / 1.1 - IE7/8/9 - Android <= 4.3 */
            "TLS_DHE_DSS_WITH_AES_256_GCM_SHA384",
            "TLS_DHE_DSS_WITH_AES_128_GCM_SHA256",
            "TLS_DHE_DSS_WITH_AES_256_CBC_SHA256")
       
        sslContextFactory.setCipherComparator(HTTP2Cipher.COMPARATOR) 
        sslContextFactory.setUseCipherSuitesOrder(true)

Not relevant, anywhere anymore.
You can get an A+ rating on ssllabs without attempting to control cipher suite order.
(Note: A+ rating is for all HTTP/1.x and HTTP/2 behaviors)

 
        sslContextFactory.setIncludeCipherSuites(ciphers:_*)

Don't include, exclude only.
You eliminate a ton of valid Cipher Suites doing it this way.
 
       
        sslContextFactory.setExcludeCipherSuites("SSL_RSA_WITH_DES_CBC_SHA",
            "SSL_DHE_RSA_WITH_DES_CBC_SHA",
            "SSL_DHE_DSS_WITH_DES_CBC_SHA",
            "SSL_RSA_EXPORT_WITH_RC4_40_MD5",
            "SSL_RSA_EXPORT_WITH_DES40_CBC_SHA",
            "SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA",
            "SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA")

Use defaults, none of those Cipher suites would be selected anyway due to default Cipher Suite exclusions in SslContextFactory AND the JVM disabled cipher suites.

    /** Default Excluded Cipher Suite List */
    private static final String[] DEFAULT_EXCLUDED_CIPHER_SUITES = {
            // Exclude weak / insecure ciphers
            "^.*_(MD5|SHA|SHA1)$",
            // Exclude ciphers that don't support forward secrecy
            "^TLS_RSA_.*$",
            // The following exclusions are present to cleanup known bad cipher
            // suites that may be accidentally included via include patterns.
            // The default enabled cipher list in Java will not include these
            // (but they are available in the supported list).
            "^SSL_.*$",
            "^.*_NULL_.*$",
            "^.*_anon_.*$"
    };

 

        sslContextFactory.addExcludeProtocols("SSLv3")

This is the in the default excluded protocols values, and also excluded/disabled at the JVM level, don't set it.

    /** Default Excluded Protocols List */
    private static final String[] DEFAULT_EXCLUDED_PROTOCOLS = {"SSL", "SSLv2", "SSLv2Hello", "SSLv3"};

 
        sslContextFactory.setRenegotiationAllowed(false)
        sslContextFactory.setEndpointIdentificationAlgorithm("HTTPS")

This is only relevant for a client initiating a connection to a destination machine with TLS.

Comment out all of the above.
It screams old-school behaviors and techniques.
Use defaults for SslContextFactory when it comes to Cipher Suites and Protocols, and don't attempt to support TLS/1.0 or TLS/1.1 Those protocols are disabled in so many places (User-Agents, Browsers, OS level, Java VM level, Server level, various hardware intermediates) now its pointless to attempt to support them.   If this is still important to you, make a different connector for legacy behaviors.

       
        sslContextFactory.setKeyStorePath(keyStore)
        sslContextFactory.setKeyStorePassword(keyPassword)
        sslContextFactory.setKeyManagerPassword(keyPassword)
        sslContextFactory.setKeyStoreType(keyStoreType)
       
        val httpsConfig = new HttpConfiguration
        httpsConfig.setSecureScheme("https")
        httpsConfig.setSecurePort(port)
        httpsConfig.setSendXPoweredBy(false)
    httpsConfig.setSendServerVersion(false)
    httpsConfig.setOutputBufferSize(32768)
        httpsConfig.addCustomizer(new SecureRequestCustomizer)
        val http1 = new HttpConnectionFactory(httpsConfig)
        val connector =
        {
            if (httpVersion != "http/2")
            {
                val ssl = new SslConnectionFactory(sslContextFactory,"http/1.1")
                val http = new HttpConnectionFactory(httpsConfig)
                new ServerConnector(server,ssl,http)
            }
            else
            {
                val http2 = new HTTP2ServerConnectionFactory(httpsConfig)
            val alpn = new ALPNServerConnectionFactory
            alpn.setDefaultProtocol(http1.getProtocol)

Use alpn.setDefaultProtocol(http1.getDefaultProtocol())


                val ssl = new SslConnectionFactory(sslContextFactory,alpn.getProtocol)
                new ServerConnector(server,ssl,alpn,http2,http1)
            }
        }
        if (address != null) connector.setHost(address)
        connector.setPort(port)
        connector.setIdleTimeout(1000L * idleTimeout)
        server.addConnector(connector)
/***** end code *****/

Then for the ThreadPool part:

/***** start code *****/
    val pool = new QueuedThreadPool(maxThreads,minThreads,idleTimeout,new ArrayBlockingQueue(queueSize))
    val server = new Server(pool)
/***** end code *****/

HTTP/2 will seriously stress out your thread pool.
Don't specify the queue, use defaults.
Don't use less then defaults for min/max threads with HTTP/2.
It's good that you are using the QueuedThreadPool, it behaves much better then a raw Executor with regards to HTTP/2.
 

Threads are 10-100, idletimeouts I have tried are from 1000 to 20000, queueSize is now 500 but I have tried both much smaller and larger values.

I have no idea what values would be sensible. The number of concurrent users is relatively low: up to a couple of hundred, rarely more. Users can do requests that take longish to complete (sometimes up to 30 or even 60 seconds) but those are very rare. The majority is relatively quick (< 50ms). Current load on the problematic servers is very low (perhaps 10 users) and as said: low memory and CPU close to zero.
No problems with HTTP/1.1 with this code/settings, problematic with HTTP/2. Strangely enough some servers seem fine (or users just did not report any problems yet).

Start by using a default QueuedThreadPool, no configuration, no min thread, no max threads, all strictly defaults.
Then monitor your usages and behaviors with HTTP/2 enabled over at least a week.
Then you can tune your maxThreads.
As to thread idle timeouts, don't bother setting them yet.
Your connector.setIdleTimeout() is far more useful to you.
 

Any help would be welcome.

Kind regards,

Silvio

Many folks have gotten HTTP/2 working just fine in embedded-jetty.
The two most common issues:
1. Not having alpn support setup properly
2. Too restrictive of an SslContextFactory that is incompatible with HTTP/2

When it comes to the ALPN suppport, you have to worry about what JVM runtime you are on first.

If you are using Java 8 then you have to align your alpn-boot version to the specific Java 8 JVM you are using.
Under Java 8, you MUST use the -Xbootclasspath on JVM startup to have alpn work properly.

However, if you are using Java 11 then you can use the jetty-alpn-java-server artifact instead (no bootclasspath requirement) to get ALPN behaviors required for HTTP/2.

You also have a conscrypt option for running ALPN + HTTP/2 on Jetty, which will work the same regardless of Java 8 or Java 11.

When it comes to too restrictive of an SslContextFactory we have the following advice:

* If you are specifying Include Cipher Suites you are too restrictive.
* Your certificates need to be RSA (not DES) and at a higher bit size (like 2048+)
* If you are attempting to get old Protocols (TLS/1.0 TLS/1.1) working, then you are introducing incompatible Cipher Suites too.
* If you are resetting the Exclude Cipher Suites you are introducing incompatible Cipher Suites.
* Avoid using Include Cipher Suites (this exists for people stuck in the past and not ever wanting to use HTTP/2)
* Avoid using Include Protocols (this exists for people stuck supporting old clients and not wanting to use HTTP/2)
* Add to the existing Exclude Cipher Suites (if needed)
* Add to the existing Exclude Protocols (if needed)
* Understand the OpenJDK JVM Crypto Roadmap and its implications - https://java.com/en/jre-jdk-cryptoroadmap.html
* Always use an Up to date (not-expired) JVM
* Learn how to use the SslContextFactory.dump() methods at runtime to know what the current state of your SslContextFactory is (including all selected cipher suites and protocols)
* Test your running server on https://www.ssllabs.com/ssltest/ to see where you are still having configuration issues with SslContextFactory

We have an example codebase that shows how to start an HTTP/2 server connector.

You can check the Jenkins project for their usage of HTTP/2 on Embedded Jetty as well. (their self executing WAR file does just this)

You could also run h2spec against your server and see if its behaving properly.

- Joakim

 


On 20-11-18 16:06, Joakim Erdfelt wrote:
How do you enable HTTP/2 in your embedded application?

Also, If you have configured your SslContextFactory, what does this configuration look like?

Lastly, what does your ThreadPool / Executor configuration look like?

Joakim Erdfelt / [hidden email]


On Tue, Nov 20, 2018 at 9:01 AM Silvio Bierman <[hidden email]> wrote:
Hello all,

We are running Jetty 9.4.14 embedded in an application that runs on a
number of Ubuntu 18.10 servers. Last night we switched from HTTP1.1 only
to HTTP2. Today we are experiencing serious problems. Users can not
access the application or are experiencing extremely long latencies. At
the same time the server is almost idle.
We have configured the Ubuntu servers as almost default setups.
OpenJDK11 with Jetty using pure Java HTTPS. We have been doing this for
years now and each time we switched from HTTP1.1 to HTTP2 things started
falling apart. In the past we had reports of people not being able to
connect at all receiving spurious messages from their browsers (some of
which are undoubtedly outdated). We expected things to have improved by
now and decided to try again, with the above result.

Are there additional things we need to take care of when using HTTP2
when compared to HTTP1.1? More threads, different queue size, more file
handles etc?

Any pointers would be very much appreciated.

Kind regards,

Silvio

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users


_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users


_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: HTTP/2 requirements

Joakim Erdfelt-8
> One additional question: do you think it is worthwhile to support HTTP/2 for newer browsers (which most of our users have) while at the same time trying to uphold support for IE9?

Last time I checked, not even https://microsoft.com/ supported anything older then MSIE11 (due to them disabling TLS/1.0 and TLS/1.1 on the server side)
Also, the last time MSIE9 had over 1% market share was July 2016 (Dropped below 1% in August 2016, currently under 0.01%)

Joakim Erdfelt / [hidden email]

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: HTTP/2 requirements

Silvio Bierman


On 20-11-18 23:07, Joakim Erdfelt wrote:
> One additional question: do you think it is worthwhile to support HTTP/2 for newer browsers (which most of our users have) while at the same time trying to uphold support for IE9?

Last time I checked, not even https://microsoft.com/ supported anything older then MSIE11 (due to them disabling TLS/1.0 and TLS/1.1 on the server side)
Also, the last time MSIE9 had over 1% market share was July 2016 (Dropped below 1% in August 2016, currently under 0.01%)


Yes, I know. Sad isn't it? Unfortunately we are stuck in this situation because some of our clients run older IE versions inside company networks (I believe they have some form of extended support). Things may change within 4-6 months though.

I tried using only defaults but got an ssllabs B score because of supported weak DH ciphers. I explicitly excluded those and am back on an A score now, only without TLS1.0 support. I am now considering a separate setup with 1.1 support for backward users only. Still no A+ score and no TLS1.3 support listed, but I think I read somewhere that ssllabs does not yet detect TLS1.3 so that may be the culprit.

I will start experimenting with HTTP/2 support with the setup you described and report back later.


Thanks again,

Silvio


_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users