Proxy server doesn't return all headers

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Proxy server doesn't return all headers

Travis Spencer
Hi There,

We are using Jetty 9.4.14.v20181114.

Our app uses a servlet filter to get requests from Jetty and handles
everything else from there. As a result, we use Jetty as a Web server
with only HTTP 1.1 support (no WebSockets, H2, JSP, or anything else).
We don't have any Jetty config files; we configure it all on startup
of our app via the embedded API.

Now, we need to start proxying some HTTP requests. To this end, we are
trying to use Jetty's ProxyServlet. This is challenging because we
don't have any servlets in our app and the servlet request/responses
are pretty obscured from the programming model of our request
handlers. Despite this, we have manged to get the proxy running and
mostly working.

The issue that we're running into, however, is that proxied requests
do not contain all of the headers from the origin server. Some of
them, Content-Type, Content-Length, etc. are not present in the
response that the proxy provides to the HTTP client (curl, Postman,
etc.).

In our request handler, this code does the proxying:

_proxyServlet.service(servletRequest, servletResponse);
servletResponse.flushBuffer();

In this snippet, _proxyServlet is an anonymous subclass of
ProxyServlet.Transparent:

_proxyServlet = new ProxyServlet.Transparent()
{
    protected String filterServerResponseHeader(HttpServletRequest clientRequest, Response serverResponse,
                                                String headerName, String headerValue)
    {
        if (_filteredResponseHeaders.contains(StringUtils.lowerCase(headerName, Locale.US)))
        {
            return null;
        }

        return headerValue;
    }

    @Override
    protected Set<String> findConnectionHeaders(HttpServletRequest clientRequest)
    {
        ImmutableSet.Builder<String> builder = ImmutableSet.builder();
        @Nullable Set<String> connectionHeaders = super.findConnectionHeaders(clientRequest);

        if (connectionHeaders != null)
        {
            builder.addAll(connectionHeaders.stream().filter(StringUtils::isNotEmpty).collect(Collectors.toSet()));
        }

        builder.addAll(_filteredRequestHeaders);

        return builder.build();
    }
};
_proxyServlet.init(new ReverseProxyServletConfig(prefix, proxyTo));

Those filter sets are just these:

Set<String> _filteredResponseHeaders = ImmutableSet.of("server", "date");
Set<String> _filteredRequestHeaders = ImmutableSet.of("cookie", "accept-encoding");

The config of the servlet has some issues ATM, but I don't think
that's the cause of the issue (might be though). Here's how it's being
configured when init is called on the proxy servlet:

private static class ReverseProxyServletConfig implements ServletConfig
{
    private final Map<String, String> _config;
    private final StaticContext _context;

    ReverseProxyServletConfig(String prefix, String proxyTo)
    {
        URI proxyToUri = URI.create(proxyTo);

        _config = ImmutableMap.of(
                "proxyTo", proxyTo,
                "whiteList", String.format("%s:%d", proxyToUri.getHost(), proxyToUri.getPort()),
                "prefix", prefix,
                "maxThreads", "16" // TODO: Remove so executor set below is used instead
        );

        _context = new StaticContext(); // Same as Jetty's except that getContextPath returns "" instead of null, so rewrite works without "null" being prepended
        _context.setAttribute("org.eclipse.jetty.server.Executor", ForkJoinPool.commonPool()); // TODO: Change executor
    }

    @Override
    public String getServletName()
    {
        return ReverseProxyController.class.getName();
    }

    @Override
    public ServletContext getServletContext()
    {
        return _context;
    }

    @Nullable
    @Override
    public String getInitParameter(String configParameterName)
    {
        return _config.get(configParameterName);
    }

    @Override
    public Enumeration<String> getInitParameterNames()
    {
        return Collections.enumeration(_config.keySet());
    }
}

If I make HTTP requests with Postman or curl, I don't get all the
response headers. BUT if I have a debugger attached when
_proxyServlet.service is called, I have seen on some occasions that
the headers will show up! This makes me think that there some async
issues here. When I read the docs, it says about ProxyServlet that
"The request processing is asynchronous, but the I/O is blocking." So,
this made me think that I just needed to wait for request processing
to complete. To do this, I tried to add this right after the call to
_proxyServlet.service:

servletRequest.getAsyncContext().complete();

That didn't work though, and in fact resulted in no body content as
well as no HTTP headers.

I also tried overriding the ProxyServlet class's service method. In my
override, I removed the async processing of the request. This also
resulted in no HTTP headers and no body content.

With the code above, and this request

curl -X POST \
  https://localhost:6749/admin/api/httpbin/post \
  -H 'Content-Type: application/json' \
  -H 'Origin: example.com' \
  -d '{
    "test": 44
}'

I see these messages in the logs:

rewriting: https://localhost:6749/admin/api/httpbin/post -> http://httpbin.org/post

Response headers HttpResponse[HTTP/1.1 200 OK]@613a2308
Connection: keep-alive
Server: gunicorn/19.9.0
Date: Sat, 17 Nov 2018 08:23:40 GMT
Content-Type: application/json
Content-Length: 628
Access-Control-Allow-Origin: example.com
Access-Control-Allow-Credentials: true
Via: 1.1 vegur

.ReverseProxyController:612 67028742 proxying to downstream:
HttpResponse[HTTP/1.1 200 OK]@613a2308
Connection: keep-alive
Server: gunicorn/19.9.0
Date: Sat, 17 Nov 2018 08:23:40 GMT
Content-Type: application/json
Content-Length: 628
Access-Control-Allow-Origin: example.com
Access-Control-Allow-Credentials: true
Via: 1.1 vegur

There's those missing headers! Why aren't those being returned to my
HTTP client? I also see the following log messages after these:

org.eclipse.jetty.client.HttpReceiver:467 Request/Response succeeded: Result[HttpRequest[POST /post HTTP/1.1]@22dbb42a > HttpResponse[HTTP/1.1 200 OK]@613a2308] null
.ReverseProxyController:631 67028742 proxying successful
org.eclipse.jetty.server.HttpChannelState:669 complete HttpChannelState@792941c2{s=ASYNC_WAIT a=STARTED i=false r=IDLE w=false}
org.eclipse.jetty.server.HttpChannel:314 HttpChannelOverHttp@1c3d6aa2{r=4,c=true,a=ASYNC_WOKEN,uri=https://localhost:6749/admin/api/httpbin/post,age=499} handle https://localhost:6749/admin/api/httpbin/post
org.eclipse.jetty.server.HttpChannelState:219 handling HttpChannelState@792941c2{s=ASYNC_WOKEN a=COMPLETE i=false r=IDLE w=false}
org.eclipse.jetty.server.HttpChannel:327 HttpChannelOverHttp@1c3d6aa2{r=4,c=true,a=COMPLETING,uri=https://localhost:6749/admin/api/httpbin/post,age=499} action COMPLETE
org.eclipse.jetty.server.HttpChannelState:860 onComplete HttpChannelState@792941c2{s=COMPLETING a=COMPLETE i=false r=IDLE w=false}
org.eclipse.jetty.server.HttpChannelState:1294 onEof HttpChannelState@792941c2{s=COMPLETED a=NOT_ASYNC i=false r=IDLE w=false}

Shortly after this, I start to see a few of these errors:

org.eclipse.jetty.io.FillInterest:134 onFail FillInterest@54bf29e8{SSLC.NBReadCB@3b86e603{SslConnection@3b86e603::SocketChannelEndPoint@45a8ac84{/0:0:0:0:0:0:0:1:54495<->/0:0:0:0:0:0:0:1:6749,OPEN,fill=FI,flush=-,to=30005/30000}{io=1/1,kio=1,kro=1}->SslConnection@3b86e603{NOT_HANDSHAKING,eio=-1/-1,di=-1,fill=INTERESTED,flush=IDLE}~>DecryptedEndPoint@6bd593aa{/0:0:0:0:0:0:0:1:54495<->/0:0:0:0:0:0:0:1:6749,OPEN,fill=FI,flush=-,to=30006/30000}=>HttpConnection@1d3bb59c[p=HttpParser{s=START,0 of -1},g=HttpGenerator@18d2abc6{s=START}]=>HttpChannelOverHttp@1c3d6aa2{r=4,c=false,a=IDLE,uri=null,age=0}}}
 java.util.concurrent.TimeoutException: Idle timeout expired: 30004/30000 ms
    at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:166) [jetty-io-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.io.IdleTimeout$1.run(IdleTimeout.java:50) [jetty-io-9.4.14.v20181114.jar:9.4.14.v20181114]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_192]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_192]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_192]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_192]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_192]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_192]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_192]

org.eclipse.jetty.io.WriteFlusher:471 ignored: WriteFlusher@2c599ac5{IDLE}->null
 java.nio.channels.ClosedChannelException: null
    at org.eclipse.jetty.io.WriteFlusher.onClose(WriteFlusher.java:502) [jetty-io-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.io.AbstractEndPoint.onClose(AbstractEndPoint.java:353) [jetty-io-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.io.ChannelEndPoint.onClose(ChannelEndPoint.java:216) [jetty-io-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.io.AbstractEndPoint.doOnClose(AbstractEndPoint.java:225) [jetty-io-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.io.AbstractEndPoint.close(AbstractEndPoint.java:192) [jetty-io-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.io.AbstractEndPoint.close(AbstractEndPoint.java:175) [jetty-io-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.client.http.HttpConnectionOverHTTP.close(HttpConnectionOverHTTP.java:195) [jetty-client-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.client.http.HttpConnectionOverHTTP.onIdleExpired(HttpConnectionOverHTTP.java:145) [jetty-client-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.io.AbstractEndPoint.onIdleExpired(AbstractEndPoint.java:401) [jetty-io-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:166) [jetty-io-9.4.14.v20181114.jar:9.4.14.v20181114]
    at org.eclipse.jetty.io.IdleTimeout$1.run(IdleTimeout.java:50) [jetty-io-9.4.14.v20181114.jar:9.4.14.v20181114]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_192]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_192]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_192]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_192]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_192]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_192]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_192]

Does anyone have any tips on what more to try or clues as to what may
be going wrong?

--

Regards,

Travis Spencer

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Proxy server doesn't return all headers

Simone Bordet-3
Hi,

On Sat, Nov 17, 2018 at 10:28 AM Travis Spencer
<[hidden email]> wrote:
> There's those missing headers!

So are they there or not?

> Shortly after this, I start to see a few of these errors:
>  java.util.concurrent.TimeoutException: Idle timeout expired: 30004/30000 ms

These are not errors, just connections that idle timeout.

> Does anyone have any tips on what more to try or clues as to what may
> be going wrong?

I frankly did not understand; you first said the header were not
there, then you said they were, then apparently not again, but you
never mentioned _which_ header was missing.
Can you put up a reproducible test case that shows exactly what you
expect and what you get instead?

--
Simone Bordet
----
http://cometd.org
http://webtide.com
Developer advice, training, services and support
from the Jetty & CometD experts.
_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users
Reply | Threaded
Open this post in threaded view
|

Re: Proxy server doesn't return all headers

Travis Spencer
On Mon, Nov 19, 2018 at 12:14 PM Simone Bordet <[hidden email]> wrote:
Hi,

Hey, Simone.

On Sat, Nov 17, 2018 at 10:28 AM Travis Spencer
<[hidden email]> wrote:
> There's those missing headers!

So are they there or not?

Not.
 
> Shortly after this, I start to see a few of these errors:
>  java.util.concurrent.TimeoutException: Idle timeout expired: 30004/30000 ms

These are not errors, just connections that idle timeout.

That's what I was hoping. Thanks for confirming.
 
> Does anyone have any tips on what more to try or clues as to what may
> be going wrong?

I frankly did not understand; you first said the header were not
there, then you said they were, then apparently not again, but you
never mentioned _which_ header was missing.

Some are there and some are not. All are included only when I have a debugger attached and am slowly going through line by line. The obvious ones that are missing are Content-Type and Content-Length.
 
Can you put up a reproducible test case that shows exactly what you
expect and what you get instead?

I can try, but I think it's something in particular with the proxy server's usage in our product :-/

_______________________________________________
jetty-users mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users