WebSockets behind a proxy

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

WebSockets behind a proxy

Phil B
Hi,

I was taking a look at using WebSockets for an application but it
appears that Aida/Swazoo doesn't know how to deal with WebSockets from
behind a proxy without a bit of work (i.e. where the connection has
been pre-upgraded by Apache.)  I was wondering if there are any
pointers as to how to best handle this scenario?

Thanks,
Phil
_______________________________________________
Aida mailing list
[hidden email]
http://lists.aidaweb.si/mailman/listinfo/aida
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: WebSockets behind a proxy

Janko Mivšek
Phil (list) je 25. 05. 2016 ob 22:29 napisal:

> I was taking a look at using WebSockets for an application but it
> appears that Aida/Swazoo doesn't know how to deal with WebSockets from
> behind a proxy without a bit of work (i.e. where the connection has
> been pre-upgraded by Apache.)  I was wondering if there are any
> pointers as to how to best handle this scenario?

It is supposed that a proxy server deals with proxying WebSocket
requests, AFAIK. So far my real-time apps are on intranets, so I didn't
yet have this case to know more about.

Best regards
Janko


_______________________________________________
Aida mailing list
[hidden email]
http://lists.aidaweb.si/mailman/listinfo/aida
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: WebSockets behind a proxy

Phil B
On Tue, 2016-05-31 at 11:34 +0200, Janko Mivšek wrote:

> Phil (list) je 25. 05. 2016 ob 22:29 napisal:
>
> >
> > I was taking a look at using WebSockets for an application but it
> > appears that Aida/Swazoo doesn't know how to deal with WebSockets
> > from
> > behind a proxy without a bit of work (i.e. where the connection has
> > been pre-upgraded by Apache.)  I was wondering if there are any
> > pointers as to how to best handle this scenario?
> It is supposed that a proxy server deals with proxying WebSocket 
> requests, AFAIK. So far my real-time apps are on intranets, so I
> didn't 
> yet have this case to know more about.
>

Janko,

Thanks for your reply and it's understandable that this may not be a
common use case.  You are correct that the proxy server deals with the
WebSockets which is a problem for Swazoo/Aida since it uses the upgrade
request to determine which requests are WebSockets. (i.e. since the
upgrade request is never seen behind a proxy, the connection is never
upgraded as far as Swazoo is concerned so it also doesn't hold it open
and all Aida sees are regular HTTP requests coming in which it doesn't
handle properly and on the client side the WebSocket fails)  Based on
some preliminary looking into this, it appears that what I need to do
is migrate the #behindProxy method to Swazoo and reconfigure things a
bit at the connection/request level to handle this scenario.

Thanks,
Phil

> Best regards
> Janko
>
>
> _______________________________________________
> Aida mailing list
> [hidden email]
> http://lists.aidaweb.si/mailman/listinfo/aida
_______________________________________________
Aida mailing list
[hidden email]
http://lists.aidaweb.si/mailman/listinfo/aida
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: WebSockets behind a proxy

Phil B
In reply to this post by Janko Mivšek
So I've done some more investigating and I think what's going on is
that the client is opening up the WebSocket with Apache, and Apache in
turn is attempting to open up a WebSocket with Aida.  It looks like it
is this second hop that is having the problem.

If I'm following what's going on correctly it appears that by the time
the request times out the SwazooBuffer has the correct request header
(i.e. the host name matches what I'd expect to see from the proxy
server and it is a connection upgrade request) in its read buffer (i.e.
its internal collection ivar) but when I attempt to #fillBuffer I still
get a StreamNoDataError: No data available. Socket probably closed.

I'm thinking (more accurately at this point: guessing) perhaps the
request is arriving in an unexpected manner from Apache? i.e. because
it's effectively relaying the request in a less direct way than it does
with non-WebSocket requests there might be a small delay between the
request starting and the header data actually becoming available that
is causing Swazoo some problems?  Note that only WebSockets are having
this problem, everything else in Aida is working fine behind the Apache
proxy.

On a related note: unfortunately, the Swazoo website appears to be gone
so any docs on its internals are no longer available.  Is there
anything (docs, diagrams, mailing list archives) that you might have
copies of that you could post on the Aida site that might be helpful in
working through this?

Thanks,
Phil

On Tue, 2016-05-31 at 11:34 +0200, Janko Mivšek wrote:

> Phil (list) je 25. 05. 2016 ob 22:29 napisal:
>
> > I was taking a look at using WebSockets for an application but it
> > appears that Aida/Swazoo doesn't know how to deal with WebSockets
> > from
> > behind a proxy without a bit of work (i.e. where the connection has
> > been pre-upgraded by Apache.)  I was wondering if there are any
> > pointers as to how to best handle this scenario?
>
> It is supposed that a proxy server deals with proxying WebSocket 
> requests, AFAIK. So far my real-time apps are on intranets, so I
> didn't 
> yet have this case to know more about.
>
> Best regards
> Janko
>
>
> _______________________________________________
> Aida mailing list
> [hidden email]
> http://lists.aidaweb.si/mailman/listinfo/aida
_______________________________________________
Aida mailing list
[hidden email]
http://lists.aidaweb.si/mailman/listinfo/aida
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: WebSockets behind a proxy

Phil B
I needed to look a little further down the stack trace: it looks like
the WebSocket upgrade request coming from Apache doesn't have the 8-
byte body being expected by HTTPGet>>readBodyFrom: which is causing
things to hang up.  So the good news is it is being recognized as a
WebSocket request, the bad news is it isn't in the form expected.

On Wed, 2016-06-15 at 07:06 -0400, Phil (list) wrote:

> So I've done some more investigating and I think what's going on is
> that the client is opening up the WebSocket with Apache, and Apache
> in
> turn is attempting to open up a WebSocket with Aida.  It looks like
> it
> is this second hop that is having the problem.
>
> If I'm following what's going on correctly it appears that by the
> time
> the request times out the SwazooBuffer has the correct request header
> (i.e. the host name matches what I'd expect to see from the proxy
> server and it is a connection upgrade request) in its read buffer
> (i.e.
> its internal collection ivar) but when I attempt to #fillBuffer I
> still
> get a StreamNoDataError: No data available. Socket probably closed.
>
> I'm thinking (more accurately at this point: guessing) perhaps the
> request is arriving in an unexpected manner from Apache? i.e. because
> it's effectively relaying the request in a less direct way than it
> does
> with non-WebSocket requests there might be a small delay between the
> request starting and the header data actually becoming available that
> is causing Swazoo some problems?  Note that only WebSockets are
> having
> this problem, everything else in Aida is working fine behind the
> Apache
> proxy.
>
> On a related note: unfortunately, the Swazoo website appears to be
> gone
> so any docs on its internals are no longer available.  Is there
> anything (docs, diagrams, mailing list archives) that you might have
> copies of that you could post on the Aida site that might be helpful
> in
> working through this?
>
> Thanks,
> Phil
>
> On Tue, 2016-05-31 at 11:34 +0200, Janko Mivšek wrote:
> > Phil (list) je 25. 05. 2016 ob 22:29 napisal:
> >
> > > I was taking a look at using WebSockets for an application but it
> > > appears that Aida/Swazoo doesn't know how to deal with WebSockets
> > > from
> > > behind a proxy without a bit of work (i.e. where the connection
> > > has
> > > been pre-upgraded by Apache.)  I was wondering if there are any
> > > pointers as to how to best handle this scenario?
> >
> > It is supposed that a proxy server deals with proxying WebSocket 
> > requests, AFAIK. So far my real-time apps are on intranets, so I
> > didn't 
> > yet have this case to know more about.
> >
> > Best regards
> > Janko
> >
> >
> > _______________________________________________
> > Aida mailing list
> > [hidden email]
> > http://lists.aidaweb.si/mailman/listinfo/aida
_______________________________________________
Aida mailing list
[hidden email]
http://lists.aidaweb.si/mailman/listinfo/aida
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: WebSockets behind a proxy

Janko Mivšek
Hi Phil,

I think Apache reverse proxy should support WebSockets explicitly,
otherwise this won't go. AFAIK this proxy store and forward HTTP
requests and responses. WebSocket Update request is still a normal HTTP
request while later over the same TCP/IP connection WebSocket packets
starts flowing, possibly on both directions simultaneously (full duplex).

To better see what is happening use the Wireshark packet sniffer and
look at the low level details, what the proxy is sending and what is
returned by Swazoo.

Also temporary switch off all exception catching in Swazoo. AFAIK it
could be in many places. Maybe some error happens inside Swazoo but it
is ignored, that is, it just close the connection immediately. .

Hope this helps a bit
Janko

Phil (list) je 15. 06. 2016 ob 23:05 napisal:

> I needed to look a little further down the stack trace: it looks like
> the WebSocket upgrade request coming from Apache doesn't have the 8-
> byte body being expected by HTTPGet>>readBodyFrom: which is causing
> things to hang up.  So the good news is it is being recognized as a
> WebSocket request, the bad news is it isn't in the form expected.
>
> On Wed, 2016-06-15 at 07:06 -0400, Phil (list) wrote:
>> So I've done some more investigating and I think what's going on is
>> that the client is opening up the WebSocket with Apache, and Apache
>> in
>> turn is attempting to open up a WebSocket with Aida.  It looks like
>> it
>> is this second hop that is having the problem.
>>
>> If I'm following what's going on correctly it appears that by the
>> time
>> the request times out the SwazooBuffer has the correct request header
>> (i.e. the host name matches what I'd expect to see from the proxy
>> server and it is a connection upgrade request) in its read buffer
>> (i.e.
>> its internal collection ivar) but when I attempt to #fillBuffer I
>> still
>> get a StreamNoDataError: No data available. Socket probably closed.
>>
>> I'm thinking (more accurately at this point: guessing) perhaps the
>> request is arriving in an unexpected manner from Apache? i.e. because
>> it's effectively relaying the request in a less direct way than it
>> does
>> with non-WebSocket requests there might be a small delay between the
>> request starting and the header data actually becoming available that
>> is causing Swazoo some problems?  Note that only WebSockets are
>> having
>> this problem, everything else in Aida is working fine behind the
>> Apache
>> proxy.
>>
>> On a related note: unfortunately, the Swazoo website appears to be
>> gone
>> so any docs on its internals are no longer available.  Is there
>> anything (docs, diagrams, mailing list archives) that you might have
>> copies of that you could post on the Aida site that might be helpful
>> in
>> working through this?
>>
>> Thanks,
>> Phil
>>
>> On Tue, 2016-05-31 at 11:34 +0200, Janko Mivšek wrote:
>>> Phil (list) je 25. 05. 2016 ob 22:29 napisal:
>>>
>>>> I was taking a look at using WebSockets for an application but it
>>>> appears that Aida/Swazoo doesn't know how to deal with WebSockets
>>>> from
>>>> behind a proxy without a bit of work (i.e. where the connection
>>>> has
>>>> been pre-upgraded by Apache.)  I was wondering if there are any
>>>> pointers as to how to best handle this scenario?
>>>
>>> It is supposed that a proxy server deals with proxying WebSocket
>>> requests, AFAIK. So far my real-time apps are on intranets, so I
>>> didn't
>>> yet have this case to know more about.
>>>
>>> Best regards
>>> Janko
_______________________________________________
Aida mailing list
[hidden email]
http://lists.aidaweb.si/mailman/listinfo/aida
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: WebSockets behind a proxy

Phil B
Janko,

On Fri, 2016-06-17 at 08:58 +0200, Janko Mivšek wrote:

> Hi Phil,
>
> I think Apache reverse proxy should support WebSockets explicitly, 
> otherwise this won't go. AFAIK this proxy store and forward HTTP 
> requests and responses. WebSocket Update request is still a normal
> HTTP 
> request while later over the same TCP/IP connection WebSocket
> packets 
> starts flowing, possibly on both directions simultaneously (full
> duplex).

I actually have it working now, so it does work.  You are correct, you
need to have a functioning reverse proxy as well as the wstunnel module
or this will not work at all.  The problem was in a difference between
the upgrade request sent by the browser and the one sent by Apache as
they are slightly different (but not in a way that seems to matter to
anything but Swazoo)...

>
> To better see what is happening use the Wireshark packet sniffer and 
> look at the low level details, what the proxy is sending and what is 
> returned by Swazoo.

Wireshark was helpful in confirming that the proxy was sending requests
that Aida was seeing but the problem was within Swazoo/Aida (i.e. it
wasn't responding) so I had to crawl around the connection code a bit
and insert a number of logging and halt statements to figure out what
was going on internally.  (if there are any docs on Swazoo internals,
they would be most welcome as it took me a bit of time to narrow it
down mainly because I didn't know where to start looking initially.
 Any available docs would be quite helpful should I need to dive into
Swazoo internals for a future issue.)

It turned out that the request #readBodyFrom: is forcing an 8-byte read
(the framing header?) for WebSocket upgrade requests which both Chrome
and Firefox appear to supply in their upgrade requests, but Apache does
not when acting as the proxy server. There is even a comment in the
#readBodyFrom: code indicating that this seemed strange so I'm not sure
why it's there, but this read was causing the request to timeout
waiting for data which never arrived.  Since it doesn't appear to be
needed so I just commented it out and everything started working.  So
far I haven't noticed any problems (i.e. WebSockets are now working
both directly to Aida as well as via a proxy) but would be interested
in knowing if there is something that I might have overlooked re: why
that 8-byte read in #readBodyFrom: would be necessary?

Thanks,
Phil
_______________________________________________
Aida mailing list
[hidden email]
http://lists.aidaweb.si/mailman/listinfo/aida
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: WebSockets behind a proxy

Janko Mivšek
Hi Phil,

Unfortunately no docs for Swazoo, on the website there was just one example.

I'm looking at the HTTPRequest>>readBodyFrom: and 8 byte read is only if
contentLength header is missing in update request. Is that so with
request from proxy? In this case it was expected that body of upgrade
request is exactly 8 bytes long, which was a case when I worked on that.
Is now different? How did you solve the problem, can you post a code?

Phil (list) je 17. 06. 2016 ob 11:15 napisal:

> Janko,
>
> On Fri, 2016-06-17 at 08:58 +0200, Janko Mivšek wrote:
>> Hi Phil,
>>
>> I think Apache reverse proxy should support WebSockets explicitly,
>> otherwise this won't go. AFAIK this proxy store and forward HTTP
>> requests and responses. WebSocket Update request is still a normal
>> HTTP
>> request while later over the same TCP/IP connection WebSocket
>> packets
>> starts flowing, possibly on both directions simultaneously (full
>> duplex).
>
> I actually have it working now, so it does work.  You are correct, you
> need to have a functioning reverse proxy as well as the wstunnel module
> or this will not work at all.  The problem was in a difference between
> the upgrade request sent by the browser and the one sent by Apache as
> they are slightly different (but not in a way that seems to matter to
> anything but Swazoo)...
>
>>
>> To better see what is happening use the Wireshark packet sniffer and
>> look at the low level details, what the proxy is sending and what is
>> returned by Swazoo.
>
> Wireshark was helpful in confirming that the proxy was sending requests
> that Aida was seeing but the problem was within Swazoo/Aida (i.e. it
> wasn't responding) so I had to crawl around the connection code a bit
> and insert a number of logging and halt statements to figure out what
> was going on internally.  (if there are any docs on Swazoo internals,
> they would be most welcome as it took me a bit of time to narrow it
> down mainly because I didn't know where to start looking initially.
>   Any available docs would be quite helpful should I need to dive into
> Swazoo internals for a future issue.)
>
> It turned out that the request #readBodyFrom: is forcing an 8-byte read
> (the framing header?) for WebSocket upgrade requests which both Chrome
> and Firefox appear to supply in their upgrade requests, but Apache does
> not when acting as the proxy server. There is even a comment in the
> #readBodyFrom: code indicating that this seemed strange so I'm not sure
> why it's there, but this read was causing the request to timeout
> waiting for data which never arrived.  Since it doesn't appear to be
> needed so I just commented it out and everything started working.  So
> far I haven't noticed any problems (i.e. WebSockets are now working
> both directly to Aida as well as via a proxy) but would be interested
> in knowing if there is something that I might have overlooked re: why
> that 8-byte read in #readBodyFrom: would be necessary?
>
> Thanks,
> Phil
> _______________________________________________
> Aida mailing list
> [hidden email]
> http://lists.aidaweb.si/mailman/listinfo/aida
>

--
Janko Mivšek
Svetovalec za informatiko
Eranova d.o.o.
Ljubljana, Slovenija
www.eranova.si
tel:  01 514 22 55
faks: 01 514 22 56
gsm: 031 674 565
_______________________________________________
Aida mailing list
[hidden email]
http://lists.aidaweb.si/mailman/listinfo/aida
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: WebSockets behind a proxy

Phil B
Janko,

On Fri, 2016-06-17 at 12:18 +0200, Janko Mivšek wrote:
> Hi Phil,
>
> Unfortunately no docs for Swazoo, on the website there was just one
> example.

OK, good to know.

>
> I'm looking at the HTTPRequest>>readBodyFrom: and 8 byte read is only
> if 
> contentLength header is missing in update request. Is that so with 
> request from proxy? In this case it was expected that body of
> upgrade 
> request is exactly 8 bytes long, which was a case when I worked on
> that. 
> Is now different? How did you solve the problem, can you post a code?
>

Literally all I did was get rid of that logic so the method now reads:

readBodyFrom: aStream
        (self headers includesFieldOfClass: HTTPContentLengthField)
ifFalse: [ ^ nil ].
        self body: (aStream nextBytes: self contentLength).

Since it wasn't being used it seemed like just ignoring the body, if it
is present for upgrade requests, is safe to do and I haven't seen any
problems as a result of this change.

Thanks,
Phil
_______________________________________________
Aida mailing list
[hidden email]
http://lists.aidaweb.si/mailman/listinfo/aida
Loading...