[dart-misc] Some troubles with dart:io client side WebSocket implementation

[Vadim Tsushko]

One of mine primary usage case for dart is all sorts of scripting to help
in my everyday job - BI development with Qlik (as qlik.com) products.
My immediate task in that area is to migrate existing infrastructure
supporting a development of QlikView BI projects to Qlik Sense.
Qlik Sense is a new product and the main interface to communicate with it
is through WebSockets (Qlik specifically claim that any client capable to
communicate through WebSockets could use their Qlik Sense API)

I have some experience with Dart WebSockets before and was relatively
confident that in that area I should not find big troubles. But in fact I
could not connect to QlikSense services with dart:io WebSockets.
Initially, I could not find any reasons for that and tried all sorts of
solutions. Strange fact for me was that I could successfully connect to
QlikSense services from within dart web applications (using dart:html
WebSocket implementation)

That was not very helpful for me though as I have to create CLI utility,
not the Web interface.

So my temporary solution is to write tiny transparent nodejs WebSocket
proxy. From dart script I interact with that proxy, and that proxy in turn
interacts with Qlik Sense service. That works for me now on development
stage, but for deployment that would be great burden indeed.

So I spent some time with WireShark to find what is different in WebSocket
handshaking of nodejs (successful) and dart:io (unsuccessful)
communication. For now I'm almost totally sure that difference and reason
why dart:io WebSocket connections break with error is that dart:io header
keys are converts to lowercase.


Successful communication looks like:


GET /app/%3Ftransient%3D HTTP/1.1
Connection: Upgrade
Upgrade: websocket
Host: 192.168.188.10
Sec-WebSocket-Version: 13
Sec-WebSocket-Key: MTMtMTQ0OTMzNjYzNzY0NQ==
Content-Type: application/json
Cookie: X-Qlik-Session=70a54c5b-a08b-4665-a0d7-1a39871ebea9; Path=/;
HttpOnly; Secure
Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits

HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: Tb5mxSaTiFApEyYAAv/ci7AMOzI=
Access-Control-Allow-Origin: ws://192.168.188.10


Unsuccessful communication looks like:


GET /app/%3Ftransient%3D HTTP/1.1
user-agent: Dart/1.14 (dart:io)
connection: Upgrade
cache-control: no-cache
accept-encoding: gzip
cookie: X-Qlik-Session=70a54c5b-a08b-4665-a0d7-1a39871ebea9; Path=/;
HttpOnly; Secure
content-length: 0
sec-websocket-version: 13
host: 192.168.188.10
sec-websocket-extensions: permessage-deflate; client_max_window_bits
content-type: application/json
sec-websocket-key: lVH5YxHL8QjS7ddjIJ/Ohg==
upgrade: websocket

HTTP/1.1 404 Not Found
Content-Length: 0
Access-Control-Allow-Origin: ws://192.168.188.10


I did not find any way to pass intact (not converted to lowercase) headers
to dart:io WebSocket or HttpClient. So finally I've tried raw Socket and
with it (writing to it capitalized Connection and Upgrade as header keys)
I've managed to get switching protocol response from the Qlik Sense
WebSocket server. So apparently from that I can proceed to create WebSocket
from manually upgraded WebSocket.

But I wonder now- is Qlik Sense WebSocket server implementation is wrong to
reject dart:io client WebSocket connections? Or dart:io WebSockets are
somehow not strictly adhere to specification?
Dart:io WebSocket client successfully connects to http://www.websocket.org/
sample server, to dart:io WebSocket servers, to nodejs ws server - but how
many `Qlik Sense like` not compatible with dart:io WebSocket servers are in
the wild?


Communication example on Wikipedia looks more like nodejs version (with
capitalized keys)

GET /chat HTTP/1.1
Host: server.example.com
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: x3JJHMbDL1EzLkh9GBhXDw==
Sec-WebSocket-Protocol: chat, superchat
Sec-WebSocket-Version: 13
Origin: http://example.com

[Vadim Tsushko]

I've created a https://github.com/dart-lang/sdk/issues/25120 for that too

[Søren]

This post might be inappropriate. Click to display it.

[Vadim Tsushko]

Thank for update. I underestand the reason and can see that it is valid.

Unfortunately it mean that porting some stuff from nodejs (or dotnet in
that case) equivalient to dart is not feasible to me. Actually I thing I
would upgrade WebSockets manually in that case but next thing I discover -
I cannot pass client certificate to Qlik Sense in a way that was done in
nodejs or dotnet.
So I probably stay put with deploing nodejs proxy service.

Post by Søren
Thanks for the report, however the current behaviour is by design.
The HTTP/1.1 specification section 4.2
<https://tools.ietf.org/html/rfc2616#section-4.2> clearly states that the
field names in HTTP headers are case-insensitive. Early on in the design of
the Dart HTTP library we decided to make all header fields lowercase, and
not try to neither keep the passed-in case nor produce the Xxx-Yyyy-Zzz
format.
Btw. the HTTP/2 specification went with all lowercase headers.
I have updated the issue https://github.com/dart-lang/sdk/issues/25120 with
this information.
Regards,
SÞren
---
SÞren Gjesse
Software Engineer, Google Denmark
CVR nr. 28 86 69 84

Post by Vadim Tsushko
One of mine primary usage case for dart is all sorts of scripting to help
in my everyday job - BI development with Qlik (as qlik.com) products.
My immediate task in that area is to migrate existing infrastructure
supporting a development of QlikView BI projects to Qlik Sense.
Qlik Sense is a new product and the main interface to communicate with it
is through WebSockets (Qlik specifically claim that any client capable to
communicate through WebSockets could use their Qlik Sense API)
I have some experience with Dart WebSockets before and was relatively
confident that in that area I should not find big troubles. But in fact I
could not connect to QlikSense services with dart:io WebSockets.
Initially, I could not find any reasons for that and tried all sorts of
solutions. Strange fact for me was that I could successfully connect to
QlikSense services from within dart web applications (using dart:html
WebSocket implementation)
That was not very helpful for me though as I have to create CLI utility,
not the Web interface.
So my temporary solution is to write tiny transparent nodejs WebSocket
proxy. From dart script I interact with that proxy, and that proxy in turn
interacts with Qlik Sense service. That works for me now on development
stage, but for deployment that would be great burden indeed.
So I spent some time with WireShark to find what is different in
WebSocket handshaking of nodejs (successful) and dart:io (unsuccessful)
communication. For now I'm almost totally sure that difference and reason
why dart:io WebSocket connections break with error is that dart:io header
keys are always converted to lower case.
GET /app/%3Ftransient%3D HTTP/1.1
Connection: Upgrade
Upgrade: websocket
Host: 192.168.188.10
Sec-WebSocket-Version: 13
Sec-WebSocket-Key: MTMtMTQ0OTMzNjYzNzY0NQ==
Content-Type: application/json
Cookie: X-Qlik-Session=70a54c5b-a08b-4665-a0d7-1a39871ebea9; Path=/;
HttpOnly; Secure
Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits
HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: Tb5mxSaTiFApEyYAAv/ci7AMOzI=
Access-Control-Allow-Origin: ws://192.168.188.10
GET /app/%3Ftransient%3D HTTP/1.1
user-agent: Dart/1.14 (dart:io)
connection: Upgrade
cache-control: no-cache
accept-encoding: gzip
cookie: X-Qlik-Session=70a54c5b-a08b-4665-a0d7-1a39871ebea9; Path=/;
HttpOnly; Secure
content-length: 0
sec-websocket-version: 13
host: 192.168.188.10
sec-websocket-extensions: permessage-deflate; client_max_window_bits
content-type: application/json
sec-websocket-key: lVH5YxHL8QjS7ddjIJ/Ohg==
upgrade: websocket
HTTP/1.1 404 Not Found
Content-Length: 0
Access-Control-Allow-Origin: ws://192.168.188.10
I did not find any way to pass intact (not converted to lower case)
headers to dart:io WebSocket or HttpClient. So finally I've tried raw
Socket and with it (writing to it capitalized Connection and Upgrade as
header keys) I've managed to get switching protocol response from the Qlik
Sense WebSocket server. So apparently from that I can proceed to create
WebSocket from manually upgraded WebSocket.
But I wonder now - is Qlik Sense WebSocket server implementation is wrong
to reject dart:io client WebSocket connections? Or dart:io WebSockets are
somehow not strictly adhere to specification?
Dart:io WebSocket client successfully connects to
http://www.websocket.org/ sample server, to dart:io WebSocket servers,
to nodejs ws server - but how many `Qlik Sense like` not compatible with
dart:io WebSocket servers are in the wild?
Communication example on Wikipedia looks more like nodejs version (with
capitalized keys)
GET /chat HTTP/1.1
Host: server.example.com
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: x3JJHMbDL1EzLkh9GBhXDw==
Sec-WebSocket-Protocol: chat, superchat
Sec-WebSocket-Version: 13
Origin: http://example.com