WebSocket API issue with obs_st messages

Developers
, ,
1

I’ve stumbled upon an interesting issue concerning the reception of obs_st messages over the WebSocket API.

If one opens a second concurrent connection and issues listen_start for the same device ID, that connection will reply with an ack along with an initial obs_st message ("source":"cache"), but no further messages. Meanwhile, the first connection will receive all of the messages.

Disconnecting the first connection does not seem to unblock the second one.

In contrast, rapid_wind messages (from listen_rapid_start) appear to be broadcast to all connections.

Is this behavior expected? If so, it would be nice if the API would expose the error rather than being silent about it.

I’m using wscat for testing:

wscat -c wss://ws.weatherflow.com/swd/data?token=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx -x ‘{“type”:“listen_start”,“device_id”:123456,“id”:“”}’ -w 3600

I’m aware that “A client should only open one websocket connection” (from the API docs) – my use case unfortunately requires two connections for two realtime displays in independent locations, and I’d prefer not to insert my own MQTT server as a point of failure. I’m hoping that this behavior is an oversight, and that there’s a tiny bit of slack on the “one connection” policy…

2

Unfortunately this behaviour has been occurring for a while. I don’t think it is to do with the “A client should only open one websocket connection” policy, but rather an issue with the Websocket itself. I am going to loop in @corrineb who might be able to shed more light

2 Likes
3

I’m interested to find this. After having used UDP for a while, I decided last friday to dive into the websockets and observed the exact same behavior. Rapid Wind comes in every 3 seconds. Obs_st only comes in once and then never again. Figured I’d come back to it later only to find this. Guess it’s not just me.

Watching…

2 Likes
4

Thanks for the reports @tris and @mabeatty1978 and for the tag @peter. @mabeatty1978 I will have the team test things using your account info and let you know what we find. @tris I looked up your account info, but it doesn’t look like you have devices attached to your account. Feel free to dm me the email address of the account you are working with and I can take a closer look.

5

Thanks for looking into this. The behaviour @tris describes has been a bit of a persistent issue for a while. I had to implement a connection watchdog in the PiConsole to reset the Websocket if this condition occurred, or else users kept reporting that the display was offline and not receiving new observations (except rapid_wind). I can reliably recreate the issue by opening two Websocket connections on two separate clients using the same internet connection

6

@corrineb—I’m seeing the same issue. I’m testing multiple versions of an application using a Web socket, and all of my data goes silent for some unexplainable reason.

I send a request to all of my devices and get an ACK, but sometimes, I get nothing back from the Websocket. Pings/Pongs work, but there is no data. Sometimes, the socket disconnects. I reconnect, but the data is not available after I get ACKs. If it makes sense to open a new request, I will do so.

In most situations I get a steady stream of OBS and Wind data from my 10 devices without issue… Nothing is changing on my side.

7

You aren’t sending a request to your devices. You’re sending a request to WF servers for data regarding your stations and corresponding devices.

Best guess - WF servers aren’t 100% working 100% of the time, like pretty much any other service on Internet…

8

@vinceskahan Thanks, but I mean the device_id for each device.

2024-06-30 13:24:58 - websockets - client - read_frame - DEBUG - < TEXT '{"type":"connection_opened"}' [28 bytes]
2024-06-30 13:24:58 - websockets - client - write_frame_sync - DEBUG - > TEXT '{"type": "listen_start", "device_id": 78532, "id": 474300}' [58 bytes]
2024-06-30 13:24:58 - websockets - client - write_frame_sync - DEBUG - > TEXT '{"type": "listen_rapid_start", "device_id": 78532, "id": 474300}' [64 bytes]
2024-06-30 13:24:58 - websockets - client - write_frame_sync - DEBUG - > TEXT '{"type": "listen_start", "device_id": 86813, "id": 701670}' [58 bytes]
2024-06-30 13:24:58 - websockets - client - write_frame_sync - DEBUG - > TEXT '{"type": "listen_rapid_start", "device_id": 86813, "id": 701670}' [64 bytes]
2024-06-30 13:24:58 - websockets - client - write_frame_sync - DEBUG - > TEXT '{"type": "listen_start", "device_id": 209487, "id": 182640}' [59 bytes]
2024-06-30 13:24:58 - websockets - client - write_frame_sync - DEBUG - > TEXT '{"type": "listen_rapid_start", "device_id": 209487, "id": 182640}' [65 bytes]
2024-06-30 13:24:58 - websockets - client - write_frame_sync - DEBUG - > TEXT '{"type": "listen_start", "device_id": 209668, "id": 498232}' [59 bytes]
2024-06-30 13:24:58 - websockets - client - write_frame_sync - DEBUG - > TEXT '{"type": "listen_rapid_start", "device_id": 209668, "id": 498232}' [65 bytes]
2024-06-30 13:24:58 - websockets - client - write_frame_sync - DEBUG - > TEXT '{"type": "listen_start", "device_id": 209669, "id": 510177}' [59 bytes]
2024-06-30 13:24:58 - websockets - client - write_frame_sync - DEBUG - > TEXT '{"type": "listen_rapid_start", "device_id": 209669, "id": 510177}' [65 bytes]
2024-06-30 13:24:58 - websockets - client - write_frame_sync - DEBUG - > TEXT '{"type": "listen_start", "device_id": 330069, "id": 419986}' [59 bytes]
2024-06-30 13:24:58 - websockets - client - write_frame_sync - DEBUG - > TEXT '{"type": "listen_rapid_start", "device_id": 330069, "id": 419986}' [65 bytes]
2024-06-30 13:24:58 - websockets - client - write_frame_sync - DEBUG - > TEXT '{"type": "listen_start", "device_id": 261322, "id": 299825}' [59 bytes]
2024-06-30 13:24:58 - websockets - client - write_frame_sync - DEBUG - > TEXT '{"type": "listen_rapid_start", "device_id": 261322, "id": 299825}' [65 bytes]

I get an ACK back as I’ve done forever, and today I’ve been getting socket disconnects, connections with ACKs back, and no data, and then other times, it’s working 100% of the time.

I’ll see if it resolves itself and build more health checks into the system. Thanks!

9

Oh - you meant ‘for’ your devices. Sure. But you need to program defensively in any event as always. What you’re running into is kinda why I always worked just off the local hub UDP broadcasts since that takes anything Internet or WF-server out of the equation, but you do lose a little bit in the data you can get of course per the APIs…

1 Like
10

I’m capturing a dozen remote devices now with plans for hundreds. While I love UDP - placing resources at every remote site isn’t viable. This has been working for years - only in the last 5 days or so has the WebSocket connection been unstable from the WeatherFlow side. When connected, it stops sending data. I’ve built a health check to restart the connection if no data is received. (I can still see ping/pong - so the connection is responsive… just no JSON is flowing)

I can have as much resilience on my side, but there’s a failure on the WeatherFlow side, and I’m trying to determine if it’s widespread or specific to me. It’s been solid for as long as I can remember.

1 Like
11

I would think folks running wfpiconsole should have seen this one by now if they’re running the default setup…

12

This has just begun to happen for me on my Weewx box. Websocket is connected, ping/pongs are responsive so the connection is remaining open, but all data just stops flowing until the Websocket connection is stopped and restarted.

This can’t be a widespread issue otherwise everyone running the piconsole would see it at the same time. It must be isolated to specific users or specific personal access tokens. Have you seen an improvement over time?

1 Like