Some deployments have limited network bandwidth and cannot tolerate the approximately 1GB/month of device data. Something on the order of 100KB/month would be reasonable.
Some analysis has been performed and (I hope) a relatively simple strategy proposed that does significantly reduce bandwidth.
1GB/month??? are you sure. That would be 23kB per minute, for just a few numbers.
Remember that it is sending formatted JSON to WF-T, so 23kB/minute is plausible with rapid_wind packets being sent every 3 seconds…
That 1GB/month (approx) came from WF support and my data confirms that is close enough.
if they say so, but sending 50 numbers each minute shouldn’t require 460 bytes per number, even if in json format.
Remember… JSON is not just the numbers, but the labels, apostrophes, brackets/braces, etc associated with each data item/group
i know, but now I’m curious how much json is really needed to send for example the wind data.
I agree… I’ll do some data capture here and see what the messages look like.
rapid_wind messages from Hub to Server:
{“serial_number”:“ST-00000418”,“type”:“rapid_wind”,“hub_sn”:“HB-00013481”,“ob”:[1685658166,1.34,357]}
Looks like it’s hitting 101 bytes at a minimum for RW messages?
And that’s just the payload… there appears to be some header info there as well… so probably more like 120bytes… which by itself is ~3.456MB per day = ~ 104MB per month… just for RW.
Unless I’m missing something.
Here is an example of 16 seconds of data transmission from the device. This does NOT include networking overheads nor does it include the response data. The program I wrote dumps a combination of text/hex so just look at the length field. The contents is a rough notion of the data.
You can see there is clearly more than 1 json message/minute!
1685661056 len 127 rate 113.60MB/31days Dest MAC b827eb276d94 Host MAC c0ee40727b15 eTyp 0800 Source IP address 10.136.0.13 Destination IP address 34.238.102.102
Contents: c0m079fsT8ff4ffeP1811C87b800002e801000fobs/ST-001078795eee{“serial_number”:“ST-00107879”,“type”:“obs_st”,“hub_sn”:“HB-00118900”,“obs”:[[1685661054,0.52,0.60,0.68,101,20,945.14,23.13,28.89,83265,6.05,694,0.000000,0,0,0,2.817,1]],“firmware_revision”:172}7c0xff3129175e67c154
1685661056 len 131 rate 114.53MB/31days Dest MAC b827eb276d94 Host MAC c0ee40727b15 eTyp 0800 Source IP address 10.136.0.13 Destination IP address 34.238.102.102
Contents: c0m079ft3f8ff4g02P18113fa42100002f0010011debug/ST-001078795eef{“serial_number”:“ST-00107879”,“type”:“debug_obs_st”,“hub_sn”:“HB-00118900”,“obs”:[[1685661054,0.52,0.60,0.68,101,20,943.35,24.16,26.73,91430,6.91,762,0.000000,0,0,0,2.817,1]],“firmware_revision”:172}7c0xec493a18e6ca7602
1685661057 len 10 rate 114.30MB/31days Dest MAC b827eb276d94 Host MAC c0ee40727b15 eTyp 0800 Source IP address 10.136.0.13 Destination IP address 34.238.102.102
Contents: c0m079fu28ff4g06P1011;y1b0000
1685661058 len 84 rate 114.59MB/31days Dest MAC b827eb276d94 Host MAC c0ee40727b15 eTyp 0800 Source IP address 10.136.0.13 Destination IP address 34.238.102.102
Contents: c0m079fu28ff4g06P1811;x820000092010011debug/ST-00107879{“serial_number”:“ST-00107879”,“type”:“light_debug”,“hub_sn”:“HB-00118900”,“ob”:[1685661056,57437,2048,0,1]}7c0x9d280bf6115dcae5
1685661058 len 177 rate 115.83MB/31days Dest MAC b827eb276d94 Host MAC c0ee40727b15 eTyp 0800 Source IP address 10.136.0.13 Destination IP address 34.238.102.102
Contents: c0m079fuc78ff4g06P1811;0b1500000bd010011debug/ST-00107879{“serial_number”:“ST-00107879”,“type”:“wind_debug”,“hub_sn”:“HB-00118900”,“ob”:[1685661056,17,106,0,0,1080,1060,1033,1046,1068,1097,1140,1141,0,8,0,0]}7c0x3487fa8d1c9057c608b010011rapid/ST-00107879{“serial_number”:“ST-00107879”,“type”:“rapid_wind”,“hub_sn”:“HB-00118900”,“ob”:[1685661056,0.17,106]}7c0xcdeec1b3a37c805b
1685661062 len 10 rate 114.70MB/31days Dest MAC b827eb276d94 Host MAC c0ee40727b15 eTyp 0800 Source IP address 10.136.0.13 Destination IP address 34.238.102.102
Contents: c0m079fw148ff4g06P1011;w90000
1685661064 len 148 rate 115.13MB/31days Dest MAC b827eb276d94 Host MAC c0ee40727b15 eTyp 0800 Source IP address 10.136.0.13 Destination IP address 34.238.102.102
Contents: c0m079fw158ff4g06P1811;xe10000091020012status/HB-00118900{“serial_number”:“HB-00118900”,“type”:“hub_status”,“firmware_revision”:“177”,“uptime”:71790,“rssi”:-65,“timestamp”:1685661062,“reset_flags”:“BOR,PIN,POR”,“seq”:7170,“radio_stats”:[26,1,0,3,33745],“mqtt_stats”:[13,16],“freq”:904000000}7c0x0c55e1a48c190d05
1685661068 len 10 rate 114.02MB/31days Dest MAC b827eb276d94 Host MAC c0ee40727b15 eTyp 0800 Source IP address 10.136.0.13 Destination IP address 34.238.102.102
Contents: c0m079fx288ff4g06P1011;v250000
1685661071 len 10 rate 113.22MB/31days Dest MAC b827eb276d94 Host MAC c0ee40727b15 eTyp 0800 Source IP address 10.136.0.13 Destination IP address 34.238.102.102
Contents: c0m079fx288ff4g06P1011;v250000
1685661074 len 28 rate 112.55MB/31days Dest MAC b827eb276d94 Host MAC 146b9cc955a2 eTyp 0800 Source IP address 10.136.0.14 Destination IP address 52.8.254.146
Contents: r27}d008931bf11200,b0d9m09:f29aVN10cbaccc42bbfc524ccc8ce238dubf;8296c0da4018a8de1e9a8cdaza791a11b
1685661074 len 148 rate 113.55MB/31days Dest MAC b827eb276d94 Host MAC c0ee40727b15 eTyp 0800 Source IP address 10.136.0.13 Destination IP address 34.238.102.102
Contents: c0m079fx298ff4g06P1811;17f70000091020012status/HB-00118900{“serial_number”:“HB-00118900”,“type”:“hub_status”,“firmware_revision”:“177”,“uptime”:71800,“rssi”:-65,“timestamp”:1685661072,“reset_flags”:“BOR,PIN,POR”,“seq”:7171,“radio_stats”:[26,1,0,3,33745],“mqtt_stats”:[13,16],“freq”:904000000}7c0xfd34b1b7b95b6418
sure it will do more than one json per minute, at least 20 json wind messages in addition to some extra for the other data. It’s hard to tell from your dump what is really send and what is generated by your program but one thing I noticed was that it mentioned around 112-115MB of data per month, which is a fraction of the 1GB/month number.
I also noticed that some fields are labeled debug, which might send more data than needed (I don’t know).
that’s a reasonable number. but not close to the 1GB/month that apparently support told Brian.
Easiest way to measure is to simply use my UDP listener (link) with the --raw option and redirect to a file.
wfudplistener --raw >> /var/tmp/someFileName.txt
Let it run for an hour or two and see how big the file gets, and do the simple math. This isn’t rocket science.
You can use the --limit and --exclude flags to include/exclude only the observation types you want, but personally I’d just take the file above grep for certain types one by one from a certain period’s capture.
grep rapid_wind /var/tmp/someFileName.txt | wc
We’re looking at the data sent to WFs serves…not the local UDP data.
Unless I have it wrong and the Hub is sending UDP to WFs severs as well…
It’s the same data under the hood. It’s just tunneled into one persistent connection.
The UDP data does not leave the subnet. It is similar to what goes to the server but different and there is more data going to the server.
Ancient WF user here. I’m pretty aware of what goes where.
I see MQTT alone clocking in at around 830MB per month. There will be other traffic such as DNS and retransmits that will lift that a bit. That makes 1GB/month a good estimate. Some telcos count traffic in different ways so the actual clocked traffic may vary according the their policies.
I believe I have a workaround for this. I basically play “man-in-the-middle”, collect all the packets and do the filtering I want to do. Not ideal, but…
At the heart, I think I want to filter the mqtt packets. They seem to start with an MQTT protocol header, then reference a target with associated json and end with ‘|0x’ and 16 hex characters. A single tcp packet can contain multiple mqtt data. I prefer to be able to modify a packet to only keep the data that I care about. I suspect it is fine if I just keep the mqtt header,json,and 19B trailer but I am not sure what that trailer is. (I am the first to admit I have not studied mqtt traffic in detail.)
Is there a reference (or someone knowledgeable with WF mqtt implementation)
fwiw, I am currently just counting outgoing traffic with the simple filtering described earlier (plus ignoring mqtt_stats); my data rate is about 0.06KB/s or ~160MB/month which is still a bit more than desired. It actually seems that most of the data is some form of 40B (and an occasional 67B) TCP packets every couple of seconds (keepalive?) Anyone know what these packets really are?
MQTT ping packets are certainly in the traces I’ve taken in the past.
I would hope that the MQTT stuff is encrypted to the point where you can’t get into the data and decode it, but perhaps (wild guess) you could set up a fake WF server with mosquitto and some DNS sleight of hand and capture get into the stream. Hard to say.
In years past there were discussions about this and the conclusions way back then were that the pipe to WF has the same data in it that UDP broadcasts to the LAN. I can’t remember any more what watchdog timer type stuff is in there as well but I think there were checks there so that the Hub would know if it needed to (later) do catchup sending of stored data to WF when connectivity to their servers was reestablished.