Managing Tempest bandwidth

Developers
1

I am exploring integrating Tempest weather stations into my product. One problem I am observing is the amount of data that is uploaded through the internet. My use-case can only tolerate about 100KB/hour of data and it seems the system wants to pass about an order of magnitude more data than that.

So, I think an ideal would be if I could configure the station with a bandwidth limit and let weatherflow decide what data to send home. (I am happy to have all the UDP data on my local network.)

Or I could collect the UDP data and massage it in whatever way is desired (including compression) and pass it (say hourly) back home. Clearly in this model wind stricks and rapid wind events would get filtered or only shared in realtime based on a dynamic threshold.

My first approach was to block the internet connection for the Tempest device. Then I found it was peppering me with DNS requests, so I blocked those too. I think I successfully blocked the device because I would get email messages telling me the device went offline and when it went back online.

I also looked through the group and see that the hub can save data for a while. So, if the device gets unblocked, then I am concerned it will try to blast a lot of data which I cannot tolerate.

Anyone else have this issue/concern? Any other thoughts?

2

ask support to limit the sampling rate drastically for you, like once ever 5 minutes. Wind/rain/lightning will be less useful, but temp, humidity, pressure, UV index, solar irradiation would be fine. Depends on your use case if that is a good idea or not.

3

I did a little more analysis collecting all the UDP json data for a minute (separated by \n) then compressed the data using gzip. I got about a factor of 7 in data reduction. The compression factors grow if I collect more data in a block.

It also occurs to me that by preventing the unit from sending data I assume I am impacting its notion of time as well as other things. So it really would be nice if WeatherFlow made me aware of some hooks that allow for this kind of configuration.

thanks

1 Like
4

Hi Brian,
Just something else to be aware of. Sometimes the Tempest or the hub may be sent an update which changes something. We are not told about it. It would require extra bandwidth for the update. You could try asking that your hub and Tempests are not updated.
If your use case only requires limited sensors you could try asking if Weatherflow can turn some off. Like Sunny suggested but restricted to particular sensors.
If weatherflow can not do that then you could insert a buck converter into the power supply to reduce the voltage to which ever power saving mode you require, which has the effect of reducing data.
Cheers Ian :slight_smile:

5

You might be picking the wrong vendor for your product. WF kinda expects Internet connectivity.

If you block the Hub from connecting to WF, it’s going to think your network is offline and (possibly) just buffer the UDP thinking it’s going to need to catch up later. You definitely want to test what the current firmware really does. My hazy recollection is the Hub might not emit UDP if you totally block communication to WF servers. Test it.

DNS and NTP can of course be faked locally, but unfortunately everything ‘used to be’ within one persistent connection to WF so pieces of the puzzle probably are not something you can just filter out or fake out one by one. Running a sniffer will tell you for sure what connection(s) to internet the hub tries to do. My recollection is the Hub establishes one connection to WF and everything runs through it.

For compression you might try bzip or xz if you want to really dial up the compression factor, once you figure out if a ‘block connection to WF’ scenario makes sense.

If you could describe your ‘product’ maybe folks could suggest approaches…

1 Like
6

I have done deeper analysis of the network traffic flowing from the hub to weatherflow servers and, while the data as it exists today is too much for my market, I do believe that WF could make it so that I can get to a data rate that I can live with.

I basically considered each type of message separately. After my “filtering” (described below) I found I was left with breakdown of: [‘rain_debug 0.000%’, ‘rapid_wind 3.880%’, ‘st_system_status 4.404%’, ‘device_status 9.120%’, ‘obs_st 8.657%’, ‘wind_debug 0.000%’, ‘debug_obs_st 0.000%’, ‘hub_status 0.951%’, ‘evt_strike 0.000%’, ‘light_debug 0.000%’] and the aggregate amount of data was reduced by an order of magnitude.

I am not claiming my process and measurements are perfect (they are approximate) or that my filtering is the ideal, but I suspect something similar with a low_data mode could be implemented in a straightforward way and have a noticeable impact.

In particular, I ignored any messages that had “debug” in the type. That is why they are 0% in the above table.

Then, I ensured that if it had been 5min since I had last sent a message of a particular type, that I let the next message go through. In other words, no data was more than 5min stale.

Then I examined the current data for a message and compared it to the prior data for that message type. If the new data was “substantively” different, I sent out the new data even if it had been less than 5min.

So the question became what is a “substantive” change. This is something WF folks are better prepared to answer but I tried the following. Basically a change in a field that was [‘timestamp’, ‘time_epoch’, ‘rssi’, ‘seq’, ‘uptime’] was NOT considered a substantive change; other field changes WERE considered substantive. The type rapid_wind which is the most frequent message required a different style of filter. I basically treated it as a substantive change IF the wind speed changed by more the 0.5km/h OR the wind direction changed by more than 30degrees with a windspeed >= 0.25km/h. I actually collected data during what is considered a windy time for where I live.

Ignoring the debug entries, the filtering reduced the remaining data by about 70% and the combination got me to less than the 100MB/month that I am targeting.

So, WF folks, can you do something to help me? I am certainly happy to try an experimental fw upgrade!

-brian

1 Like
7

Again, without knowing what you’re building it’s hard to say, but if you can build yourself any kind of standalone embedded-system-like weewx setup, you ‘could’ just simply emit MQTT periodically and include only the things you want to emit. For example, a weewx system set to 5-minute archive period would emit a tiny amount of data 12x/hour.