Download Cached Data

I am wanting to use Tempest WeatherFlow stations in an area where wifi is not an option. My plan was to visit them every few days and use my mobile hot spot to connect them to the internet. I’ve managed to set up the station to link to my hot spot data and I’ve linked the station to the IFTTT that exports data in 1 minute increments.

The issue I’m having is that the IFTTT only uploads data when station is connected to network. I need a way to download data that is cashed on the station or in WU after data has refreshed. I’ve played around with work arounds, but am not having any luck. Any suggestions?

Do you have the ability to have a small PC that can store the UDP data? Such as a raspberry pi.

One way to go would be to use WeatherFlow2MQTT, import to Home Assistant, then export to InfluxDB for all historical data. There are more streamlined ways, but since I have all of the running at the moment, that is the route I would go and then grab the data when I needed it.

I don’t have anything, but I could invest in something if that would make this all easier.

In case it helps, here are more details:

I’m going to install seven stations at seven houses in rural Mexico. I’m giving each household a cell phone and will be adding buying cell data (in Mexico people buy MB of data as needed rather than having a monthly plan) at least once a week for them to use the phone’s hotspot to upload data to weather underground. Best case scenario would be if I could log on to weather underground after they have synced data and download the data myself.

I did my best to understand your second response. Is raspberry pi needed to use WeatherFlow2MQTT? Looking at forums I pieced together that I needed to create a PWS. I did this, though the station is offline on the PWS but online on the weatherflow app, and thus I’m waiting to see if time will sort this out.

Thanks,

Sara

Hi @sara.dykman
There is not a simple solution that does exactly what you want, unless you can just leave the phones hot spots active full time would work.
There are several ways that it can be done though.
Basically in my experience if the hub does not upload the data to Weatherflow because there was no wifi for a while it only uploads an hour or two of the most recent data with some sensors loading more than others. Somewhere there is a specific amount mentioned but I just recall about an hour or twos worth. That means you will need the hotspot available every hour for several minutes (more than 10 minutes is my guess) to have enough time to upload its history for the past hour.
To guess how much data will be used through your mobile hotspot see this topic:

So if you simply left a mobile hotspot running at each location would be the most reliable method but perhaps too expensive depending on your purpose.
Now if you do that I am suggesting the data may not be at Weather Underground because that only works if it is live and it does not work as well as getting the historical data directly from Weatherflow servers.
There are several topics describing why Weather Underground is sometimes unreliable if loaded directly from the Tempest and will not be the one minute values.
So then all the history is available from Weatherflow servers online using several different options.
If you want all one minute data history then it can be accessed using the API which is described here:
https://weatherflow.github.io/Tempest/
For one minute data you would need to download one days worth at a time.
Several people have created different methods of using the API to display their data. Some are written in python using a raspberry pi computer or other languages or other types of computers.
Or you can use a link in your browser to pull the data straight into the browser window. The developers API area has a description where you can test doing that.
Or you can view the history simply using a web page and look at graphs or history data similar to if you were using the App.
If you actually decide to progress with your plans then I can describe more detail about how to access the history using a raspberry pi computer.
I keep having more ideas that may complicate your understanding and I just give you the links to consider but they are not actual solutions without some more work.

cheers Ian

Suggestions from Ian and Glenn look good to me.

If you can enable the hotspots often enough that the hub can play catch up and get the data to Weatherflow’s servers, do that. (Can the Hub really just cache data for an hour or two?) BTW, I highly recommend USB batteries with passthrough charging like the official Hub Power Bank to ensure the Hub will keep receiving data as long as the Tempest is sending it.

If you cannot do that, then I’m afraid you’d have to do something more labor intensive and fragile like configure a Raspberry Pi to listen for the Hub’s realtime UDP broadcasts, store them, and somehow send that data somewhere when the hotspot is available. I’d hope the Pi could act as both a WiFi hotspot and see the UDP broadcasts. Part of the fragility here is that during any Pi downtime you’d lose data. It seems likely that you could use a “USB cellular modem” with the Pi to have it connect to the Internet occasionally and upload its data.

How much data? The daily “device” data from the Weatherflow APIs Ian mentioned, including minute-by-minute observations, is usually about 125 kilobytes per day, and compresses via gzip to about 25 kilobytes. Full UDP captures might take more space since they’re more frequent than once a minute when the Tempest battery is in good shape.

In addition to being a lot less work, providing Internet to the Hub periodically and having it send to Weatherflow also would allow you to take advantage of Nearcast/Rain Check for what should be more accurate precipitation data. For daily “station” info I usually find the Nearcast final daily precipitation values are available by about 7am the next day.

Let me bring some extra details (some can be found on this page)

  • The Hub can store about a week of data from one Tempest
  • The amount of data uploaded by the Hub from a single Tempest device is about 1 GB per month or about 30-35 MB per day (understand with the 3 second wind interval)
  • Next to the hub the Tempest itself can cache also a few hours of data if it can’t reach the hub but it is rather limited (if memory isn’t to bad, about 2 hours)

in both caching systems once the memory is full it deletes oldest and continues to cache recent data …

Hope this helps better dimension the solution.

2 Likes

I was Incorrect. Sorry I forgot that in my experience that my hub also lost power in that situation. You should trust what Eric said.
cheers Ian :slight_smile:

Dumb question - Does the Hub cache the three second rapid_wind data for later transmission, or only the slower obs_* information? That would surely save a lot of bandwidth in a location without a 24/7 Internet connection…

1 Like

Why then, does my sister’s station (45970) only upload 6-6.5 hours of data after her station gets Wi-Fi access again? I’ve seen something similar on my stations but I have more than just a Tempest attached. Her station looses Wi-Fi at 11:00pm and gets it back at about 6:00am and the whole gap does not get back filled.

2 Likes

Isn’t this a hub from beta testing ? Those had smaller buffers. It is only when production models came out the full buffer memory was used.

I should test this again but I do think yes.

No, it is a production hub.

What about a pi pulling in udp packets → weewx → script to copy and restart the weedb each time. The back end could zip them together as needed.

If the pi could be replaced with a better machine with a built in hotspot you could simplify and the weewx db could store lots more than a week. Even with a pi.

I’m used to weewx so when u have a hammer…

Oh ya. You can get weewx to sync direct to wu also.

Thanks for all the thoughts and suggestions. I will be in Mexico next week. For your replies, my plan is to use old cell phones to keep stations online with mobile hot spot data. Then to access the data I will go to the station website and pull needed data. It looks like data on the weather underground site can pull up historic data by day, week, and month, so I should be able to get what I need. It won’t be streamline, but I think it will work.

The data from weatherflow using each stations web link will be more detailed and accurate than from Weather Underground. I simply mention this because if you encounter problems with Weather Underground remember the full history is on the Tempest web site like this as an example Tempest or Tempest
cheers Ian :slight_smile:

the full history is on the Tempest web site (iladyman)

and the Tempest RESTful APIs. Amazingly, even the minute-by-minute device observation data seems to be available for a long time, hundreds of days!

The Hub can store about a week of data from one Tempest (eric)

I am thankful for that. Just this past week, I went away for five days, and my Hub lost Internet access on the second day away (a physical cable problem with my WiFi router meant my house only had local WiFi for days). Less than an hour after I returned home & fixed the Internet problem, my Hub backfilled the four days of missing data to the cloud. :slightly_smiling_face:

2 Likes

Thanks @iladyman. When I look at the Tempest graphs with historic data I find that as time passes the detail decreases. After one week, I can only get measurements at 30 minute intervals. Judging from the example you sent, the 30 minute interval is only available for a month and then it jumps to every 3 hours.
This means I need to be collecting specific data before one week lapses, which is difficult for me because of internet availability. There isn’t anyway to get more specific historic data?

When I look at the Tempest graphs with historic data I find that as time passes the detail decreases. After one week, I can only get measurements at 30 minute intervals. Judging from the example you sent, the 30 minute interval is only available for a month and then it jumps to every 3 hours.

This means I need to be collecting specific data before one week lapses, which is difficult for me because of internet availability.

If you want historic data, look at the /observations/device/ APIs documented at Tempest API . As I noted above, minute-by-minute observations are available going back many, many days – just use a larger “day_offset” value to get older data. As long as the hub managed to both collect the data from the Tempest unit and send it to Weatherflow’s servers, you should be able to retrieve that data at a much later date.

(I don’t think Weatherflow has made any assurances about how far long they’ll make detailed device observation available, so I make sure to save local copies, waiting long enough for the RainCheck corrected data to be included.)

Thanks @peter4! I was able to successfully use API to create a CSV file which I put into excel. It all looks good, but I can’t figure out how to format the “timestamp” column in excel to reflect date/time info. How do I interpret the timestamp? Is it a date and time, as I would expect?

Sara

I think I answered the above on my own. Looks like I need to use this formula in excel: =(TIME STAMP CELL/86400)+DATE(1970,1,1).

1 Like