Add ability to download historical data within the app or web interface

[thegumnut]

Improve the “historical data” functionality (within the app and/or the web user interface itself - not an API to another app) to allow the user to download historical data. Currently the data can be viewed but not downloaded. This would allow analysis of the data (in other applications for example).

Allowing the user to select a date range to download data would be nice to have.

Note that the current “history” functionality appears to be limited to a day-by-day scrolling (in the app) and a max/min over a pre-defined date range (in the web interface).

Note also that the IFTTT functionality that exists is limited to downloading live data (i.e. not historical data).

There is another (year old) feature request very similar to this, but this request is specifically to add this feature to a user-accessible interface.

[eric]

What you describe exists via several channels
or locally via UDP or remotely calling data from the WeatherFlow servers

it is all described in this page

[Lux4rd0]

I’ve built a data exporter/importer as part of my WeatherFlow data collector (as a few others have done on this forum as well…) Curious how you’d like to see that data? In a CSV file? I may be able to add a feature to create a file of your data from a command line (via Docker) perhaps. Everything in that link that @eric provided is a way of getting after your JSON data in the cloud. Lots of data exists all over the place

[thegumnut]

Thanks @eric . The page you linked refers to the developer tools (of which there are indeed many). But my request is specifically for a download feature to be added to the app or web interface as I tried to make clear. While I guess many owners are weather geeks, probably not so many are software geeks, and most probably don’t even know what JSON, UDP, API even mean, let alone how to use these protocols.

[thegumnut]

Thanks for the response @Lux4rd0 ! The historical data in a CSV file would be perfect. If you have anything I can use as a starting point to work with, that would be great.

By the way - and this is perhaps a question showing my ignorance - but I don’t find (yet) clear details of where the historical data actually resides and how much of it there is (and at what time resolution). I can create a CSV file of the current data stream, but I’m really looking for whatever historical data there is (either locally on the hub or residing on a server…).

I appreciate your offer to help!

[sunny]

there really is no need to add this to the app. There is a useable web interface available when you click on the api in top of this screen. It allows you to download the historical data in cvs. Yes that page is geared towards developers, but it really is easy enough to download the data from there.

[Lux4rd0]

@sunny - yep! I forgot about the ability to switch between JSON and CSV. Something like:

https://swd.weatherflow.com/swd/rest/observations/device/<<device_id>>?day_offset=1&format=csv&token=<<token>>

or with some time_start & time_end dates certainly would do that nicely! However - dumping that out for more than a few days may be a little tedious And per minute granularity falls off if you start selecting bigger swaths of time outside of 5 days.

Certainly a great first place to try things out and see if the data you’re looking for is something worth playing with. It’s essentially all of the per minute data from your device without any derived metrics or event data.

"device_id","type","bucket_step_minutes","timestamp","wind_lull","wind_avg","wind_gust","wind_dir","wind_interval","pressure","temperature","humidity","lux","uv","solar_radiation","precip","precip_type","strike_distance","strike_count","battery","report_interval","local_daily_precip","precip_final","local_daily_precip_final","precip_analysis_type"
122367,obs_st,1,1623715200,0.07,1.23,2.32,348,3,989.4,31,48,10477,0.63,87,0,0,0,0,2.79,1,0,0,0,1,
122367,obs_st,1,1623715260,0.14,0.97,2.19,0,3,989.4,31,48,10392,0.62,86,0,0,0,0,2.79,1,0,0,0,1,
122367,obs_st,1,1623715320,0,0.61,1.78,353,3,989.4,31,48,10299,0.62,85,0,0,0,0,2.79,1,0,0,0,1,

Here’s a sample of my data if you want to just plug it into Excel as a CSV.

(If you have like 5 Tempests, a few Air and Skys and 1200 days of data - this probably won’t work so well for you…)

[sunny]

yeah, doing this by hand and selecting the high res data, by downloading it day by day, wouldn’t be fun. However, I suspect that you simply could do this once a day, and just keep up to date.

[Lux4rd0]

Tempest is fun enough that doing this by hand only offsets the overall experience by just a little bit

[thegumnut]

Thanks for the feedback @Lux4rd0 and @sunny . I hadn’t really paid attention to the API button on this webpage before. OK, so now I can download (and have downloaded some) data manually and no, it really isn’t fun (even allowing for the “Tempest fun offset”. Given I’m looking to get all the historical data that’s available, it could get a lot less fun in a hurry. And I don’t think this approach is really addressing my original request for a user-friendly method. Maybe I could sit down and automate this, but that’s really not what I had in mind with the original feature request: something that’s easily accessible in the UI.

Given that both the app and the web interface allow a user to display historical data in a graphical format, and to scroll through it - so the data is available to generate the graph - it seemed pretty straightforward to have a “download” button right there to simply download the data to a local file.

[sunny]

you want to have an interface for manual downloading data in cvs, someone else wants it in userfriendly json, and perhaps yet someone else wants it in some binary format. Others want it to integrate with some other program. There are so many options, be happy there is at least some way to download it.
And what I was saying, once you downloaded it all you probably want to download the extra data the next day, and the day after that etc, so you might want to automate it anyway. For that purpose the API is there. My suggestion, use it.

[eric]

the API as proposed are pretty standard so everyone can handle the data rom as wished as said Sunny since there are so many end use cases you just can’t cover them all. Did you have a look into third party integrations section, maybe there is already something or you are a little patient and Lux4rd0 might come along soon? Not sure if he can do it quickly or not but he can for sure.

[Lux4rd0]

I’m going to play around and see what I can do but to @sunny and @eric points - for example - what if you have three Tempest, six Skys, and three Airs - and you wanted to pull a CSV file - how would you like that? One file with it listed by Device name, Station name, Station ID? How do you want to map fields that don’t exist across each of the devices? One file for each day and each device? etc.? You get the idea - as a community forum, what’s easy for one may be complex for another. I’ve loved being able to share and build interesting dashboards but what was once a couple of lines of code to listen in on local UDP sockets for raw JSON has turned into 7226 lines of code to provide “flexibility”. - but that’s my fun. LOL.

Like I mentioned, let me play with what I already have done and see about getting a CSV file dished out.

[Lux4rd0]

@thegumnut Try this example:

docker run --rm \
  --name=weatherflow-collector-remote-export \
  -e WEATHERFLOW_COLLECTOR_COLLECTOR_TYPE=remote-export \
  -e WEATHERFLOW_COLLECTOR_FUNCTION=export \
  -e WEATHERFLOW_COLLECTOR_TOKEN=127d99f7-3s88-4d7d-bd73-0ba962d181ec \
  -e WEATHERFLOW_COLLECTOR_EXPORT_DAYS=20 \
  -v $(pwd):/weatherflow-collector/export \
  lux4rd0/weatherflow-collector:latest

Change the WEATHERFLOW_COLLECTOR_TOKEN to your own personal access token and the days to the number of previous days you want to export. It’ll create a CSV file for each of your devices.

More details here:

lux4rd0/weatherflow-collector

[Lux4rd0]

[thegumnut]

Thanks @Lux4rd0 ! I’ve finally found some time to do some self-training on GitHub, Docker, Grafana… (none of which I was familiar with.) It’s been an interesting journey and thanks for sending me on it! Your scripts worked perfectly and I managed to generate a CSV with all the data from my Tempest (back to December last year when I installed it). Next step is to do some data analysis - starting with calculating the actual solar radiation to compare it with my predictions (made when I was looking to install solar panels). I might post a tentative result here if I get one.

[thegumnut]

[Lux4rd0]

Nice - I’m glad that it was helpful!

[nintanga1]

Where is it i can’t find it?

[alexanderholsteinson]

This is a must have feature