Archive Time Buckets - 180/30/5/1 minute bucket ranges

Developers
1

Hopefully a quick question on pulling previous archive data when using the time filters. When I read:

“Observation data at a one minute time resolution is available for a time range that is five days or less.” I took that to mean “5 days of one-minute time resolution data” - but it looks like it’s really “up to 5 days”. Is there an easier way of knowing how many days out I should request data to get each of the resolutions available? If I go out past 30 days, it’s 180-minute buckets. From 6-30 days is 30-minute buckets. from 1-5 days it’s 5 minutes. And 1 day or less it’s 1 minute. Does that seem correct? Will there ever be 1-minute intervals for more than the last 24 hours?

Thanks!

2

Can you share the link to where you were reading this? I couldn’t find it from searching.

The last time I played with this, the length of the archive bucket depended on the length of the data window you were requesting, not how long ago that window occured. For example, you can get one minute data for observations recorded last week/month/year, as long as you don’t request more than 24 hours worth of data at once.

3 Likes
3

actually you can get 1 minute data as well.
If you go to Tempest API and authorize, find your device id and request for example data from 40 days ago (https://swd.weatherflow.com/swd/rest/observations/device/110506?day_offset=40&format=csv)
I’ll get:

"device_id","type","bucket_step_minutes","timestamp","wind_lull","wind_avg","wind_gust","wind_dir","wind_interval","pressure","temperature","humidity","lux","uv","solar_radiation","precip","precip_type","strike_distance","strike_count","battery","report_interval","local_daily_precip","precip_final","local_daily_precip_final","precip_analysis_type"
110506,obs_st,1,1614729600,0,0,0,0,15,1028.6,4.1,94,0,0,0,0,0,0,0,2.48,1,0,,,0,
110506,obs_st,1,1614729660,0,0,0,0,15,1028.6,4,94,0,0,0,0,0,0,0,2.49,1,0,,,0,
110506,obs_st,1,1614729720,0,0,0,0,15,1028.6,4,94,0,0,0,0,0,0,0,2.49,1,0,,,0,
110506,obs_st,1,1614729780,0,0.11,0.45,178,15,1028.5,4,94,0,0,0,0,0,0,0,2.48,1,0,,,0,
110506,obs_st,1,1614729840,0,0,0,0,15,1028.5,4,94,0,0,0,0,0,0,0,2.49,1,0,,,0,
110506,obs_st,1,1614729900,0,0,0,0,15,1028.5,4,94,0,0,0,0,0,0,0,2.49,1,0,,,0,
110506,obs_st,1,1614729960,0,0,0,0,15,1028.5,4,94,0,0,0,0,0,0,0,2.48,1,0,,,0,

The forth number is the unix timestamp in seconds, and you can see it goes up by 60 seconds.

2 Likes
4

it is mentioned in the description of the api call. Tempest API

1 Like
5

This response makes total sense. Appreciate the nudge it thinking about this differently. What I was doing was keeping my “time_end” date “now” and my “time_start” rolling back over time (by days). The buckets I mentioned were seen as I went from 1 day to N days out. I didn’t chunk the request by individual days. :slight_smile:

I was also thinking of this just in an export fashion - where I was thinking “give me everything” and not being so targeted. But this makes way more sense from an “actively asking for information in real-time” for the exact time that I want. Which - LOL - now has me re-envisioning just using a live JSON connection instead of importing data into my TSDB. (I’ll probably do both!!)

Also makes sense why there’s an offset for days. Thanks @sunny - I was always calculating a timestamp before the request!

Thank you for the nudge in the right direction!!

2 Likes
6

Hi @Lux4rd0 ,
Your thoughts are what I have found works well for me so I posted where I am at with my approach, see:

cheers Ian :slight_smile:

1 Like
7

For sure!! I spent a little bit of time on this yesterday, and it worked out well for me:

weatherflow_api_only
weatherflow_api_only1714×699 142 KB

About 90% of what I wanted to build works well with live API data - the rest deals more with transformations and usability. And to test using OAuth rather than tokens for better-shared security. But it helped me get a better idea of what I can do at least!!

Nice job on your build as well!!

1 Like
8

I fix some issues when I convert the json from the API into my json. I notice that you are displaying lots of zero minimum wind readings. They may be correct, but there is an issue with minimum wind speed and glitches of zero which are known errors. So if I am calculating the minimum wind speed for a 10 min period then I do not include a single zero reading into the calculation for that period. As my code reads the values to create my own json file of converted values it only uses a zero speed if it followed another zero.
Cheers Ian :slight_smile:

2 Likes
9

good… even better because you seem to be using a fixed period of 10 minutes, unlike weatherflow’s implementation which changes the period depending on the zoom level of the graph.

10

I use different time periods depending on the time span of the graph and how crowded the icons become. I mentioned 10 minutes as an example of my technique to reduce the zero wind errors.

11

in my opinion wind lull should be defined as the minimum wind during some period (say 10 minutes). When zooming out, one should average those values. Weatherflow currently takes the minimum of a larger period. As a result of this, if you want to see how lull varies over the seasons, you are out of luck, as it is just a graph of zero’s.

1 Like
12

I’d love to understand this scenario more as long-term metrics is something I’m certainly interested in but just now getting some data. (Is this from local 3-second local polling or just the 1-minute averages?)

Where are the zeros happening? (My example is 3 months at 6-hour increments. Do I need to dive into 1-minute data? And over what time?

wind_3_months_6_hours
wind_3_months_6_hours2170×894 213 KB

13

your example looks great.
but look at what weatherflow does to the lull. When I zoom out it becomes all zero.
thats because there is apparently always some zero measurement during the day.
Weatherflow’s graph just doesn’t communicate the perceived lull during the day.
Note that a zero during the day, might be a single bad measuremend as @iladyman mentioned or it might be a real period sometime during the day with zero wind.

but yours doesn’t show that problem. You also display more data, which is good (weatherflow only displays 32 timestamps in the graph).

Untitled-1
Untitled-1802×938 247 KB

14

Your top page of 4 graphs had minimum wind showing flat line of zero. But this latest chart does not show the zeros and is called lulls rather tthan minimum. I do not know what you are using for the lull amounts.
For my graphs first I ask myself what is important to me. I want to know how good the wind is for paragliding. We talk about gust factor which is how variable the speed is and when it is too gusty I do not fly. I look at the graph and I can see how gusty it is by comparing the average and the maximums over time. Then the minimum causes me to have insufficient lift to soar but I also see it in the detailed chart rather than reading the single amount.
When I was younger studying power available for wind mills then I would require different detail. The power is proportional to the square of the speed so using an average and a max and min over a time period is not exactly the power available. If I wanted to know available power then I would use the 3 sec data and add the squares, and also I might adjust the power for direction due to the windmill not turning as quick as the gusts from different directions.
If I was to use Sunny’s suggestion I would create another variable named something like average minimum to display.
It depends on your purpose. I guess when I am paragliding I can look at my graph of 1min resolution at the lowest of the average wind speeds as my indication if I will not have enough lift to stay up. And I can ignore the minimum sourced from the lowest 3 sec speed.
The reason I ignore independent zero minimum speeds is due to glitches which are errors. From my testing I discovered glitches on Skys occuring a couple of times a day or so. I have not tested Tempests for glitches.
When the wind is strong all day I do not want a glitch to cause a zero minimum for the day.
Cheers Ian :slight_smile:

2 Likes
15

when I was younger, I studied physics and during my study we had to do about 100 different experiments. One involved a windmill. If I remember correctly (but I might be mistaken) the power from the windmill is proportional with the wind speed to the third power, not the square of the wind speed.

anyway, I couldn’t quickly find a general definition of wind lull, but it is supposed to be a short (perhaps a few minutes) period of low wind. Weatherflow’s way of taking the minimum during a 24 hour period (when zoomed out), doesn’t match that definition. My proposal would be to take a sliding window of say 10 minutes and take the minimum during that period. (that still updates every minute). When aggregating the data (when zooming out), just average those measurements.

16

Oh yes I was slack with my memory…
I was wrong, yes it is more than only the square:

thanks Sunny as always :slight_smile:

1 Like