# Smart Weather Archive for Cheap Local Storage
**URL:** https://community.tempest.earth/t/smart-weather-archive-for-cheap-local-storage/617
**Category:** Tempest Owners
**Tags:** wfarchive, archivesw
**Created:** [March 19, 2018, 3:29am UTC](https://community.tempest.earth/t/smart-weather-archive-for-cheap-local-storage/617 "2018-03-19T03:29:45Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@GaryFunk](https://community.tempest.earth/u/GaryFunk)
#### Post date: [March 19, 2018, 3:29am UTC](https://community.tempest.earth/t/smart-weather-archive-for-cheap-local-storage/617/1 "2018-03-19T03:29:45Z")
The one big thing that’s missing from the Smart Weather is local storage. The system should be cheap, readily available and easy to configure.
The “Smart Weather Archiver” runs on a RaspberryPi 3 B+ and stores all the UDP data sent by a WeatherFlow “Smart Weather Station.”
---
### Author:  [@michael](https://community.tempest.earth/u/michael)
#### Post date: [March 19, 2018, 4:37am UTC](https://community.tempest.earth/t/smart-weather-archive-for-cheap-local-storage/617/2 "2018-03-19T04:37:36Z")
Do you mean storing historical data locally?
If so - my solution is to have Logstash ‘catch’ the udp broadcast packets, transform them and then post the data to an Elasticsearch host. Granted - not ‘cheap’ or ‘easy to configure’ - but it works well enough for my purposes (especially since I had elasticsearch already running anyway).
---
### Author:  [@dan.gealt](https://community.tempest.earth/u/dan.gealt)
#### Post date: [March 19, 2018, 4:44am UTC](https://community.tempest.earth/t/smart-weather-archive-for-cheap-local-storage/617/3 "2018-03-19T04:44:32Z")
Hi Gary,
Are you thinking ALL data including the Rapid Wind and how long do you foresee keeping the data?
Dan.
---
### Author:  [@GaryFunk](https://community.tempest.earth/u/GaryFunk)
#### Post date: [March 19, 2018, 4:55am UTC](https://community.tempest.earth/t/smart-weather-archive-for-cheap-local-storage/617/4 "2018-03-19T04:55:07Z")
@michael, Yes, locally.
@dan.gealt, Most of the data.
I am thinking the obs\_air, obs\_sky, obs\_breathe packets along with lightning strike. I see no reason to store rapid wind. Storage time length will depend on the size of the storage media.
Once storage starts running running low old data could be reduced to “low” and “high” records.
---
### Author:  [@vreihen](https://community.tempest.earth/u/vreihen)
#### Post date: [March 19, 2018, 11:54am UTC](https://community.tempest.earth/t/smart-weather-archive-for-cheap-local-storage/617/5 "2018-03-19T11:54:36Z")
I’m partial towards weewx, and do not like re-inventing the wheel. It can run on a Raspberry Pi, Docker container, VirtualBox VM, and just about anywhere else where a Python interpreter is available. It does basic archiving to a Sqilite or MySQL database.
For a presentation/archiving layer, my choice is MeteoTemplate. (No offense to @weather34 intended, because his code would be my first choice for real-time viewing.) Jachym has put a lot of time/effort into MeteoTemplate, and is a stickler for proper/accurate reporting and statistics. MT also stores incoming data into a MySQL database, which can be pre-populated from WU data if you have any historical stuff there. With advanced warning that I have not installed updates in almost a year and some of the blocks are broken, here’s a link to my MT site: [http://ae3.homelinux.net:81/weather/indexDesktop.php](http://ae3.homelinux.net:81/weather/indexDesktop.php)
Of course, the “turnkey” thing is another issue. There’s nothing stopping anyone from bundling up a Raspbian/Docker/VirtualBox image and sharing it. Not something that I can commit to, but I am crazy enough to try writing a shell script to install/configure all of the required stuff on a raw Raspbian image once my new batch of Pi 3 B+'s arrive in a few days…
---
### Author:  [@dan.gealt](https://community.tempest.earth/u/dan.gealt)
#### Post date: [March 19, 2018, 12:05pm UTC](https://community.tempest.earth/t/smart-weather-archive-for-cheap-local-storage/617/6 "2018-03-19T12:05:25Z")
I have worked with a few udp loggers, but they either are free and require too much work or they are rather pricey and still don’t fit the weather instrumentation / data requirements very well.
I like the idea from @vreihen a lot and I, too, have ordered the new rPi (not sure when it might get here, though) and would certainly try that route.
---
### Author:  [@Weather34](https://community.tempest.earth/u/Weather34)
#### Post date: [March 19, 2018, 12:51pm UTC](https://community.tempest.earth/t/smart-weather-archive-for-cheap-local-storage/617/7 "2018-03-19T12:51:18Z")
no offense taken its good to have a choice otherwise it would be pretty mundane . ive always said meteotemplate is a good solution for historical analyst data not many seem to acknowledge that comment. for me its all about realtime and thats reflected in what i do im not the guy to discuss historical analysis data, spent 15 years working in database maintenance for a large network food distribution company in the uk i litterally hate numbers in that respect … for what its worth the template i design uses mysql via meteobridge its all automated based on one simple insert mysqli script and one simple automated daily truncate script to keep the database not becoming excessive and easily maintained for future maintenance if required.my host loves me as it automatically truncates and exports all daily data into a monthly,yearly csv file and starts a new one at midnight it never grows beyond 800k i use it quite successfully for two years and there is a full back option for those who dont have enknowledge of sql to use data stored from wu. so cut me out of this one ill leave it to you guys to discuss about best practices for stored data.
---
### Author:  [@bpaauwe](https://community.tempest.earth/u/bpaauwe)
#### Post date: [March 19, 2018, 3:19pm UTC](https://community.tempest.earth/t/smart-weather-archive-for-cheap-local-storage/617/8 "2018-03-19T15:19:13Z")
I think I’m with @vreihen too even though I’ve never used weewx ti seems like about as complete/turnkey as you’re going to find.
Storage, in general is cheap, at least for this type of data. I’ve been storing my electric use / solar production info in mysql database for almost 10 years now and at 1 minute intervals (and also hours, days, months, and years) I’m still under 1GB of disk space. So \< $100 for an SSD, a RPi and weewx and you’re probably set for 50 years worth of data storage. The hardware will probably fail before you run out of storage space.
I’ve been collecting weather data in a mysql database also since about 2013 and that’s about 300MB. I use my own software to grab the data and push into the database.
---
### Author:  [@GaryFunk](https://community.tempest.earth/u/GaryFunk)
#### Post date: [March 19, 2018, 4:00pm UTC](https://community.tempest.earth/t/smart-weather-archive-for-cheap-local-storage/617/9 "2018-03-19T16:00:36Z")
I am thinking the hardware will be a Raspberry Pi.
The software will include MySQL and Python.
The back end will listen for UDP and add the packets to tables. It will add any new fields to the tables as necessary. It will create any new tables as necessary when a new device is added.
The front end will allow for quires such as “high temperature from date to date,” “low temperature on date,” “average precipitation from data to date,” “total strike count from date to date.”
Data will be returned in JSON format.
---
### Author:  [@dsj](https://community.tempest.earth/u/dsj)
#### Post date: [March 19, 2018, 9:53pm UTC](https://community.tempest.earth/t/smart-weather-archive-for-cheap-local-storage/617/10 "2018-03-19T21:53:02Z")
Note, we considered adding an SD card slot to the Hub for the purpose of storing data locally. We rejected that idea due to the cost and complexity it would add, coupled with our strategy of making the data open and easy to get at so developers like you could do that (and more) yourself. It’s really amazing to see that happening!
---
### Author:  [@vinceskahan](https://community.tempest.earth/u/vinceskahan)
#### Post date: [March 19, 2018, 10:07pm UTC](https://community.tempest.earth/t/smart-weather-archive-for-cheap-local-storage/617/11 "2018-03-19T22:07:53Z")
> There’s nothing stopping anyone from bundling up a Raspbian/Docker/VirtualBox image and sharing it.
I’ve had my docker/vagrant setups for weewx in simulator mode up on github for years now, if you wanted to fiddle with weewx on the side while awaiting hardware delivery.
A couple examples:
- Docker is [GitHub - vinceskahan/weewx-docker: weewx in docker for a variety of os](https://github.com/vinceskahan/weewx-docker)
- Vagrant/Virtualbox is [GitHub - vinceskahan/weewx-ubuntu-vagrant: DEPRECATED - DO NOT USE - see my weewx-vagrant repo instead](https://github.com/vinceskahan/weewx-ubuntu-vagrant)
Of course you’d want to do a more long-lived variant with a fully edited weewx.conf file and whatever skin(s) you want installed. I have other repos up there if you wanted to surf for other examples. Scripting an install end-to-end is pretty straightforward.
I have over 1M records in my weewx database which is still under 200MB in size. Data goes back to Nov-2006 just to give you an idea of how small the sqlite3 database for weewx is.
---
### Author:  [@GaryFunk](https://community.tempest.earth/u/GaryFunk)
#### Post date: [March 19, 2018, 10:47pm UTC](https://community.tempest.earth/t/smart-weather-archive-for-cheap-local-storage/617/12 "2018-03-19T22:47:52Z")
> [@dsj](#):
>
> we considered adding an SD card slot to the Hub for the purpose of storing data locally.
If you had there would have to be a way to query the data from the Hub. It would be a support nightmare as not everyone would be happy.
I will build one to function for my needs and give others the ability to modify the output. Even then not everyone will like it.
---
### Author:  [@todayweather4](https://community.tempest.earth/u/todayweather4)
#### Post date: [March 20, 2018, 12:41am UTC](https://community.tempest.earth/t/smart-weather-archive-for-cheap-local-storage/617/13 "2018-03-20T00:41:57Z")
does the Hub have a port for the SD card? or do you talk about the new Hub? because I’ve check my hub and did not find any port fir sd card, so I’m confused.
---
### Author:  [@GaryFunk](https://community.tempest.earth/u/GaryFunk)
#### Post date: [March 20, 2018, 12:45am UTC](https://community.tempest.earth/t/smart-weather-archive-for-cheap-local-storage/617/14 "2018-03-20T00:45:10Z")
@todayweather4
Here is what David wrote.
> [@dsj](#):
>
> Note, we considered adding an SD card slot to the Hub for the purpose of storing data locally. **We rejected that idea** due to the cost and complexity
---
### Author:  [@GaryFunk](https://community.tempest.earth/u/GaryFunk)
#### Post date: [March 29, 2018, 8:08am UTC](https://community.tempest.earth/t/smart-weather-archive-for-cheap-local-storage/617/15 "2018-03-29T08:08:45Z")
Hardware:
RaspberryPi 3 B+
Software:
Rasbian Stretch Lite
Python
MariaDB
Using Python I now have code that:
listens to port 50222 in non-exclusive mode
writes data to a SQL table
The next step is to actually parse the UDP data into the proper SQL statement.
Then turn the code into a real application and sell it for $1.98.
---
### Author:  [@dan.gealt](https://community.tempest.earth/u/dan.gealt)
#### Post date: [March 29, 2018, 1:28pm UTC](https://community.tempest.earth/t/smart-weather-archive-for-cheap-local-storage/617/16 "2018-03-29T13:28:44Z")
Pi 3 B+ arriving here today. Where do you want me to send the $1.98 ??? 👍
---
### Author:  [@bpaauwe](https://community.tempest.earth/u/bpaauwe)
#### Post date: [March 29, 2018, 3:22pm UTC](https://community.tempest.earth/t/smart-weather-archive-for-cheap-local-storage/617/17 "2018-03-29T15:22:54Z")
> [@GaryFunk](#):
>
> Hardware:
> RaspberryPi 3 B+
>
> Software:
>
> Rasbian Stretch Lite
> Python
> MariaDB
>
> Using Python I now have code that:
> listens to port 50222 in non-exclusive mode
> writes data to a SQL table
>
> The next step is to actually parse the UDP data into the proper SQL statement.
>
> Then turn the code into a real application and sell it for $1.98.
You’re missing a couple of steps
- ???
- Profit!
Good job!
Are you writing the database to an SD card? If so, do you have any concerns with that?
---
### Author:  [@GaryFunk](https://community.tempest.earth/u/GaryFunk)
#### Post date: [March 29, 2018, 3:23pm UTC](https://community.tempest.earth/t/smart-weather-archive-for-cheap-local-storage/617/18 "2018-03-29T15:23:31Z")
This thing needs a name. We have to call it something.
---
### Author:  [@GaryFunk](https://community.tempest.earth/u/GaryFunk)
#### Post date: [March 29, 2018, 3:42pm UTC](https://community.tempest.earth/t/smart-weather-archive-for-cheap-local-storage/617/19 "2018-03-29T15:42:02Z")
Yes, it writes to the SD Card. I recommend a very fast 32GB card. I have a lot of testing to do as you know.
I’m still not sure Python is the best choice.
---
### Author:  [@dan.gealt](https://community.tempest.earth/u/dan.gealt)
#### Post date: [March 29, 2018, 4:47pm UTC](https://community.tempest.earth/t/smart-weather-archive-for-cheap-local-storage/617/20 "2018-03-29T16:47:10Z")
To be serious . . . “Smart Weather Archiver” . . . and to be not so serious . . . “Weather Pi - baked on premises”.
[Next page](https://community.tempest.earth/t/smart-weather-archive-for-cheap-local-storage/617.md?page=2)