If you’re open to trying something radically different, there is a pending pull request from @tkeffer that might resolve corrupted packet issues…
1 Like
thanks @vreihen will give his master a shot
@tkeffer i switched to your branch of the UDP driver, i’m still randomly catching an error that shuts down weewx. this happens on both versions of the driver any ideas where i should be looking?
Aug 2 23:51:11 weewx weewx[20332] CRITICAL main: Caught unrecoverable exception:
Aug 2 23:51:11 weewx weewx[20332] CRITICAL main: **** ‘utf-8’ codec can’t decode byte 0xaa in position 2: invalid start byte
Aug 2 23:51:11 weewx weewx[20332] CRITICAL main: **** Traceback (most recent call last):
Aug 2 23:51:11 weewx weewx[20332] CRITICAL main: **** File “/usr/share/weewx/weewxd”, line 154, in main
Aug 2 23:51:11 weewx weewx[20332] CRITICAL main: **** engine.run()
Aug 2 23:51:11 weewx weewx[20332] CRITICAL main: **** File “/usr/share/weewx/weewx/engine.py”, line 188, in run
Aug 2 23:51:11 weewx weewx[20332] CRITICAL main: **** for packet in self.console.genLoopPackets():
Aug 2 23:51:11 weewx weewx[20332] CRITICAL main: **** File “/usr/share/weewx/user/weatherflowudp.py”, line 292, in genLoopPackets
Aug 2 23:51:11 weewx weewx[20332] CRITICAL main: **** for udp_packet in self.gen_udp_packets():
Aug 2 23:51:11 weewx weewx[20332] CRITICAL main: **** File “/usr/share/weewx/user/weatherflowudp.py”, line 318, in gen_udp_packets
Aug 2 23:51:11 weewx weewx[20332] CRITICAL main: **** m0 = m0.decode(‘utf-8’)
Aug 2 23:51:11 weewx weewx[20332] CRITICAL main: **** UnicodeDecodeError: ‘utf-8’ codec can’t decode byte 0xaa in position 2: invalid start byte
Aug 2 23:51:11 weewx weewx[20332] CRITICAL main: **** Exiting.
Is your Pi connected to the network wired or wirelessly? My initial thought is that the UDP packet is being mangled/corrupted due to network congestion on the local subnet. The UDP protocol is a blind send, with no handshaking or packet verification…
it’s Ubuntu running on a QNAP NAS, hard wired. i’m slowly commenting out sensor maps to see if it is a specific map causing the issue
this server ran interceptor for accurite atlas for months before converting it to WF
once i get through he sensor maps and if it is still happenign will purge the VM and start over
[WeatherFlowUDP]
driver = user.weatherflowudp
log_raw_packets = False
udp_address =
# udp_address = 0.0.0.0
# udp_address = 255.255.255.255
udp_port = 50222
udp_timeout = 90
share_socket = False
[[sensor_map]]
outTemp = air_temperature.ST-00015203.obs_st
outHumidity = relative_humidity.ST-00015203.obs_st
pressure = station_pressure.ST-00015203.obs_st
#lightning_strikes = lightning_strike_count.ST-00015203.obs_st
#avg_distance = lightning_strike_avg_distance.ST-00015203.obs_st
outTempBatteryStatus = battery.ST-00015203.obs_st
windSpeed = wind_speed.ST-00015203.rapid_wind
windDir = wind_direction.ST-00015203.rapid_wind
#luxXXX = illuminance.ST-00015203.obs_st
#UV = uv.ST-00015203.obs_st
rain = rain_accumulated.ST-00015203.obs_st
windBatteryStatus = battery.ST-00015203.obs_stThe error is happening in the line that decodes the raw UDP packet, long before the sensor_map comes into play.
Is the QNAP connected to a gig port, or 100Mbps?
Are you running WeeWX in a Docker container on the QNAP, as a virtual machine, or native on QNAP’s Linux platform???
gig port, running as a VM
the interesting thing is the error changes after changing to @tkeffer updated version
master
Jul 27 07:59:39 weewx weewx[4992] CRITICAL main : Caught unrecoverable exception:
Jul 27 07:59:39 weewx weewx[4992] CRITICAL main : **** a bytes-like object is required, not ‘str’
Jul 27 07:59:39 weewx weewx[4992] CRITICAL main : **** Traceback (most recent call last):
Jul 27 07:59:39 weewx weewx[4992] CRITICAL main : **** File “/usr/share/weewx/user/weatherflowudp.py”, line 310, in genLoopPackets
Jul 27 07:59:39 weewx weewx[4992] CRITICAL main : **** m0 = str(m[0],‘utf-8’).replace(",null",",None") # Python 3
Jul 27 07:59:39 weewx weewx[4992] CRITICAL main : **** UnicodeDecodeError: ‘utf-8’ codec can’t decode byte 0xaa in position 2: invalid start byte
updated driver
Aug 2 23:51:11 weewx weewx[20332] CRITICAL main : Caught unrecoverable exception:
Aug 2 23:51:11 weewx weewx[20332] CRITICAL main : **** ‘utf-8’ codec can’t decode byte 0xaa in position 2: invalid start byte
Aug 2 23:51:11 weewx weewx[20332] CRITICAL main : **** Traceback (most recent call last):
Aug 2 23:51:11 weewx weewx[20332] CRITICAL main : **** File “/usr/share/weewx/weewxd”, line 154, in main
I’d run the same code on a real computer rather than on a NAS, just to take that variant out of the picture. If you have a raspi toss that one online and see if you can see the NAS and the pi both fail simultaneously. That would say a lot I’d think, as would the pi working and the NAS failing…
i might have a pi2 laying around, the NAS is not a cheap one, its quad core with 16gb of ram
Not questioning the speed or quality of the NAS, but I’ve always though that running stuff on a storage appliance is a bit of a kludge. Trying to throw in the same software on more of a reference platform usually helps figure out if it’s the software or the platform.
If you have a PC/Mac/Laptop/etc. that you can even spin up a Vagrant/VirtualBox VM on, that would work fine. But a model-B pi runs weewx fine too if you don’t throw too many heavyweight skins like Belchertown at it.
had a brand new B still int he box, working on it now. weewx ran fine with the interceptor driver for months
The interceptor driver uses TCP (with handshake and error correction), AFAIK…
so this is interesting both the PI and VM crashed
Aug 3 16:40:25 weewx weewx[1576] DEBUG weewx.reportengine: Report 'RSYNC' not enabled. Skipping.
Aug 3 16:42:54 weewx weewx[1576] INFO weewx.engine: Main loop exiting. Shutting engine down.
Aug 3 16:42:54 weewx weewx[1576] INFO weewx.engine: Shutting down StdReport thread
Aug 3 16:42:54 weewx weewx[1576] DEBUG weewx.engine: StdReport thread has been terminated
Aug 3 16:42:54 weewx weewx[1576] DEBUG weewx.restx: Shut down CWOP thread.
Aug 3 16:42:54 weewx weewx[1576] DEBUG weewx.restx: Shut down PWSWeather thread.
Aug 3 16:42:54 weewx weewx[1576] DEBUG weewx.restx: Shut down Wunderground-PWS thread.
Aug 3 16:42:54 weewx weewx[1576] DEBUG weewx.restx: Shut down StationRegistry thread.
Aug 3 16:42:54 weewx weewx[1576] CRITICAL __main__: Caught unrecoverable exception:
Aug 3 16:42:54 weewx weewx[1576] CRITICAL __main__: **** 'utf-8' codec can't decode byte 0xaa in position 2: invalid start byte
Aug 3 16:42:54 weewx weewx[1576] CRITICAL __main__: **** Traceback (most recent call last):
Aug 3 16:42:54 weewx weewx[1576] CRITICAL __main__: **** File "/usr/share/weewx/weewxd", line 154, in main
Aug 3 16:42:54 weewx weewx[1576] CRITICAL __main__: **** engine.run()
Aug 3 16:42:54 weewx weewx[1576] CRITICAL __main__: **** File "/usr/share/weewx/weewx/engine.py", line 188, in run
Aug 3 16:42:54 weewx weewx[1576] CRITICAL __main__: **** for packet in self.console.genLoopPackets():
Aug 3 16:42:54 weewx weewx[1576] CRITICAL __main__: **** File "/usr/share/weewx/user/weatherflowudp.py", line 292, in genLoopPackets
Aug 3 16:42:54 weewx weewx[1576] CRITICAL __main__: **** for udp_packet in self.gen_udp_packets():
Aug 3 16:42:54 weewx weewx[1576] CRITICAL __main__: **** File "/usr/share/weewx/user/weatherflowudp.py", line 318, in gen_udp_packets
Aug 3 16:42:54 weewx weewx[1576] CRITICAL __main__: **** m0 = m0.decode('utf-8')
Aug 3 16:42:54 weewx weewx[1576] CRITICAL __main__: **** UnicodeDecodeError: 'utf-8' codec can't decode byte 0xaa in position 2: invalid start byte
Aug 3 16:42:54 weewx weewx[1576] CRITICAL __main__: **** Exiting.
PI
Aug 3 21:42:54 raspberrypi weewx[10348] INFO weewx.engine: Main loop exiting. Shutting engine down.
Aug 3 21:42:54 raspberrypi weewx[10348] INFO weewx.engine: Shutting down StdReport thread
Aug 3 21:42:54 raspberrypi weewx[10348] DEBUG weewx.engine: StdReport thread has been terminated
Aug 3 21:42:54 raspberrypi weewx[10348] CRITICAL __main__: Caught unrecoverable exception:
Aug 3 21:42:54 raspberrypi weewx[10348] CRITICAL __main__: **** 'utf-8' codec can't decode byte 0xaa in position 2: invalid start byte
Aug 3 21:42:54 raspberrypi weewx[10348] CRITICAL __main__: **** Traceback (most recent call last):
Aug 3 21:42:54 raspberrypi weewx[10348] CRITICAL __main__: **** File "/usr/share/weewx/weewxd", line 154, in main
Aug 3 21:42:54 raspberrypi weewx[10348] CRITICAL __main__: **** engine.run()
Aug 3 21:42:54 raspberrypi weewx[10348] CRITICAL __main__: **** File "/usr/share/weewx/weewx/engine.py", line 188, in run
Aug 3 21:42:54 raspberrypi weewx[10348] CRITICAL __main__: **** for packet in self.console.genLoopPackets():
Aug 3 21:42:54 raspberrypi weewx[10348] CRITICAL __main__: **** File "/usr/share/weewx/user/weatherflowudp.py", line 292, in genLoopPackets
Aug 3 21:42:54 raspberrypi weewx[10348] CRITICAL __main__: **** for udp_packet in self.gen_udp_packets():
Aug 3 21:42:54 raspberrypi weewx[10348] CRITICAL __main__: **** File "/usr/share/weewx/user/weatherflowudp.py", line 318, in gen_udp_packets
Aug 3 21:42:54 raspberrypi weewx[10348] CRITICAL __main__: **** m0 = m0.decode('utf-8')
Aug 3 21:42:54 raspberrypi weewx[10348] CRITICAL __main__: **** UnicodeDecodeError: 'utf-8' codec can't decode byte 0xaa in position 2: invalid start byte
Aug 3 21:42:54 raspberrypi weewx[10348] CRITICAL __main__: **** Exiting.
UDP does not guarantee message delivery, but it does guarantee message integrity (through a checksum). So, most likely it is not the problem.
For whatever reasons, the base station is emitting datagrams that are not in UTF-8. I suppose that’s its right, but it is easy enough to create JSON encoded in UTF-8, or even 7-bit ASCII, by using escaped sequences. Personally, I’ve never heard of a version of JSON that did not use UTF-8.
I don’t know why the driver tries to decode the bytes object before decoding the JSON: the function json.loads() can take a bytes object. I guess just blindly copied from the original version of the driver.
It would also be interesting to see what is in those packets.
Give the version in my fork a try. It will log any datagrams it cannot decode.
I no longer have a Tempest, so I can’t check it. Hopefully, it works.
thanks @tkeffer. just installed the updated fork.
Thanks a lot for creating this excellent driver. It has been working well for my Tempest and Weewx 4.1.1. I have only one wish which I’m not sure the driver can be upgraded to handle this.
I usually use it as loop mode in Weewx, so it work perfectly with Belchertown skin. As I understand Tempest will send two different data set at different intervals. The wind data will come as a more frequent data set; and the others, e.g. temp, humdity, etc… will come as another data set at a less frequent interval.
Is it possible that the driver can send a one-format dataset at the wind data interval; instead of two different dataset. For example, wind data will come every 3 second; and other data (including wind data) will come every minute, if I’m not wrong. What I wish to have is the driver provides only, or as an option, the second dataset at 3 second interval. Off course wind data will change every 3 second, but other data will remain same until they change.
The archive data in Weewx will remain the same at 5 minute average update (which has nothing to do with this driver, I believe).
With this new option, it is easier to write a web interface/script to deal with only one-format dataset and have all the data refreshed every 3 second.
Thank you.
here is the log. strangely it did not exit weewx this time
Aug 5 23:45:13 weewx weewx[2741] INFO user.weatherflowudp: Unable to decode packet b'UU\xaa\xaa\xf4\x00\x00\x00\x00\x00P\x90\x08\x01\x05ZZM-SS7009D8-S\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00045c0696d834\x00\x00\x00D\xc0%\x13@\x98=xD\n\x00\x00\x00,\xdf\xff\xff799990301\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00V43.91.1.04\x00****\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x08\x00\x00\x00\x00\x01\x00\x9eP\x00@\x1f(#\x00\x00192.168.1.250\x00\x00\x00\x00\x00\x00\x00192.168.1.1\x00\x00\x00\x00\x00\x00\x00\x00\x00255.255.255.0\x00\x00\x00\x00\x00\x00\x0004:5c:06:96:d8:34\x00\x00\x0015AP73YX500BBDH\x00'
Aug 5 23:45:14 weewx weewx[2741] INFO user.weatherflowudp: Unable to decode packet b'UU\xaa\xaa\xf4\x00\x00\x00\x00\x00\x01\x90\x01\x19\x01bSD-H2605\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00c0210d777c6d\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00799990198\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00V8.0.1.70\x00\x00\x007999\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x01\x00\x00\x01\x00\x01\xb6P\x00@\x1f(#\x03\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00ZMD13DID9000738\x00'
Aug 5 23:45:14 weewx weewx[2741] INFO user.weatherflowudp: Unable to decode packet b'UU\xaa\xaa\xf4\x00\x00\x00\x00\x00\x01\x90\x01\x11\x01bSD-H2901\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x008420963204d8\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00799990198\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00V8.0.1.70\x00\x00\x007999\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x01\xb6P\x00@\x1f(#\x03\x00192.168.1.5\x00\x00\x00\x00\x00\x00\x00\x00\x00192.168.1.1\x00\x00\x00\x00\x00\x00\x00\x00\x00255.255.255.0\x00\x00\x00\x00\x00\x00\x0084:20:96:32:04:d8\x00\x00\x00ZMD13GNDA265955\x00'
Aug 5 23:45:14 weewx weewx[2741] INFO user.weatherflowudp: Unable to decode packet b'UU\xaa\xaa\xf4\x00\x00\x00\x00\x00\x01\x90\x01\x11\x01bSD-H2901\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00c0210deef639\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00799990198\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00V8.0.1.70\x00\x00\x007999\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x01\xb6P\x00@\x1f(#\x03\x00192.168.1.8\x00\x00\x00\x00\x00\x00\x00\x00\x00192.168.1.1\x00\x00\x00\x00\x00\x00\x00\x00\x00255.255.255.0\x00\x00\x00\x00\x00\x00\x00c0:21:0d:ee:f6:39\x00\x00\x00ZMD13F6DA001127\x00'
Aug 5 23:45:14 weewx weewx[2741] INFO user.weatherflowudp: Unable to decode packet b'UU\xaa\xaa\xf4\x00\x00\x00\x00\x00\x01\x90\x01\x11\x01bSD-H2901\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00c0210de8e5f6\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00799990198\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00V8.0.1.70\x00\x00\x007999\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x01\xb6P\x00@\x1f(#\x03\x00192.168.1.46\x00\x00\x00\x00\x00\x00\x00\x00192.168.1.1\x00\x00\x00\x00\x00\x00\x00\x00\x00255.255.255.0\x00\x00\x00\x00\x00\x00\x00c0:21:0d:e8:e5:f6\x00\x00\x00ZMD13EMDA000799\x00'
Aug 5 23:45:14 weewx weewx[2741] INFO user.weatherflowudp: Unable to decode packet b'UU\xaa\xaa\xf4\x00\x00\x00\x00\x00\x01\x90\x01\x11\x01bSD-H2901\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00c0210dedd7c2\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00799990198\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00V8.0.1.70\x00\x00\x007999\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x01\xb6P\x00@\x1f(#\x03\x00192.168.1.10\x00\x00\x00\x00\x00\x00\x00\x00192.168.1.1\x00\x00\x00\x00\x00\x00\x00\x00\x00255.255.255.0\x00\x00\x00\x00\x00\x00\x00c0:21:0d:ed:d7:c2\x00\x00\x00ZMD13F6DA000432\x00'
Aug 5 23:45:14 weewx weewx[2741] INFO user.weatherflowudp: Unable to decode packet b'UU\xaa\xaa\xf4\x00\x00\x00\x00\x00\x01\x90\x01\x19\x01bSD-H2605\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00c0210d878e16\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00799990198\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00V8.0.1.70\x00\x00\x007999\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x01\x00\x00\x01\x00\x01\xb6P\x00@\x1f(#\x03\x00192.168.1.7\x00\x00\x00\x00\x00\x00\x00\x00\x00192.168.1.1\x00\x00\x00\x00\x00\x00\x00\x00\x00255.255.255.0\x00\x00\x00\x00\x00\x00\x00c0:21:0d:87:8e:16\x00\x00\x00ZMD13E4D9002345\x00'
The unknown datagram is gibberish to me, although it does contain hints of an IP origin. Perhaps it means something to the WF support team.
The program didn’t exit this time because the patch just logs the offending datagram instead of exiting.
In any case, as I said, I no longer have a Tempest, nor am I the primary author of the driver, so I don’t think I can be of much more help.
1 Like
This would be a great question for @wfsupport or maybe @dsj directly - ask them what is that UDP packet and where is it referenced in the UDP API documentation. Alternately ask if it’s indicative of a hardware problem on your station, as I think you’re the only person who’s reported this kind of thing.
Where I’m concerned is the ip addresses in those messages. They continually change. Almost wonder if you got hacked somehow.
I see 192.168.1.1 (your router?) but also see hosts of 192.168.1.7, 192.168.1.10, 192.168.1.46, 192.168.1.8, 192.168.1.5 as well as strings saying V43.91.1, V8.0.1.70, SD-H2901, SD-H2605 almost like something is walking your network using udp/50222 and itemizing your network.
Which would be bad. Real bad.
Definitely contact @wfsupport pronto. Consider trying to match up the ip addresses above with known boxes on your network, as well as those version strings with hostnames or ids or firmware versions etc. for stuff on your network like your router or access points etc.
But it was me and I saw that, at a minimum I’d reset my cable modem and access point and run a network sniffer to look for corrupted systems. Looks like weewx is doing a great job of acting like a sniffer for that need, FWIW.
(update - quick google search says SD-H2901 and SD-H2605 are wifi cameras (beware - they phone home to china!, V43.91.1 is a Zmodo camera firmware version, V8.0.1.70 lines up with a VisualCADD version and a Chinese app ACDSee Pro) so I’m guessing your wifi cams are walking all over your network big time).
1 Like