Tempest obs_st not being transmitting and rapid_wind missing transmissions over UDP when offline

Developers Third-Party Integrations
, ,
1

I currently have my tempest connected in standard form through a router with internet access and am able to access the UDP data stream via an Arduino connected to the same router. The rapid_wind data streams at 3 seconds on average as expected (Table 1)~88% stream at 3 seconds, ~6% stream at 2 seconds, and ~6% stream at 4 seconds) and obs_st steams at 60 seconds.

Table 1: Seconds between rapid_wind data stream internet

Seconds between data receive Percent occurrence
2 ~6
3 ~88
4 ~6

However, when I remove internet access (i.e. disconnect the cat5 from the cable modem) the rapid_wind data stream at 4.1 seconds on average (Table 2) and obs_st does not stream at all.

Table 2: Seconds between rapid_wind data stream without internet

Seconds between data receive Percent occurrence
2 ~1.2
3 ~60.6
4 ~1.2
5 ~4.2
6 ~28
7 ~4.8

When I reconnect internet, the rapid_wind data stream returns to normal, and the obs_st backfills before returning to normal as well.

Here are some things I have tried to be sure I can replicate the behaviour under different circumstances:

  1. Two different routers (Linksys e900 with dd-wrt, and Netgear Orbi with stock firmware.)
  2. Monitor stream using Netcat - same behaviour.
  3. Stream UDP to Tempest iOS app - this causes the app to crash.
  4. Stream BLE to Tempest iOS app - this causes the app to crash.

Has anyone else run into this issue while using their tempest offline? I am looking to run several Tempests in the field without access to internet, but it seems that the one minute data does not stream unless the device can phone home.

Many thanks for any tips or suggestions on what to try next!

2

interesting find. I would expect udp to be send out, even when the hub doesn’t have internet. This might be a bug. How does it look with the app open and connected via bluetooth?

3

Let me try to bring some details (won’t answer all questions but might already help understand)

The hub sends out data via UDP as long it has an IP assigned (lan), regardless it can send data to the servers (wan side) If for some reason the hub doesn’t have an ip (lan) assigned, the UDP data stream stops.
Someone will recheck this in the lab to be sure during the recent upgrades no bug creeped in.

The backfill behavior is expected. The HUB transmits the st_obs message over UDP and to our servers at the same time. (hence why this backfilling happens when you re establish internet).

Now the app part is more dark. Can you elaborate a bit more how you do points 3 and 4 please. Also what version of the app do you use ?
Does the phone have a connection to the internet (via cellular or wifi ?) when you try points 3 and 4 ?

Thanks for your help in understanding your problems. This will help technicians to give correct feedback and or solve any bugs.

4

Thank you for the prompt to double check the app behaviour, @sunny. I’ve got some more clarity on that listed below.

@eric, Thank you for that information. I will try my best to touch on the points you make.

When I disconnect internet, I leave the LAN setup completely in tact. I only remove the cable modem from the mix. When I log into the router, I can see that the hub still maintains it’s DHCP assigned IP address. Also, I still receive some data, just not the obs_st and rapid_wind less frequently.

Yes, the backfill makes sense. I should still expect to see the obs_st without internet and then again when I reconnect internet - so in a dataset spanning a period without internet followed by a period with internet, I should expect to see each obs_st data stream twice, once during the normal 1 minute cycle, and once as a backfill. Am I understanding that correctly?

I will do my best to explain each wireless connectivity scenario I have tried with the app.

Here’s my version information:

  • iPhone 7 Plus (MN632LL/A)
  • iOS 14.7 (18G5042c)
  • Tempest app version 4.19 (501)
  • Hub firmware revision 170
  • Station hardware revision 1
  • Station firmware revision 156
  1. With the setup disconnected from WAN, and with both the hub and phone still connected to WiFi/LAN (and bluetooth enabled), the app will load the screen in Figure 1. When I tap the station icon in the upper right, the icon will turn dark blue from a light blue. (Figure 2) The app will then freeze similar to the behaviour in this post

IMG_6761.PNG
IMG_6761.PNG1242×2208 115 KB

Figure 1: Hub and phone connected to LAN without WAN access.

IMG_6762.PNG
IMG_6762.PNG1242×2208 115 KB

Figure 2: Hub and phone connected to LAN without WAN access after station icon as been tapped.

  1. With the hub connected to WiFi/LAN without internet access, I am able to access the data stream over bluetooth so long as my phone is able to phone home using cellular. (Figure 3)

IMG_6763.PNG
IMG_6763.PNG1242×2208 280 KB

Figure 3: Hub connected to WiFi without internet, but phone connected to internet via cellular, and to hub via bluetooth.

  1. However, with my phone is airplane mode (with cellular and WiFi antennas disabled, and bluetooth enabled, the app does not display data and freezes. (Figure 4)

IMG_6764.PNG
IMG_6764.PNG1242×2208 112 KB

Figure 4: Hub connected to WiFi without internet, and phone connected only to bluetooth (WiFi and cellular disabled).

  1. If I start with tempest app with only cellular enabled, allow the data to load via bluetooth, and then disable cellular, I am able to keep receiving data via bluetooth without internet. (Figure 5) If I background the app of restart it, I will get the same results as in 3 above.

IMG_6766.PNG
IMG_6766.PNG1242×2208 276 KB

Figure 5: Hub connected to WiFi without internet, and phone connected to cellular and bluetooth to start data stream, then cellular disabled.

I have also tried each of the scenarios again with my VPN disabled and the results are the same.

Finally, I was fortunate enough to have a storm pass through while I was collecting data on my Arduino. I noticed that the tempest app correctly showed lightning and rain start events leading up to and during the storm, but the UDP data did not show any evt_precip or evt_strike data streams.

I am attaching the UDP data stream with the hub connected to WiFi. Ignore the DEBUG line.

20210701_UDPcolletion.txt (169.4 KB)

I also tested for the lightning and rain event signals with the hub and phone connected to the WiFi first with internet to load the data, then I disconnected internet and the lightning events as displayed in the app went from ever ~4-10 seconds at 20-35 km to every 60 seconds at 1 km (which would be the same data supplied by obs_st that I collect in my UDP backfill). This makes it seem as thought the evt_precip and evt_strike data streams are only available when online. Is this correct? (Okay, I see that @eric mentions this will be fixed in upcoming firmware in this post. Thanks!) I am on firmware revision 156 already - thanks @sunny.

5

but you are already an 156 so this should work for you.

6

You’re right! I am so used to seeing 143, that it didn’t even register.

Well, there goes that theory. Thank you for pointing that out.

7

The st_obs should continue to be sent over UDP when the Internet is down. I’ve investigated the issue and the reason for the is because the HUB is busy trying to connect to our servers and isn’t able to process the data from the Tempest in time. We’ll have a fix for the issue in the next firmware release.

The backfill behavior is expected. The HUB transmits the st_obs message over UDP and to our servers at the same time.

4 Likes
8

Has there been a firmware update to address this? Is there any workaround to poke the hub to make it send obs_st again? I can reliably make this happen (stop sending obs_st and device_status when the internet is down) and i want to keep getting those messages no matter what the internet happens to be doing at the time.

thanks!

jake

3 Likes