- 20: EVEs, Site stn1_west, Thu 12-Sep-2002 16:40:22 MDT, Stn1 working again
Station 1 died Sep 11, at about 00:05 MDT (interesting timing).
Steve Oncley checked that day, and found the power supply to
be dead, again.
I went up there today with a new supply. After connecting the new
supply, the battery dragged the voltage from 13.9 to under 6 V.
Since the battery was probably damaged, or would take a long time
to get to sufficient voltage to power the station, I removed
the battery and lugged it back.
I also taped and raised off the ground all extension cord junctions
from the transformer to stn1.
The station was up at 13:00 MDT, but archive and covar were not
restarted until 16:00 and 16:30.
Archive had been killed when I made a change a few days ago,
and covar was getting errors writing to nc_server, because
nc_server had closed the connection due to lack of data.
- 16: EVEs, Site all, Thu 05-Sep-2002 08:47:03 MDT, pam stations up
9/4/02 afternoon ~14-15:30 MDT.
The PAM stations were down because both A/C power supplies
died; probably from lightning.
The dead supplies were replaced with identical units at
both Stn1/3. The stations started up normally and RF to
ASTER was ok.
I added a ground to help protect the replacement supplies.
- 11: EVEs, Site all, Tue 03-Sep-2002 15:14:30 MDT, pam stations out
stn3 died on Aug 31, 18:14 MDT, and stn1 died Sep 1, 10:13 MDT.
John and Tony are going up there today (Sep 3) to figure out
what is wrong.
- 6: EVEs, Site all, Fri 16-Aug-2002 18:46:18 MDT, EVE bug causing serial errors and lost data
There is an EVE bug related to the new contents of
fast-out channel 4. EVE is still reporting a length of 2, and it
should be 3, because it also contains a 1 byte station number.
I verified this, both by looking at the fast out data, and looking
at the code in fastOutput.c
This is why we see large numbers of serial errors in the
check_aster output.
It also causes more problems for station 3 than station 1.
After reading the timetag, channel number=4 and length=2 (incorrect)
it then reads the next 2 bytes as data and expects the next byte to
be 0x03 (ETX). In the case of station 1, the next byte is 0x1,
and so the ingestor records a serial error, then looks forward
for the ETX, which is the next byte. So for station 1 no data is lost,
but we see serial errors.
For station 3, the ingestor gets an ETX when it expects one, but
it is actually the station number because both are 0x3.
The next byte is actally the ETX, but it treats that as part of
the next sample's timetag. This next sample will then be garbage,
which is why we're losing sonic data, and seeing CSAT3 diag values
of 16 (indicating a missing sample).
- 3: EVEs, Site none, Wed 31-Jul-2002 13:12:44 MDT, EVE processing load estimate w/10hz sonics
7/25/02
Just for giggles, I decided to look at the impact of running the
'COVARDP' stuff and a couple of other permutations for the Niwot cfg.
NOTE the config used was preliminary and necessarily involved simulating
3 'uw' sonics instead of the actual csat/2-ati because of limitations
with running the 'fake-sensor' ingest.
Test #1
---------
The fake uw_sonic ran at '10hz' and was paralleled in to the 3 different
sonic ports needed for Niwot. Two had a 'krypton' and thus had covar/avgs
for that in addition to the normal parameters. The third had
covar/avgs for only the normal components. No other sensors were
attached (gps, 2-trh, baro). The response of the console was ok and
EVE was able to keep up.
Note the inefficiency/cost of running the covardps processing really
ate cpu cycles: 55% for processing the sonics, plus 31% for
interrupt servicing.
NAME ENTRY TID PRI total % (ticks) delta % (ticks)
-------- -------- ----- --- --------------- ---------------
Fastout _fastoutTask 13c874 12 1% ( 190505) 1% ( 438)
UW1-AIn _uwSonicIn 139bac 12 20% ( 2766422) 20% ( 6160)
UW2-AIn _uwSonicIn 13597c 12 20% ( 2736774) 20% ( 6107)
UW3-AIn _uwSonicIn 133bfc 12 15% ( 2062943) 15% ( 4614)
tIdleCpu _idleCpu 130538 250 9% ( 1240177) 9% ( 2776)
INTERRUPT 31% ( 4317957) 32% ( 9961)
TOTAL 96% (13525715) 97% ( 30534)
2012 207 08:40:02 01
u v w ts h2o us vs ws cnt freq
UW1= 5.27 -5.27 1.06 25.00 4.46 20.0 20.0 20.0 3141 10.5
0.02 -0.02 0.00 0.00 0.02 0.0 0.0 0.0 0 0.0
0.00 0.00 0.00 0.00
u v w ts h2o us vs ws cnt freq
UW2= 5.28 -5.28 1.06 25.00 5.65 20.0 20.0 20.0 3135 10.5
0.02 -0.02 0.00 0.00 0.02 0.0 0.0 0.0 0 0.0
0.00 0.00 0.00 0.00
u v w ts h2o us vs ws cnt freq
UW3= 5.28 -5.28 1.06 25.00 20.00 20.0 20.0 3127 10.4
0.02 -0.02 0.00 0.00 0.02 0.0 0.0 0.0 0 0.0
Test #2
--------
Then I decided to try the 'same' config without covars and only
averaging of the sonic counts:
NAME ENTRY TID PRI total % (ticks) delta % (ticks)
-------- -------- ----- --- --------------- ---------------
Fastout _fastoutTask 140c80 12 1% ( 1454) 1% ( 477)
UW1-AIn _uwSonicIn 13dfb8 12 8% ( 7769) 8% ( 2582)
UW2-AIn _uwSonicIn 139d88 12 9% ( 8645) 9% ( 2868)
UW3-AIn _uwSonicIn 138008 12 6% ( 6334) 6% ( 2132)
tIdleCpu _idleCpu 12aee8 250 38% ( 35068) 38% ( 11750)
INTERRUPT 32% ( 30115) 32% ( 10004)
TOTAL 95% ( 91588) 95% ( 30525)
Notice the huge increase of cpu cycles freed up, and of course as expected
the interrupts stayed about the same.
Test #3a/b
--------
Knowing how scientists are, I thought I'd try two 20hz sonic's with full
covardps/avgs: as expected, it does not work
Then I tried two 20hz sonics without any covar/avgs: EVE is on the
borderline, but appears to be able to keep up even though almost 50% of
the time is spent in interrupt processing as expected.
Conclusion:
-----------
I tried the gps and it was a minimal load. The sbus sensor
processing should not add much to the cpu load.
EVE should be OK with all 3 sonics running at 10hz even with or without
covars/etc (including fasoutput). Based on an estimate of raw character
processing and the final test above, it might be possible, to run 1 sonic
at 20hz and the other 2 at 10hz provided there is no covar stuff.
5 hz = piece-of-cake.
For the actual Niwot config, we will include the COVAR's
plus fastout to compare EVE's computations with the base.