NIWOT02: Logbook Entries

NIWOT02: EVEs Messages: 5 Entries..

Return to Logbook Contents Page
Entry Date Title Site Author #Graphics
20 Thu 12-Sep-2002Stn1 working againstn1_westmaclean
16 Thu 05-Sep-2002pam stations upallmilitzer
11 Tue 03-Sep-2002pam stations outallmaclean
6 Fri 16-Aug-2002EVE bug causing serial errors and lost dataallmaclean
3 Wed 31-Jul-2002EVE processing load estimate w/10hz sonicsnonemilitzer


20: EVEs, Site stn1_west, Thu 12-Sep-2002 16:40:22 MDT, Stn1 working again

Station 1 died Sep 11, at about 00:05 MDT (interesting timing). 

Steve Oncley checked that day, and found the power supply to
be dead, again.

I went up there today with a new supply.  After connecting the new
supply, the battery dragged the voltage from 13.9 to under 6 V.

Since the battery was probably damaged, or would take a long time
to get to sufficient voltage to power the station, I removed
the battery and lugged it back.

I also taped and raised off the ground all extension cord junctions
from the transformer to stn1.

The station was up at 13:00 MDT, but archive and covar were not
restarted until 16:00 and 16:30.

Archive had been killed when I made a change a few days ago,
and covar was getting errors writing to nc_server, because
nc_server had closed the connection due to lack of data.

16: EVEs, Site all, Thu 05-Sep-2002 08:47:03 MDT, pam stations up
9/4/02 afternoon ~14-15:30 MDT.
The PAM stations were down because both A/C power supplies
died; probably from lightning.
The dead supplies were replaced with identical units at
both Stn1/3.   The stations started up normally and RF to
ASTER was ok.
I added a ground to help protect the replacement supplies.
11: EVEs, Site all, Tue 03-Sep-2002 15:14:30 MDT, pam stations out
stn3 died on Aug 31, 18:14 MDT, and stn1 died Sep 1, 10:13 MDT.

John and Tony are going up there today (Sep 3) to figure out
what is wrong.
6: EVEs, Site all, Fri 16-Aug-2002 18:46:18 MDT, EVE bug causing serial errors and lost data
There is an EVE bug related to the new contents of
fast-out channel 4. EVE is still reporting a length of 2, and it
should be 3, because it also contains a 1 byte station number.

I verified this, both by looking at the fast out data, and looking
at the code in fastOutput.c

This is why we see large numbers of serial errors in the
check_aster output. 

It also causes more problems for station 3 than station 1.

After reading the timetag, channel number=4 and length=2 (incorrect)
it then reads the next 2 bytes as data and expects the next byte to
be 0x03  (ETX).  In the case of station 1, the next byte is 0x1,
and so the ingestor records a serial error, then looks forward
for the ETX, which is the next byte. So for station 1 no data is lost, 
but we see serial errors.

For station 3, the ingestor gets an ETX when it expects one, but
it is actually the station number because both are 0x3.  
The next byte is actally the ETX, but it treats that as part of 
the next sample's timetag.  This next sample will then be garbage,
which is why we're losing sonic data, and seeing CSAT3 diag values 
of 16 (indicating a missing sample).






3: EVEs, Site none, Wed 31-Jul-2002 13:12:44 MDT, EVE processing load estimate w/10hz sonics
7/25/02 

Just for giggles, I decided to look at the impact of running the
'COVARDP' stuff and a couple of other permutations for the Niwot cfg.

NOTE the config used was preliminary and necessarily involved simulating
3 'uw' sonics instead of the actual csat/2-ati because of limitations
with running the 'fake-sensor' ingest.


Test #1
---------
The fake uw_sonic ran at '10hz' and was paralleled in to the 3 different
sonic ports needed for Niwot.  Two had a 'krypton' and thus had covar/avgs
for that in addition to the normal parameters.  The third had
covar/avgs for only the normal components.   No other sensors were
attached (gps, 2-trh, baro).   The response of the console was ok and
EVE was able to keep up.
Note the inefficiency/cost of running the covardps processing really
ate cpu cycles: 55% for processing the sonics, plus 31% for
interrupt servicing.


NAME          ENTRY         TID   PRI   total % (ticks)  delta % (ticks)
--------     --------      -----  ---   ---------------  ---------------
Fastout      _fastoutTask 13c874   12     1% (  190505)    1% (     438)
UW1-AIn      _uwSonicIn   139bac   12    20% ( 2766422)   20% (    6160)
UW2-AIn      _uwSonicIn   13597c   12    20% ( 2736774)   20% (    6107)
UW3-AIn      _uwSonicIn   133bfc   12    15% ( 2062943)   15% (    4614)
tIdleCpu     _idleCpu     130538  250     9% ( 1240177)    9% (    2776)
INTERRUPT                                31% ( 4317957)   32% (    9961)
TOTAL                                    96% (13525715)   97% (   30534)

2012 207 08:40:02 01
        u       v       w      ts    h2o     us     vs     ws    cnt   freq
UW1=   5.27  -5.27   1.06  25.00   4.46   20.0   20.0   20.0   3141  10.5
       0.02  -0.02   0.00   0.00   0.02    0.0    0.0    0.0      0   0.0
       0.00   0.00   0.00   0.00
        u       v       w      ts    h2o     us     vs     ws    cnt   freq
UW2=   5.28  -5.28   1.06  25.00   5.65   20.0   20.0   20.0   3135  10.5
       0.02  -0.02   0.00   0.00   0.02    0.0    0.0    0.0      0   0.0
       0.00   0.00   0.00   0.00
        u       v       w      ts    h2o     us     vs     ws    cnt   freq
UW3=   5.28  -5.28   1.06  25.00  20.00          20.0   20.0   3127  10.4
       0.02  -0.02   0.00   0.00   0.02    0.0    0.0    0.0      0   0.0


Test #2
--------
Then I decided to try the 'same' config without covars and only
averaging of the sonic counts:

NAME          ENTRY         TID   PRI   total % (ticks)  delta % (ticks)
--------     --------      -----  ---   ---------------  ---------------
Fastout      _fastoutTask 140c80   12     1% (    1454)    1% (     477)
UW1-AIn      _uwSonicIn   13dfb8   12     8% (    7769)    8% (    2582)
UW2-AIn      _uwSonicIn   139d88   12     9% (    8645)    9% (    2868)
UW3-AIn      _uwSonicIn   138008   12     6% (    6334)    6% (    2132)
tIdleCpu     _idleCpu     12aee8  250    38% (   35068)   38% (   11750)
INTERRUPT                                32% (   30115)   32% (   10004)
TOTAL                                    95% (   91588)   95% (   30525)

Notice the huge increase of cpu cycles freed up, and of course as expected
the interrupts stayed about the same.


Test #3a/b
--------
Knowing how scientists are, I thought I'd try two 20hz sonic's with full
covardps/avgs: as expected, it does not work

Then I tried two 20hz sonics without any covar/avgs: EVE is on the
borderline, but appears to be able to keep up even though almost 50% of
the time is spent in interrupt processing as expected.


Conclusion:
-----------
I tried the gps and it was a minimal load.  The sbus sensor
processing should not add much to the cpu load.

EVE should be OK with all 3 sonics running at 10hz even with or without
covars/etc (including fasoutput).   Based on an estimate of raw character
processing and the final test above, it might be possible, to run 1 sonic
at 20hz and the other 2 at 10hz provided there is no covar stuff.
5 hz = piece-of-cake.

For the actual Niwot config, we will include the COVAR's
plus fastout to compare EVE's computations with the base.