NIWOT02: Logbook

NIWOT02: Chronological Logbook Entries

Return to Logbook Top Page
Entry Date Title Site Author #Graphics
33 Thu 14-Nov-2002sonic data swapped 3m.s1 & 6m.s1stn1_westoncley1
32 Wed 23-Oct-2002merging SDR dataallmaclean
31 Wed 23-Oct-2002SDR dataallmaclean
30 Tue 22-Oct-2002chronologyallmaclean
29 Mon 07-Oct-2002Hydra visit/ teardowndelany
28 Mon 07-Oct-2002Hydra visitdelany
27 Mon 07-Oct-2002Time delays for the Hydra intakesalldelany
26 Mon 07-Oct-2002Final Intercomparison using Brittscalstn2_centraldelany
25 Mon 23-Sep-2002LICOR INTERCOMPARISON SYSTEMstn2_centraldelany
24 Tue 17-Sep-2002Site visitdelany
23 Tue 17-Sep-2002Hydra Check liststn2_centraldelany
22 Fri 13-Sep-2002Stn1 CSAT up / Batt addedstn1_westmilitzer
21 Thu 12-Sep-2002CSAT at stn1 downstn1_westmaclean
20 Thu 12-Sep-2002Stn1 working againstn1_westmaclean
19 Thu 12-Sep-2002Cal cycle changestn2_centraldelany
18 Thu 12-Sep-2002Hydra visit and Check liststn2_centraldelany
17 Tue 10-Sep-2002Hydra visitstn2_centraldelany
16 Thu 05-Sep-2002pam stations upallmilitzer
15 Wed 04-Sep-2002Hydra sample period increased againstn2_centraldelany
14 Wed 04-Sep-2002Hydra sample period increasedstn2_centraldelany
13 Wed 04-Sep-2002Hydra operationalstn2_centraldelany
12 Wed 04-Sep-2002fixed opto22 command problemstn2_centralmaclean
11 Tue 03-Sep-2002pam stations outallmaclean
10 Tue 03-Sep-2002opto22.s2stn2_centralmaclean
9 Thu 22-Aug-2002Changed datel averagingstn2_centralmaclean
8 Fri 16-Aug-2002Changed freewave frequency keystn1_westmaclean
7 Fri 16-Aug-2002Replaced broken TRH ventilator hornstn2_centralmaclean
6 Fri 16-Aug-2002EVE bug causing serial errors and lost dataallmaclean
5 Fri 16-Aug-2002Freewave configallmilitzer
4 Fri 16-Aug-2002Replaced Ironicsnonemaclean
3 Wed 31-Jul-2002EVE processing load estimate w/10hz sonicsnonemilitzer
2 Fri 28-Jun-2002Adam testingnonemaclean
1 Fri 28-Jun-2002Logbook creatednonemaclean

33: LOG, Site stn1_west, Thu 14-Nov-2002 16:55:52 MST, sonic data swapped 3m.s1 & 6m.s1
Prompted by Sean, I've determined that the ATI sonic signals were swapped 
between the 3m and 6m sensors at stn1.  My confirmation is a scatterplot of
T.6m with the signals called tc.3m and tc.6m, which I've shown in the attached
file for stn1 and stn3.  It is obvious that T.6m is better related to the 
signal currently labeled tc.3m.  Sean also observes that the wind and 
temperature profiles are better behaved using "swapped" data.

Thus, I'm about to change prep.config to reflect this situation.  We'll have
to rerun covars soon.

32: LOG, Site all, Wed 23-Oct-2002 09:48:40 MDT, merging SDR data

Here are the scripts used in preparing and merging the SDR data with the RF
data.  They are on $PAM/scripts unless noted otherwise.  Passing the -4
option fixes the channel 4 length problem.

set_project NIWOT02

cd to_some_scratch_space

mkdir -p stn3/disk1

cd stn3/disk1
# read flash drive
cp /mnt/pccard/niwot* .

# this script creates directory new_names, containing symbolic
# links back to the actual SDR files.  The symbolic links have names
# like 020815.00000 indicating the actual start date of the file
assign_date_sdr -4

cd new_names

# strip ETX from SDR files, pass data into ingestor, which does
# necessary sscanf's on ASCII channels based on channel_config, and
# generates samples that ASTER data system can handle.  Script expects
# to read files called YYMMDD.HHMMSS so that it can determine the date.
# script runs archive, which connects to the ingestor and archives
# the files on $RAWDATADIR/all
convert_sdr -4 stn3 0208*

# Merge the sdr and rf data sets. This script assumes the sdr data is on 
# /scr/isff/aster/projects/NIWOT02/raw_data/sdr
# the rf data is on
# /scr/isff/aster/projects/NIWOT02/raw_data
# It creates new merged files on
# /scr/isff/aster/projects/NIWOT02/raw_data/new


31: LOG, Site all, Wed 23-Oct-2002 09:27:37 MDT, SDR data
The SDR disks were not monitored closely during NIWOT02, since we had
RF communications.

However the SDR data will fill some holes:
1. Aug 9-14 at station 3 when the SDR was operating but
   the RF was not.
2. At the beginning of the project EVE, was writing channel 4 data
   (containing days since Jan 1, 1970 and station #) with a wrong
   length value of 2, when it should have been 3.  This caused
   the RF ingest program to loose data

John gave me two SDR (IBM microdrives) disks from each station.
Disk2 from station 1 only contains July lab data. Disk5 from stn1
has data starting Aug 14.

The two disks from station 3 have useful August data.  It appears that
the only disk swap was at stn3, on Aug 14.

Since the SDRs contain the early data, they must have quit writing
once they filled up, and (fortunately) did not delete old files.

Here are ls -l of each disk, followed by the time values provided by
$PAM/scripts/sdr_show_date -4

Stn1, disk #2
  ls -l:
  -rwxrwxr-x    1 maclean  sssf     19370467 Jan  1  2018 niwot101.dat
  -rwxrwxr-x    1 maclean  sssf     19611939 Jan  2  2018 niwot102.dat
  -rwxrwxr-x    1 maclean  sssf      6307840 Jan  2  2018 niwot103.dat

  file          date
  niwot101.dat  020725.221635   had chan 4 length problem (2 should be 3)
  niwot102.dat  020726.061629   had chan 4 length problem (2 should be 3)
  niwot102.dat  020726.141637   had chan 4 length problem (2 should be 3)

  Since the dates are July, this disk is of lab data.

Stn1 disk #5

  total 348072
  -rwxrwxr-x    1 maclean  sssf     55336649 Aug  5  1980 niwot102.dat
  -rwxrwxr-x    1 maclean  sssf     55223871 Aug  6  1980 niwot103.dat
  -rwxrwxr-x    1 maclean  sssf     55311905 Aug  7  1980 niwot104.dat
  -rwxrwxr-x    1 maclean  sssf     55262628 Aug  8  1980 niwot105.dat
  -rwxrwxr-x    1 maclean  sssf     55541780 Aug  9  1980 niwot106.dat
  -rwxrwxr-x    1 maclean  sssf     53141504 Aug  9  1980 niwot107.dat
  -rwxrwxr-x    1 maclean  sssf     26583040 Aug 11  1980 niwot108.dat
  drwxrwxr-x    2 maclean  sssf         8192 Aug 14 12:50 recycled

  file          date
  niwot102.dat 020814.203341    had chan 4 length problem (2 should be 3)
  niwot103.dat 020815.204005    had chan 4 length problem (2 should be 3)
  niwot104.dat 020816.204015    had chan 4 length problem (2 should be 3)
  niwot105.dat 020817.204011    had chan 4 length problem (2 should be 3)
  niwot106.dat 020818.204022    had chan 4 length problem (2 should be 3)
  niwot107.dat 020819.204023    had chan 4 length problem (2 should be 3)
  niwot108.dat 020820.194505    didn't need fix
  recycled/dd12.01z  020622.080124  IHOP data?
  recycled/dd13.32z  020622.003231  IHOP data?

  The RF archive contains data from 020814.212108.  However it suffers
  from the channel 4 problem so converting these files will gain some

Stn3 disk #1

  total 309760
  -rwxrwxr-x    1 maclean  sssf     41435136 Jan  1  2021 niwot10g.dat
  -rwxrwxr-x    1 maclean  sssf        57344 Dec 21  1914 niwot10h.dat
  -rwxrwxr-x    1 maclean  sssf     55762860 Jan  2  2010 niwot301.dat
  -rwxrwxr-x    1 maclean  sssf     55782410 Jan  3  2010 niwot302.dat
  -rwxrwxr-x    1 maclean  sssf     55777648 Jan  4  2010 niwot303.dat
  -rwxrwxr-x    1 maclean  sssf     55774014 Jan  5  2010 niwot304.dat
  -rwxrwxr-x    1 maclean  sssf     52591448 Jan  6  2010 niwot305.dat

  file          date
  niwot10g.dat  020730.232109   had chan 4 length problem (2 should be 3)
  niwot10h.dat  020731.173038   had chan 4 length problem (2 should be 3)
  niwot301.dat  020809.205547   had chan 4 length problem (2 should be 3)
  niwot302.dat  020810.205454   had chan 4 length problem (2 should be 3)
  niwot303.dat  020811.205456   had chan 4 length problem (2 should be 3)
  niwot304.dat  020812.205501   had chan 4 length problem (2 should be 3)
  niwot305.dat  020813.205507   had chan 4 length problem (2 should be 3)

  RF archive contains data from 020814.212236, so these files will help.

Stn3 disk #6
  -rwxrwxr-x    1 maclean  sssf     55546305 Jan  7  2010 niwot306.dat
  -rwxrwxr-x    1 maclean  sssf     55795404 Jan  8  2010 niwot307.dat
  -rwxrwxr-x    1 maclean  sssf     36233216 Jan  8  2010 niwot308.dat
  -rwxrwxr-x    1 maclean  sssf        98304 Jan  8  2010 niwot309.dat
  -rwxrwxr-x    1 maclean  sssf     55787828 Jan  9  2010 niwot30a.dat
  -rwxrwxr-x    1 maclean  sssf     55787385 Jan 10  2010 niwot30b.dat
  -rwxrwxr-x    1 maclean  sssf     55791484 Jan 11  2010 niwot30c.dat
  -rwxrwxr-x    1 maclean  sssf     15966208 Jan 12  2010 niwot30d.dat

  file          date
  niwot306.dat  020814.193618   had chan 4 length problem (2 should be 3)
  niwot307.dat  020815.193600   had chan 4 length problem (2 should be 3)
  niwot308.dat  020816.193606   had chan 4 length problem (2 should be 3)
  niwot309.dat  020817.111148   had chan 4 length problem (2 should be 3)
  niwot30a.dat  020817.111449   had chan 4 length problem (2 should be 3)
  niwot30b.dat  020818.111501   had chan 4 length problem (2 should be 3)
  niwot30c.dat  020819.111505   had chan 4 length problem (2 should be 3)
  niwot30d.dat  020820.111511   had chan 4 length problem (2 should be 3)
        IO error reading niwot30d from microdrive

  recycled/dd12.01z  020622.080124  IHOP data (we have these files on CD)
  recycled/dd13.32z  020622.003231  IHOP data

30: LOG, Site all, Tue 22-Oct-2002 15:20:26 MDT, chronology

Here is a chronology based upon the data archive, system logs
(and anybody's notes or recollection?):

Aug  7 17:41 MDT  Reboot of aster. No more reboots are seen in the system
                  log until Aug 24  stn1,3 ingest timeouts until Aug 14 15:18
                  Therefore aster must have been installed in the CUFF trailer
                  on Aug 7.

Aug  9 14:55 MDT  Date of earliest station 3 (east) SDR data

Aug 14 14:33 MDT  Date of earliest station 1 (west) SDR data

Aug 14 11:54 MDT  First cosmos message in system log: first archive file
Aug 14 15:18 MDT  last select timeouts for stn1 and 3, receiving RF data


29: CO2, Site , Mon 07-Oct-2002 16:17:21 MDT, Hydra visit/ teardown
30 Sep''02, Arrived 10:05
Steve S, Kirt, Tony, John, Steve O, also Jielun

All three pumps OK
Descicant 2/3 pink, 1/3 blue
All aspiration flouting ball between 6.0 and 6.5

Licor A: sample/cal 6.5 in cal
         dry air 6.5
         Ref 7.5 
         P = 1.2 when pump off when in calibrate mode
           = 5.7 when in sample mode

Licor B:  sample/cal 6.5+ in cal 
          dry air 6.5
          Ref 3.0
          P = 1.4 when pump off in cal mode
            = 5.4 when in sample mode
Licors powered off at 10:21
No Brittcal gases

Ref 1500/5.6 no flow        LK 438863
Cal2 1400/5.9 no flow       270552
Spare ref capped            REE 222
Cal3 1660/5.8 no flow       K361224
Cal1 1510/6.1 no flow       414576

All Hydra components removed.
except the intake lines. The lines into the hydra box were clamped tightly 
in the strain relief. The top lines, A 1-9, were tagged with black panduit ties.

28: CO2, Site , Mon 07-Oct-2002 16:09:49 MDT, Hydra visit
24 Sep''02, Tuesday
Tony Delany and Don Lenschow
Both RAA pumps and dry air pump OK
All aspiration floating balls between 6.0 and 6.5. All adjusted to 6.25
Ref  1380/4.2
Cal2 1520/6.2
Spare ref Capped
Cal3 1750/5.8
Cal1 1630/5.6

Licor A: sample/cal on sample 65
         dry air 62
         ref 5, increased to 6
         P = 5.8 on sample
Licor B:  sample/cal on sample 64
          dry air 60
          ref 2, increased to 6
          P on sample 5.1

Hydra delay times determined. See other LOGBOOK entry
27: CO2, Site all, Mon 07-Oct-2002 10:07:33 MDT, Time delays for the Hydra intakes
24 Sep''02. Tony D and Don L visited Niwot02 site and undertook to measure the
time delays for the Hydra.

The AC power for the relays, and hence solenoids, was switched off. 
The AC power for the Opto22 and the 12 VDC for the sample valve and the 
sample pump were maintained.
The AC daisy-chain and the corresponding connections from the solenoids 
were disconnected. Zip cords were connected in their stead. Thus on the flip 
of a switch on the power block a single solenoid could be activated.

The flow through the buffer volumes was continuous and the choice of 
solenoid merely changed the choice of sampled volume. 
The solenoids were activated at the instant of an Opto22 transition and the 
instruction to breathe into the intake was also given, by radio at the 
instant of an Opto22 transition. Generally the buffer volume was sampled 
for 80 x 2 seconds befor the breath instruction was given. 
then 30 seconds of breathe was sampled. A 10 - 15 minute period was allocated 
to allow the pulse of CO2 to travel the length of the Hydra intake line.
 A table of delay times is gien in 
26: CO2, Site stn2_central, Mon 07-Oct-2002 09:47:22 MDT, Final Intercomparison using Brittscal
Sean Burns took these data.

       Tank Pressures with Valves
       Cylinder    Closed     Open
       --------    -------    ----- 
       A             1200     1200
       B             1320     1350
       C                0     1250
       D             1600     1600

       System       Time*     Event               IC Gas    Line
       ------      --------   -------------      --------  ------
       NCAR-A      14:24:45    Connect to LiCorA  -- Hydra was on S5 
       NCAR-A      14:25:30    step 4-way           A
       NCAR-A      14:28:30    step 4-way           B
       NCAR-A      14:31:30    step 4-way           C
       NCAR-A      14:34:30    step 4-way           D
       NCAR-A      14:37:30    3-way back to hydra          hydra on S6.

       NCAR-B      14:41:02    3-way to ICS         -       
       NCAR-B      14:41:30    step 4-way           A  
       NCAR-B      14:45:00    step 4-way           B
       NCAR-B      14:48:00    step 4-way           C
       NCAR-B      14:51:00    step 4-way           D
       NCAR-B      14:54:00    3-way back                   hydra on ?

       Time Check:  at 14:38:46 switch from s6-s7
       Note: Hydra Cal was just finishing as i arrived (~14:20).

       USGS Time Check-- 14:20:00 USGS = 15:20:04 (watch time, which was
                                             about 3 sec behind GPS time)

  USGS        15:17:20   3-way to ICS         on L5, P~509 torr
  USGS        15:18:30   run start            A  P?, CO2~431.1
  USGS        15:21:34   step 4-way           B  P~546 torr, CO2~403.8
  USGS        15:24:34   step 4-way           C  P~518 torr, CO2~389.5
  USGS        15:27:34   step 4-way           D  P~450, CO2~457.0
  USGS        15:28:30   still on D --> increased P to be near 515 torr.
  USGS        15:31:34   3-way back   USGS Time=14:06:01  on L7

  CU          16:09:40   to man, C2 on                 L4
  CU          16:11:24   3-way to ICS         
  CU          16:12:00   step 4-way           A   (P~8 psi)
  CU          16:16:00   step 4-way           B   (P~12 psi)
  CU          16:20:00   step 4-way           C   (P~24 psi)
  CU          16:24:00   step 4-way           D   (P~18 psi)
  CU          16:25:23   still on D,  increased P to 20 psi.   
  CU          16:28:00   step 4-way           A
  CU          16:30:00   step 4-way           B
  CU          16:32:00   step 4-way           C
  CU          16:36:00   3-way back to CU
  CU          16:37:15   C2 on, to auto            L1

  Time Check:  at 16:38:24 switch from L2-L3

        *times shown are from my watch which is off from GPS by about
         3-4, GPS= 19:41:00, watch=19:41:03.


  All the tank pressures and flows on the Hydra looked fine.  
...all the Britt intercomparison details are also posted on the MMM niwot
ridge webpage under "Project Timeline and Updates".  ie,

25: CO2, Site stn2_central, Mon 23-Sep-2002 11:32:02 MDT, LICOR INTERCOMPARISON SYSTEM
Sean Burns took these data.

        Tank Pressures with Valves
Cylinder    Closed     Open
--------    -------    ----- 
A                0     1250
B             1400     1490
C                0     1300
D                0     1610

System       Time*     Event               IC Gas    Line
------      --------   -------------      --------  ------
CU          15:00:28   to man, C2 on                 L1
CU          15:02:01   3-way to ICS         
CU          15:02:35   step 4-way           A  (P~4 psi)
CU          15:05:00   step 4-way           B  (P~12 psi)
CU          15:07:30   step 4-way           C  (P~20 psi)
CU          15:11:00   step 4-way           D  (P~20 psi)
CU          15:14:30   3-way back to CU
CU          15:16:00   C2 on, to auto                L4

USGS        15:33:30   3-way to ICS  USGS Time=14:33:30  on L10
USGS        15:33:40   run start            A  P~4, CO2~429.7
USGS        15:37:00   step 4-way           B  P~4, CO2~404.4
USGS        15:40:00   step 4-way           C  P~10, CO2~390.1
USGS        15:43:00   step 4-way           D  P~6, CO2~450.9
USGS        15:46:01   3-way back    USGS Time=14:06:01  on L7

NCAR-A      16:22:30   Connect to LiCorA  -- Hydra was on S4 
NCAR-A      16:23:30   step 4-way           A
NCAR-A      16:27:00   step 4-way           B
NCAR-A      16:29:30   step 4-way           C
NCAR-A      16:32:00   step 4-way           D
NCAR-A      16:35:00   3-way back to hydra          hydra on S2 (?).

NCAR-B      16:37:00   3-way to ICS         A       hydra on S6.
NCAR-B      16:38:00   step 4-way           A  (slight pressure adjust)
NCAR-B      16:41:00   step 4-way           B
NCAR-B      16:43:30   step 4-way           C
NCAR-B      16:46:00   step 4-way           D
NCAR-B      16:48:30   3-way back                   hydra on S4

Time Check:  at 17:00:03 switch from s4-s5
             at 17:01:23 switch from s5-s6

*times shown are from my watch which has been adjusted to be within
1 sec of GPS time.

24: LOG, Site , Tue 17-Sep-2002 15:40:22 MDT, Site visit
On 16 Sept Sean and Britt visited the Niwot site.
Here are Seans comments:

Sean Burns and Britt Stephens took these data.

Cylinder   Init. P (before opening valves)  Pressure (valves open)
A                0                                        1400
B             1310                                        1500
C             1400                                        1400
D             1405                                        1800

System      Time          Event          ICS Gas    Line
NCAR-A      15:04:00    Connect to LiCorA  -- Hydra was on S3  
NCAR-A      15:04:20    step 4-way           A
NCAR-A      15:07:23    step 4-way           B
NCAR-A      15:09:43    step 4-way           C
NCAR-A      15:11:53    step 4-way           D
NCAR-A    ---Hydra Cal started at about 3:11:58 and went till 15:24:30.
NCAR-A      15:24:40    stayed on D          D
NCAR-A      15:28:55    step 4-way           A  (went back to A for 3-min cal gas)
NCAR-A      15:32:03    3-way back to hydra          hydra on S6.

NCAR-B      15:35:55    3-way to ICS         A       hydra on S1 or S9.
NCAR-B      15:38:55    step 4-way           B
NCAR-B      15:41:55    step 4-way           C
NCAR-B      15:44:59    step 4-way           D
NCAR-B      15:48:35    3-way back                   hydra on S1

USGS        16:10:11    run start            A       on L7, USGS Time~= 15:01:24
USGS        16:13:02    step 4-way           B
USGS        16:15:52    step 4-way           C
USGS        16:18:50    step 4-way           D
USGS        ---checked LiCor response by switching from D to about 5-10sec 
               LiCor readout went from 455 to 430.
USGS        16:24:00    3-way back                   on L13 (?), USGS Time= 15:17:30
USGS        note:  time check, Licor USGS=15:14:00 when my watch= 16:20:30

CU          16:40:08   to man, C2 on                 L6
CU          16:47:45   3-way to ICS         A
CU          16:51:05   step 4-way           B
CU          16:54:05   step 4-way           C
CU          16:57:05   step 4-way           D
CU          17:01:40   C2 on, to auto                L2

*the times shown are from my watch which i know is about 5 minutes
fast (i should have corrected for this).  a comparison of my watch
time and GPS time is: GPS Time=16:12:00,  My Watch=16:16:29

next time i will adjust my watch to GPS time before doing the


23: CO2, Site stn2_central, Tue 17-Sep-2002 09:15:27 MDT, Hydra Check list
                              HYDRA CHECK LIST

Both RAA Gast pumps operating?
Dryer pump operating?

All aspiration floating balls between 6 and 6.5?. If not then adjust.

Note cylinder pressure and regulator pressure
Ref                 (eg 1750/4.9)
Cal 2               (eg 1560/5.9)
Spare Ref              capped
Cal 3               (eg 1830/6.2)
Cal 1               (eg 1760/6.1)

Hydra box: open top and watch the little lights to see that the cycle is 
progressing properly:
 ... cal3, cal2, cal1, Sample/s9,s8,s7,s6,s5,s4,s3,s2,s1, cal3, cal2, etc ...

Look at flow tubes on inside of door. Note values and,if necessary,adjust knob
     below guage to set at correct value.
Look at pressure at pump output. Note value and, if necessary, adjust the knob
     on the pressure controller T'd with the pressure guage.
Licor 1: Dry air                 =            (should be 65   )
         Ref                     =            (should be  5   )
         sample/cal              =            (should be 66   )
         P (when in sample mode) =            (should be  6.0 )

Licor 2: Dry air                 =            (should be 65   )
         Ref                     =            (should be  5   )
         sample/cal              =            (should be 66   )
         P (when in sample mode) =            (should be  6.0 )

Both tubes of desiccant.  Blue OK, If totally pink, replace.

Undertake a Brittcal on both Licors. 
Keep track of the "Sample" / "Calibration" sequence.
Fit each entire Brittcal into a single "Sample" period as overlapping with a 
"Calibration" causes some problems.

22: Sonic, Site stn1_west, Fri 13-Sep-2002 16:33:11 MDT, Stn1 CSAT up / Batt added
13Sep02, ~14:00 MDT

CSAT up.  At least it still was ~1.5 hours after doing the work.
It was still down upon arrival. Power cycled everything first (see
below) but it was still not coming in despite second
power cycle on the serializer.   When jumpered directly using
a different cable, the sonic responded and I could re-set it to 10hz.
Replaced serializer; then power cycled to make sure both came up OK.

Charge controller (and thus low-voltage-disconnect) and tiny 7AH battery
added.  I checked that the battery was getting charged and able to run
station by itself as well.

21: Sonic, Site stn1_west, Thu 12-Sep-2002 16:55:59 MDT, CSAT at stn1 down

The CSAT3 at 1m at station 1 is not working properly.  I am quite
sure it was generating good data after I powered up the station today,
but within the first hour EVE no longer reported data from it.

The CSAT3 command shows all NAs, and no csat data is coming
through fastout.

I tried several things, none of
which helped:
	power cycled the serializer several times
	power cycled eve
	power cycled the serializer again
	power cycled the sonic
	blew out some moisture from the front panel connector

If you do a talk to it from eve, it is transmitting binary
gibberish as it should, but somehow eve is not recognizing the
records.  It is as if it isn't sending the end-of-record bytes.

Using these commands:
it replies:

ET= 10 ts=i XD=d GN=443a TK=1 P=5 FK=0 RN=1 IT=1 DR=102 rx=2 fx=038 BX=0 AH=1  AT=0 RS=1 BR=0 RI=1 GO=00000 HA=0 6X=3 3X=2 PD=2 SD=0 ?d sa=1
WM=o ar=0 ZZ=0 DC=6  ELo=020 020 020 ELb=020 020 020 TNo=898 d TNb=998 JD= 007
C0o=-2-2-2 C0b=-2-2-2 RC=0 tlo=8 8 8 tlb=8 8 8 DTR=01740 CA=1 TD=  duty=028     AQ= 10 AC=1 CD=0 SR=1 X=0 MX=0 DT=02320 DTC=01160 RD=o ss=1 XP=2 RF=018 DS=007 SN0538 25oct01 HF=005 JC=3 CB=3 MD=5 DF=05000 RNA=1 rev 3.0a cs=10382 &=0 os= 

The only thing I know to look for is RS=1, which is correct.

To restart it, did:

It started spewing data, but EVE is not recognizing it.

20: EVEs, Site stn1_west, Thu 12-Sep-2002 16:40:22 MDT, Stn1 working again

Station 1 died Sep 11, at about 00:05 MDT (interesting timing). 

Steve Oncley checked that day, and found the power supply to
be dead, again.

I went up there today with a new supply.  After connecting the new
supply, the battery dragged the voltage from 13.9 to under 6 V.

Since the battery was probably damaged, or would take a long time
to get to sufficient voltage to power the station, I removed
the battery and lugged it back.

I also taped and raised off the ground all extension cord junctions
from the transformer to stn1.

The station was up at 13:00 MDT, but archive and covar were not
restarted until 16:00 and 16:30.

Archive had been killed when I made a change a few days ago,
and covar was getting errors writing to nc_server, because
nc_server had closed the connection due to lack of data.

19: CO2, Site stn2_central, Thu 12-Sep-2002 16:17:51 MDT, Cal cycle change
Gordon changed the calibration period and duration and the reset interval
At 14:19:
       the calibration period was changed from every 30 min to every 60 min.
       the calibration durations were changed from 2 min to 4 min                      the reset interval was changed from every 2.5 hrs to 5 hrs.
These changes do not modify the total cycle time of 1 hour.

At 16:15 this was checked back at FLAB and seen to be so.

18: CO2, Site stn2_central, Thu 12-Sep-2002 11:44:39 MDT, Hydra visit and Check list
Steve O and Tony D arrived at the NCAR tower at 11:30
RAA aspiration, all balls between 6 and 6.5
Cal2 valved off 1560/ 19 psi
Ref             1740/5 psi

The new AC cal valves had been plumbed in the day before but not electrically.
At 11:54 AC to all black relays off. (AC to Opto22 left on)
Electrical reconnection (pink relays replaced by black, DC jumpers removed, 
AC jumpers installed, TM10 wiring to Licor box barrier strips disconnected, 
New valve wiring connected to barrier strips) completed and AC power restored 
at 12:45.

12:24 Licor 1 P = 5.8 sample/cal = 65
      Licor 2 P = 5.0 sample/cal = 65

13:15 Licor 1 Britt-cal CalA regulator adjusted to bring Britts intake P to 5.8
                        CalB                  ditto
                        CalC                  ditto
                        CalD                  ditto
13:35 Licor 2 Britt-cal CalA regulator adjusted to bring Britts intake P to 5.0
                        CalB                  ditto
                        CalC                  ditto
                        CalD                  ditto
    Note the Britt-cal for Licor 2 interupted by Cal cycle. Extended Britt-cal

14:02 Filled descicator tubes with blue Dryrite, powered pump and adjusted 
     dry air flows to 6 for both Licors

Finished removing all the ss nipples on front of intake filters.

14:32 Returned from last of filter nipple removals
                              CHECK LIST

Both RAA Gast pumps OK
Dryer pump OK

All aspiration floating balls between 6 and 6.5

Ref                 1750/4.9
Cal 2               1560/5.9
Spare Ref           capped
Cal 3               1830/6.2
Cal 1               1760/6.1

Hydra box: open top and watch the little lights to see that the cycle is 
progressing properly:
           Sample/s1,s2,s3,s4,s5,s6,s7,s8,s9  , cal1 , cal2 , cal3 , 

Licor 1: Dry air =                 55
         Ref =                      5 ( incr to 7)
         sample/cal =              62 ( incr to 65)
         P (when in sample mode) =  5.8

Licor 2: Dry air =                 58
         Ref =                      5 
         sample/cal =              68
         P (when in sample mode) =  5.0

Descicant  Blue OK OK

14:46    Leave

17: CO2, Site stn2_central, Tue 10-Sep-2002 10:57:23 MDT, Hydra visit
Tony D and Britt S visited Niwot site on Thurs 5 Sep''02
Arrived 14:50 

Both Gast RAA Hydra aspirator pumps running.
All floating balls  between 550 and 650. Adjusted all balls to 60 +_ 3
on the 102-05N (st). This corresponds to 3100 ccpm uncorrected for P and T.

Gas cylinders, Ref  1850/4 psi
               Cal2 1700/7.2 psi

15:00 Inlets swopped A1 <-> A4. This was done to bring the 
Licor A NCAR 1m sample into time synch with the Licor B NCAR 1m sample.

Licor A sample/cal flow = 50 on N032-15(st), increased to 65 ( =  145     ccpm)
Licor B sample/cal flow.= 40 on N032-15(st), increased to 65 ( =  145     ccpm)
NB caltable for N032-15 (ST):
05 =8, 10 =12, 20 =27, 30 =45, 40 =65, 50 =97, 60 =125, 65 =145 ml min-1

Licor A ref flow = 3 on N032-15(st), increased to 5 ( =    8   ccpm)
Licor B ref flow.= 4 on N032-15(st), increased to 5 ( =    8  ccpm)

Licor A KNF pump P = 5.5 psi
Licor B KNF pump P = 4.5 psi

During the period 15:33 to 15:38 observed sequence sampl/cal1/cal2/cal3/sample.
Looked OK

!6:40 - 16:51 Britt's calibrator to Licor B, ran CalA, CalB, CalC and CalD
!6:53 - 17:03 Britt's calibrator to Licor A, ran CalA, CalB, CalC and CalD

16: EVEs, Site all, Thu 05-Sep-2002 08:47:03 MDT, pam stations up
9/4/02 afternoon ~14-15:30 MDT.
The PAM stations were down because both A/C power supplies
died; probably from lightning.
The dead supplies were replaced with identical units at
both Stn1/3.   The stations started up normally and RF to
ASTER was ok.
I added a ground to help protect the replacement supplies.
15: CO2, Site stn2_central, Wed 04-Sep-2002 13:08:41 MDT, Hydra sample period increased again
At 13:08 MDT the Hydra sample period was increase from 40s to 80s
The number of cycles between cals was decreased from 4 to 2 thus maintaining 
the total cal-cal as constant.

14: CO2, Site stn2_central, Wed 04-Sep-2002 11:09:46 MDT, Hydra sample period increased
At 10:41 the Hydra sample period was increase from 20s to 40s
The number of cycles between cals was decreased from 8 to 4 thus maintaining 
the total cal-cal as constant.
13: CO2, Site stn2_central, Wed 04-Sep-2002 10:45:43 MDT, Hydra operational
On Thursday, 28 Aug''02 the Licors were installed with the Hydra and the system set to run over the weekend.The data inspected on Tuesday showed little 
variation. The system apparently was not cycling through the different intakes.
On 3 Sept, Tuesday the system was inspected. 
-the DC had been disconnected when the DC power was run through the BOB. 
Thus theOpto22 relay for the sample valve and the sample pump operated but 
here was  resulting operation. Power restored.
-the serial message from ther ADAM was not received properly and so the opto22 
ran intermittently sometimes operating for several cycles and the switching off and restarting. Gordon changed the buffer timing and the system operated well.

Inspection of the data on Wednesday indicated that the Hydra operated properly 
- the calibration is still incomplete owing to the single cal gas feeding the 
manifold valving. This will be rectified when the new manifold valves are 
- there is apparently a 7-8 sec delay after sample valve change before the 
analyser responds.
-there is not time within the 20 sec period for the analyser to come to steady
state with the new sample.
The latter two issues will be addressed by increasing sample flowrate, 
shortening the sample manifold to analyser lines and increasing the sampler

12: CO2, Site stn2_central, Wed 04-Sep-2002 09:12:10 MDT, fixed opto22 command problem
When Tony was up at the hydra yesterday he saw the opto22 relays behaving
erratically.  It would work for 5 minutes or so, then quit, then work again.

I think the problem was in rserial.  

If rserial's input is not from a terminal (in this case rserial's 
input is from a pipe), rserial was adding its own 2 second delay before
sending the data. rserial is also not reading in chunks of lines,
it is just reading what is available.  Because of these two
factors I believe a series of opto22 commands like this:

	   20 seconds
	   20 seconds

could be received at the opto22 like this:
	   2 seconds...
	   18 seconds
	   2 seconds ...
	   2 seconds...
	   16 seconds

I think this confused the opto22.

The delay has been removed from rserial, and from the CO2 data, it
looks like the opto22 has been working correctly since then:
Sep 3 22:37 GMT.

11: EVEs, Site all, Tue 03-Sep-2002 15:14:30 MDT, pam stations out
stn3 died on Aug 31, 18:14 MDT, and stn1 died Sep 1, 10:13 MDT.

John and Tony are going up there today (Sep 3) to figure out
what is wrong.
10: CO2, Site stn2_central, Tue 03-Sep-2002 14:18:38 MDT, opto22.s2
The variable opto22.s2 contains the value of the last 4 hexadecimal
digits of the command ">00JXXXX", that was sent to the opto22 valve 
controller  on the hydra.

Note that opto22.s2 is value of the command that was sent. An
indication of whether the opto22 actually received the command
and activated the given relays is not directly available in the
present configuration.

To see which relays were switched convert opto22.s2 to hex. The bits
indicate which relays.

decimal hex     meaning
8193    2001    main sample valve (2000hex) and first valve open
8194    2002    sample and second valve open
8196    2004    sample and 3rd valve
8200    2008    sample and 4th
8208    2010    sample and 5th
8224    2020    sample and 6th
8256    2040    sample and 7th
8320    2080    sample and 8th
8448    2100    sample and 9th valve open

4608    1200    main cal valve (1000hex) and 1st cal valve open
5120    1400    main cal and 2nd cal valve open
6144    1800    main cal and 3rd cal valve

0       0       opto reset (every 2 hours), command=">00B"
-65535  FFFF    opto config, command=">00GFFFF",
                all ports configured for output

9: Adams, Site stn2_central, Thu 22-Aug-2002 10:21:13 MDT, Changed datel averaging
The 20hz analog data on cosmos was coming in at 19.9 Hz, and the
channel 0 samples showed a Datel Status of 1:

data_dump -c 0 -A -f cos020822.160000 
16:18:30.180 0 16 Datel status: 1
16:18:31.190 0 16 Datel status: 1
16:18:32.190 0 16 Datel status: 1
16:18:33.200 0 16 Datel status: 1
16:18:34.200 0 16 Datel status: 1
16:18:35.210 0 16 Datel status: 1
16:18:36.210 0 16 Datel status: 1
16:18:37.220 0 16 Datel status: 1
16:18:38.220 0 16 Datel status: 1

Datel status = 1 means that one 200hz conversion interrupt is
being missed every second. 

We are sampling 14 analog channels on cosmos. The datel code
used 14 channels as a cutoff, such that for 1-14 channels
it averaged 8 samples for each output.  

It apparently can't quite keep up, so I've changed the cutoff to 
13 channels, and so now it will average 4 samples per outout for 
the 14 channels.  That seems to work, we're seeing 20hz analog
channels, and no Datel status messages in channel 0.

8: Communications, Site stn1_west, Fri 16-Aug-2002 19:11:26 MDT, Changed freewave frequency key

Before I figured out why we were getting so many serial errors 
(see entry 6), I assumed the problem was with the RF communications.

I changed the frequency key from F to the following values on
the master radios:

stn1  ttyUSB0  571-2101  freq_key=0
stn3  ttyUSB1  571-2095  freq_key=1

7: TRH, Site stn2_central, Fri 16-Aug-2002 19:09:48 MDT, Replaced broken TRH ventilator horn
At 15:30 MDT the ventilator tube, with the broken horn on the 1meter
TRH was replaced.
6: EVEs, Site all, Fri 16-Aug-2002 18:46:18 MDT, EVE bug causing serial errors and lost data
There is an EVE bug related to the new contents of
fast-out channel 4. EVE is still reporting a length of 2, and it
should be 3, because it also contains a 1 byte station number.

I verified this, both by looking at the fast out data, and looking
at the code in fastOutput.c

This is why we see large numbers of serial errors in the
check_aster output. 

It also causes more problems for station 3 than station 1.

After reading the timetag, channel number=4 and length=2 (incorrect)
it then reads the next 2 bytes as data and expects the next byte to
be 0x03  (ETX).  In the case of station 1, the next byte is 0x1,
and so the ingestor records a serial error, then looks forward
for the ETX, which is the next byte. So for station 1 no data is lost, 
but we see serial errors.

For station 3, the ingestor gets an ETX when it expects one, but
it is actually the station number because both are 0x3.  
The next byte is actally the ETX, but it treats that as part of 
the next sample's timetag.  This next sample will then be garbage,
which is why we're losing sonic data, and seeing CSAT3 diag values 
of 16 (indicating a missing sample).

5: Communications, Site all, Fri 16-Aug-2002 18:45:12 MDT, Freewave config
Copied from an email from John:
 The RF seems less than optimal but the cockpits
didn't appear to be losing anything.  Time will tell.
The connects look like this:
Stn1 West:
  FW2105 (slave@pam) - ~9m antenna 64-degreesTrue - FW2101 (master@base) - usb0
  uses lower antenna (short/thin cable) at base
Stn3 East:
  FW2088 (slave@pam) - ~9m antenna 47-degreesTrue - FW2095 (master@base) - usb1
  uses upper antenna (long/thick cable) at base
The upper base antenna could be raised quite a bit, but the lower not.
Otherwise we could go with the 2.4 FWs which could work better in the cluttered
treeish conditions, but they'd require some programming and other antennas.

NOTE: We need another Freewave to USB de9/m-f PC cable to
replace the one I borrowed from Andy.

4: Adams, Site none, Fri 16-Aug-2002 18:35:58 MDT, Replaced Ironics
Replaced sick ironics (SN 129043) in cosmos with SN 422016.

Serial channels are now coming in.

3: EVEs, Site none, Wed 31-Jul-2002 13:12:44 MDT, EVE processing load estimate w/10hz sonics

Just for giggles, I decided to look at the impact of running the
'COVARDP' stuff and a couple of other permutations for the Niwot cfg.

NOTE the config used was preliminary and necessarily involved simulating
3 'uw' sonics instead of the actual csat/2-ati because of limitations
with running the 'fake-sensor' ingest.

Test #1
The fake uw_sonic ran at '10hz' and was paralleled in to the 3 different
sonic ports needed for Niwot.  Two had a 'krypton' and thus had covar/avgs
for that in addition to the normal parameters.  The third had
covar/avgs for only the normal components.   No other sensors were
attached (gps, 2-trh, baro).   The response of the console was ok and
EVE was able to keep up.
Note the inefficiency/cost of running the covardps processing really
ate cpu cycles: 55% for processing the sonics, plus 31% for
interrupt servicing.

NAME          ENTRY         TID   PRI   total % (ticks)  delta % (ticks)
--------     --------      -----  ---   ---------------  ---------------
Fastout      _fastoutTask 13c874   12     1% (  190505)    1% (     438)
UW1-AIn      _uwSonicIn   139bac   12    20% ( 2766422)   20% (    6160)
UW2-AIn      _uwSonicIn   13597c   12    20% ( 2736774)   20% (    6107)
UW3-AIn      _uwSonicIn   133bfc   12    15% ( 2062943)   15% (    4614)
tIdleCpu     _idleCpu     130538  250     9% ( 1240177)    9% (    2776)
INTERRUPT                                31% ( 4317957)   32% (    9961)
TOTAL                                    96% (13525715)   97% (   30534)

2012 207 08:40:02 01
        u       v       w      ts    h2o     us     vs     ws    cnt   freq
UW1=   5.27  -5.27   1.06  25.00   4.46   20.0   20.0   20.0   3141  10.5
       0.02  -0.02   0.00   0.00   0.02    0.0    0.0    0.0      0   0.0
       0.00   0.00   0.00   0.00
        u       v       w      ts    h2o     us     vs     ws    cnt   freq
UW2=   5.28  -5.28   1.06  25.00   5.65   20.0   20.0   20.0   3135  10.5
       0.02  -0.02   0.00   0.00   0.02    0.0    0.0    0.0      0   0.0
       0.00   0.00   0.00   0.00
        u       v       w      ts    h2o     us     vs     ws    cnt   freq
UW3=   5.28  -5.28   1.06  25.00  20.00          20.0   20.0   3127  10.4
       0.02  -0.02   0.00   0.00   0.02    0.0    0.0    0.0      0   0.0

Test #2
Then I decided to try the 'same' config without covars and only
averaging of the sonic counts:

NAME          ENTRY         TID   PRI   total % (ticks)  delta % (ticks)
--------     --------      -----  ---   ---------------  ---------------
Fastout      _fastoutTask 140c80   12     1% (    1454)    1% (     477)
UW1-AIn      _uwSonicIn   13dfb8   12     8% (    7769)    8% (    2582)
UW2-AIn      _uwSonicIn   139d88   12     9% (    8645)    9% (    2868)
UW3-AIn      _uwSonicIn   138008   12     6% (    6334)    6% (    2132)
tIdleCpu     _idleCpu     12aee8  250    38% (   35068)   38% (   11750)
INTERRUPT                                32% (   30115)   32% (   10004)
TOTAL                                    95% (   91588)   95% (   30525)

Notice the huge increase of cpu cycles freed up, and of course as expected
the interrupts stayed about the same.

Test #3a/b
Knowing how scientists are, I thought I'd try two 20hz sonic's with full
covardps/avgs: as expected, it does not work

Then I tried two 20hz sonics without any covar/avgs: EVE is on the
borderline, but appears to be able to keep up even though almost 50% of
the time is spent in interrupt processing as expected.

I tried the gps and it was a minimal load.  The sbus sensor
processing should not add much to the cpu load.

EVE should be OK with all 3 sonics running at 10hz even with or without
covars/etc (including fasoutput).   Based on an estimate of raw character
processing and the final test above, it might be possible, to run 1 sonic
at 20hz and the other 2 at 10hz provided there is no covar stuff.
5 hz = piece-of-cake.

For the actual Niwot config, we will include the COVAR's
plus fastout to compare EVE's computations with the base.

2: Adams, Site none, Fri 28-Jun-2002 13:24:06 MDT, Adam testing
These were the serial numbers of VME cards that worked well during

  backplane #3
  matrix: SN 9013
  ironics iv3234:  SN129043
  datel: 320689

  backplane: #4
  matrix: SN 940211101 
  ironics iv3234: SN 422016
  datel: SN 320589

This third adam is working now in the staging area:
  backplane #1
  matrix: 9009
  ironics iv3234: 820003
  datel  400190
  Teca: #4  (NSF 21690)
  power supply appears new, from MIT?

This datel does not work (LED does not come on): 440690

1: LOG, Site none, Fri 28-Jun-2002 13:22:14 MDT, Logbook created
initial log file /net/aster/projects/NIWOT02/logbook/tklog.log created by USER maclean