ARM91: OPERATIONS Messages: 81 Entries..

ragwort crashed while adding barometer 
 
I think I killed the barometer board while connecting it to the system, which then 
sent bad data which crashes ragwort.  The barometer is back out of the system. 
 
 

Time reset on ASTER 
 
Before doing this, I shut down everything (even ingest).  Then I used the 
date command (as su) and restarted everything.  I think everything is going 
now - check_aster, raw_data, and xcockpits look okay. 
 
Note that I couldn't reach NIST, so I used the time difference I remembered 
from yesterday.  I also reset cocklebur's time at the same time (from the 
ASTER xclock display), which caused cocklebur to be about 1.5 seconds slow. 
 
I think overall that we are within 5 seconds of UTC, so I'm happy. 
 
 

ARM IOP #1 
 
	All systems go for ASTER. There is some 
spiking on the 4m UW sonic, mostly in the u component. 
The fast q at 10m is off scale according to cockpit. 
The psp.in is also off scale on cockpit. Will do some 
further investigating. 
 
 

RESTART COVAR_RAGWORT: 
 
	Covar_ragwort was restarted to add the 
pressure data to the mean calculations. The script 
covar_ragwort had defined the did's for pressure but 
they were not in the call to covar. 
This was done at 16:58 GMT. 
 

POWER CRASH! 
 
	A power glitch happened at 20:50 GMT. 
The UPS did not kick in so ASTER went down. 
The power glitch caused the bridge to lock up. 
After recycling power to the bridge, went out to 
array to reset each ADAM. 
System was up and running at 21:40 GMT. 
 
Will do some tests on the UPS to see why it failed. 
 

UPS TEST: 
 
	The test mode on the UPS worked ok. 
This automatically shuts off AC to the UPS input 
and runs the inverter for ~20 secs. 
Why it did not work before is a mystery. 
 

covar output is now split into one file for each julian date.  
This is done via crontab at 5 minutes past midnight 
everyday. 
 
The output goes in a file with the Julian date appended, eg: 
 
comos.covar.048  ragwort.covar.048  marigold.covar.048 
 
These files may be found in the usual place,ie. 
$ASTER/projects/ARM91/results/covar/ 
 
Two scits are used for this : scripts/covar_daily and 
scripts/covar_split. covar_daily is called by crontab. 
This script in turn calls covar_split. 
 
 

IOP #1 DONE: 
 
	The first IOP has ended at 01:00 GMT on 
julian day 50. 
 

DAILY PLOTS for JD 49: 
 
	The daily plots, for data from 
ragwort and marigold, have been cut-off at 
about 22:00 GMT. This maybe due to a bug in 
the script to split the covar data files into 
day files. I was running graph routines after 
20:00 GMT for day JD 49 while still in JD 49. 
Maybe executing plots during the last cycle of 
syncserver for the day has some effect. 
Need to talk to Charlie about it. 
 

GRAPH_COV: 
 
	The version of graph_cov was destroyed. 
The plots for day JD 49 used graph_cov~. Need to  
confirm with Steve or Charlie about the latest version. 
 

COVAR FILES for RAGWORT & MARIGOLD: 
 
	Ran covar_split manually on marigold and 
ragwort for day 49. Plots now have 24 hours. 
 

COVAR TASKS: 
 
	Killed and restarted covar tasks to take 
into account the new pathlength parameter for the 4m 
h2o sensor. The sensor has not been hooked up yet. 
 

disk full - archiving stopped manually. 
 
I will restart archiving after backup is done (and reset xgsetroot!) 
 

archiving again 
 
 

Covar's started - ragwort was off 
 
I've just restarted all covar processes after the UPS test reboot this morning. 
I've noticed that covar on ragwort was never restarted last night (probably after 
the Krypton was added), so last night and this morning will be missing. 
 
 

Connecting to 30th St. Instructions 
 
aster:/home/aster> kermit 
kermit> set line /dev/cua0    connects kermit to the modem - ignore WARNING  
kermit> set speed 9600        sets baud rate  
kermit> c                     short for connect  
 ...stuff appears saying that control-\ is the control character for kermit with 
   ^\? - list kermit commands 
   ^\h - hangup line 
   ^\c - return to kermit...  
 
attd 14978747       characters NOT echoed - you are typing in the blind  
		    commands our modem to dial the modem connected to the  
                    SPARCstation on Santigo's desk  
 
 wait about 30 secs. for CONNECTED message  
RET RET  hit a few returns if you need to get the login prompt on pinot  
 
pinot login: 
 
 ...login as normal - pinot shares login passwords with stout...  
 
rlogin stout  if you want to be on stout  
 
 after logging out of pinot, hangup the line with:  
control-\ h 
[NO CARRIER] 
returning to local system 
kermit> quit 
aster:/home/aster>  
 
 

UW TOWER: 
 
	The uw tower is back in a vertical position (23:50 GMT). 
 

SYSTEM STATUS: 
 
	System looks good. All Adams responding and 
sending data. 
 

Project default name 
 
	the project default namme was changed from FLAT90 to ARM91. 
 
 

bkfiles_kill done on jd052 
 
I hope that this was okay - I did it from home 
(thanks Charlie for leaving tape_store.dir in /tmp!) 
Remember that the tape in the drive has data - I think I'll 
"offline" it just to be safe. 
 
Good night, aster. 
 
 
 

crontab edited to correct ups_stat path 
 
The crontab entry was trying to find ups_stat in /home/aster/bin4, 
but the subdirectory is really /sun4.  Hopefully, ups_stat will now 
work - we'll see tomorrow! 
 
 

Backup done and files killed - bad tape? 
 
This was done from home again.  This time I did not "mt offline", 
since we may want to append to this tape if no one gets out there 
tomorrow.  Note that this tape had some problems.  I got an I/O error 
the first time I ran bkfiles.  Then I rewound and did tape_init, which 
aborted with another I/O error and a core dump.  I rewound and tried 
tape_init again, which appeared to work normally.  Finally, I manually 
did "tape_store dummy rad prop psyc uw comp" (I hope I got the order 
right!), which also appeared to work - with the directory looking normal. 
 
I would have thrown out this tape and used another blank tape if I had 
been there, but I wasn't about to drive 2 hours just to recover today's 
data during a non-IOP.  Hopefully, whatever the problem was cleared itself. 
 
 

Power cycled on Exabyte drive 
 
Hopefully, this will also help recover from the problems I had last night. 
It is almost (but not quite) time for a tape cleaning. 
 
 

Printer for sodar started as per Joel's phone call. 
 
You simply toggle the print mode in the VT320 setup menu (access by F3) 
Instructions are at the bottom of page 2 in Joel's instructions. 
 
This occured about 1 1/2 hours ago at 5:06 (local) 
 
 

Exabyte cleaned 
 
This was done in attempt to get tar to work.  I actually did it twice since the  
first time seemed too fast.  This finished up one cleaning cartridge which had 
other cleaning dates of: 
1/17/91 
1/25/91 
2/24/91 (now) 
 
 

Ran bk_files, but got the following error message when 
extracting the table of contents: 
 
advancing to next file 
advancing to next file 
advancing to next file 
tar: tape read error: I/O error 
error reading tar file, status = 3 
Archive to tape completed 
 
Assuming that the data were actually on the tape per Martin 
and Oncley, I removed the backed-up data. 
 
 

Modified shocov so that it will accept multiple 
a:v:r specifications. The values are output  
in tabular form, with the columns in the same  
order as the command line switches. If a parameter 
is not found for a particular time step, "NA" 
(not availible) is put out. 
 
These tables will be useful for Splus, 
and (via xgsetup) can be fed into 
xgraph. 
 
I modified graph_rad to work with the new 
shocov - old method took 6:16, new method  
takes 1:10. 
 
All existing scripts should work (slowly) 
unmodified, since the shocov output for a single 
parameter is the same as before. 
 

everything nominal as they say; the previously mentioned 
dead sensors (v.ati.comp, t.10m.uw) have not come 
back to life. 
 
About a dozen spikes on *.10m.uw over the last three  
hours. Other sonics look clean 
 

sodar backed up 
 

COCKPIT-ragwort DOWN: 
 
	Cockpit for ragwort was not running. 
message on window refers to a major opcode of request failed: 
7 (X_ReparentWindow). 
 
restarted cockpit. 
 

SODAR PRINTER OFF: 
 
	Sodar printer was disabled today at about 
22:35 GMT. 
 

DISK FULL!!! 
 
	The archive data was not transferred to disk 
last night; therefore, the disk filled up. Archiving 
stopped at about 06:00 GMT. Charlie tryed to run bkfiles from 
Boulder this morning but had no luck. 
 

EXABYTE TROUBLES: 
 
	The exabyte drive has gone south for the 
winter. We can not get it to initialize a tape. The 
only message back is "status = 138". The power has been 
recycled, the head has been cleaned, and 3 different tapes 
have been tryed. People are looking into the options of 
borrowing a drive from STOUT or renting one. 
 
Because the data set we wish to archive is during an iop, 
the archive processes have been shut down so that we do not 
run out of disk space. 
 

COVAR for COSMOS: 
 
	Covar_cosmos was restarted to include the 
gill and ati sonics. This took place at 19:55 GMT. 
 

EXABYTE round TWO! 
 
	After playing around with the exabyte; cycling power, 
trying tar commands, etc., I was able to get it to tar the backup 
files to a tape. A tar read showed all files were on tape. 
I then proceeded to try bkfiles again. Of course, everthing worked 
ok! We will still replace our drive with the loaner from stout. 
 

ARCHIVE GOING: 
 
	Restarted archiving of data after getting bkfiles 
to run, time was 20:38 (GMT). 
 

COSMOS A/D STOPPED: 
 
	It was noticed in cockpit that the analog channels 
from cosmos were not running. A "data_off" /"data_on" was 
required to get it going again. This occurred at about 
00:55 GMT. 
 
Note that the gill file starts late because of this and that 
the ncar file is short because the fast T wasn't recorded initially. 
(spo) 
 
 

Wrong baro filename 
 
A bad timetag caused one of the baro files last night to have 
the wrong name - I've just renamed it back to baro058080008, since 
data_dump said that that was the first good timetag. 
 
 

EXTABYE TEST: 
 
	Tyed running "tape_store" on exabyte. 
It did not work; bombed in initializing. So I 
tryed a separate "tape_init", this worked. 
Tryed "tape_store" again, everything worked up until 
it tryed to read the directory. 
Will try again tonight. 
 

PREPROCESS.CONFIG CHANGE: 
 
	Changed limits for licor sensors, in preprocess.config, 
to conform with psp sensors. 
 

IOP # 3: 
 
	IOP #3 starts at 03:00 GMT. 
ASTER is up and running. All sensors are responding 
and looking good. The sodar printer was restarted for 
the IOP. 
 

TAPE ARCHIVE for JD 58: 
 
	Archiving of data for JD 58 went ok. 
The exabyte ran smoothly. I'll leave power on and 
try using the drive later tonight. 
 

EXABYTE TESTS: 
 
	Ran the exabyte with "tape_store" 6 times. 
The first test was with a "used" tape. Tape error messages 
resulted. With a new tape, each run was sucessful (6 runs 
on on tape). 
	The drive rack was pulled out; however, I do not think 
that had anything to do with it. 
 

INGEST STOPPED on ALL ADAMS: 
 
	At 7:52 GMT, the ingestors stopped on all adams. 
Upon examination of syslog, error messages from ingest: 
ingest> Error reading stream message, rval:0, errno:54 
 
Archiving was running again at 08:00 GMT. 
covars were back up at 08:05 GMT. 
 
Possible cause: I was working on the exabyte drive at 
the time of the crash. Maybe I bumped a switch on the optical 
bridge. 
 

graph_cov and graph_mean sped up 
 
These now have the multivariable shocov calls and are tolerably fast. 
 
 

Disabled sodar printer: 
Auto print mode -> normal print mode 
 
 

checked aster - everything looks okay 
 

SODAR PRINTER: 
 
	The sodar printer was started up for the IOP. 
 

FALSE IOP! 
 
	Due to a miss communication over the operational 
status messaages, ARM did not go operational this morning. 
WISP was operational. 
 THAT IS ALL! 
 

COVAR MARIGOLD: 
 
Covar for marigold was stopped and started to use 
new calibrations for pygreometer temperatures. 
 

HUMIDITY INTERCOMPARISON: 
 
	The Vaisala and Rotronic instruments 
were added to the array today. They are interface 
to cosmos. In addition, the spare psychrometer was 
added as a reference to compare against. The height 
from the ground to the air intake is 1.4m. 
Below are the critical parameters are interfacing. 
 
ADAM: cosmos 
 
sensor			channel		gain		filter 
 
tdry,twet.comp		202             ---		--- 
 
rh.vaisala.comp		101		10 /w offset 5	.5hz 
rh.rotronic.comp	102		10 /w offset 5	.5hz 
t.rotronic.comp		103		5		.5hz 
 
			calibration		did # 
 
rh.vaisala.comp		.001525925, 50.0	530 
rh.rotronic.comp	.001525925, 50.0	540 
t.rotronic.comp		.003051757, 0.0		541 
 
 
 

CONFIG CHANGES: 
 
	The necessary changes have been made to the configuration 
files to handle the new sensors. The changes to archive will 
start at 00:00 GMT for day 63. These changes are removal 
of sub-directory /psyc/varo, addition of vaisala.4m.uw to 
uw/uw04m, and creation of sub-directory comp/humidity. 
No changes were required yesterday for the addition of 
temperature information on the pygs (we were already collecting 
that data). 
 

POWER OUTAGE: 
 
	There was a power outage today around 
19:00 GMT. Everything was backup at 19:35 GMT. 
 

LOGBOOK FIX!!! 
 
	The default for projects works now. 
The problem was the new version for the script 
logbook was not moved to $ASTER/`arch'. 
 
NOTE: logbook does not want to work under 
cocklebur, Charlie should have the answer. 
 

When attempting to remove archive files, 
I was requested to override protection 664 
for each file.  The protection was -rw-rw-r. 
Shouldn't this have allowed me as aster to  
remove the files without quibling? 
 
 

IOP #4: 
 
	ARM went operational, along with WISP, 
at 18:00 GMT. 
 
	The system is up and running. 
Sensor status loooks ok except for the following: 
	7m tdry is reading low. I have Cathy checking 
the calibration coefficients. 
	The 4m Krypton is bad. This may have to do 
with water on the windows since it is raining out here. 
As a first pass, all other sensors are looking good. 
 
followup: 
	 The coeficients for the 7m psyc. are ok. 
Due to weather conditions, we will not be able to 
lower the tower to work on the sensor. 
 
	The 4m Krypton goes in and out. As mentioned 
before, this is probable due to the windows getting 
wet. 
 

COVAR TASKS: 
 
	the covar tasks were restarted to use the new 
version of preprocess which allows for proper calibration 
of pyg temperatures. 
Also, they were started in root. 
 

ANOTHER SCRIPT: 
 
	Graphpak is a combination of the 
basic graph packages we use. This way you only 
need to remember one script name but more input 
parameters. Below is the usage format. 
 
graphpak   wind <-s -d>> 
 
note: for psyc or wind you must add the extra parameters 
 

Error in graphing u w for JD063: 
 
aster:/home/aster>graph_cov 063 u w 
Error in file `/tmp/graph_cov.22438' at line 816: 
  Cannot read second coordinate 
Problems found with input data. 
 
 
 

COVAR RESTARTED: 
 
At 00:15, covar files were moved to .old, 
e.g. cosmos.covar -> cosmos.covar.old, and 
the covar processes were stopped and  
restarted.  The purpose was to allow writing 
cosmos.covar.old, etc. to tape and removing 
them from the disk.  The three .covar.old files 
take up 33.8 megabytes or about 5.5% of /home. 
 
 

Leaving for dinner in Platteville. 
 
 

Back from dinner. 
 
ASTER processes all seem normal and 
cockpit displays look okay. 
 
 

Daily ops 
the pitch and roll were switched from the 10m sonic 
to the 4m sonic at 1:00, and the soil sample was taken. 
 
 

REPLACEMENT ROTRONIC: 
 
	The rotronic sensor being 
used in the humidity comparison was 
removed and a new was put in. The old 
one was known to act funny at high 
humidity. This was done at 20:45 GMT. 
 

ARCHIVE TASKS: 
 
	The archive tasks were restarted because 
they were last started under root which asks each 
time you wish to delete a file. 
 

SODAR BACKUP: 
 
	The sodar backup was perfomed around 
21:00 GMT. 
 

Forgot to kill archive files, file system 
full from 0355 to 0635, so raw data was lost 
for this time period on jd 067. 
 
 

NEW GRAPH ROUTINES: 
 
	Over the last week a variety of 
new scripts have been written to do graphs. 
Below is a discription of some of those routines. 
 
GRAPHPAK: graphpak was designed to combine a number 
of the standard graphic packages we use. These are 
graph_rad, graph_ts, graph_baro, compsyc, and compwind. 
The input string to the graphpak is the same format as 
has been used for the other routines above. The only 
difference is you have to tell it what you want to 
plot. 
 
USAGE: graphpak   wind <-s -d>> 
 
EXAMPLES: "graphpak 67 psyc -w"    "graphpak 63 baro" 
 
note: for psyc and wind, you must add the additional parameter. 
 
 
 
XYCOS, XYRAG, XYMAR: These packages provide xy plots of any 
parameter coming in on their respective ADAM. The switches 
available let you do differences vs one parameter or time, 
simple x vs y, and one parameter vs time. 
 
USAGE: xycos    <-d -dt> 
 
EXAMPLES: "xymar 66 pyg.in.rad pyg.out.rad -dt" 
	  "xyrag 54 u.4m.uw null" 
 
notes:  parm1 and parm2 are the x and y components. The 
	proper input string is the "did" configuration 
	name. As an example, for the 4m Krypton sensor the 
	input string would be: q_h2o_uv.4m.uw 
	if null is used as the second parameter, a time 
	series plot of parm1 results. 
	Switches -d or -dt are optional, if not used the 
	plot is parm1 vs parm2. 
	switch -d: parm1-parm2 vs parm1 
	switch -dt: parm1-parm2 vs time 
 
 
PLOTQ: plotq is a special package for comparison 
plots between the Krypton sensors and the Vaisala 
sensor at the 4m height. The format for this script 
is similar to the xy scripts. Only one input parameter 
is required, y. the x component is defaulted to the 
vaisala component. If you want a plot of the krpton at 
4m vs 10m, use xyrag. 
 
USAGE: plotq   <-d -dt> 
 
EXAMPLE: "plotq 23 q.4m"    "plotq 45 q.10m -dt" 
 
notes: parm possibilities are "q.4m" and "q.10m". 
 

Cleaned up bad entry in marigold.covar.067 
associated with the break in data archiving 
between 0400 and 0630. 
 
 

Possible errors in covar files. 
 
At ~5:40 I cleaned up an error in 
marigold.covar from day 067. 
I did this between the 5 min writes 
to the covar files, but the size of 
the file also generated a 'device 
full' warning during the editing 
process.  Perhaps this introduced  
another error in the covar file while 
I was cleaning it up.  I just tried 
to run comp_uv10m 68 and it bombed, 
possibly while running covar_split. 
 
 

I regenerated missing pieces of marigold.covar 
from the archive files. Note that for jd067  
the time tags are out of order; should not 
bother shocov though. 
 

RAGWORT- analog channel test: 
 
	From 20:10 to 20:40 GMT, the analog 
channels on ragwort were being checked with 
a voltage source. 
 

IOP #6: 
 
	This IOP started at 21:00 GMT. It is a 
WISP IOP inwhich ARM will participate. 
The sodar printer was started. 
 

COVAR_MARIGOLD: 
 
	Covar_marigold was restarted since it 
was not writing any data to the file. It stop 
writing to this file back on day 68. 
 

Reconstructed marigold.covar's for 
JD's 68, 69, 70, but not the root 
marigold.covar.   
 
 

RH ROTRONIC: 
 
	The rotronic rh response went to 
100 % at about 7:00 GMT and has not come back 
down. The vaisala has a similar response but it 
comes back down. 
 
Follow-up: Upon examination, the rotronic has lost 
its transducer. This may have been the result of high 
winds last night or the airflow from the fan. The 
transducer did not have its filter cover on. 
 

TAPE DRIVE: 
 
	The tape drive did not want to initialize 
a tape. I had to recycle power to get it to work. 
2 tapes were initialized. 
 

SODAR work: 
 
	The sodar printer was stopped and the 
data backed up on floppy. 
 

COMPUTING RH from PSYC: 
 
	Found mistake in psyhum code which 
computes rh from tdry,twet. Will run some plots 
on vaisala/rotronic comparison units. 
 

RAGWORT INGEST SHUTDOWN: 
 
	Ragwort ingest shutdown a few minutes 
ago. Error message refers to: INGEST_SOCKET_ZERO_READ 
for ragwort. I logged into ragwort and everything looked ok. 
Ragwort proceeded to spit out bad time tag messages from recDatel. 
Ragwort was back up and archiving at 20:16 GMT. 
 
followup: Forgot to "data_on" ragwort; therefore, data collection 
didn't start until 20:35 GMT. 
 

GRAPH_COV error: 
 
	Graph_cov did not want to work for 
"u w" and "v w". Error meesage refers to a 
temp. file inwhich second coordinate could not 
be read.