FLAT90: Logbook Entries

FLAT90: PROBLEMS Messages: 40 Entries..

Return to Logbook Contents Page
Entry Date Title Site Author #Graphics
47 Sat 15-Sep-1990DAISY noneOperator
50 Mon 17-Sep-1990suw1, sat3 and barometers noneclm
53 Mon 17-Sep-1990sat3 back on line noneclm
54 Mon 17-Sep-1990suw1 decoding properly noneclm
56 Mon 17-Sep-1990fixed baro decodes noneclm
62 Mon 17-Sep-1990aster system crash noneOperator
63 Tue 18-Sep-1990system crash noneclm
64 Tue 18-Sep-1990all ingestors go down noneclm
80 Fri 21-Sep-1990Checked sonics for spikes nonetwh
87 Fri 21-Sep-1990Checked sonics for spikes: nonetwh
115 Mon 24-Sep-1990wrong data rate on uw fast t's nonetwh
117 Mon 24-Sep-1990system crash noneclm
129 Tue 25-Sep-1990aster crash! noneclm
145 Wed 26-Sep-1990aster fixed noneclm
169 Fri 28-Sep-1990ati.fluxes program nonesrs
205 Wed 03-Oct-1990Ati sonic cabling problem was discovere nonejmw
223 Thu 04-Oct-1990NOAA data nonesrs
250 Sat 06-Oct-1990No signal from 7 m prop-vane. nonetwh
251 Sat 06-Oct-1990Killed marigold and ragwort archive pro nonetwh
252 Sat 06-Oct-19907 m prop-vane fixed by cycling power nonetwh
253 Sat 06-Oct-1990Began to archive data at 18:46 on nonetwh
254 Sat 06-Oct-1990I mistakenly began to append hot wire nonetwh
257 Sat 06-Oct-1990No data incoming on marigold analog cha nonetwh
264 Sun 07-Oct-1990Restarted marigold analog channels with nonetwh
267 Sun 07-Oct-1990Inventory of sensor performance: nonetwh
268 Sun 07-Oct-1990NOAA sonic power cycled. noneacd
271 Mon 08-Oct-1990Sensor performance inventory: nonetwh
282 Wed 10-Oct-1990Serial breakout box swap at propvane to noneacd
368 Fri 19-Oct-1990Checked sonics, fast T's and fast press nonetwh
376 Fri 19-Oct-1990Inventory of sensors nonetwh
401 Sun 21-Oct-1990Most data look good this morning. nonetwh
425 Tue 23-Oct-1990data system hangs at about 0445 noneclm
427 Tue 23-Oct-1990exabyte drive is misbehaving noneclm
429 Tue 23-Oct-1990exabyte working now noneclm
553 Tue 06-Nov-1990Steve Semmer's automatic program to sav nonetwh
559 Wed 07-Nov-1990The program to write ati.covar and uw.c nonetwh
608 Tue 13-Nov-1990Difficulty with morning archive-to-tape noneacd
611 Wed 14-Nov-1990Difficulty with the exabyte. noneacd
613 Wed 14-Nov-1990Attempt to solve the difficulty with xs noneacd
614 Wed 14-Nov-1990Problem with the mariold archive. noneacd


47: PROBLEMS , Site none, Sat 15-Sep-1990 21:24:08 GMT, DAISY
DAISY 
 
	Tryed to get DAISY running today. It had a hard time getting past 'sync' task waiting to sync the clock. If it succeeded there, it would hangup trying to download code to DATEL. 
Tryed swapping cpu card MARIGOLD with DAISY by changing the network addresses. This did not have 
any major effects. Tryed changing rom chips, still no success. At this point, decided to configure 
MARIGOLD to handle fast analog data so Steve O. could check to hot film output. This meant 
changing MARIGOLD's startup.cmd file. The old file is named "startup_old.cmd". 
 
	Today sucked! Maybe tomorrow will be brighter. 
 
		S.R. Semmer 
 
50: PROBLEMS , Site none, Mon 17-Sep-1990 02:37:50 GMT, suw1, sat3 and barometers
Noticed that we are not getting data for sat3 (marigold:202). 
 
Also noticed that the ascii decoding is not working 
for suw1 (marigold:201) and the barometers. Data files contain the ascii 
values rather than decoded values. 
 
53: PROBLEMS , Site none, Mon 17-Sep-1990 04:20:26 GMT, sat3 back on line
cycled power on sat3 and it started running again. 
 
54: PROBLEMS , Site none, Mon 17-Sep-1990 04:21:42 GMT, suw1 decoding properly
suw1 is decoding properly now 
 
56: PROBLEMS , Site none, Mon 17-Sep-1990 04:35:44 GMT, fixed baro decodes
 
Fixed barometer decode problem, although I'm not sure how. 
 
Barometer message gives pressure in hundreths of millibars, 
so pressures were in the range of 83000. This is larger 
than the 16 bits availible in a short, so had to break 
the decoded message into two shorts, the first with 
the units value and the second with the fractional 
part.  
 
Really must fix the decoding scheme so that arbitrary 
data types can be produced. 
 
 
62: PROBLEMS , Site none, Mon 17-Sep-1990 21:40:12 GMT, aster system crash
system crash at about 1755 - 
probably caused by xgraph waiting 
for window to be located, at 
same time exabyte was being accessed 
and heavy system load. 
 
63: PROBLEMS , Site none, Tue 18-Sep-1990 03:28:13 GMT, system crash
system crashed again, during exabyte operations. 
Also had just spooled a job to the laser printer 
 
64: PROBLEMS , Site none, Tue 18-Sep-1990 15:04:19 GMT, all ingestors go down
18-sep-90 ~0750 GMT all ingestors stopped, "no activity 
from adam". Some sort of network glitch possible. No 
power faults were seen by the ups, and all adams 
were still running all tasks except "sync". 
 
80: PROBLEMS , Site none, Fri 21-Sep-1990 01:51:14 GMT, Checked sonics for spikes
Checked sonics for spikes  
and found: 
 
SAT2  spike code 01 
SAT3  spike code 03 
 
87: PROBLEMS , Site none, Fri 21-Sep-1990 14:42:58 GMT, Checked sonics for spikes:
Checked sonics for spikes: 
 
SAT2  spike code 01 
SAT3  spike code 03 
 
 
115: PROBLEMS , Site none, Mon 24-Sep-1990 22:41:18 GMT, wrong data rate on uw fast t's
DUE TO METALOG CRASH, DATE SHOULD BE:Sun Sep 23 21:09:19 GMT 1990 
---------------------------- 
 
AIR fast T's on the uw tower at 4 and 7 m were 
being sampled at 1 hz.  Charlie changed this to 
20 hz.  twh 
 
 
117: PROBLEMS , Site none, Mon 24-Sep-1990 22:45:43 GMT, system crash
DUE TO METALOG CRASH, DATE SHOULD BE: Mon Sep 2 00:25:00 GMT 1990 
---------------------------- 
 
System CRASH when running exabyte. Took a long 
time to get the adams to reboot - problems with 
gmt_tt_set, and datel board. Put debugging 
satements in gmt_tt_set, didn't help. 
Removed taskDelay following spawn of gmt_tt_set,  
and this seemed to solve that problem. Then found that 
Tick_init was initializing SYSTEM_TT from the rtc AFTER gmt_tt_set 
was called, so clock got set wrong. Moved Tick_init before 
spawn(gmt...) in mx320_common_start.cmd 
 
Now, datel was misbehaving. Seemed like the timetags were 
getting trashed. Found that Ticker() was doing a long word 
write to the timetag location on the datel DPR. Matt on 
Friday said explicitly not to do long word writes to the datel 
DPR; that only word writes were legal (can't even do a byte write). 
Datel seems to be working now, but I'm sure other bugs will evolve. 
 
Finally recovered from these disasters at 05:00 gmt, and restarted all. 
 
 
129: PROBLEMS , Site none, Tue 25-Sep-1990 16:13:22 GMT, aster crash!
system crash - running exabyte. Happened when a print job 
was created with enscript while exabyte was also writing. 
 
Everything up and running now. 
 
145: PROBLEMS , Site none, Wed 26-Sep-1990 20:07:06 GMT, aster fixed
system fixed, back up and running - found that SCSI board 
was causing the crashes. 
 
169: PROBLEMS , Site none, Fri 28-Sep-1990 01:50:53 GMT, ati.fluxes program
ati.fluxes program 
 
	Tryed running ati.fluxes. It kept sending 
messages with respect data not found for all dids. 
Cancelled job!!! 
 
205: PROBLEMS , Site none, Wed 03-Oct-1990 01:33:23 GMT, Ati sonic cabling problem was discovere
Ati sonic cabling problem was discovered and fixed 
yesterday. Cross-talk was occurring between the ATI 
sonics because the caables were strung too closely 
together through the tower. After re-wiring the sonic 
cables to maximize seperation, the cross-talk  
disappeared. The problem manifested itself as a 
burst of spikes building up to peak velocities 
above 10m/s, lasting for approximately 20s, with 
a period initially of 3min 13s. Later secondary 
bursts occurred at different frequencies. These 
periods were the beat frequencies between the 
various sonics. Ski asked Herb about it. Herb's 
reply was that all sonic cables must be seperated by 
a minimum of 4 in to eliminate the problem. 
 
223: PROBLEMS , Site none, Thu 04-Oct-1990 18:33:25 GMT, NOAA data
NOAA data 
 
	The noaa sonic still has spikes 
after the calibration code was changed. Upon 
looking at the data on the w component, discovered 
that the software must convert -00 to 00. 
As a result an negative data between -0.00 and 
-0.99 will get converted to 0.00 to 0.99. 
This was confirmed by using raw data from rserial 
and playing with xgraph. 
 
	Possible solutions are to change output 
of noaa instrument, ATI says that can be done, or 
more modifications to software. 
 
250: PROBLEMS , Site none, Sat 06-Oct-1990 16:48:53 GMT, No signal from 7 m prop-vane.
No signal from 7 m prop-vane. 
 
No output on xwind and no response 
from cshow cosmos:203 
 
251: PROBLEMS , Site none, Sat 06-Oct-1990 17:19:44 GMT, Killed marigold and ragwort archive pro
Killed marigold and ragwort archive processes 
at 17:13.  Killed cosmos archive at 17:30. 
The morning backup failed(?) when 
reading extracting the table of contents from 
the backup tape: 
 
load archive tape for JD 279 
hit return when exabyte light comes on 
 
reading directory 
Tape in /dev/nrst9 is not a labeled ExeTOC tape. 
It may be that this tape has not been initialized 
Do you want to initialize it? [y/n]y 
initializing tape... 
writing initial toc... 
tape has been initialized 
skipping to file 1... 
writing tape... 
writing files from directory ati 
writing files from directory uw 
writing files from directory noaa 
rewinding tape... 
skipping forward... 
extracting table of contents 
tar: tape read error: I/O error 
error reading tar file, status = 3 
Archive to tape completed 
 
Since I am not sure that the data have been saved at  
this point, I will not clean up the disk.  It is now 
97% full, so I stopped archiving.  I am continuing  
to attempt to call Charlie. 
 
 
252: PROBLEMS , Site none, Sat 06-Oct-1990 18:20:44 GMT, 7 m prop-vane fixed by cycling power
7 m prop-vane fixed by cycling power 
to the sensor.  This was done by 
momentarily unplugging the signal cable 
at the breakout box. 
 
 
253: PROBLEMS , Site none, Sat 06-Oct-1990 18:49:37 GMT, Began to archive data at 18:46 on
Began to archive data at 18:46 on 
cosmos, ragwort and marigold.  The 
exabyte tape drive was just mixed up, 
which was fixed by powering it down 
and back up.  Then I re-executed  
bkfiles to complete the backup and 
cleaned up the disk. 
 
 
254: PROBLEMS , Site none, Sat 06-Oct-1990 19:10:33 GMT, I mistakenly began to append hot wire
I mistakenly began to append hot wire 
data to the existing hot wire data tape 
for day 279 without skipping the existing 
file.  I had only turned on the tape 
recording from daisy for a minute or less 
before I realized my mistake and turned it 
back off.  I will start a new tape with 
the hope that the data on the first tape can 
be recovered. 
 
 
257: PROBLEMS , Site none, Sat 06-Oct-1990 20:37:42 GMT, No data incoming on marigold analog cha
No data incoming on marigold analog channels. 
 
 
264: PROBLEMS , Site none, Sun 07-Oct-1990 15:34:46 GMT, Restarted marigold analog channels with
Restarted marigold analog channels with 
the command `data_on marigold' 
 
 
267: PROBLEMS , Site none, Sun 07-Oct-1990 16:28:55 GMT, Inventory of sensor performance:
Inventory of sensor performance: 
 
7 & 15 m prop-vanes out 
10 m wet bulb looks frozen 
All sonics very noisy.  No response 
  from NOAA sonic. 
Fast T's good. 
IR H2O/CO2 noisy. 
UV hygrometer looks good.  Output 
  = 43 (units?) 
Fast p's okay. 
Hot bridge will not balance, so 
  turned off power. 
 
 
268: PROBLEMS , Site none, Sun 07-Oct-1990 17:53:43 GMT, NOAA sonic power cycled.
NOAA sonic power cycled. 
 
At 17:45 the power out at the NOAA tower was recycled and the power wiring 
checked.  
 
Back at the base it was noted that although a new file started  
no new data aquired. 
 
271: PROBLEMS , Site none, Mon 08-Oct-1990 16:49:41 GMT, Sensor performance inventory:
Sensor performance inventory: 
 
Prop-vanes: 
  4 m speed, 7 m speed & dir, 15 m speed out. 
  All instruments look physically ok. 
Psychs: 
  All are operating; cannot judge wet bulb 
  performance with temps.  
Pressure sensors: 
  No heavy accumulation of snow on ports, 
  but some clear ice.  Signals look good. 
Sonics: 
  uw sonics look good.  Some small spiking 
  seen in file ~oncley/sonic/maint/uw_spikes. 
  ati sonics spiking heavily. 
  noaa sonic inoperative. 
Fast T's: 
  4 m T's on both uw and ati towers indicate 
  50 deg.  Tony indicates that it is not that 
  warm out there, so that must be an output 
  of 5 V, i.e. a broken sensor. 
  7 and 13 m T's good. 
IR H2O/CO2: 
  Very noisy. 
UV H2O: 
  Looks okay. 
NOAA Profiler: 
  Still shut down after paper jam. 
Sodar: 
  Running, but snow has damped signal. 
   
Towers are too icy for climbing. 
 
 
282: PROBLEMS , Site none, Wed 10-Oct-1990 16:47:25 GMT, Serial breakout box swap at propvane to
Serial breakout box swap at propvane tower. 
 
The 7m propvane was again dropped out. The serial BOB at the base of the 
propvane tower was replaced with a spare  at 10:15. This solved the problem. 
 
 
 
368: PROBLEMS , Site none, Fri 19-Oct-1990 00:06:04 GMT, Checked sonics, fast T's and fast press
Checked sonics, fast T's and fast pressures; 
all look reasonable except the v channel on 
the 7m ati sonic.  Although noise spikes are 
not being shown by rserial, this channel has 
a noticeable noise component of ~0.5 m/s at 
~5 hz.  This morning this channel had a very 
large offset, ~20 m/s. 
 
 
376: PROBLEMS , Site none, Fri 19-Oct-1990 16:13:23 GMT, Inventory of sensors
Inventory of sensors 
shows that most are working okay. 
Exceptions are the noise still 
present on 7 m ati v component. 
The 4 m ati u and 7 m ati v components 
appear to have the wrong bias compared 
to the other sonics. 
There is some spiking on the 7 m uw sonic. 
 
 
401: PROBLEMS , Site none, Sun 21-Oct-1990 15:29:07 GMT, Most data look good this morning.
Most data look good this morning.   
 
Sonic spike counts last night were all close 
to zero!!! 
 
4 m ati u component has a different offset 
from other sonics. 
 
1 m wet bulb looks phony. 
 
Fast pressures, particularly 4 and 13 m 
appear to have some high frequency noise. 
 
 
425: PROBLEMS , Site none, Tue 23-Oct-1990 14:58:26 GMT, data system hangs at about 0445
data system hangs at about 0445 
 
all ingestors stopped and ADAMs hung, due to 
somenetwork problem. The fiber optic hub was hung;  
had to cycle the power on it to get cocklebur  
back on the network. 
 
All of the ADAMs were hung; could not rlogin to them. 
When given the 'i' command (from field terminal), 
they would list about 5 tasks, stopping with 
the port mapper daemon, and then I would have 
to interrupt the shell to get back to the prompt. 
When I tried to rlogin into aster from the adam, vxWorks 
would reboot. 
 
It looks, once again, like some network bugaboo, and 
probably exercises some weakness in the Matrix 
network code. 
 
If we could inject some noise onto the copper  
ethernet, perhaps we could simulate the failure 
mode 
 
 
 
427: PROBLEMS , Site none, Tue 23-Oct-1990 16:23:06 GMT, exabyte drive is misbehaving
exabyte drive is misbehaving  
 
every tape write generates an error message  
in the system log: 
 
vmunix: st1:  illegal command 
vmunix: st1:  failed cmd =  a  0  0  80  c  0 
vmunix: st1 error:  sense key(0x5): illegal request 
vmunix:            sense = f0  0  5  0  0  0  1  12 
                           0  0  0  0  0  0  0  0  0 
                           0  0  0  0  0  0  17 
 
429: PROBLEMS , Site none, Tue 23-Oct-1990 16:29:05 GMT, exabyte working now
exabyte working now 
 
Powered down tape for 2 minutes - now 
seems to be working okay. 
 
553: PROBLEMS , Site none, Tue 06-Nov-1990 01:09:20 GMT, Steve Semmer's automatic program to sav
Steve Semmer's automatic program to save 
the 5 minute averages does not append the 
22:00 files after midnight.  Perhaps this 
is a bug associated with looking for the  
22:00 files of the current day rather than 
the previous day (just a guess). 
 
 
559: PROBLEMS , Site none, Wed 07-Nov-1990 00:16:46 GMT, The program to write ati.covar and uw.c
The program to write ati.covar and uw.covar 
has a bug at midnight.  The entries that 
start at 23.666.. and end at 24.000 are  
labeled with the day that starts at 24.000 
rather than the day before. 
 
 
608: PROBLEMS , Site none, Tue 13-Nov-1990 16:42:19 GMT, Difficulty with morning archive-to-tape
Difficulty with morning archive-to-tape 
 
The morning bkfiles routine did not result in the production 
of a tape contents sheet nor a tape label 
 
611: PROBLEMS , Site none, Wed 14-Nov-1990 00:49:49 GMT, Difficulty with the exabyte.
Difficulty with the exabyte. 
 
The morning tape did not produce either a cover or a directory listing. 
This evening the exabyte refused to recognize that the tape was a prepared tape 
and asked to initialize it. I submitted a new tape which was then initialized 
and apparently accepted data. 
 
 
613: PROBLEMS , Site none, Wed 14-Nov-1990 02:37:59 GMT, Attempt to solve the difficulty with xs
Attempt to solve the difficulty with xstrip lethargy 
 
The xstrip process was becoming very slow. To solve we undertook to: 
 kill the archiving process on marigold and then sequentially brought it back 
 This did not improve matters. 
 kill the cov_ati and the sonic spiker on marigold and then bring them back 
 We could not bring back the spiker but decided that it was not important. 
 
None of these activities did any good and the xstrip is even slower. 
Every thing except the spiker is operating so we have left the problem 
for consultation with Steve S or Charlie. 
 
 
614: PROBLEMS , Site none, Wed 14-Nov-1990 17:34:24 GMT, Problem with the mariold archive.
Problem with the mariold archive. 
 
In the morning it was noted that the ati files had stopped at 02:30, 
just after we left last night. 
The marigold showed that it was archiving but the data wasnt being stored. 
We phoned Steve S and tried to restart. We ran through the archive kill 
restart routine, then reset marigold by key and tried again. 
We still have the problem