CASES99: ADAMS Messages: 25 Entries..

  Cosmos Matrix card s/n 940211101
   16 channel Ironics serial card s/n 129043
   Datel Dvme 601 A/D card s/n 400190
   signal conditioning card # 3
   NCR power supply s/n 000307
   chassis # 4 (missing insulation on enclosure lid)
   Teca cooler # 4 s/n 8940208


At first we had Datel #440690 in this adam. It would work for
about 1/2 hour and then the A/D data (Tbox.cosmos) would quit.
Saw HSTSTA errors on console once.

Datel #440690 was in daisy in the field.
Swapped in Datel #400190 which was in cosmos in the field.  

The adam failed twice over the weekend in this configuration.
(I mxreset it from home once).
On Monday am (10:00 MST) I plugged the adam's power cable into
a different socket on bottom of the power conditioning box.

At 15:30  MST Monday, replaced coax network cable.  The collision
light on cosmos' transceiver was flickering constantly. Jiggling
the coax connector at the adam end would stop the collisions.

final resting place for all ADAM cards at teardown


   Ragwort Matrix card s/n 9006
   16 channel Ironics serial card s/n 821003
   Datel Dvme 601 A/D card s/n 320689
   signal conditioner card # 2
   NCR power supply s/n 000371
   chassis # 1  (missing insulation on enclosure lid)
   Teca cooler # 1 s/n 9121083  NSF 23024


   Daisy Matrix card s/n 9010
   8 channel Ironics serial card (chicory) s/n 014029
   Datel Dvme 601 A/D card s/n 440690
   NCR power supply (belongs to PSU) s/n 002672
   chassis # 4 (missing insulation on enclosure lid)
   Teca coller # 4 s/n 8940208


   Cosmos Matrix card s/n 940211101
   16 channel Ironics serial card s/n 129043
   Datel Dvme 601 A/D card s/n 400190
   Datel Dvme 641 expansion card s/n 001231288
   signal conditioning cards # 3, # 7, and a no number board (belongs to PSU)
   NCR power supply s/n 000307
   chassis - no number (belongs to PSU)
   Teca cooler s/n 9421060 (belongs to PSU)


   Marigold Matrix card s/n 9009
   8 channel Ironics serial card (Edelweiss) s/n 67356
   Datel Dvme 601 A/D card 320589
   NCR power supply (marked dead) s/n 000298
   chassis # 2
   Teca cooler # 3


   Fester Matrix card s/n 9013
   16 channel Ironics serial card s/n 821022
   Datel Dvme 601 A/D card s/n 000940694
   signal conditioning card # 6
   NCR power supply s/n 000238
   chassis # 3
   Teca cooler # 2

Inspection at ~1500, 26 Oct''99, indicated that the disc inserted in the morning
to hold open the lid of Ragwort had been placed so that it held the radiation 
shield of the ADAM rather than the lid. This was fortuatous as this spacing 
resulted in good daytime ventilation (max Tbox ~ 32 C for a hot day ). This
arrangement will be maintained for nighttime ventilation

Throughout the nighttime the comb held the lid of Ragwort open ~ 1 cm. 
The temperature only fell to 20C. 
At ~ 0900 the comb was removed and the disc was substituted holding the lid
open ~ 5 cm.

All the ADAMS were reset at 16:09 local.
At this time, I do not know why. They came back
up without the need for human input.

The ragwort lid spacer, the comb, was repalced with a
floppy disk at ~10:00 local.

At 1730 the comb spacer for Ragwort was changed from vertical to horizontal.
Note that even when vertical the spacing was not large because plastic comb
was deformed until nearly bent double.

The high today is predicted to be high 70s. So Cathy
and I changed the comb orientation from horizontal
to vertical.

Daisy stopped around 16:10 CDT, Oct 18.  We could not 'ping' daisy,
so Steve recycled power at 17:59 CDT.  It was around 16:10 that
we were working on the tower, but it is not apparent how we could
have caused this crash.  We lowered the 20m boom at 16:05 and then
went up to 50m to move the radiometers to the end of their boom.

Rebooted cosmos and ragwort at 15:18 GMT in order to properly restart
covars.

Also stopped and restartd serial ingest of PAM data on toucan at 
15:20 GMT for the same reason.

During the day, tests were run to determine the source of noise
on analog channel 123, h2o.k.20m, on cosmos. Below is the list of
tests and the outcome.

(1) switched h2o.5m with h2o.k.20m. So h2o.5m signal was going
into channel 123 and h2o.k.20m was going into 108.
result: noise problem stayed with channel 123.

(2) changed analog/filter board
result: noise on 123

(3) removed special control board near power supply
result: noise on 123

(4) reversed h2o.5m and h2o.k.20m. switched analog breakout boxes
result: noise still on channel 123

Ragwort is back on the air. Tom and I decided to leave it
on tonight. There is a slight chance of rain.

Ragwort was taken off the air last night because of possible
rain. The lid has to be left open because of heating. Hopefully,
we will change chassis in the next few days.

Bring up the console window on ASTER so you can watch the adam/eve bootup
messages.

	ping cosmos

If it is alive, you can probably bring it up with mxreset:

	mxreset cosmos

Otherwise you will have to power cycle it.	

In the console window you should see: 

Oct 12 16:52:03 cosmos  System reset!
Oct 12 16:52:29 cosmos  adatsSock SO_SNDBUF len=4096
Oct 12 16:52:29 cosmos  adatsSock SO_SNlast message repeated 1 time
Oct 12 16:52:30 aster cosmos[753]: cosmos Beginning connection on port 1030
Oct 12 16:52:31 cosmos  cosmos booted
Oct 12 16:52:35 aster cosmos[753]: cosmos STREAM socket connected to aster:33008, 1 active connections
etc...etc...

Then do the same for daisy,marigold,ragwort.

If an adam is pingable, but mxreset does not work from ASTER, then try:

	rlogin cosmos
	VxWorks login: aster
	Password: asterize
	-> mxreset

To restart the PAM ingest processes, log into toucan, and do:
	sio_shutdown -s

This should kill 'em all.  If curious, check:
	ps -ef | fgrep stn

If things are OK, then restart them with:
	serial_ingest -z

Installed new Ironics 3234 card at ragwort. pp.50m shows spikes due
to dropped characters, which I assume is due to the ironics 1624
not keeping up.

Ragwort died today at 11:34 CDT (16:34 GMT).

The LMON light on ragwort's port in the chem shelter fiber box 
was out.  Before I realized it was probably a cooling problem,
I just switched the breaker on and off.

Ragwort came up but
	siostatus yarrow
showed non-zero values in the msg ERRORS column.

I then did a mxreset and it came up with the serial card working.
It then died at 19:08 GMT.

At 23:00 I climbed the tower.  The power supply was dead - the
three red lights were out.

I redid the ventilation cardboard so that the air flows over the
power supply in addition to the 4 VME cards (but not the signal-cond
card).  Also ripped out the insulation under the top of the adam.

Finally looked at Tbox.ragwort.  A plot of Tbox.ragwort shows that
the cooler must have failed at about 06:40 CDT this morning.
Tbox went from its 10 minute cycling at about 27 degC up to 35 degC 
at 11:30 when ragwort failed.

After restarting ragwort at 18:30 Tbox shot up to 34C, so at 20:00
the lid was left propped open about 5 inches.

Daisy is sick.

Configuration:

  daisy
  chicory
  datel
  chassis #4

Did lots of switching cards around.  

Indications also pointed to cooling problems, because the adam would 
work fine and then die 1 half hour after closing the lid.
(Daisy's lid is missing insulation, which should keep it cooler.)

The new matrix card has its own problems.  After a while it would
choke on a bad sample (chan==0,length==0).  This appears to be
some sort of memory corruption problem.

The new ironics cards may work now, but perhaps run hot.
They have 4 meg of RAM (the others have 1 meg).

Ventilation was improved by duct taping cardboard in the fan "horn"
with gaps only at the ends so that the cool air is directed only
to the 4 cards on the left (matrix,ironics,serial breakout and datel)
and to the power supply on the right.

Here is what is working:

  daisy matrix (S/N #9010
  chicory ironics (#014029)
  daisy's original datel (#440690)
  PSU power supply
  chassis #4

Swapped the matrix board in ragwort with daisy's.

Results:

  daisy could not talk to datel (HSTSTA errors)

  ragwort is sick, will not stay up. 
  After a while it does not see ironics board and mxreset does not
  cure it. A power cycle will work for a while

Then on 10/1
  put the daisy matrix board back in daisy:  works

  put the PSU matrix board into ragwort.  The EPROM is configured
  for 1 meg RAM without PPP.
  
  The PSU ragwort booted, but then "siostatus yarrow" showed msg errors.
  After a while it died.  Same symptoms as the ragwort matrix.


   

Power to tower was down last night due to a rabbit's chewing.   

I was distracted so didn't monitor adams much today. After power
was restored the adams were up except for ragwort.

Ragwort was dead (status light orange, Ironics halted). 

Replaced the Ironics 1624 in ragwort (yarrow) with a new iv3234.

Lots of rain last night with lightning.  Must have had power outages
because the adams all rebooted at the same times.  Toucan reboots
cause breaks in the PAM archive.

The log from the UPS in the trailer showed some entries for these
times also.

01:01:50	cosmos  ?? (probably not a power outage)
01:55:00	cosmos,daisy,marigold,ragwort,toucan
02:20:30	cosmsos,daisy,marigold, toucan
02:21:30	cosmos,daisy,marigold,ragwort
03:10:50	cosmos,daisy,marigold,ragwort,toucan
12:01:30	cosmos,daisy,marigold,ragwort,toucan

Everything was running in the morning, except the serial channels
on ragwort.  "siostatus yarrow" showed errors in the "msgs" column.
mxreset would not bring the serial channels back.  Power cycling
ragwort from the base of the tower did the trick.

The last network fix to ragwort was swapping the fiber transceivers
with cosmos.  That must have helped because we haven't had any
unexplained reboots since then. 

cosmos:
  ran overnight
daisy: 
  ran overnight
marigold:
  ran overnight
ragwort:
  dead this morning.
  From archive:
     died at 06:18 GMT (01:18 am CDT)
     restarted itself at 06:25 GMT.
     died at 08:30 GMT
     restarted itself at 11:43 died immediately
     rebooted at 14:43 (9:43) by human

  Log file on toshiba was empty - serial
  connector fell off after setting it up.

  Swapped fiber transceiver with the one from cosmos.  Rebooted at
  15:47 GMT

  

cosmos:
  OK
daisy: 
  crashed and auto rebooted again.
  Switched to other two fibers in cable, from green/blue to orange/brown.
marigold:
  running fine and dandy.

  Then at 7:10 pm CST (00:11 GMT) it went nuts, flooding the net 
  with traffic (removing its fiber from the hub quieted things down).
  Did not look at its console output.  A power cycle cured things.
ragwort:
  crashed and auto rebooted once or twice. When visited on tower,
  console was not responding, and status light was dull orange.
  Cable to Toshiba was loose, so no info was available.

  Rebooted ragwort with toshiba attached to console. After a while
  saw "med0: excessive collision" errors.

  Switched to other two fibers in cable, from orange/brown to green/blue.


Also, turned off the chem shelter air conditioner, in case the
problem could be power related.
The AC is on the same transformer as ragwort,daisy,marigold.

cosmos:
  Fixed its problems this morning.  Software bug in krypton calibration
  function couldn't handle empty lines in the cal file and would
  infinitely loop, hanging whatever host was running covar, cockpit, or
  prep.  Cosmos is now ingesting all its channels.
daisy: 
  rebooted itself this morning for no apparent reason.  
marigold:
  running fine and dandy.
ragwort:
  On Friday, 9/24, replaced the 40m fiber borrowed from ISS with
  a 4-fiber 300m cable.  The 40m cable is still tie-wrapped to the tower.
  This was done to fix problem losing fiber link (LMON light on Dyna FL/4
  port would go out and ragwort would reboot).

  ragwort died this morning.   The halt light on the Ironics card was red.
  The light on the Matrix was dim red I believe.  Power glitch?
  Rebooted, with the toshiba logging the console.

Just noticed that the 200hz data from cosmos was actually coming in at
199hz.  The datel couldn't quite keep up, and was missing one
sample.  Cosmos is sampling 15 analog channels.  

The datel was computing averages of 8 samples when it is doing 17
channels or less.  If it does more than 17 channels, it averages
4 high-rate samples.  I lowered the cutoff from 17 to 14 channels.
Therefore it will be averaging 4 samples on cosmos, and 8 samples
on the other adams.

Cosmos (#940211101) started acting funny on 6/16.  It would seldom 
finish the reboot sequence successfully, or would die soon after booting.
Removing the datel did not help.  This was in backplane #4,
datel #440690.

Swapped in daisy (#9010) and all worked fine.

On 6/18 swapped cosmos back in and it seems to be working again...

It ran until 18:00 6/19, and died. The error message had scrolled of the
procom screen because of many "write to datel pipe" errors.

On Monday 6/21, it would hang during the startup sequence because it
couldn't talk to the Datel's DPR, failing to get the appropriate response
back after trying to start the Datel executive.