CuPIDO: GOES Messages: 20 Entries..

As now mentioned 3 other times, we cycled power on the GOES transmitter last
night (~20:15 on 3 Aug), which got rid of "SOH (start-of-header)" error 
messages.  GOES operation is now okay.

The GOES data recovery from station 8 has been spotty since
the morning of Jul 25.

At 17:00 MST  I visited the site and installed
new software, which provides status information
in /var/tmp/goes.log, and does a soft reset
of the transmitter if it is not responding.

This site has a SE model 1200, 
software build 10/14/2004 16:08:26.
The GOES GPS is on top of a T post.  
RF cable surge arrestor grounded with the solar panel.

Measured 7.5 watts during transmission at 00:48:30Z.

Status file shows a good GOES clock, within 100 msec
of the system clock.  Saw no errors during 5 or 6
transmissions.

Did not reboot viper or power cycle the GOES transmitter.

Left 17:58 MST

The script that telnets to NESDIS started failing at
7/27 04:10 MST, with an error:

Remote is disabled or nonexistent
Connection closed by foreign host.


At around 4 pm I called the NESDIS operations center (phone
number is on the white board).  They confirmed the
problem and rebooted a bank of modems to get things
going again.

Missing data filled in except for small gap at 16:45 MST 7/27.

Taped GOES cooler; found one tiny trace of water in bottom

GOES transmitter model 120; no surge arrester

Visited station 9 15:20 - 15:33 MST

Swapped USB disk
Taped GOES cooler; no moisture in cooler
Model 120 GOES transmitter

Repaired and tightened fence; charger okay.

Visited site 8 14:17 - 14:47 MST

Swapped USB disk; taped GOES cooler

Model 1200 GOES transmitter connected to ttyS8

Visited site 7 13:35 - 13:50 MST

Swapped USB disk; taped GOES cooler lid (no moisture in cooler).

Model 110 GOES transmitter

Station 5 – Station web plots down

Station 5 was visited and it was found that the GOES system had become 
non-functional. It had gone down during the previous day’s monsoon storms 
and not been able to re-synch with the data system, which had continued running
(no notable data losses were found). Water was noted in the bottom of the GOES 
container (cooler). The system was restarted successfully with a reboot.

~1200 the data drive was also exchanged station 5. Next exchange prior to 
July 30. 

17jul06 15:45-17:00

Purpose: routine maintenance


GOES pvc elbows installed, raised off ground to help improve ventillation.

	Note: this SE1200 s/n 1031, with gps antenna s/n 2537881
	took very long to set its clock.  After 45 minutes a power cycle
	and reboot of the data system did work and the transmitter's
	clock was set allowing the dsm to proceed and telemeter data.

	Check this unit in Boulder for abberations.

16Jul06 9:05-9:40 
PVC elbows for cable feedthroughs were installed to help vent heat from
the GOES cooler.  Box was raised a bit off the ground

12Jul06, 14:15-18:40

Purpose: Station Reporting has been down / internittent

Data System:
	Data was still being collected locally: swapped usbdisks
	Stored log files in:
	/data/projects/CuPIDO/logs/
	e_adamlog_071206.txt and e_sysmessages_071206.txt 
	These were actually grabbed before doing much of the following,
	however should be examined.

Goes:	Observed 1 transmit then 2 missed per wattmeter!
	Rebooted data system while recovering local storage (and logs);
	then
	Observed 1 transmit then 2 missed per wattmeter; same!
	did notice some IO exceptions in logs
	Swapped transmitters:
		SE110 1005
		SE110 1006 in (same unit from st7 previously)
	Observed 1 transmit then 2 missed per wattmeter; same!
	did notice some IO exceptions in logs
	Installed newest software / rebooted:
	Observed continuous transmissions.

Implication: is that the newest (test SE110 model with 242 bytelen)
	may be the solution but not the prev. (test SE110 model with 244
	bytelen).   Both version of this software include Gordon's 'late
	message into statsproc fix.'

Recommendation:
	add more log messages that indicate 'goes xmit handshake is ok'
	or otherwise.x

Antenna:	Was loose and ~10-15deg off in both az/el....ie bad.
	Had a fair challenge getting this re-aimed correctly and locked down.

Goes Box Drilled:
	Speculation that transmitters are getting too hot in the very
	recent 105+ weather, I widened the holes in the cooler and glued
	in 2" pvc elbows which made nice cable feed throughs.  They may
	still allow some rain to splash in but shouldn't be very much,
	while allowing a small amount of ventillation.
	Raised box 1-2" above ground.

Rnet:	Cleaned

9Jul06, 9:30-10:30

Purpose: Station reporting over goes still down.



1) Still no goes output upon arrival.
   Adam was working, but saw some goes related errors in log messages.
   The spare from Boulder arrived last night, so
   Decided to swap transmitters:
   Removed   SE110 - 
   Installed SE110 - 1005 (unit sent from Boulder)

2) Bad Surge Arrester / Replaced
   Initially saw 2w - ish output (as from SE120 yesterday)...yuk.
   Removed old version surge arrester and determined it was the culprit.
   recalling that the stud on it had been loose and it appears that
   when i tightened it up yesterday that caused a signal loss problem.
   Replaced it with the last of the 'newer' units.

   Observed 12W output,min. refl.

2) Corrected the antenna angle
   It had been slightly off and a bit loose.
   Took quite a bit of fiddling to get it fixed.
   Still need a better mounting scheme!!

8Jul06, 13:15-14:20;

Purpose: Intermittent communications since yesterday/rain(maybe), and
low rf output since my last visit.

Discovered a few problems:
1) No goes output upon arrival.

	The cooler had some water in the bottom and the connectors for
	the sio and power were 'on the bottom'....possibly the root
	cause.  Raised the connectors, and transmitter, tightened
	loose surge supressor ground strap, added lug to the ground
	strap/rod.

	A simple power cycle didn't get goes working again.  There
	were several 'wrong id' messages in the syslog file but the
	ck_goes kept looking good.  Thought that after cycling the
	goes a reboot was prudent and that appeared to work, getting
	7-Watts out of the transmitter per the wattmeter a couple of
	times.  ....however,
	upon arriving at the base 1-hour later still no messages....

2) Adam was working, but many gps NMEA errors in the log off and on
beginning on the 6th, but especially since the rain yesterday.


Second visit:    16:30-18:15
	More errors in log including 'wrong id'....wondering if errors
	of this type never truly get cleared.  evidence to that effect
	is that a reboot recovered goes and obsevered 7w out.
	- Decided to try a different transmitter...the questionable
	one from st8: SE120-3515.  after doing that and getting
	things running, saw only 2w output; yuk!
	Put the other SE110 back in and saw 7W output again.
	- Swapped microStrip out
	- Installed yaggi in

	Things appeared working when i left..but it didn't last long...

8Jul06, 10:20 - 11:30

Purpose: Rlw.out and derived Tsfc was erroneously high since heavy rain event 
on the 5th.

Discovered a few problems:
1) Rlw.in
	Scratch on the dome, but shouldn't be a problem. A picture was taken.

	Split in the cable insulation probably allowed water to
	penetrate in.  Fortunately the 'drip loop' prevented the water
	from flowing all the way to the connector.  No water in it now
	that we could tell.  Taped it and the main connector because
	of....

2) Rlw.out
	This is it: considerable water in the Amp-9 connector.
	It appears that the rain was forced in through the back side
	of the connector under the heat shrink tubing which protruded
	into the shell and no doubt past the RTV.  Also, these units
	don't have the plenum and the connectors are just free
	hanging, perhaps a bit vulnerable.  Taped the connector, cleaned the            corrosive accumulation with Q-tips and dried it all out.

Uploaded USB disk: 58% full for 19Jun - 8Jul

Neighbor visited:
	Giovanni Panza 818-2815, nice guy and very interested in project.

6jul06 12:20-15:45

Purpose: station was not reporting for most of last night and today.

Upon arrival the data system was still running, collecting and
archiving the data.  However there were indications of continuing
system problems on both the goes and the USB drive.  In addition the
/var/log/messages indicated some 'ttyS5 opened' despite the logger now
being on ttyS8.

GOES: ck_goes showed the 'wrong channel' problem.  Due to consistency
of problems over the last few days, decided to swap the transmitter
and put in the spare unit: SE1200-1031. This continued to show other
problems despite rebooting the daq.  Errors included 'wrong goes-id'
and the normal clock not yet set.  Tried using the wattmeter and never
saw a message go out even when the ck_goes appeared normal beyond 1/2
hour and more when the gps should have locked.

Footnote: I tried using my pc test software version to both
transmitters directly and had problems.  Neither completely worked for
setting the id, and clock for the se120.  I hadn't tried it on this
particular portable previously and am not sure which version of the
code was on it, however it should have worked.

Ultimately after trying both transmitters again, I left the SE1200
running and rebooted the system a few times in the process including
full box power cycling.  USB problems continued intermittently.  I had
intended to take the old transmitter to st9 and put it there,
retrieving the other working goes unit to put in st8, however, Greg
reported that st8 began reporting while on the way so we decided to
let it run and see what happens overnight.

This station may have been zapped by a storm discharge (it is very
close to a modest sized water tank of the park), and/or it has box
wiring or other hardware problems.

The GOES Xmitter was elevated off the bottom of the cooler, and holes were drilled in the cooler.

The GOES set up was checked at Station 4, where strange behaviour had been noted before operations. There was between 5 and 10 mm of water in the cooler but because the cooler was tilted the transmitter was mostly dry. It appears that water, via capillary action, curls under the lid and accumulates in these coolers (all sites now have holes drilled, except Site 9.

Holes were drilled in the cooler to drain it and the xmitter was elevated on some flat rocks. A rock was placed on top as well. 

This site has an SE110 xmitter (Serial# 1007) and antenna #8 (square).

5jul06 13:15-14:30

SE110-1002, microStrip 5
Forward power low at 6W.

Attached ground strap, bit didn't have a proper screw terminal so just cable tied it as best i could.
moved antenna down from top of t-post and adjusted the angle.  The u-bolts are stripped and will need to be replaced if there is any more adjustments made.

3Jul06 11:50 - 

Purpose: station was 'down'

In reality the station was still recording data but the goes transmitter was
locked up.  The adam.log showed that it had stopped transmittion at 0530UTC (consistent with the 1030pm web plot stoppage).

GOES came back up when power cycled.
Surge protection was added to the system to protect against this potential cause of the disruption.

The 512 Mb Cruzer usbstick (unnumbered) was replaced with 512Mb Cruzer #2 and data was uploaded to aster.

The GOES system had stopped transmitting on Site 8, resulting in no web plots. It was confirmed that the system had written local data during the outage. The GOES system responded after power cycling the GOES and rebooting the Viper. This same procedure lasted only 8 hours or so the previous night and so bears watching.

The pathology is not evident using ck_goes because you get the same error when GOES is up as when it is down at this site. That is, the test signal provides an error message that includes 'invalid channel' (it takes one GOES cycle for this particular error to appear after a reboot because the GOES transmitter hasn't received clock information - resulting in a different error). This is a false indication of an error and so other methods of verifying GOES x-mission should be used.