CuPIDO: data-system Messages: 36 Entries..

How to change calibration coefficients on CR10.

A basic station has only two coefficients for Rnet
so it is pretty simple.  The first value is for positive Rnet,
the second is for negative Rnet, but both values that
you enter are positive numbers.  (Following a value by
'C' changes the sign to negative, but both of the Rnet
values should be positive).

rserial /dev/ttyS5 localhost

hit return until you get a "*"

7H

should get a ">" prompt.

*4

should get a MODE prompt.

Hit returns to step through the coefficients.

Because of the way rserial works, when a certain coeficient 
is being displayed, the CR10 is reading input for the next
coefficient, so you have to anticipate things and enter
coef # N+1 when it is displaying coef N.

Repeated carriage returns will cycle through the coefficients
so you can double check,

At 5:23 tonight, station 6 died.  We realized this at 7:30 from the Inn, and
did a late-dusk scramble up the hill to check up on it.  The data system 
processes had stopped for a reason we were not able to determine.  We ended
up just rebooting the station and all seemed fine.  Perhaps the crash was
due to a different type of 512Mb memory stick?  (Though Gordon is sure that
this stick has been used elsewhere.)

A mystery...we'll keep an eye on it.

Swapped USB disk.  Checked data from logger, TRH, prop-vane, barometer.

Returned to station 5 this morning to remove the Pocketec from its case for
better cooling.

Gordon rewired the CR10X multiplexer bypass with 22 gauge wire and amp pins
between the bulkhead connecter and the CR10X patch panel:

Rnet Amp pin
4 		clear -> AM16 shield
1		grey -> CR10 differential port 2H
2		green -> CR10 differential port 2L

Kept shield through original  cable

I duct-taped the lid of the GOES cooler and swapped the USB disk.

gordon replaced the 4 GB USB memory stick at station 4 with the 60 Gb Pocketec
hard disk to increase the service interval at this station.

st6 Visit: Rnet / logger outage

20jul06 7:40-9:15

Purpose: Bad Rnet since station zapped 2 nights ago.

Logger readings were kaput for the installed rnet as well as the spare, known
good sensor.   Upon a power cycle the readings momentarily looked good but
steadily setting into pegged out values.

Rather than change the software (a temporary until power cycle fix) to determine
whether the bad component was the am416 mux or the cr10 adc (batt voltage was
still good), SteveS recommended directly wiring the rnet to the mux input 
channel on the cr10.  That worked because there is only 1 sensor here.  
I had to 'twist wires' together to cobble a jumper to have them long enough 
to reach the adc-diff channel 2 input.   This worked and the readings began 
to appear reasonable however, the security of the connection needs to be 
improved to ensure it remains good.

19Jul06 14:10-14:45

Purpose: Station reporting was intermittent and then down last night/today

Upon arrival:
Station power OK
DAQ OK
usbstick OK, data was being archived.
GOES down.
No obvious reason in messages or adam.log
	See adam/message logs in:
	/data/projects/Cupido/logs/st9_071906_xxx.log

Reinstalled software
	Just for giggles.
	Did see the 'memory error' that I've seen before when doing this,
	even though Gordon reports that shouldn't be a problem.
usbstick swapped

Rebooted: everything went ok and reporting started right up without
	any problem, or needing power cycle, etc.

Note:	Did observe data_stats 'late message received' error after the
	reboot.  Perhaps that had something to do with it.

19Jul06 8:30-9:30

Purpose: Station reporting went down last night during Tstorm.

DAQ down upon arrival although station power and the Viper was up and running.
The usbdisk was 'dead' to the system.
Basically minimal log files since 4z.

Some small amount of water in the bottom of the goes box.

Halted data system and powered down.

Goes pvc elbows installed to improve ventillation.

Booted and everything went ok.  Discovered the usbstick data is there but
stopped ~18jul06 21:40, thus we lost data overnight.

Stopped adam, swapped usbstick,
ck_goes was good.
Restart and everything looks ok.

Conclusion:
	I believe the station took a lightning surge and 'burped'.
	Decided not to reinstall the software just to see if it'd recover
	without that even though the 'zap' probably fouled it's dynamic
	memory if not other.
	Grounding here has always been a concern.  We installed 2 rods
	roughly 25' away but i'm thinking that the ground strap from the 
	elec box to mast lug should be removed with a hunch that ground
	strokes could be putting a surge on the ground.

Rnet:  I tried to clean it although it appeared basically clean.
	When doing so, i noticed water vapor condense briefly inside the dome.
	Didn't think much of it other than perhaps we need to change dessicant.
	However..
	when i looked at rserial i thought the values were 350-ish and ok,
	but since it's been back online the rnet data has been bad.
	returning with spare sensor.....

The data system went offline, perhaps due to the USB stick going offline) at 
2145Z or so 18 July during the day's monsoon 
storms. It is unclear if the storms were the cause but grounding issues are 
being investigated. The power was up at the system and the aspirator and other
systems were running. The GOES was fine as well (although it had nothing to 
send). The system was returned to functionality after the investigation.

17jul06 15:45-17:00

Purpose: routine maintenance

rnet cleaned 16:15
	appeared clean prior to 'cleaning'

17jul06 17:50-18:30

Purpose: routine maintence

USB Data storage retrieved and swapped
	Had trouble with the initial usb stick used
	(#13 which has been bad, questionable at several other sites previously
	but Greg wanted to try it one last time
	....io errors once again
	....performed minor sledge-hammer adjustment on this stick at
	base later on.)
	Installed 4G unit instead of .5 since no other available
	...swap it out for flux sites.


GOES pvc elbows installed, raised off ground to help ventillation.



Software upload.
	Double checked that the newest software version is installed here
	because it does have a SE110, and wanted to confirm the usbstick
	errors were not caused by the viper code for some strange reason.

17jul06 15:45-17:00

Purpose: routine maintenance

USB data storage retrieved and swapped

GOES pvc elbows installed, raised off ground to help improve ventillation.

See also rad cleaning, goes note

USB Drive fill times by station type – CUPIDO 16 July 2006

					 90%
				Space	Drive	Time to	
		Drive 	4hr fl	Used	Space	fill
		size	size	/day	Avail	USB drive
                (Mb)    (Mb)     (Mb)    (Mb)   (days)
		------	------	------	------	------	
Flux w/LiCor	4000	55.62	333.72	3600	10.79	Stn 5, SSE		Basic		 512	1.73	10.38	460.8	44.39	Stns 1,4,6,7,8,9
Flux		4000	27.80	166.8	3600	21.58	Stns 2,3 and 10
									
Since the only difference between St 5 and the other flux stations is the LiCor
it is somewhat surprising that the data volume is doubled. Will ask Gordon.

15Jul06 11:25 - 12:20

Purpose: Station appeared down beginning last night a 9pm after our visit
there.

Data System:
	The data files were on the usbdisk but we need to look at them a bit.
	GOES and it's stats_proc were down.
	Saw many errors in the messages/adam logs including goes length mis-
	match, wrong sequence number, stats_proc late messages and many
	gps errors of the type "clk_fault".  The statsproc for goes had died.
	See logs in: /data/projects/CuPIDO/logs/st2_071506_[adam,sys].log
	Looked at gps with minicom and daq off....seemed rather unusual with
	a lot of 2-5 char appearing short messages and then some normal ones.
	Checked software install and it appears OK in comparison with the
	source usbdisk; ie the upload last night seemed good.
	Just for giggles did a full box power cycle.
	Upon rebooting everything came back, goes looked good, etc.
	Note that in the ~5 hours since then even the periodic outages we
	had been noticing in the goes data (late messages in statsproc)
	have gone away for now.

Soil Sample Collected in tupperware for Dr.Oncley's enjoyment back in Boulder.

Moved Electric Fencer.
	Based on a hunch, moved fencer away from the solar/goes stand so that
	the ground rod is not dual-use with the goes surge arrester....perhaps
	that wide-band noise from the pulsing was clobbering the goes-sio.
	Better config. now, but must check this later since the new ground
	was just twisted tightly around one of the rebar fence posts and that
	could oxidize and mess up the fencers ground

14Jul06 14:00 - 17:30

Purpose: Intermittent outages last night / routine maintenance

DAQ was running.  USB was at 40% full since 3jul.

Installed new software,
Memory Error: Eep Block 0x002a0000 taken from free block had.. 0x0000064!!"
similar to what was seen at station 6 yesterday.  otherwise ok
Uploaded data

NOTE: Tower lowered 14:40 - 15:46 on 2 occassions
during cleaning and repositioning of the tsfc.

Other
Installed electric fence.....horses observed hanging around today.

Installed pvc cable feedthrough (vents) for goes box and raised it.
Antenna alignment very good.   SE110 1007

13Jul06 13:10-14:15

Purpose: Station plots had been down...ie goes down.

DAQ on arrival was ok, still recording.
This site still had the old version software, so suspecting that being the
usual 'slow sensor into stats-proc problem' and just for giggles i rebooted
and watched the logs.  1st xmit ok, 2nd/3rd missing and no indication in
either log file.   as expected.

Installed new software.  Retrieved local storage
Reboot.
everything ok with continuous goes 10W messages
This time with Gordon's new code it did provide an expected error message:
"statsproc Late Message TimeTag 20:47:30 expected"
and later looking at the data plots we are seeing holes in the 'slow logger'
values occassionally.  The logger messages were coming in nevertheless to
local storage.

Footnote: Rnet was clean and level,  RainGauge ditto.

The data USB drive was exchanged at NNW (10) this morning at approximately 1130am LT.

7Jul06 12:05-12:40

Purpose: installed new software
see next log entry

The latest software (.xml and GOES alts) was installed at ~ 1725.

The latest software was installed and the USB stick (data download) was exchanged. The data was copied to Aster this evening.

Weepholes were drilled in the lower plate of the electronics box as well.

Gordon's new software was installed and the 512Mb USB was exchanged (first exchange at st 4 of the experiment) - at approximately 1145 LT. This data was synched on Aster this evening.

2 weep holes were drilled in the lower plate corners of the electronics box.

5jul06 13:15

Purpose: install new software

Data system was still running and collecting data.

Data uploaded to aster.
New software installed and the xml also

5jul06 11:30-12:30


GOES Notes:
box here also had a bit of water in it despite being under the pv array.
SE120-3513, microstrip 9
However: measured 15W forward power!

5jul06 13:15

Purpose: install new software

Data system was still running and collecting data.

Data uploaded to aster.
New software installed and the xml also

5jul06 9:25-10:30

Purpose: station was down since last night 8gmt when a thunderstorm was in the area and rain.

Data system was still running and collecting data.

GOES was not responding to the dsm queries.  A power cycle got things going again.
Note: some water in the goes cooler box despite being under the pv array makes me concerned that this could be an issue so i taped the sio connector which was on the bottom of the box.

Data uploaded to aster.
New software installed but not the xml due to it's unique 'ttyS8' for logger.

4Jul06 10:00

Installed new code per my edit of Gordon's edit.  He crafted edit to improve situation when a 'slow' sensor's late message causes no goes output.  I changed the size limit on the data message to allow up to a full message (ie from st5 including licor) to get sent without dropping the packet.

3Jul06 14:50 - 15:50

Purpose: station was 'down'

In reality the station was still recording data but the goes transmitter was note getting data from the Viper.
The problem was the new xml file.   It had '244' characters to be sent and the viper code disallowed it via an exception condition:
"statsproc: SampleOutputBase: /dev/ttyS3 IOExeception /dev/ttyS3: send message too long 244 bytes"

Replaced the cupido.xml with the old version that doesn't send the Licor data and everything is working.

New GOES software was installed using the 'install' script. This installation failed to bring up the GOES transmission (data was being collected however, see next entry regarding LiCor). We will try again this afternoon because the USB software stick used in the failed attempt had some bad memory and needs to be reformatted.

The most recent version of cupido.xml was copied to the Viper at Site 10. The new C++ GOES code will be inserted during the next visit.

Weepholes were drilled today (corners of bottom plate).

Weep holes were drilled (two corners in bottom plate) in the electronics box as part of that ongoing effort.

2Jul06, 12:50-14:50 MST

Purpose: station down

Data System:
A USB failure caused the station to go down, not goes.

Yesterday, a new xml was installed but the station has been down since our visit.

Data system reported that the new usb disk was "Write Protected" and could not write to it: Lexar #12.
Checked it back at aster: nothing unusual with file protections EXCEPT:
I had put a copy of the new cupido.xml on it via root account.....doubt that
would be a problem and there were a couple of data files on it.

Swapped in Cruzer #1 and removed Lexar #12.

Rebooted and everything fine.

GOES:
Still need to reposition the antenna and pv's more, but there was a
good 10W output per the wattmeter and the angle looks close enough.

2Jul06, 12:50-14:50 MST

Purpose: station down, shoot boom angle, install new xml

Data System:
A USB failure caused the station to go down, not goes despite low sig-strength.
The USB indicated 'read-only' and the log file shows that it ran for awhile but died at 18Z on 1Jul06.
Most likely due to heat not the storms that occurred later.
Cruzer 4g-3 had been in the drive.

Several mount, boot, different usb attempts later finally got things working.
Also was able to upload the new xml.
When it worked, I installed the disks in the lower connector and maybe things had cooled and/or the 'reconnecting and fiddling' with the cables re-established conductivity perhaps.
IE. this station may go down for the same reason again.

GOES:
10W output just fine after restarting nidas.
Re-configured the arrangement a bit:
lowered antenna so it wouldn't wobble in the wind, and
secured the flimsey cabling inside the funky cooler box by cable tying the
surge arrestor to the body of the transmitter.
SE120 -  microStrip3
everything appears good.

RMY Boom Direction:
Shot boom angle and put it into the rmy: 277.
Verified consistency with observed conditions

CSAT pointing towards 179

2Jul06, 9:30-10:30 MST

Swapped and uploaded data:
out Cruzer 4g-4, in MiniCruzer

New Software installed per edit / build this morning after talking with Gordon.
This attempt was to overcome possible goes/station lock by late statistics processing.

Oops!  This install was later determined to be the wrong code which caused the station to go down....remains to be seen if data loss....

7/1/06 13:30-15:35 local

Purpose: Station had been down, Rnet odd

Data System:
- Initial checkout: showed data was being collected but many 'stats_proc' errors of bad time tag in the log.  I copied the log to the bison pc.
Aside, procomm connect to the adam had garbage characters instead of printable.  Restarted procomm 3 times before it was normal...may have been port?
ck_goes showed the se120 was not responding.  A power cycle fixed that except the next ck_goes had errors such as 'clock not loaded.'   Nevertheless, i watched at least 2 more cycles and there was no transmit.  A third ck_goes also failed.
At this point I rebooted the viper and after that the goes did work.
- installed new xml file:
- swapped usb: cruzer #512-1 out; lexar #512-11 in
- installed new xml loaded
- bootup OK
- goes seems OK
- Logger down.

Logger Problems:
Initially we thought we'd seen the logger reporting in a data_stats, however after the new xml was loaded, it was not responding.  Tests revealed it was working on ttyS2, ttyS6, ttyS8 and the trh for the most part was working on ttyS5 although not every time.  Determined ttyS5 (cable/board?) is bad and moved the logger to ttyS8 and it worked.  This mean't editing the xml file:
NOTE NOTE NOTE: st1 has a unique xml file locally that puts the logger on ttyS8 insteal of ttyS5 !!


Note: Gordon said that the 'stats_proc' timetag errors were probably from a slow responding sensor not getting the values transferred in time.  This may also have caused the goes lock-up.  So it is possible that the logger on a bad port may have been the root problem.  Perhaps even the wierdness on the rnet....we'll keep watching.