PLOTSCATTER: A Program to retrieve errors from radar data

***********************************************

1 Introduction

Plotscatter is a program developed by Wen-Chau Lee to retrieve the majority of errors in ELDORA data due to the Inertial Navigation System (INS) or antenna mounting errors. This includes errors in aircraft altitude, speed, drift, vertical velocity, pitch and roll, radar range delay, antenna spin angle (rotation angle) and tilt angle. This routine is also applicable to P-3 data.

1.1 Overview

Plotscatter is an iterative program that utilizes the radar echoes returned from the earth's flat surface to calculate the corrections and is written in Fortran (F77) and contains calls to NCAR Graphics. In addition, the data for plotscatter is in the form of ground echo files generated by the translator. Please refer to Appendix E for a description of the format of the ground echo files. Running plotscatter involves repeating the following sequence of steps multiple times.

1. Creating ground echo files

2. Running plotscatter

3. Interpreting results and adding to the correction factor files (one for each radar).

There is a perl script available to automate the iterative process.

Once the corrections have been calculated (either interactively or automated), they can be applied to the data via the translator or solo.

1.2 Choosing Tilt Angles

The tilt angle for waveguide antenna should be known accurately other than the mounting error. In the mathematical derivation, there are 8 unknowns and only 7 equations. Therefore, one unknown has to be pre-determined before the set of equations can be solved. It is decided that the tilt angle can be pre-determined. A list of possible tilt angle corrections (e.g., -0.5 to 0.5 degree in 0.1 deg increment) is prescribed to obtain corresponding corrections on other 7 variables. Since some unknowns should have the same corrections for fore and aft radars, the tilt angle corrections are determined by finding the closest match between fore and aft radars among these runs. The experience from past experiments (e.g., VORTEX and LakeIce) suggested that for ELDORA data, the tilt angle corrections are aft .30° and fore -.20°. All other errors are calculated based on the assumption that the initial tilt angles are correct. The tilt angles can be adjusted if the results are not satisfactory. Tilt angle errors on the P-3 antenna can be found using similar procedures.

1.3 Criteria For Choosing Data

In order to maximize the accuracy of plotscatter, the data must be carefully chosen. The data should reflect a five to ten minute data set where the earth surface is flat. Additionally, the desired aircraft altitude should be anywhere above 3 km and the aircraft should not be climbing or descending. Because the echoes returned from the surface are based on values with high reflectivity and near zero velocity, the data should not be in a stormy environment, where reflectivity of 45 dBZ and higher are near the surface, as the echoes returned from the storm may be confused with the earth surface.

1.4 Accuracy

Due to the intrinsic accuracy of the instruments (e.g., INS, radar altimeter, etc.), the accuracy of each parameter is shown below:

The accuracy shown in the table suggests that drift and ground speed have the largest uncertainty.

1.5 Plotscatter Modes (Separate; Combine)

There are two modes in which to run plotscatter: separate and combined. Initially, each radar should be run separately with the initial tilt angles until the correct tilt angle is established and the results are satisfactory. If necessary, the tilt angles can be adjusted (usually by .05° - 1°). This process identifies radar dependent correction factors, such as, tilt angle, rotation angle, and range delay. Once the results returned from running each radar separately are sufficient, then plotscatter should be run in combined mode to obtain the final set of corrections on radar independent parameters, such as drift, pitch, ground speed, aircraft altitude. The procedure converges when changes in each parameters on successive iterations fall within the tolerance in the above table.

2 Running Interactively

2.1 Necessary Files

2.1.1 Translator Script

The first step in running plotscatter involves generating the files that contain the gates of each beam that intersect the earth surface, hereafter referred to as ground echo files. These files are generated by the translator and have a naming convention similar to sweep files except the prefix is 'gde' as opposed to 'swp'. Please refer to Appendix A for sample translator scripts.

2.1.2 Plotscatter Input File

In addition to the ground echo files, plotscatter uses an input file to set run-time parameters. The file must be named scatter.in and a brief description is listed in the table below:

The values listed as 'User Specify' are the values that are usually edited before running plotscatter. Please refer to Appendix C for a sample scatter.in file.

2.1.3 CFAC Files

The CFAC Files are ascii files, one for each radar, that contain the corrections (generated in plotscatter) to be applied to the data. The environment variable CFAC_FILES must be set in the translator script in order to apply the corrections to the data. Please refer to Appendix C for sample CFAC files.

2.1.4 Data

The data can be in either ELDORA field format or DORADE and can be accessed via tape or disk file. Due to the iterative nature of plotscatter, it is more time efficient to put the data on disk.

2.2 Running plotscatter

2.2.1 Separate Mode

In order to run plotscatter separately on each radar, the radarflag must be set to either 1 (fore) or 2 (aft) in scatter.in.

The following is a list of steps to be followed to run plotscatter by hand.

1. Initialize cfac files with all values set to 0.0 except for the initial tilt angles.

2. Save CFAC files to reflect the current iteration and radar (i.e.: cfac.aft.1, cfac.fore.1, cfac.aft.2..etc).

3. Create ground echo files by running the translator. Make sure that the translator script includes the CFAC_FILES environment variable.

4. Edit scatter.in to include the proper ground echo file names and radar (1 = fore; 2 = aft).

5. Run plotscatter by typing the name of the executable (plotscatterN). The values of the coefficients will scroll by on the screen and it is helpful to scan these to make sure that none of the values are set to NaN, a sign indicating that something is wrong.

6. Save the gmeta file to reflect the current iteration and radar (i.e.: gmeta.aft.1, gmeta.fore.1, gmeta.aft.2...etc).

7. View the gmeta file produced by plotscatter. If the plots look satisfactory, then eyball the averages (throwing out the outliers) for the following plots and add the averages to the values in the CFAC files. The following table lists the plotscatter output variable and the corresponding cfac file entry:

7. Repeat steps 2 - 6 until all the corrections are converging and within the acceptable tolerance.

2.2.2 Combine Mode

Running plotscatter in combine mode is similar to running each radar separately except for the scatter.in radarflag is set to 0 and the CFAC files are initialized with the final CFAC files from running plotscatter in the separate mode. The CFAC files should contain radar dependent corrections (i.e., tilt, rotation angle, range delay) while radar independent variables (i.e., pitch, drift, ground speed and altitude) can be set to zero or the average value of the two separate runs.

In the following example, the radar independent corrections are the average of the two separate runs.

3 Running Automated Version

3.1 Introduction

The perl script (plotscatter.pl) is a wrapper script that runs plotscatter, but automatically handles all the steps required in the interactive version.

3.2 Necessary Files

3.2.1 Translator Script

This is a template translator script that plotscatter.pl uses to generate the translator script that creates the ground echo files. The following environment variables must exist in the template script:

setenv CFAC_FILES

setenv DORADE_DIR

setenv SOURCE_DEV

setenv TIME_LIMITS

The values to these variables are updated by plotscatter.pl to reflect the current data set so it doesn't matter what the actual values are in the template script. Please refer to Appendix A for an example of translator scripts that produce ground echo files.

3.2.2 plotscatter.pl input file

This file is an ascii file where the user can set all the parameters necessary to run plotscatter.pl. The file is divided into 3 sections:

· translator variables: used to create the translator script

· scatter.in variables: used to create the plotscatter input file (scatter.in)

· threshold/tolorance variables: used to determine when to stop updating corrections factors

A brief description is listed in the table below:

The values listed as 'User Specify' are the values that are usually modified by the users before running plotscatter.pl. Please refer to Appendix D for a sample input file for plotscatter.pl and a more detailed description of the parameters.

3.2.3 Data

The data can be in either ELDORA field format or DORADE and can be accessed via tape or disk file. Due to the iterative nature of plotscatter, it is more time efficient to put the data on disk.

3.3 Running plotscatter.pl

3.3.1 Recommended Directory Structure

Since plotscatter.pl generates a variety of output files, it is important that the files generated from the single and combine mode are in separate directories to avoid files inadvertently being renamed. Listed below is a sample directory structure:

/VORTEX/950507/0057/single:

where all the output files from the single run are contained

/VORTEX/950507/0057/combine:

where all the output files from the combine run are contained

/VORTEX/950507/0057/files:

where all the necessary files to run plotscatter.pl are contained (i.e.: translator script, input file..etc).

3.3.2 Command Line Arguments

If plotscatter.pl is run without any arguments in the command line, then the following list of options will appear:

options:

-a: tilt angle aft

-f: tilt angle fore

-i: batch input filename

-m: mode (separate or combine)

-A: *aft cfac file (use if mode=combine)

-F: *fore cfac file (use if mode=combine)

-p: plotscatter executable (optional)

*note: Option -A and -F should be the final

cfac file from running plotscatter.pl in separate

mode.

Example:

plotscatter.pl -a.30 -f-.20 -i1234.inp -mcombine

-A../single/1234.cfac.aft -F../single/1234.cfac.fore

3.3.3 Running plotscatter.pl

The first line in plotscatter.pl executes perl ( #!/localbin/perl ). Therefore, if the perl executable is in a different path, this line may need to be altered to reflect the correct path.

Additionally, plotscatter.pl contains the default location of the plotscatter executable (plotscatterN) with the following line:

# default name of plotscatter executable

$opt_p = "/scr/science19/snorman/src/plotscatter/scatterdir/plotscatterN";

Although the path name of the executable can be changed with the -p command line option, the default path must be changed in plotscatter.pl.

Listed below are the steps required to run plotscatter.pl. The examples are utilizing the recommended directory structure discussed previously.

1. Assemble all the necessary files in /VORTEX/950507/0057/files

2. Go to the directory where the output from the single mode should go

eg: cd /VORTEX/950507/0057/single

3. Edit the plotscatter.pl input file with the appropriate values ( date, time, data location..etc).

4. Run plotscatter.pl in single mode:

eg: plotscatter.pl -a.30 -f-.20 -i../files/0057.inp -msingle

The example above assumes that location of plotscatter.pl is included in the default path.

5. Run plotscatter.pl in combine mode:

Once the results from the single run are satisfactory, go to the directory where the output from the combined mode should go and run plotscatter in combined mode:

eg: cd /VORTEX/950507/0057/combine

plotscatter.pl -a.30 -f-.20 -i../files/0057.inp -mcombine

-A../single/0057.cfac.aft -F../single/0057.cfac.fore

The -A and -F options reflect the final cfac files from the single run and are necessary to run in combined mode.

APPENDIX A: Sample translator script to create ground echo files

#!/bin/csh

#

#script to ground echo files from an eldora dorade tape

#

setenv AC_NETCDF_FILES /scr/science22/dmg/raf_cdf/vortex/07may95.cdf

#

setenv AC_NETCDF_ALIASES "LATC < XLATC LONC < XLONC PALT < PALT HGME < HGME \

PITCH < NOT ROLL < NOT VEWC < XVEWC VNSC < XVNSC THDG < NOT \

UIC < GVEW VIC < GVNS WIC < VSPD"

#

setenv INPUT_FORMAT "DORADE_FORMAT"

#

setenv DORADE_DIR ../data

#

setenv SOURCE_DEV /dev/rmt/1bn

#

setenv OUTPUT_FLAGS "GECHO_DATA"

#

setenv NCP_THRESHOLD_VAL "NCP 0.25"

#

setenv ALTITUDE_LIMITS "-5. < 22.5"

#

setenv COMPRESSION_SCHEME "HRD_COMPRESSION"

#

setenv FIRST_GOOD_GATE 2

#

setenv TIME_LIMITS "05/08/95:00:57 < 05/08/95:01:03"

#

setenv DERIVED_FIELDS "DEFAULT(VR > VRB1 VRB1 > VG VG > VT) \

DUAL_PRT(VS > VQB1 VQB1 > VG VG > VT)"

#

# the correction files

setenv CFAC_FILES "TF > 0057.cfac.fore \

TA > 0057.cfac.aft"

#

# field for gates that intersect the ground

setenv GECHO_VEL "VG"

#

setenv BATCH_MODE

#

# fix for record size bug

setenv OPTIONS "DESC_SEARCH"

#

/localbin/xltrs

APPENDIX B: Sample plotscatter Input File (scatter.in)

gde/gde.950508005700.TF-ELDR.0

gde/gde.950508005700.TA-ELDR.0

points.out

2 radarflag !(0=both,1=fore,2=aft)

ELDORA radar !char string (7chars)

0.1 stdw !std. deviation of vertical wind

.02 stdbeta !std. deviation of pitch angle

.10 stdb2 !std. b2

10.0 stdc !std. c

.01 mu !lagrangian multiplier

100.0 mu1 !lagrangian multiplier for variable e (2nd time)

1.0 mu2 !lagrangian multiplier for deltabeta and deltaw

1.0 stdval(1) !is the data within this many std.

1.0 stdval(2) ! deviations?

10 imin !plot the scans between these

20 imax ! values (scan imin to scan imax are plotted)

.true. direct_sol !true is lagrangian multiplier,false=gauss

124 number of scans to process

APPENDIX C: Sample CFAC Files

FORE Radar:

azimuth_corr = 0.000

elevation_corr = 0.000

range_delay_corr = 107.420

longitude_corr = 0.000

latitude_corr = 0.000

pressure_alt_corr = -0.051

radar_alt_corr = 0.000

ew_gndspd_corr = -0.900

ns_gndspd_corr = 0.000

vert_vel_corr = 0.000

heading_corr = 0.000

roll_corr = 0.000

pitch_corr = -1.240

drift_corr = -0.030

rot_angle_corr = 2.610

tilt_corr = -0.200

AFT Radar:

azimuth_corr = 0.000

elevation_corr = 0.000

range_delay_corr = 141.250

longitude_corr = 0.000

latitude_corr = 0.000

pressure_alt_corr = -0.042

radar_alt_corr = 0.000

ew_gndspd_corr = 1.590

ns_gndspd_corr = 0.000

vert_vel_corr = 0.000

heading_corr = 0.000

roll_corr = 0.000

pitch_corr = -1.380

drift_corr = -0.320

rot_angle_corr = -1.080

tilt_corr = 0.300

APPENDIX D: Sample plotscatter.pl input file

#***************************************************

# This is a sample input file used in plotscatter.pl

#***************************************************

#

#***translator variables****************************

trans_fname = ../files/dor2gde ! name of template translator script

date_begin = 05/08/95 ! begin date MM/DD/YY

time_begin = 00:57:00 ! begin time HH:MN:SS

date_end = 05/08/95 ! end date MM/DD/YY

time_end = 01:03:00 ! end time HH:MN:SS

dor_dir = gde ! directory for the ground echo files

data_fname = ../files/0057.data ! data filename or device filename

#fskip = 8 ! number of files to skip if data is on tape

#

#***scatter.in variables***************************

radar_name = ELDORA ! radar 7 character string

stdw = 0.1 ! std deviation of vertical wind

stdbeta = .02 ! std. deviation of pitch angle

stdb2 = .10 ! std. b2

stdc = 10.0 ! std. c

mu = .01 ! lagrangian multiplier

mu2 = 1.0 ! lagrangian multiplier for deltabeta and deltaw

mu1 = 100.0 ! lagrangian multiplier for variable e

! (2nd time)

stdval1 = 1.0 ! is the data within this many std.

stdval2 = 1.0 ! deviations?

imin = 10 ! plot the scans between these

imax = 20 ! values (scan imin to scan imax are plotted)

direct_sol = .true. ! true is lagrangian multiplier,false=gauss

scan_count = 500 ! number of scans to process

#

#***threshold / tolorance variables***************************

num_iter = 5 ! number of iterations

alt_av_thresh = .20 ! altitude average threshold

alt_std_thresh = .20 ! altitude std threshold

alt_tol = 10 ! altitude tolorance (m)

scan_thresh = 10 ! scan threshold

fore_range_tol = 20 ! fore range tolorance (m)

aft_range_tol = 20 ! aft range tolorance (m)

grndspd_tol = .40 ! ground speed tolorance (m/s)

pitch_tol = .10 ! pitch tolorance (deg)

drift_tol = .10 ! drift tolorance (deg)

rot_tol = .10 ! rotation angle tolorance (deg)

APPENDIX E: Ground Echo File Format