RAF's primary datasets consist of synchronous time series data. Data are written to the netCDF file one record per second, even for data at higher rates.
Data acquisition of numerous instruments at high-rate (from 1 to 5K samples per second) for 10 to 11 hours can produce large data files. Typical, raw packed data files from the NCAR C-130 and GV are 2-3 GB, and unpacked with no data reduction would be twice that size. Files reduced to 1 sample per second (sps) are typically 100-150 MB.
Note: For all size-distributions (PMS1D, PMS2D, RDMA) the zeroth or first (depending on how you count) bin is undefined, do not use this value. Attributes for the various size distribution variables should include FirstBin and LastBin.
There is also an occasional need to look at data at the sample rate.
RAF uses a naming convention consisting of a project designator followed by the flight type, followed by a 2 digit flight number, followed by an indicator of sample rate ( 'h' to indicate high-rate data, low-rate data has no indicator), followed by ".nc". Prior to about 2005/2006, field projects were designated with a 3 digit number. With the adoption of ADS-3 / nidas in 2005/2006, a switch was made to upper-case project names.
Flight types used by RAF are "ff", "tf", and "rf".
In our files RPO stands for "Right Pod Outboard".
Time is the unlimited dimension, and every variable (except base_time) has the Time dimension. No more than one record is to be written per second; all high-rate data are to be contained "per record", though data averaged down slower than 1 sps may be written (e.g., when you have 1 sample every 10 seconds).
A new dimension has been added called Async. Async is a triplet containing the location of the asynchronous data in a separate binary "heap" file. The first value contains the offset in bytes into the file of the first record for that second, the second number tells the record length, and the third is the number of records for that particular second. Async will always be in the 3rd (or Vector) dimension, e.g., float 2D-P_LPI(Time, sps1, Async). Async was added to facilitate future capabilities of being able to provide satellite images, PMS-2D data, and other asynchronous products into one "heap" file. (Note: I have yet to use this feature, so implementation of file name, etc., are incomplete).
int Time(Time) ; Time:long_name = "time of measurement" ; Time:standard_name = "time" ; Time:units = "seconds since 2004-08-02 16:46:11 +0000" ; Time:strptime_format = "seconds since %F %T %z" ;
// global attributes: :institution = "NCAR Research Aviation Facility" ; :Address = "P.O. Box 3000, Boulder, CO 80307-3000" ; :Phone = "(303) 497-1030" ; :creator_url = "http://www.eol.ucar.edu" ; :Conventions = "NCAR-RAF/nimbus" ; :ConventionsURL = "http://www.eol.ucar.edu/raf/Software/netCDF.html" ; :ConventionsVersion = "1.3" ; :Metadata_Conventions = "Unidata Dataset Discovery v1.0" ; :standard_name_vocabulary = "CF-1.0" ; :ProcessorRevision = "4927" ; :ProcessorURL = "http://svn/svn/raf/trunk/nimbus" ; :date_created = "2010-06-28T20:27:34 +0000" ; :ProjectName = "HIPPO-3" ; :Platform = "N677F" ; :ProjectNumber = "HIPPO-3" ; :FlightNumber = "rf08" ; :FlightDate = "04/10/2010" ; :TimeInterval = "19:27:23-03:36:43" ; :InterpolationMethod = "Linear" ; :latitude_coordinate = "LATC" ; :longitude_coordinate = "LONC" ; :zaxis_coordinate = "GGALT" ; :time_coordinate = "Time" ; :geospatial_lat_min = -14.33873f ; :geospatial_lat_max = 20.13709f ; :geospatial_lon_min = -170.7218f ; :geospatial_lon_max = -154.5442f ; :geospatial_vertical_min = 3.815f ; :geospatial_vertical_max = 13780.87f ; :geospatial_vertical_positive = "up" ; :geospatial_vertical_units = "m" ; :wind_field = "WSC WDC WIC" ; :landmarks = "39.9083 -105.116 BJC,61.1741 -149.996 ANC,19.6475 -156.013 KOA,-43.4893 172.532 NZCH" ; :Categories = "Position,Thermodynamic,Aircraft State,Atmos. State,Liquid Water,Uncorr\'d Raw,Wind,PMS Probe,Housekeeping,Chemistry,Radiation,Non-Standard" ; :time_coverage_start = "2010-04-10T19:27:23 +0000" ; :time_coverage_end = "2010-04-11T03:36:43 +0000" ;
coordinates is required starting with version 1.3. These
are the aircrafts "best" position variables.
TimeInterval is required by ncplot and ncpp, but I hope to get away from that.
All variables contain the basic attributes units, long_name, and _FillValue (missing_value is being deprecated). In addition we provide some other attributes; not all are used for every variable:
standard_name we have adopted the CF CF conventions section 3.3 use of standard_name where applicable.
SampledRate is the rate (sps) at which the variable was sampled onboard the aircraft. This does not apply to derived variables.
DataQuality attempts to inform the user of the quality of these data. Some values are Bad, Preliminary and Good
Category Since a file can contain hundreds of variables with not very descriptive names, this is provided to help create sub-lists of variables. Comma separated list.
CalibrationCoefficients are the values used to produce engineering units from a measurement's DC voltage. It is used by the analog/digital group. These values have already been applied! They are present for documentation.
Dependencies are the input variables that were used to produce this derived value. Like CalibrationCoefficients , these are present for documentation.
For processing purposes, variables in our files are organized into 3 sorted lists. The first sorted list consists of Analog or Digitally-sampled variables. These only have calibration coefficients applied to them. The second group is raw data from "block" probes (e.g., PMS1D, IRS, GPS, and a few from group 1 that need special processing). The third group consists of purely derived variables. I have included one example from each of the 3 groups (which also show different dimension schemes):
float PITCH(Time) ; PITCH:_FillValue = -32767.f ; PITCH:units = "degree" ; PITCH:long_name = "IRS Aircraft Pitch Angle" ; PITCH:standard_name = "platform_pitch_angle" ; PITCH:valid_range = -180.f, 180.f ; PITCH:actual_range = -3.31724 15.88532f ; PITCH:Category = "Analog" ; PITCH:SampledRate = 50 ; PITCH:TimeLag = -180 ; PITCH:TimeLagUnits = "milliseconds" ; PITCH:DataQuality = "Preliminary" ; PITCH:CalibrationCoefficients = 0.26f, 1.f ; float CFSSP_RPC(Time, sps1, Vector16) ; CFSSP_RPC:_FillValue = -32767.f ; CFSSP_RPC:units = "#/cm3" ; CFSSP_RPC:long_name = "FSSP-100 Concentration (per cell)" ; CFSSP_RPC:Category = "PMS Probe" ; CFSSP_RPC:SerialNumber = "FSSP076" ; CFSSP_RPC:DataQuality = "Preliminary" ; CFSSP_RPC:Dependencies = "2 AFSSP_RPC TASX" ; CFSSP_RPC:FirstBin = 1 ; CFSSP_RPC:LastBin = 15 ; CFSSP_RPC:CellSizes = 2.f, 5.f, 8.f, 11.f, 14.f, 17.f, 20.f, 23.f, 26.f, 29.f, 32.f, 35.f, 38.f, 41.f, 44.f, 47.f, 2.f, 4.f, 6.f, 8.f, 10.f, 12.f, 14.f, 16.f, 18.f, 20.f, 22.f, 24.f, 26.f, 28.f, 30.f, 32.f, 1.f, 2.f, 3.f, 4.f, 5.f, 6.f, 7.f, 8.f, 9.f, 10.f, 11.f, 12.f, 13.f, 14.f, 15.f, 16.f, 0.5f, 1.f, 1.5f, 2.f, 2.5f, 3.f, 3.5f, 4.f, 4.5f, 5.f, 5.5f, 6.f, 6.5f, 7.f, 7.5f, 8.f ; CFSSP_RPC:CellSizeUnits = "micrometers" ; CFSSP_RPC:CellSizeNote = "CellSizes are upper bin limits as diameter." ; CFSSP_RPC:HistogramNote = "Zeroth data bin is an unused legacy placeholder." ; CFSSP_RPC:DepthOfField = 2.7f ; CFSSP_RPC:BeamDiameter = 0.18f ; CFSSP_RPC:Density = 1.f ; CFSSP_RPC:PLWfactor = 1.e-06f ; CFSSP_RPC:DBZfactor = 1000000.f ; float WIC(Time, sps25) ; WIC:_FillValue = -32767.f ; WIC:units = "m/s" ; WIC:long_name = "GPS-Corrected Wind Vector, Vertical Gust Component" ; WIC:standard_name = "upward_air_velocity" ; WIC:actual_range = -3.393437 5.762867f ; WIC:Category = "Wind" ; WIC:DataQuality = "Good" ; WIC:Dependencies = "9 TASX VEWC VNSC PITCH ROLL THDG ATTACK SSLIP VSPD" ;
|PMS1D||# bins||Counts name||Concentration name||DMT Converted||# bins||Counts name||Concentration name|
|2D-C (entire-in algo)||32||A1DC_xxx||C1DC_xxx||Fast2DC||64||A1DC_xxx||C1DC_xxx|
|2D-C (center-in algo)||64||A2DC_xxx||C2DC_xxx||Fast2DC||128||A2DC_xxx||C2DC_xxx|
DMT converted probes can record either 10, 20, 30 or 40 bins of data. To maintain backwards compatability, we pad the front of the array with a zero and increase the number of bins by one (11, 21, 31 or 41).
NOTE: Always ignore the very first bin of any size-distribution histogram in our netCDF files.
e.g. The bin limits for CS100[n] are CellSizes[n-1] and CellSizes[n], lower and upper respectivily.
float AS100_LPC(Time, sps1, Vector31) ; AS100_LPC:_FillValue = -32767.f ; AS100_LPC:units = "count" ; AS100_LPC:long_name = "FSSP-100 Raw Accumulation (per cell) - DMT" ; AS100_LPT:actual_range = 0.f, 3914.f ; AS100_LPC:Category = "PMS Probe" ; AS100_LPC:SerialNumber = "FSSP122" ; AS100_LPC:SampledRate = 10 ; AS100_LPC:DataQuality = "Good" ; float CS100_LPC(Time, sps1, Vector31) ; CS100_LPC:_FillValue = -32767.f ; CS100_LPC:units = "#/cm3" ; CS100_LPC:long_name = "FSSP-100 Concentration (per cell) - DMT" ; CS100_LPC:Category = "PMS Probe" ; CS100_LPC:SerialNumber = "FSSP122" ; CS100_LPC:DataQuality = "Good" ; CS100_LPC:Dependencies = "5 AS100_LPC TASX REJAT_LPC OVFLW_LPC FRNG_LPC" ; CS100_LPC:FirstBin = 3 ; CS100_LPC:LastBin = 30 ; CS100_LPC:CellSizes = 0.7f, 1.4f, 2.35f, 3.9f, 5.45f, 7.f, 8.55f, 10.1f, 11.65f, 13.2f, 14.75f, 16.3f, 17.85f, 19.4f, 20.95f, 22.5f, 24.05f, 25.6f, 27.15f, 28.7f, 30.25f, 31.8f, 33.35f, 34.9f, 36.45f, 38.f, 39.55f, 41.1f, 42.65f, 44.2f, 45.75f ; CS100_LPC:CellSizeUnits = "micrometers" ; CS100_LPC:CellSizeNote = "CellSizes are upper bin limits as diameter." ; CS100_LPC:HistogramNote = "Zeroth data bin is an unused legacy placeholder." ; CS100_LPC:DepthOfField = 2.3f ; CS100_LPC:BeamDiameter = 0.2f ; CS100_LPC:Density = 1.f ; CS100_LPC:PLWfactor = 1.e-06f ; CS100_LPC:DBZfactor = 1000000.f ; float CONCF_LPC(Time) ; CONCF_LPC:_FillValue = -32767.f ; CONCF_LPC:units = "#/cm3" ; CONCF_LPC:long_name = "FSSP-100 Concentration (all cells)" ; CONCF_LPT:actual_range = 0.f, 437.7162f ; CONCF_LPC:Category = "PMS Probe" ; CONCF_LPC:SerialNumber = "FSSP122" ; CONCF_LPC:DataQuality = "Good" ; CONCF_LPC:Dependencies = "1 CS100_LPC" ;
National Center for Atmospheric Research
Research Aviation Facility