NCAR-RAF netCDF Conventions

Version 1.3

National Center for Atmospheric Research

Research Aviation Facility

NOAA AOC (GIV)

University of Wyoming (King Air)

Table of Contents

Appendices

Introduction

The goal is to support synchronous time series including the following properties:

RAF's primary datasets consist of synchronous time series data. Data are written to the netCDF file one record per second, even for data at higher rates.

Data acquisition of numerous instruments at high-rate (from 1 to 5K samples per second) for 10 to 11 hours can produce large data files. Typical, raw packed data files from the NCAR C-130 and GV are 2-3 GB, and unpacked with no data reduction would be twice that size. Files reduced to 1 sample per second (sps) are typically 100-150 MB.

1.0.1 Vector data

Instruments which measure particle size distributions produce histograms or vector data. These instruments include the suite of PMS-1D probes, the RDMA and quantitative data reduction from the PMS-2D probes.

Note: For all size-distributions (PMS1D, PMS2D, RDMA) the zeroth or first (depending on how you count) bin is undefined, do not use this value. Attributes for the various size distribution variables should include FirstBin and LastBin.

1.0.2 Multi-rate files

RAF's standard data product is a netCDF file where all measurements are output at 1 sps. Certain projects require high-rate data. RAF has standardized at 25 sps for high-rate data. In general we don't want to put out variables sampled at 1 sps at 25 sps, as there is no information in the data, and it would take up excess space.

There is also an occasional need to look at data at the sample rate.

NetCDF File Component Descriptions

1 Filename

RAF uses a naming convention consisting of a project designator followed by the flight type, followed by a 2 digit flight number, followed by an indicator of sample rate ( 'h' to indicate high-rate data, low-rate data has no indicator), followed by ".nc". Prior to about 2005/2006, field projects were designated with a 3 digit number. With the adoption of ADS-3 / nidas in 2005/2006, a switch was made to upper-case project names.

e.g.

PREDICTtf01.nc would be low-rate data from test flight 1 from the PREDICT project.
131ff01.nc would be low-rate data ferry flight 1 from project 131.
803rf17h.nc would be high-rate data from research flight 17 of project 803.

Flight types used by RAF are "ff", "tf", and "rf".

2 Data types

All data are stored using type float, except for base_time which is of type int.

3 Naming conventions

Variable, dimension and attribute names should begin with a letter and be composed of letters, digits, and underscores.

3.1 Variable names

The underscore has special meaning for variable names. Anything following an underscore is considered a suffix for grouping variables from the same instrument or probe. There is no standard for suffix names, but RAF regularly uses about a dozen specific ones for describing an instrument's mounting location on the aircraft.

e.g.,

AFSSP_RPO, CFSSP_RPO, PLWC_RPO, CONCF_RPO all belong to the same intrument.

In our files RPO stands for "Right Pod Outboard".

3.2 Dimension names

Dimension names we use are: although I don't believe it matters what they are called.

4 Dimensions

4.1 First dimension - Time

Time is the unlimited dimension, and every variable (except base_time) has the Time dimension. No more than one record is to be written per second; all high-rate data are to be contained "per record", though data averaged down slower than 1 sps may be written (e.g., when you have 1 sample every 10 seconds).

4.2 Second dimension - data rate

The second dimension, if present, represents the number of samples per second. The size of this dimension will vary according to the output rate of the variable (e.g., RAF uses 1, 5, 10, 25, 50, 250, and 1000). If the dimension does not exist, then there is only one sample for the second.

4.3 Third dimension - vector length

The third dimension, if present, is the vector length of the data. (This will be present only for variables with vector data). Only PMS-1D probes are producing vector data at this time.

A new dimension has been added called Async. Async is a triplet containing the location of the asynchronous data in a separate binary "heap" file. The first value contains the offset in bytes into the file of the first record for that second, the second number tells the record length, and the third is the number of records for that particular second. Async will always be in the 3rd (or Vector) dimension, e.g., float 2D-P_LPI(Time, sps1, Async). Async was added to facilitate future capabilities of being able to provide satellite images, PMS-2D data, and other asynchronous products into one "heap" file. (Note: I have yet to use this feature, so implementation of file name, etc., are incomplete).

5 Variables

http://www.cfconventions.org/.

5.1 Time

Time follows the recomendations set forth in the CF conventions section 4.4. 
        int Time(Time) ;
                Time:long_name = "time of measurement" ;
                Time:standard_name = "time" ;
                Time:units = "seconds since 2004-08-02 16:46:11 +0000" ;
                Time:strptime_format = "seconds since %F %T %z" ;

Pre version 1.3 time was represented as follows, it is still included for backwards compatibility but is being deprecated.

Time is stored as a base_time and time_offset pair. base_time is standard Unix time (seconds since Jan 1, 1970), and is typically the time stamp of the first record in the file. base_time has only one value, and does not contain the time dimension. time_offset is added to base_time to give the time of the current record. 

        int base_time ;
                base_time:units = "seconds since 1970-01-01 00:00:00 +0000" ;
                base_time:long_name = "Start time of data recording." ;
        float time_offset(Time) ;
                time_offset:long_name = "Seconds since base_time." ;
                time_offset:units = "seconds since 2004-08-02 16:46:11 +0000" ;

5.2 Missing data

At this time we use the _FillValue attribute.

Prior to 01/27/2003 we used missing_value as the missing value. missing_value (pre 1998 spelled MissingValue) is being deprecated.

6 Attributes

6.1 Global attributes

// global attributes:
                :institution = "NCAR Research Aviation Facility" ;
                :Address = "P.O. Box 3000, Boulder, CO 80307-3000" ;
                :Phone = "(303) 497-1030" ;
                :creator_url = "http://www.eol.ucar.edu" ;
                :Conventions = "NCAR-RAF/nimbus" ;
                :ConventionsURL = "http://www.eol.ucar.edu/raf/Software/netCDF.html" ;
                :ConventionsVersion = "1.3" ;
                :Metadata_Conventions = "Unidata Dataset Discovery v1.0" ;
                :standard_name_vocabulary = "CF-1.0" ;
                :ProcessorRevision = "4927" ;
                :ProcessorURL = "http://svn/svn/raf/trunk/nimbus" ;
                :date_created = "2010-06-28T20:27:34 +0000" ;
                :ProjectName = "HIPPO-3" ;
                :Platform = "N677F" ;
                :ProjectNumber = "HIPPO-3" ;
                :FlightNumber = "rf08" ;
                :FlightDate = "04/10/2010" ;
                :TimeInterval = "19:27:23-03:36:43" ;
                :InterpolationMethod = "Linear" ;
                :latitude_coordinate = "LATC" ;
                :longitude_coordinate = "LONC" ;
                :zaxis_coordinate = "GGALT" ;
                :time_coordinate = "Time" ;
                :geospatial_lat_min = -14.33873f ;
                :geospatial_lat_max = 20.13709f ;
                :geospatial_lon_min = -170.7218f ;
                :geospatial_lon_max = -154.5442f ;
                :geospatial_vertical_min = 3.815f ;
                :geospatial_vertical_max = 13780.87f ;
                :geospatial_vertical_positive = "up" ;
                :geospatial_vertical_units = "m" ;
                :wind_field = "WSC WDC WIC" ;
                :landmarks = "39.9083 -105.116 BJC,61.1741 -149.996 ANC,19.6475 -156.013 KOA,-43.4893 172.532 NZCH" ;
                :Categories = "Position,Thermodynamic,Aircraft State,Atmos. State,Liquid Water,Uncorr\'d Raw,Wind,PMS Probe,Housekeeping,Chemistry,Radiation,Non-Standard" ;
                :time_coverage_start = "2010-04-10T19:27:23 +0000" ;
                :time_coverage_end = "2010-04-11T03:36:43 +0000" ;

coordinates is required starting with version 1.3. These are the aircrafts "best" position variables.
TimeInterval is required by ncplot and ncpp, but I hope to get away from that.

6.2 Variable attributes

All variables contain the basic attributes units, long_name, and _FillValue (missing_value is being deprecated). In addition we provide some other attributes; not all are used for every variable:

standard_name we have adopted the CF CF conventions section 3.3 use of standard_name where applicable.

SampledRate is the rate (sps) at which the variable was sampled onboard the aircraft. This does not apply to derived variables.

DataQuality attempts to inform the user of the quality of these data. Some values are Bad, Preliminary and Good

Category Since a file can contain hundreds of variables with not very descriptive names, this is provided to help create sub-lists of variables. Comma separated list.

CalibrationCoefficients are the values used to produce engineering units from a measurement's DC voltage. It is used by the analog/digital group. These values have already been applied! They are present for documentation.

Dependencies are the input variables that were used to produce this derived value. Like CalibrationCoefficients , these are present for documentation.

For processing purposes, variables in our files are organized into 3 sorted lists. The first sorted list consists of Analog or Digitally-sampled variables. These only have calibration coefficients applied to them. The second group is raw data from "block" probes (e.g., PMS1D, IRS, GPS, and a few from group 1 that need special processing). The third group consists of purely derived variables. I have included one example from each of the 3 groups (which also show different dimension schemes): 

        float PITCH(Time) ;
                PITCH:_FillValue = -32767.f ;
                PITCH:units = "degree" ;
                PITCH:long_name = "IRS Aircraft Pitch Angle" ;
                PITCH:standard_name = "platform_pitch_angle" ;
                PITCH:valid_range = -180.f, 180.f ;
                PITCH:actual_range = -3.31724 15.88532f ;
                PITCH:Category = "Analog" ;
                PITCH:SampledRate = 50 ;
                PITCH:TimeLag = -180 ;
                PITCH:TimeLagUnits = "milliseconds" ;
                PITCH:DataQuality = "Preliminary" ;
                PITCH:CalibrationCoefficients = 0.26f, 1.f ;

        float CFSSP_RPC(Time, sps1, Vector16) ;
                CFSSP_RPC:_FillValue = -32767.f ;
                CFSSP_RPC:units = "#/cm3" ;
                CFSSP_RPC:long_name = "FSSP-100 Concentration (per cell)" ;
                CFSSP_RPC:Category = "PMS Probe" ;
                CFSSP_RPC:SerialNumber = "FSSP076" ;
                CFSSP_RPC:DataQuality = "Preliminary" ;
                CFSSP_RPC:Dependencies = "2 AFSSP_RPC TASX" ;
                CFSSP_RPC:FirstBin = 1 ;
                CFSSP_RPC:LastBin = 15 ;
                CFSSP_RPC:CellSizes = 2.f, 5.f, 8.f, 11.f, 14.f, 17.f, 20.f, 23.f, 26.f, 29.f, 32.f, 35.f, 38.f, 41.f, 44.f, 47.f, 2.f, 4.f, 6.f, 8.f, 10.f, 12.f, 14.f, 16.f, 18.f, 20.f, 22.f, 24.f, 26.f, 28.f, 30.f, 32.f, 1.f, 2.f, 3.f, 4.f, 5.f, 6.f, 7.f, 8.f, 9.f, 10.f, 11.f, 12.f, 13.f, 14.f, 15.f, 16.f, 0.5f, 1.f, 1.5f, 2.f, 2.5f, 3.f, 3.5f, 4.f, 4.5f, 5.f, 5.5f, 6.f, 6.5f, 7.f, 7.5f, 8.f ;
                CFSSP_RPC:CellSizeUnits = "micrometers" ;
                CFSSP_RPC:CellSizeNote = "CellSizes are upper bin limits as diameter." ;
                CFSSP_RPC:HistogramNote = "Zeroth data bin is an unused legacy placeholder." ;
                CFSSP_RPC:DepthOfField = 2.7f ;
                CFSSP_RPC:BeamDiameter = 0.18f ;
                CFSSP_RPC:Density = 1.f ;
                CFSSP_RPC:PLWfactor = 1.e-06f ;
                CFSSP_RPC:DBZfactor = 1000000.f ;

        float WIC(Time, sps25) ;
                WIC:_FillValue = -32767.f ;
                WIC:units = "m/s" ;
                WIC:long_name = "GPS-Corrected Wind Vector, Vertical Gust Component" ;
                WIC:standard_name = "upward_air_velocity" ;
                WIC:actual_range = -3.393437 5.762867f ;
                WIC:Category = "Wind" ;
                WIC:DataQuality = "Good" ;
                WIC:Dependencies = "9 TASX VEWC VNSC PITCH ROLL THDG ATTACK SSLIP VSPD" ;

Appendices

A  References

[NetCDF]
NetCDF Software Package, from the UNIDATA Program Center of the University Corporation for Atmospheric Research.
[UDUNITS]
UDUNITS Software Package, from the UNIDATA Program Center of the University Corporation for Atmospheric Research.

B  PMS1D Probe Variables Description

B.1 Variable names

Each PMS probe has two histograms stored in the netCDF file. First is the raw counts and this variable always starts with the letter 'A' (for "Actual"). Second is the concentration array which always starts with the letter 'C'. Below you will see the variable names for the orginal PMS probe and for the DMT converted electronics.

PMS1D# binsCounts nameConcentration nameDMT Converted# binsCounts nameConcentration name
FSSP16AFSSP_xxxCFSSP_xxxS10031AS100_xxxCS100_xxx
PCASP16APCAS_xxxCPCAS_xxxS20031AS200_xxxCS200_xxx
F30032AF300_xxxCF300_xxxS30031AS300_xxxCS300_xxx
CDP31ACDP_xxxCCDP_xxx
UHSAS100AUHSAS_xxxCUHSAS_xxx
260X64A260X_xxxC260X_xxx------
2D-C (entire-in algo)32A1DC_xxxC1DC_xxxFast2DC64A1DC_xxxC1DC_xxx
2D-C (center-in algo)64A2DC_xxxC2DC_xxxFast2DC128A2DC_xxxC2DC_xxx

B.2 Number of bins

The original PMS probes always had the first (zeroth) bin set to zero or a value which was not part of the size-distribution histogram (e.g. the FSSP-100 recorded 16 bins of which the last 15 contained data). However we wanted all raw data available to an investigator. For consistency this required the concentration array to be the same size, so it's first (zeroth) bin is also set to zero.

DMT converted probes can record either 10, 20, 30 or 40 bins of data. To maintain backwards compatability, we pad the front of the array with a zero and increase the number of bins by one (11, 21, 31 or 41).

NOTE: Always ignore the very first bin of any size-distribution histogram in our netCDF files.

B.3 Cell sizes

Cell sizes are stored as an attribute of the counts array. They are upper bin limits. These are diameters. In the example below, there are 31 cell sizes to go with the 30 used values of the size-distribution. The first cell size is actually the lower bin limit for the first bin of data. Also note the attributes FirstBin and LastBin, these should be used to determine which bins are considered valid, both are inclusive.

e.g. The bin limits for CS100[n] are CellSizes[n-1] and CellSizes[n], lower and upper respectivily. 

        float AS100_LPC(Time, sps1, Vector31) ;
                AS100_LPC:_FillValue = -32767.f ;
                AS100_LPC:units = "count" ;
                AS100_LPC:long_name = "FSSP-100 Raw Accumulation (per cell) - DMT" ;
                AS100_LPT:actual_range = 0.f, 3914.f ;
                AS100_LPC:Category = "PMS Probe" ;
                AS100_LPC:SerialNumber = "FSSP122" ;
                AS100_LPC:SampledRate = 10 ;
                AS100_LPC:DataQuality = "Good" ;

        float CS100_LPC(Time, sps1, Vector31) ;
                CS100_LPC:_FillValue = -32767.f ;
                CS100_LPC:units = "#/cm3" ;
                CS100_LPC:long_name = "FSSP-100 Concentration (per cell) - DMT" ;
                CS100_LPC:Category = "PMS Probe" ;
                CS100_LPC:SerialNumber = "FSSP122" ;
                CS100_LPC:DataQuality = "Good" ;
                CS100_LPC:Dependencies = "5 AS100_LPC TASX REJAT_LPC OVFLW_LPC FRNG_LPC" ;
                CS100_LPC:FirstBin = 3 ;
                CS100_LPC:LastBin = 30 ;
                CS100_LPC:CellSizes = 0.7f, 1.4f, 2.35f, 3.9f, 5.45f, 7.f, 8.55f, 10.1f, 11.65f, 13.2f, 14.75f, 16.3f, 17.85f, 19.4f, 20.95f, 22.5f, 24.05f, 25.6f, 27.15f, 28.7f, 30.25f, 31.8f, 33.35f, 34.9f, 36.45f, 38.f, 39.55f, 41.1f, 42.65f, 44.2f, 45.75f ;
                CS100_LPC:CellSizeUnits = "micrometers" ;
                CS100_LPC:CellSizeNote = "CellSizes are upper bin limits as diameter." ;
                CS100_LPC:HistogramNote = "Zeroth data bin is an unused legacy placeholder." ;
                CS100_LPC:DepthOfField = 2.3f ;
                CS100_LPC:BeamDiameter = 0.2f ;
                CS100_LPC:Density = 1.f ;
                CS100_LPC:PLWfactor = 1.e-06f ;
                CS100_LPC:DBZfactor = 1000000.f ;

        float CONCF_LPC(Time) ;
                CONCF_LPC:_FillValue = -32767.f ;
                CONCF_LPC:units = "#/cm3" ;
                CONCF_LPC:long_name = "FSSP-100 Concentration (all cells)" ;
                CONCF_LPT:actual_range = 0.f, 437.7162f ;
                CONCF_LPC:Category = "PMS Probe" ;
                CONCF_LPC:SerialNumber = "FSSP122" ;
                CONCF_LPC:DataQuality = "Good" ;
                CONCF_LPC:Dependencies = "1 CS100_LPC" ;

bin#0123....2930
n/aFirstBinLastBin

For further information contact:

Chris Webster
National Center for Atmospheric Research
Research Aviation Facility
(303) 497-1044

email
Last update: Mon Jun 29 03:43:37 MDT 2009