Chapter 6. Overview of ISS Data Streams
- Table of Contents
- Ingest
- Archive
- Data Transfers
Ingest
Most of the ingest processes run under the ingest scheduler, using the configuration file /iss/configs/iss/is.config. In order to isolate the scheduler from hardcoded directory and executable paths, it is always run from the top of the data directory, /iss/ds. Thus all of the data paths in the configuration file are relative to that directory. Likewise none of the executables have absolute and instead rely on the program being on the PATH. The /iss/etc/init.d/ingest module takes care of setting the path correctly and running is in the top of the data directory.
The continuous data streams, surface and the POP serial moments, have their own modules under /iss/etc/init.d for stopping, starting, and configuring their ingest processes. They still run under the ingest scheduler so that they will be restarted if they should crash, but they can be stopped and started manually through their module scripts. The scripts just send the appropriate command to is with zrun. Likewise, is does not automatically start those ingestors on startup, like it does for the other ingest processes, so they can be enabled or disabled in the site configuration outside of is. Those ingestors start (if enabled) when the init.iss module calls their particular modules, for example surface_met or moments.
 | The moments module controls the ingest of the profiler moments over the serial line, while the soap module controls the periodic IDL plots of the SOAP data. |
When the ingest scheduler starts a surface or moments ingest process, it actually calls the control script in /iss/etc/init.d instead of running the ingest program directly. This allows the control script to pass configured information on the command line without modifying the is.config file. For example, the surface ingest program, cam_ingest, needs the site location and serial port to be passed on the command line. Likewise the moments ingest program, popserial, needs a serial port device.
Profiler Consensus Data
Not done yet.
Profiler Serial Output Moments Data (SOAP)
Not done yet.
NCAR Improved Moments Processing
Not done yet.
GLASS Sounding Files
The GLASS PC maps the ISS workstation data directory served by Samba to the K: drive. Before a sounding, the GLASS software retrieves the current surface data from the file K:\campbell\cr10data.txt. After a sounding, the D data file is written to K:\class (/iss/ds/class on the workstation). The class ingest process running under the ingest scheduler checks for new files there and ingests them with glass_ingest. Ingested sounding files are then moved to the done_10sec subdirectory.
Surface Data
As mentioned in the section above, the surface ingest program cam_ingest runs under the ingest scheduler, but it is started and stopped through the surface control script. There are two links in /iss/etc/init.d, one for each of the possible surface station platforms: surface_met and surface_two. Each can be started, stopped, and configured individually through its control script. The scripts are links to the surface script, which controls the appropriate surface platform (surface_met or surface_two) according to the name by which it was run.