Status of Data Provision

S-Pol IHOP_2002, Oklahoma Panhandle
May/Jun 2002


Thu Dec 12 10:50:59 MST 2002 --- Primary processing completed

All S-Pol data have been merged and processed to sweepfiles. Sweepfiles are all available on /scr/rz1 and /scr/rz2 (May and June, respectively). Sweepfiles are being copied to the Mass-Store for general distribution.

Wed Dec 11 13:31:50 MST 2002 --- Primary data processing near completion

One or two recent glitches in the processing, but work is still on track to complete merging of the primary S-Pol data stream by tomorrow (Thursday). Sweepfiles are being created as progress is made, and all sweepfiles should be available on /scr/rz1 (May) or /scr/rz2 (June) by Thursday.

Follow-up work will include completing copies of the sweepfiles on the MSS, placing merged S-Pol PIRAQ data on the MSS, and writing merged data to tape (for redundancy in the data); all these steps will be complete by Monday or Tuesday (17 Dec). Also yet to be done is creating redundancy in the realtime S-Pol sweeps.

The placement of sweepfiles on the Mass-Store constitutes delivery of the S-Pol data set to the community at large.

Mon Dec 9 08:46:51 MST 2002 --- Primary data processing

S-Pol PIRAQ data have been merged through 27-May (June is already completed), and creation of sweepfiles is beginning for 23-27 May. The process has been somewhat delayed over the weekend due to problems with mead, one of the ATD servers.

Data are still being loaded for the remainder of May. It is likely that the entire S-Pol primary merge/reprocessing will be done by the end of this week, or possibly as late as next Monday, if the hardware all holds up.

The RAIDZone known as /scr/rz1 has rebuilt itself, and has seemed reliable over the past 4 days.

Wed Dec 4 10:36:12 MST 2002 --- RAIDZone 1 Failing

The RAID system designated RAIDZone 1 (or /scr/rz1) is failing. A disk failed and the system is rebuilding, but without RAID protection. RAIDZone1 contains reprocessed sweepfiles for May (most, but not all, have been backed-up to the NCAR Mass-Store), and the original realtime sweepfiles for June.

The realtime June files are considered very important. These exist nowhere else, and efforts are being made to copy these to another RAID system, and then to the MSS.

Also on rz1 are many of the Zebra files from other platforms. These files also exist on backup tape (with about 95% certainty), so can likely be re-loaded.

Rz1 is currently running extremely slowly. Priority tasks are:

Other work will continue to the extent possible. This problem will likely cause a one to two day delay in processing, assuming the RAIDZone can be rebuilt properly and brought back as a RAID level-5 system.

Tue Dec 3 09:48:10 MST 2002 --- Processing May primary S-Pol data

The JetStore disk is holding up well. No further problems have been noted since the rebuild.

The processing queue was stuffed and monitored over the Thanksgiving holiday. Most things proceeded well. Some small mistakes were made that will cost an extra day or two of processing. Net results are that processing is done for 13-May thru 22-May, with the exception of 18-May (data prior to 13-May do not need reprocessing since dual recording was not used). Data are being updated on the ATD RaidZone (/scr/rz1), and will be moved to the NCAR Mass-Store System (MSS) shortly.

It is expected that we'll begin loading the final batch of S-Pol tapes on Thursday. This final batch will include the P and Q tapes for the 9 remaining days in May. If all goes well, processing will be done by the end of next week (around 13 Dec, or 10 days from now). It's not possible to move any faster than this, since we're I/O bound, and we already use as many as five processors for our work.

Mon Nov 25 08:45:20 MST 2002 --- Proceeding despite flakey JetStore

The JetStore successfully rebuilt itself over the weekend. It is no longer complaining about a damaged disk. This is a bit unsettling -- intermittent failure is a difficult issue with which to work. But work is now proceeding, after the loss of two-and-a-half days that would have been used for computational processing.

Data for all of June have now been moved to the archive location on the RAIDZone. Files for the end of June still need to be moved to the Mass-Store. Raw P/Q tapes for mid-May are being loaded for merging and processing.

Fri Nov 22 13:38:49 MST 2002 --- JetStore RAID Failure

[The following directly affects the merging of the primary S-Pol data stream]

Failure of a RAID disk and subsequent need to rebuild the RAID has caused an interruption in the merging of S-Pol PIRAQ data streams. Currently, the merge and generation of final sweepfiles has been completed through the end of June, but data for 25 and 26 June has not been moved to its final repository.

Work will continue once the RAID rebuilds. The RAID may then run without full recovery capability. This will be OK for data merging, since we view this RAID as temporary space, anyway. The greatest impact will be on future use of the RAID. Should the RAID fail when full re-build is not possible, we could loose up to 3 days of work.

Work is likely suspended through the end of today, but may be taken up again over the weekend.

Thu Nov 21 12:57:25 MST 2002 --- Merged S-Pol Data

[The following applies to the primary S-Pol data stream, only.]

Data through the 26th of June have been merged to make the most complete S-Pol data set possible. Most of the merged data have been converted to sweepfiles, with the exception of 21-26 June (the sweepfile conversion will take place overnight for these days).

Data thru 12 UTC 20-Jun have been sent to the NCAR Mass-Store, and are available on-line [Post-script: the mass-store listing routine has experienced intermittent time outs, so the Web-based access tools may not reflect the proper level of completion for this data set].

It is expected that data through the 26 June will be fully processed and available by Saturday.

Work is continuing on the Web-based S-Pol Project Report for IHOP (this page is part of that report). Latest additions include panoramic photos from the S-Pol site.

Fri Nov 15 15:06:06 MST 2002 --- Merged S-Pol Data

[The following applies to the primary S-Pol data stream, only. No statement is made concerning NEXRAD radar mosaic, third-party data sets, or Zebra data.]

Merging of the two data streams from the individual S-Pol RDAs (Radar Data Acquisition systems) is continuing. Currently, data from 1-June-2002 12:00 UTC through 20-Jun-2002 12:00 UTC are available on the ATD local network. Those same data are being loaded to the NCAR Mass-Store System (MSS), and will be available for ftp distribution to the user community, probably by 19-Nov-2002 (Jean Hurst is managing this part of the operation).

Additional tape data have been loaded for merging and NCP thresholding, and will be merged over the coming weekend. Data for 20 thru 23 June will be merged.

It is very likely that all June data will be available on the MSS prior to Thanksgiving. The May data will then be processed.

--- Bob Rilling --- / NCAR Atmospheric Technology Division
Created: Fri Nov 15 15:05:40 MST 2002
Last modified: