Follow-up work will include completing copies of the sweepfiles on the MSS, placing merged S-Pol PIRAQ data on the MSS, and writing merged data to tape (for redundancy in the data); all these steps will be complete by Monday or Tuesday (17 Dec). Also yet to be done is creating redundancy in the realtime S-Pol sweeps.
The placement of sweepfiles on the Mass-Store constitutes delivery of the S-Pol data set to the community at large.
Data are still being loaded for the remainder of May. It is likely that the entire S-Pol primary merge/reprocessing will be done by the end of this week, or possibly as late as next Monday, if the hardware all holds up.
The RAIDZone known as /scr/rz1 has rebuilt itself, and has seemed reliable over the past 4 days.
The realtime June files are considered very important. These exist nowhere else, and efforts are being made to copy these to another RAID system, and then to the MSS.
Also on rz1 are many of the Zebra files from other platforms. These files also exist on backup tape (with about 95% certainty), so can likely be re-loaded.
Rz1 is currently running extremely slowly. Priority tasks are:
Other work will continue to the extent possible. This problem will likely cause a one to two day delay in processing, assuming the RAIDZone can be rebuilt properly and brought back as a RAID level-5 system.
The processing queue was stuffed and monitored over the Thanksgiving holiday. Most things proceeded well. Some small mistakes were made that will cost an extra day or two of processing. Net results are that processing is done for 13-May thru 22-May, with the exception of 18-May (data prior to 13-May do not need reprocessing since dual recording was not used). Data are being updated on the ATD RaidZone (/scr/rz1), and will be moved to the NCAR Mass-Store System (MSS) shortly.
It is expected that we'll begin loading the final batch of S-Pol tapes on Thursday. This final batch will include the P and Q tapes for the 9 remaining days in May. If all goes well, processing will be done by the end of next week (around 13 Dec, or 10 days from now). It's not possible to move any faster than this, since we're I/O bound, and we already use as many as five processors for our work.
Data for all of June have now been moved to the archive location on the RAIDZone. Files for the end of June still need to be moved to the Mass-Store. Raw P/Q tapes for mid-May are being loaded for merging and processing.
Work will continue once the RAID rebuilds. The RAID may then run without full recovery capability. This will be OK for data merging, since we view this RAID as temporary space, anyway. The greatest impact will be on future use of the RAID. Should the RAID fail when full re-build is not possible, we could loose up to 3 days of work.
Work is likely suspended through the end of today, but may be taken up again over the weekend.
Data thru 12 UTC 20-Jun have been sent to the NCAR Mass-Store, and are available on-line [Post-script: the mass-store listing routine has experienced intermittent time outs, so the Web-based access tools may not reflect the proper level of completion for this data set].
It is expected that data through the 26 June will be fully processed and available by Saturday.
Work is continuing on the Web-based S-Pol Project Report for IHOP (this page is part of that report). Latest additions include panoramic photos from the S-Pol site.
Merging of the two data streams from the individual S-Pol RDAs (Radar Data Acquisition systems) is continuing. Currently, data from 1-June-2002 12:00 UTC through 20-Jun-2002 12:00 UTC are available on the ATD local network. Those same data are being loaded to the NCAR Mass-Store System (MSS), and will be available for ftp distribution to the user community, probably by 19-Nov-2002 (Jean Hurst is managing this part of the operation).
Additional tape data have been loaded for merging and NCP thresholding, and will be merged over the coming weekend. Data for 20 thru 23 June will be merged.
It is very likely that all June data will be available on the MSS prior to Thanksgiving. The May data will then be processed.