CSU has received one FireWire disk with data from 08-Jul-04 to 16-Jul-04. The data from that disk has been off-loaded at CSU.
Disk creation has been slowed due to a disturbing tendency for the FireWire filesystem to "automatically" set files to zero-length. This has happened at least four times to major portions of the data set. There has been no systems administration support available at EOL/ATD to address this issue.
This weekend, I decided to re-format the removable disks to a type that is not usable under Windows (but is nicely usable under Linux or Unix). This, of course, meant starting over. Currently, data are created from 17-Jul-04 thru 01-Aug-04, and the disks seem stable. Tape backups are being created, and the disks will be ready for CSU within a day.
EOL/RDP has found some system admin help, and the administrator is scheduled to review the problem with removable disks tomorrow. I leave for the field early Saturday, and my availability is very limited for the remainder of the week.
Parameter List -- A near-final list of S-Pol parameters has been developed. The list will include either 18 or 20 variables, depending upon inclusion of AIQ and NIQ. This list is longer than the minimum requested by CSU scientists, and will be generated to ensure that full polarization matrix variables are availble for use by EOL (formerly ATD) engineers and future users of the NAME dataset. It is expected that the final data set size will now exceed 1.5 TB.
The capability to generate one of the required variables was added by SE staff just yesterday. Tests show good results, and data generation will start immediately.
Zdr Bias -- Zdr bias has been well-characterized by vertical pointing data collection. However, the NAME data set frequently shows areas of negative Zdr. Some time was spent reviewing this situation. The negative Zdr regions are most often an effect of differential attenuation, perhaps partial beam blockage, and sometimes a true signature (cause of negative Zdr has yet to be fully characterized). As a further check of Zdr bias/calibration, we've had our software engineer extract the "DW" parameter (see the parameter list) to allow application of Hubbert's full matrix Zdr bias estimate.
Note that there was a brief time near the end of the project when the vertical transitter pulse experienced some problems, and Zdr data are bad for this time period.
0.8° Missing Segments -- The missing segments of 0.8 degree scans required more extensive review. While many of the segments were recoverable by ignoring the scan transition flag, in other cases, the data for these segments is actually missing. Data were lost due to unreadiness of the VIRAQ S-band processor during the start of a sweep. New-volume-initiation tasks within the processor take several seconds; if the end of the previous volume leaves the antenna very near where the new volume should start, then there is not sufficient time for the initial tasks to complete, and beams are lost.
Data Transfer -- Several 250GB USB/Firewire external hard disks have been ordered, and should arrive around Wednesday next week. These disks will be used on a rotating basis for transport of data to CSU, a mechanism that should be faster than ftp transfer of data. Only 750 to 1000 GB of storage will be available, not enough to allow transfer of the entire data set.
SMN Data on Kanguru Disk -- Kanguru Corp has evaluated the damaged disk containing SMN data. They suggest that the disk itself is physically damaged, and recommend a data recovery service. This is obviously not a good development. We will pursue the data recovery.
Known Problems -- A list of "known problems" is being developed.
This is the first public post-NAME status report on S-Pol and SMN data availability.
Main benchmarks include:
Suitable sweepfiles can be generated for distribution. Testing is proceeding to ensure that complete 0.8 degree scans are generated. During a preliminary run on 4 days of data, several 0.8 deg sweeps appear as incomplete, but these were mostly accompanied by radar processor resets. Different limits on fixed angle tolerance will be tested for the remaining problematic scans.
Selection of those S-Pol parameters for inclusion with the distribution data set must be made. PIs will be contacted regarding details.
Preliminary work has been done to provide sweep lists for S-Pol in NAME. Further work is required to structure sweep information metadata into easily digested tables.
A list of "known problems" is being formalized. See the appropriate section of this report.
As S-Pol sweepfile data sets are completed, they will be loaded to the NCAR MSS for distribution. ATD will also work with CSU to provide a more direct method for ftp of those sweepfile data sets. The publicly available list of data available on the Mass-Store is not updated reliably, due to the growing size of the ATD data archive; work must be done on code to improve this situation, but cannot currently be made a priority.
The Guasabe radar digital processor came on-line 10-Jun-2004; Los Cabos on 16-Jul-2004. Both radars are still logging data to removable disk, with the most recent data expected to be retrieved in the next week or two. Trouble was experienced with two of the removable disks, raising fears of significant data loss. Data were recovered for one of these disks (loose internal connector likely), but the disk for Guasabe for the time period 12-Jul to 29-Jul is still not recoverable. The manufacturer has been contacted for advice (Jon Lutz is handling this issue).
Please note that we have some reasonable hope for the recovery of the data on the problematic Guasabe disk. The disk did record data in the field, as verified at the time the disk was retrieved. The disk spins smoothly, but is not recognized as formatted by any operating system. Our suspicion is that the internal disk interface has a damaged component, and that the manufacturer will be able to assist in data recovery.
There are known time and calibration corrections required for some of the SMN data. A summary of known issues is being prepared. There are likely yet-to-be-determined issues, as well.
The ATD radar data translator (xltrs) has been tested and found to work on the SMN data. The translator was not tested to apply a clock offset, nor was a calibration/reflectivity correction applied.
The mass-store pathname for the raw SMN data files is
/ATD/STOR/2004/NAME2004/SMN_radars/raw. Users of NCAR computer
systems can download these files via the msrcp command.
A full listing of current mass-store SMN data sets is available.
Preparation of S-Pol for the next project (RICO) has impacted, and will continue to impact delivery of data for NAME. This is particularly true in the case of software readiness, and is somewhat true for availability of Data Management staff.