NEXRAD Data Quality Optimization

FY98 Annual Report



Submitted to the

WSR-88D Operational Support Facility
NEXRAD Data Quality Optimization

December 15, 1998



Cathy Kessinger, Scott Ellis, Joseph Van Andel, Don Ferraro and R. Jeffrey Keeler

National Center for Atmospheric Research
Atmospheric Technology Division/Research Applications Program
Boulder, CO 80307-3000





Table of Contents

Cover Page of Document

Executive Summary

1.0 Overview of the NEXRAD Data Quality Optimization Program

1.1 Introduction

1.2 The AP Clutter Mitigation Scheme

1.2.1 The "Radar Echo Classifier"

1.2.2 Clutter Suppression

1.2.3 Reflectivity Compensation to Correct Clutter Filter Bias

1.2.4 Clutter Residue

1.3 Implementation Plan for the AP Clutter Mitigation Scheme

1.4 References

2.0 Anomalous Propagation Detection Algorithm

2.1 Introduction

2.2 Review of Pratte and Ecoff Work

2.3 Selection of Case Studies

2.4 The Fuzzy Logic AP Classifier

2.5 Histograms of Data Characteristics

2.6 Truthing of Cases

2.7 APCAT Improvements

2.8 Optimization of the AP Detection Algorithm

2.8.1 Methodology

2.8.2 Detailed Examination of Data from KAMA

2.8.2.1 A Region of AP Echo

2.8.2.2 A Region of Precipitation Echo

2.8.3 Summarized Examination of Case Studies

2.8.3.1 KLSX, July 7, 1993 at 0334 UTC

2.8.3.2 KTLX, July 1, 1994 at 1106 UTC

2.8.3.3 KHGX, October 19, 1994 at 0530 UTC

2.8.3.4 KDDC, July 13, 1993 at 0655 UTC

2.8.3.5 KLOT, October 19, 1995 at 2331 UTC

2.8.3.6 KFTG, September 21, 1995 at 0206 UTC

2.8.3.7 KLSX, July 24, 1993 at 1009 UTC

2.8.3.8 KLSX, July 11, 1993 at 0131 UTC

2.8.3.9 KLOT, October 18, 1995 at 1005 UTC

2.8.3.10 KNQA, July 6, 1997 at 0454 UTC

2.8.3.11 KLWX, August 17, 1997 at 0958 UTC

2.8.3.12 KLOT, October 19, 1995 at 0011 UTC

2.8.3.13 KFTG, January 11, 1995 at 1548 UTC

2.8.4 Conclusions

2.9 Work in Progress and Future Work

2.9.1 Further Analysis and Optimization of the APDA

2.9.2 Clutter Filter Maps

2.9.3 Use of Polarimetric Data

2.10 References

 

3.0 Princeton University Evaluation of the AP Detection Algorithm

3.1 Introduction

3.2 Analysis Results

3.3 Conclusions

3.4 References

4.0 Compensating Reflectivity for Clutter Filter Bias

4.1 Introduction

4.2 Two Variations of the Simple Gaussian Correction Model

4.2.1 The "Table Look-Up" SGCM

4.2.2 The "Direct" SGCM

4.3 Results of the SGCM Tests Using Memphis A1 Data

4.4 Discussion

4.5 Clutter Residue Tracking

4.6 References

5.0 Convective Precipitation Detection Algorithm

5.1 Introduction

5.2 Preliminary Algorithm Description

5.3 Preliminary Algorithm Results

5.4 Future Work

5.5 References

6.0 Memphis Archive 1 Data

6.1 Introduction

6.2 Data Analysis and Methodology

6.3 Discussion of Results

6.4 Comparison of the AP Detection Algorithm Results

6.5 Summary and Future Work

6.6 References

7.0 Data Quality Instrumentation

7.1 The Archive 1 Data Acquisition (A1DA) Unit

7.1.1 Introduction

7.1.2 A1DA #1 Modifications for Use with S-Pol

7.1.2.1 S-Pol Modifications

7.1.2.2 A1DA Modifications

7.1.2.3 Time Series Data Collection in PRECIP-98

7.1.2.4 Recommendations and Future Plans

7.1.3 A1DA #2 Installation at KCRI

7.1.3.1 Recommendations and Future Plans

7.2 KCRI RDA Instrumentation

7.2.1 Introduction

7.2.2 Functionality

7.2.3 Recommendations and Future Work

Appendix A
 


NEXRAD Data Quality Optimization

FY98 Annual Report

Cathy Kessinger, Scott Ellis, Joseph Van Andel, Don Ferraro and R. Jeffrey Keeler

National Center for Atmospheric Research

Atmospheric Technology Division/Research Applications Program

Boulder, CO 80307-3000



Submitted to the

WSR-88D Operational Support Facility

NEXRAD Data Quality Optimization



December 15, 1998


Cover Image: Reflectivity (left panel) and radial velocity (right panel) plots from the Amarillo, TX WSR-88D (KAMA) on 25 May 1994 at 0035 UTC. The 0.5 degree elevation angle is shown. The region of anomalous propagation (AP) echo is to the north of the radar, while precipitation echoes are to the south of the radar. The cyan-shaded radial velocity values (indicating 0 m/s) to the north of the radar contain the AP echoes. KAMA data provided by courtesy of Dr. M. Steiner, Princeton University.


Executive Summary

This document summarizes the work done by NCAR during FY-98 for the NEXRAD Data Quality Optimization program. This program has two main tasks: Specification of an AP Clutter Mitigation Scheme and the KCRI Testbed Instrumentation for the WSR-88D at Norman, OK.

An AP Clutter Mitigation Scheme is proposed that will eventually be implemented as part of the Open Systems Architecture. The AP Clutter Mitigation Scheme will use the Radar Echo Classifier to detect the presence of AP ground echoes, precipitation echoes, clutter residue and other radar scatterers as necessary. The output from the AP Detection Algorithm will determine where clutter filters will be enabled or disabled. The first step in the implementation process of the AP Clutter Mitigation Scheme will be to augment the clutter bypass map with the locations of AP echoes and allow the forecaster to manually edit the map, if necessary. The last step in the implementation process will be full automatic control of the ground clutter filtering process.

In this report, the preliminary optimization of the AP Detection Algorithm (APDA) is presented. Various tests of the membership functions and input variables are performed over a data set consisting of 60 individual scans. One test, termed Test #4, slightly outperformed the other tests. Individual cases are analyzed and the algorithm results examined. An independent evaluation of an early version of the APDA shows that the algorithm has significant skill in detecting AP echoes. A preliminary Precipitation Detection Algorithm is presented. Compensation of the reflectivity values within precipitation echo is required for the AP Clutter Mitigation Scheme and preliminary results of this work are presented.

For the instrumentation work, the installation of the Archive 1 Data Acquisition (A1DA) unit on the Norman, OK WSR-88D (KCRI) and the NCAR S-Pol radars is completed. The A1DA allows the collection of time series data. Time series (Archive 1) data will be necessary for the clutter residue characterization work. Also, one example case compares visually edited spectral domain data to the Archive 2 data for the same location and time. In addition, the Testbed Instrumentation system was installed on KCRI and allows the archival and display of various engineering parameters needed to monitor the operating system of the WSR-88D.


1.0 Overview of the NEXRAD Data Quality Optimization Program

1.1 Introduction

This document summarizes the work done by the National Center for Atmospheric Research (NCAR) for the Operational Support Facility (OSF) on the Data Quality Optimization program. The goals of this program are to improve the quality of the base data from the Weather Surveillance Radar-1988 Doppler (WSR-88D) systems with special emphasis on the problem of ground clutter return due to anomalous propagation (AP) conditions and to provide data quality instrumentation to the OSF.

To meet program goals of improving data quality (Saffle, 1997), an "AP Clutter Mitigation Scheme" is outlined in this document and is discussed briefly in Section 1.2 and comprehensively in Sections 2, 3, 4, and 5. The AP Clutter Mitigation Scheme consists of 1) computer algorithms based on "fuzzy logic" (Kosko, 1991) for the classification and detection of various types of radar echoes, 2) an automated system of ground clutter filter specification and control, 3) compensation of reflectivity within regions of precipitation echo that are biased by application of the clutter filter and 4) regions of AP echo that must be "tracked" by the statistical characterization of the clutter residue (or some other method) to allow filters to be enabled when needed and disabled when not. The AP Clutter Mitigation Scheme is a planned enhancement to the WSR-88D Open Systems Architecture beginning with Open Build 2 in the year 2000, as currently scheduled. The Implementation Plan is discussed in Section 1.3.

To meet program goals of providing data quality instrumentation, the Archive 1 Data Acquisition (A1DA) unit was installed on the NCAR S-Pol radar and the WSR-88D at Norman, OK (KCRI). In addition, the Testbed Instrumentation system was installed on KCRI. This work is described in Section 7. The Testbed Instrumentation system acquires, stores and monitors various environmental and internal system parameters to allow for hardware and software revision testing. The A1DA allows collection of time series data from the radar. The S-Pol radar collected time series data near Melbourne, FL during the summer. These data are needed to meet program goals for the Range-Velocity Mitigation program (Frush et al., 1998) also funded by the OSF. For the AP Clutter Mitigation Scheme, some of the Memphis WSR-88D time series data collected with the A1DA during July 1997 are used to do a preliminary analysis of the benefits of spectral domain processing for AP detection. This work is described in Section 6.

Personnel and budget summaries for the Data Quality Optimization program are given in a separate, attached FY-98 Administrative Report.

1.2 The AP Clutter Mitigation Scheme

Radar return from AP clutter is an unpredictable event that often contaminates precipitation measurements, causing the generation of erroneous, radar-derived rainfall estimates used in hydrology products and confounding operational applications of WSR-88D data. Atmospheric conditions in which AP echoes are observed by radar are caused by time-variable radar refractivity profiles that modify propagation characteristics of the planetary boundary layer. In this section, the AP Clutter Mitigation Scheme is discussed briefly with complete descriptions of completed work in following sections.

A schematic flow chart of the AP Clutter Mitigation Scheme within the Open Systems Architecture is shown in Fig. 1.1. Within the scheme, the "Radar Echo Classifier" and "Reflectivity Compensation" algorithms are run. Output from the AP Detection Algorithm (APDA), one of the programs within the Radar Echo Classifier, is used to determine where clutter filters are necessary. This information is input into an "augmented clutter filter map" that the "Master System Control" uses to specify the clutter filters. Output from the APDA and possibly the "Precipitation Detection Algorithm" will be used by the Precipitation Pre-processing Subsystem (PPS) (O'Bannon, 1997) to ascertain data quality. In later builds of the Open Systems, information from the AP Clutter Mitigation Scheme will be input from the Radar Data Acquisition (RDA) unit to improve data quality.

1.2.1 The "Radar Echo Classifier"

The "Radar Echo Classifier" is a collection of fuzzy logic algorithms that will classify the type of echo observed by a WSR-88D. Current detection algorithms include the near-final version of the AP Detection Algorithm (Section 2) and a preliminary version of the Precipitation Detection Algorithm (Section 5). Other algorithms may be implemented to detect clear air return, clutter residue, birds, chaff, or other types of echoes. The Radar Echo Classifier is planned to be flexible such that additional detection algorithms can be added as needed. Figure 1.2 is a schematic showing the flow chart for the data input into these algorithms and follows through the final output of the detection algorithms. "Features" of the data are computed over a two dimensional region of the base data stream and assist in the recognition process for all detection algorithms. For instance, the APDA uses five features: the "texture" of the signal-to-ratio (TSNR) field, the mean radial velocity (MVE) field, the standard deviation of the radial velocity (SDVE) field, the vertical difference of the reflectivity (GDZ) and the mean spectrum width (MSW) field. Other algorithms will use different or similar features. A "membership function" is applied to the features for calculation of the "interest field." Interest fields are a measure of the likelihood that the particular feature is a positive indicator for the presence (or absence) of the desired phenomena. After application of the appropriate threshold to the interest field, the final output of the particular detection algorithm is obtained and a radar "product" generated. The Radar Echo Classification algorithms will be straightforward to modify and amenable to real-time operation in the WSR-88D Open Systems environment. Products from these algorithms will be available for use by all algorithms within the Open RPG system.

Mattias Steiner, Princeton University, has done a preliminary evaluation of the NCAR AP Detection Algorithm by using the functional description of the algorithm (Appendix A) to create a working version of the algorithm there. Using test cases that he has evaluated with other AP detection algorithms (Steiner and Smith, 1997), Steiner found that the NCAR algorithm performed with significant skill. The work done at NCAR on this task is described in Section 3. In his preliminary evaluation to the OSF, Steiner noted the following about the NCAR APDA:

    "The fuzzy-logic classifier clearly shows potential to recognize ground and AP clutter. A threshold of maybe 0.6 ... appears to be doing a good job at separating weather from clutter."

1.2.2 Clutter Suppression

Automation of WSR-88D clutter suppression through selective use of clutter filters and point clutter censoring is an ultimate goal of the Data Quality work. Automation will remove the task of clutter filter control for AP suppression from the responsibility of the National Weather Service (NWS) staff and should provide a standard means of clutter filter control for all WSR-88D systems. Automation will be achieved in real-time by using the output from the AP Detection Algorithm to augment the clutter filter map defined for normally-propagating (NP) ground clutter (currently called the "bypass map") such that clutter filters are enabled or disabled where appropriate for each volume coverage pattern (VCP). The AP clutter suppression will be specified on a gate-by-gate basis in this manner. Figure 1.3 shows an example bypass map for the Open Systems Architecture. This automated scheme must track the evolution of AP echo regions and adjust the clutter suppression as necessary. To prevent application of clutter filters within precipitation regions not affected by AP return, the output from precipitation detection algorithms may be used selectively to enable or disable clutter suppression.

As currently envisioned (but not necessarily rejected as an option), clutter filters will not be enabled at all times and at all locations. Rather, clutter filters will always be enabled at locations having NP clutter and will be enabled and disabled as needed for regions of AP clutter. Figure 1.4 shows a schematic of the clutter filter control process as currently envisioned. For each radar volume of data, the AP Detection Algorithm (APDA) is run and checked to see if a new AP product has been produced. If so, the product is sent to the clutter filter map and clutter filters are activated. If there are no new AP clutter products, then the age of old AP products are checked to see if the time has expired. If the time limit has expired (this limit is user-specified), then the locations contained within the AP product are removed from the clutter filter map. Figure 1.5 illustrates a sequence of AP detections and how they are used to enhance the clutter filter map. The AP product for the current radar volume is the darkest shade of red with lighter shades of red denoting increasing time history. Notice that the detections are cumulative and "age off" as the time limit expires. Note that this scheme does result in a one volume delay between detection of the AP and the application of appropriate clutter filters to remove the contamination from AP.

1.2.3 Reflectivity Compensation to Correct Clutter Filter Bias

Reflectivity compensation is necessary in any AP mitigation scheme using clutter filtering. Because clutter filters negatively bias those radar reflectivity components that are located near zero radial velocity, these measurements must be compensated whenever clutter filters are applied to suppress either AP or NP clutter. Furthermore, when strong clutter is suppressed, a residue may remain causing a positive bias to the radar reflectivity. Clutter residue removal is expected to be an integral component of the automated processing.

The amount of precipitation bias created by the clutter filter depends on the radar echo velocity and width of an assumed Gaussian-shaped spectrum model as well as the clutter filter response. The filtered radial velocity and spectrum width define the biased zone of reflectivity. The reflectivity correction is then defined from a "lookup" table that has been previously generated from the known filter functions convolved with parameterized Gaussian weather model (Cornelius et al., 1995; Pratte et al., 1997; Ellis et al., 1999). Simulations indicate that a tolerable level of random compensation error (<3dB) can be achieved if the reflectivity loss in stratiform precipitation is no greater than ~10 dB, which corresponds to one of the WSR-88D medium suppression filters. Higher suppression levels yield greater compensation errors that may not be usable in hydrologic applications. Section 4 describes the work done on reflectivity compensation.

1.2.4 Clutter Residue

Use of a ground clutter filter can leave a "residue" in the filtered base data fields. Understanding the statistical characteristics of this clutter residue is required to facilitate the development of a fuzzy logic algorithm that can detect clutter residue. This work is needed since detection of the presence of AP clutter is difficult after the ground clutter filters are applied. Characterizing clutter residue will require time series data. This work is planned to occur during FY-99.

Examination of data from mountainous regions supports the conjecture that the "Clutter Residue Editing Map" (CREM) improves system performance. A CREM identifies the residual clutter that the filter fails to remove for NP situations. The Terminal Doppler Weather Radar (TDWR) program (Turnbull et al., 1989) has successfully used CREMs to remove clutter residue and to improve system and algorithm performance. Data within the CREM area may be corrected as part of the reflectivity compensation procedure (described below), censored, or flagged for further attention. Any one of these schemes will improve the base data quality and the meteorological products.

1.3 Implementation Plan for the AP Clutter Mitigation Scheme

A phased, two-stage implementation is currently planned for the AP Clutter Mitigation Scheme within the Open Systems Architecture of the NEXRAD WSR-88D systems. The first implementation will be within the "Open Systems Radar Product Generator (ORPG)" with a later implementation within the "Open Systems Radar Data Acquisition (ORDA)" system. Figure 1.6 shows the WSR-88D user control panel. Initially, the AP Clutter Mitigation Scheme will occur within the Radar Product Generator (RPG). The enhanced clutter filter map will be accessed through the "Clutter Regions" icon shown within the "Applications" panel to the right.

For the initial ORPG work, a phased implementation of the AP Clutter Mitigation Scheme is planned because testing the full AP Clutter Mitigation Scheme (including reflectivity compensation, residue tracking, and clutter filter control procedures) cannot be performed in a timely manner due to funding limitations. An initial set of Configuration Change Requests (CCRs) for these enhancements will be submitted in 1999, with implementation planned for the Open Build 2. However, the implementation may be delayed to future builds if modifications to the Precipitation Pre-processing Subsystem (PPS) algorithm (O'Bannon, 1997) prove to be a major task. The first CCR will specify a new AP recognition product that could be used in all Open RPG algorithms. The second CCR will specify that the PPS algorithm will use the AP product rather than the "tilt test" in providing estimates of rainfall rate and amounts. The tilt test is a confirmed weak point of the PPS. The third CCR will implement the reflectivity compensation algorithm. The fourth CCR will specify that the clutter bypass map will be "enhanced" with the locations of the AP clutter as defined by the AP Detection Algorithm product. Application of appropriate ground clutter filters using the enhanced clutter filter map will be activated manually. For these CCRs, automation of the clutter filter control is not achieved. This task will be specified with later CCRs.

With the inauguration of the Open RDA WSR-88D, additional RDA processing will allow a higher performance AP Clutter Mitigation Scheme. Access to both the filtered and unfiltered base data streams will allow better clutter suppression, filter bias and residue compensation, and tracking of the AP affected areas. Eventually, the ORDA processor system will likely allow Doppler spectral domain processing. The increase in real time computing power will allow combining AP and precipitation recognition schemes with new operational range-velocity ambiguity mitigation techniques being proposed (Frush and Doviak, 1998; Frush and Daughenbaugh, 1999; Sachidananda and Zrnic, 1997; Sachidananda et al., 1997; Sachidananda et al., 1998; Torres, 1998). Polarimetric data may also be available late in that decade allowing even greater recognition and mitigation performance. At this time, submission of CCRs for this type of work is premature.

1.4 References

Cornelius, R., R. Gagnon, and F. Pratte, 1995: Optimization of WSR-88D Clutter Processing and AP Clutter Mitigation. Final Report to the Operations Support Facility (OSF), 182 pp.

Ellis, S.M., F. Pratte, C. Frush, 1998: Compensating reflectivity for clutter filter bias in the WSR-88D. 15th IIPS, Jan. AMS, Dallas, TX.

Frush, C.F., and R. Doviak, 1998: NEXRAD range-velocity: Exploring selected mitigation techniques. Final Report to the Operations Support Facility (OSF).

Frush, C.F., and J. Daughenbaugh, 1999: Reduction of radar range ambiguities: A performance evaluation, 15th IIPS, January, AMS, Dallas, TX.

Keeler, R.J., 1997: AP clutter processing for Nexrad, COST-75 Final Seminar, March, Locarno, Switzerland.

Kessinger, C., S. Ellis and J. VanAndel, 1998: Detection of anomalously propagated ground clutter for the WSR-88D, 15th IIPS, January, AMS, Dallas, TX.

Kosko, B. 1991: Fuzzy systems and neural networks. Prentice Hall, NJ.

O'Bannon, T., 1997: The Enhanced Precipitation Pre-processing Subsystem (PPS) algorithm, 14th IIPS, January, AMS, Phoenix, AZ.

Pratte, F., D. Ecoff, J. VanAndel, and R.J. Keeler, 1997: AP clutter mitigation in the WSR-88D. Preprints, 28th Radar Meteor. Conf., AMS, Austin, TX, 7-12 Sep. 1997.

Sachidananda, M., and D.S. Zrnic, 1997: Phase coding for resolution of range ambiguities in Doppler weather radar, 28th AMS Radar Meteoro. Conf., Austin, TX, 246.

Sachidananda, M., D.S. Zrnic, and R.J. Doviak, 1997: Signal Design and Processing Techniques for WSR-88D Ambiguity Resolution, Part 1, National Severe Storms Laboratory.

Sachidananda, M., D.S. Zrnic, R.J. Doviak, and S. Torres, 1998: Signal Design and Processing Techniques for WSR-88D Ambiguity Resolution, Part 2, National Severe Storms Laboratory, June 1998, 105 pp.

Saffle, R.E. 1997: Nexrad Product Improvement Plan, 14th IIPS, January, AMS, Phoenix, AZ.

Steiner, M. and J. Smith, 1997: Anomalous propagation of radar signals - challenges with clutter, 28th AMS Radar Meteoro. Conf., Austin, TX, 501.

Torres, S., 1998: Ground clutter canceling with a regression filter. NSSL Report, 1998

Turnbull, D., J. McCarthy, J. Evans, and D. Zrnic, 1989: The FAA Terminal Doppler Weather Radar (TDWR) Program. Preprints, 3rd Intl. Conf. on Avia. Wea. Sys., Anaheim, 29 Jan.-3 Feb., Amer. Meteor. Soc., 414-419.


2.0 The Anomalous Propagation Detection Algorithm

2.1 Introduction

Ground clutter contamination within radar data has two sources: normally-propagated (NP) ground clutter from stationary targets such as buildings, trees, or terrain, and anomalously-propagated (AP) ground clutter that arises from particular atmospheric conditions within the planetary boundary layer that duct the radar beam to the ground. Because NP targets are stationary and "unchanging," they are relatively easy to remove with the use of clutter "bypass" maps that are part of the WSR-88D data quality control system. However, AP clutter can evolve as atmospheric conditions change and is not always present. Recognition of the presence of AP clutter is the responsibility of the National Weather Service (NWS) forecasters, who must then modify the clutter filter maps to specify additional regions of clutter filter application. Automatic detection of the AP ground echoes and the specification of ground clutter filters to remove the AP echoes will augment current WSR-88D capabilities, improve the quality of the base data fields and derived products and relieve the forecasters of this responsibility.

In Section 1, the AP Mitigation Scheme was described briefly. Within this scheme, the first step necessary for improving WSR-88D data quality is to design an algorithm to detect the presence of AP clutter echoes. NCAR has developed a fuzzy logic algorithm to perform this task that is called the Anomalous Propagation Detection Algorithm (APDA). In this section, details of the fuzzy logic algorithm development are given, software modifications are described, the methodology for the analysis of case studies is described, and the near-final optimization of the APDA is presented.

2.2 Review of Pratte and Ecoff work

This year's effort was preceded by a period of inactivity on the project after the departure of both Pratte and Ecoff in April 1997. When Kessinger and Ellis began working on the project in late 1997, their first priority was to understand the work done by Pratte and Ecoff so that the algorithm development could continue and be completed.

Initially, Pratte and Ecoff evaluated three techniques for potential usefulness in detecting AP ground clutter contamination (Cornelius et al., 1995; Pratte et al., 1997; Keeler et al., 1998a). These techniques included a neural network (NN) scheme, a fuzzy logic classifier and an empirical, rule-based technique. After extensive testing, the fuzzy logic classifier was selected as the algorithm of choice due to the ease of implementation and the small differences in statistical performance between the fuzzy logic classifier and the neural network classifier. It was at this decision point that Pratte and Ecoff departed NCAR.

During their testing, Pratte and Ecoff selected two sets of WSR-88D scans. These sets of scans were referred to as the "alpha" and "beta" scan sets. The alpha scans were used to train the neural network (NN) AP detection algorithm while the beta scan set was used to evaluate the trained NN algorithm. The fuzzy logic classifier and the empirical scheme were run using these same scan sets. Currently, eleven of the nineteen scans of the alpha and beta scans are included in the dataset used to evaluate the fuzzy logic AP Detection Algorithm (APDA). Because of the addition of the vertical difference of reflectivity feature in the APDA (see Section 2.4), alpha and beta scans were excluded from the current study when the higher elevation angle scan could not be obtained due to not having the A2 tapes on hand. Because Pratte and Ecoff truthed only small regions of each elevation angle, the alpha and beta dataset was re-truthed following the current methodology that is described in Section 2.6. The alpha and beta scans are listed in Table 2.1.

2.3 Selection of Case Studies

To enable easier selection of additional AP case studies, the WSR-88D data contained on the in-house, Archive 2 (A2) data tapes were examined and cataloged. Table 2.2 (pages one and two) contains the Archive 2 (A2) tape catalog.

Time lapse movies in MPEG format were made of all the A2 tapes to facilitate case study selection. For most, the movies consist of the lowest three elevation angles: 0.5, 1.5, and (typically) 2.3 deg elevation. For the Memphis KNQA data taken during the Archive 1 Data Acquisition (A1DA) effort in July 1997, an additional set of movies were made with only the 0.5 deg elevation angle. All movies have two panels that show the reflectivity and radial velocity data at each elevation angle. A list of dates and time for which movies were made is in Table 2.3. Two example movie frames are shown in Fig. 2.1.

The movies were examined and notes made regarding the weather or AP events that were occurring. Table 2.4 (pages one, two and three) gives a short summary of events noted. As a result of the review process, additional cases were selected and included within the AP data set. A total of 60 radar volumes comprise the dataset used to evaluate the AP Detection Algorithm. Within these scans, a variety of weather conditions existed: from clear air return to individual convective cells to large, organized convective complexes with regions of stratiform precipitation. The amount of AP contamination varied from very minor to major areal extent. The 60 scans of the dataset are summarized in Table 2.5.

As shown in Table 2.5, data from nine radars are used in the evaluation process, covering selected areas from the Rocky Mountains (KFTG) to the Atlantic seaboard (KLWX), and from the Great Lakes (KLOT) to the Gulf Coast of Texas (KHGX). As shown by Smith et al., (1996), this area of the United States has the highest occurrences of AP clutter contamination, especially during the summer months. Eighteen of the scans were individual scans with little time history, while 42 of the scans had extended temporal coverage. Eleven of the scans were from the alpha and beta datasets. The extended temporal coverage was done to examine the feasibility of "tracking" AP clutter for the automated AP Mitigation Scheme (Keeler et al, 1999). An analysis of the evolution of an AP event should assist in this work.

2.4 The Fuzzy Logic AP Classifier

A fuzzy logic classifier (also called recognizer) scheme is a technique for scaling various input fields into a common reference frame such that each input indicates the presence or absence of the event that is being detected. For our purposes, the fuzzy logic classifier scheme is being used to detect the presence of AP ground clutter. The framework of the fuzzy logic AP Detection Algorithm developed at NCAR is shown in Fig. 2.2 and covers the processes from the input derived fields to the final output product of the detection algorithm. In this section, an overview is given of the processes used within the AP Detection Algorithm. Additional details are contained in following sections.

Briefly, a fuzzy logic classifier uses various derived fields (formally known as "features") as input, scales them to a common reference frame by use of a "membership function," and then computes a weighted sum of the resultant "interest" fields. After application of a threshold, the final output product of the detection algorithm is obtained and contains the locations of the AP ground clutter. The purpose of the membership function is to scale all input fields to be between zero and unity. In the interest fields, values that approach unity are indicative of the presence of the quantity and that is desired to be detected and values that approach zero are indicative of the absence of that quantity. For example, AP clutter typically has radial velocity values near 0 m s -1 . Regions of the radial velocity interest field will have values near 1 when the radial velocity field is near 0 m s -1 .

As determined by Pratte and Ecoff, the derived fields that showed the most promise for detecting the presence of AP clutter are the "texture" of the signal-to-ratio (TSNR) field, the mean radial velocity (MVE) field, the mean spectrum width (MSW) field, and the standard deviation of the radial velocity (SDVE) field. The TSNR field is similar to the texture of the reflectivity field but was much easier to compute for these tests. Steiner and Smith (1997) compared various AP detection algorithms and found that the vertical gradient of reflectivity showed promise in being able to detect AP clutter. For this reason, the vertical difference of the reflectivity (GDZ) between the 1.5 and the 0.5 deg elevation angles has been added to the APDA. To simplify the calculation, the difference in height between the two elevation angles is not considered; therefore the resultant field is not truly a spatial gradient, but more of a range-dependent reflectivity difference. The equations used for calculating the input fields are shown in Eqs. 2.1-2.5,

     (Eq. 2.1)
     (Eq. 2.2)
     (Eq. 2.3)
     (Eq. 2.4)
     (Eq. 2.5)

where N is the number of values within a specified region, i is the index of each value, SNR is the signal-to-ratio, DZ is the reflectivity value at a range gate, SW is the spectrum width value at a gate, Vi is the radial velocity value at a gate, and V is the mean radial velocity within the specified region.

Pratte and Ecoff determined an initial set of membership functions to be applied to the MVE, TSNR, MSW and SDVE fields. Those membership functions are shown in Fig. 2.3. Membership functions are constructed to distinguish those radar characteristics typical of "clutter" from those characteristics that are typical of "not clutter" (i.e., echoes from weather, clear air, birds, etc.). To understand the differences in radar characteristics between clutter and not-clutter, histograms of the variables were constructed for both cases. This process is described in more detail in Section 2.5.

Determination of the regions of clutter versus regions of not-clutter is called "truthing" the data set. The results of the truthing process are used to statistically score the performance of the AP Detection Algorithm. Improving the statistical scores of the algorithm should lead to a more optimum algorithm that has good performance in the field. For this reason, the process of determining the truth is very important in this algorithm development. To decide the truth regions, a human expert (Kessinger or Ellis) examined all scans of the radar data set and decided which gates had clutter and which had not-clutter. Polygons are drawn and values set within the expert truth field. This process is described in more detail in Section 2.6.

The statistical scores computed within the NCAR AP Clutter Analysis Tool (APCAT) are the Probability of Detection (POD), the False Alarm Ratio (FAR), the Critical Success Index (CSI) (Donaldson, et al., 1975), and the Percent Correct. They are defined as

     (Eq. 2.6)
     (Eq. 2.7)
     (Eq. 2.8)
and
     (Eq. 2.9)

where TP is the True Positive count, FN is the False Negative count, FP is the False Positive count and TN is the True Negative count as shown for the contingency table in Fig. 2.4. The results from the AP Detection Algorithm are scored against the truth field as determined by a human expert.

2.5 Histograms of Data Characteristics

To determine the characteristics of AP ground clutter radar echoes and echoes from targets that are not-clutter, histogram plots are made for the derived variables using the truth field to distinguish between the two types of radar echoes. Figure 2.5 shows examples of histograms of the variables currently used in the AP Detection Algorithm as derived from data taken by the Dodge City, KS (KDDC) WSR-88D radar. Fields shown include the texture of signal-to-noise ratio (TSNR), mean radial velocity (MVE), vertical difference of reflectivity (GDZ), mean spectrum width (MSW), and the standard deviation of the radial velocity (SDVE). Histograms are plotted for regions identified as clutter and for regions identified as not-clutter by the experts.

After construction of the histograms, they are examined to find the maximum differences in the values of the variables that occur within regions of clutter and regions of not-clutter. When differences are large, this indicates a variable that can be used to distinguish clutter from not-clutter. For instance, clutter has high values of TSNR (Fig. 2.5a, page one) due to its high spatial variability while not-clutter has low values of TSNR since it has low variability; likewise, clutter has MVE values near 0 m s -1 Fig. 2.5a while not-clutter has a wide range of MVE values. Similarly, GDZ has a large magnitude in clutter Fig. 2.5c) compared to not-clutter within precipitation echoes. For clear air return, GDZ can also have a large magnitude. The MSW (Fig. 2.5d, page two) and SDVE Fig. 2.5e) fields typically have smaller values in clutter compared to not-clutter. This process is used to define the features used in the AP Detection Algorithm. The histograms also aid in defining the membership functions of the features used in the algorithm.

2.6 Truthing of Cases

"Truthing", or manually reviewing and marking the AP and precipitation areas that will be used for comparison with the fuzzy logic AP Detection Algorithm, is an important activity. Comparison with truth as defined by expert humans is crucial in optimizing the fuzzy logic algorithm for its best performance.

To determine locations of AP clutter, the expert begins by examining the various radar fields and their derived products using the NCAR SOLO program (Oye et al., 1995). Only the 0.5 deg elevation angle is truthed but the other elevation angles are examined for additional insight into the case. All regions of the 0.5 deg scan are characterized as either clutter, not-clutter (precipitation, clear air returns, or anything else), or clutter residue. An attempt was made to define regions where clutter filters were applied for stationary targets (i.e., NP clutter) by use of the clutter "bypass" map. The bypass map defines the location of stationary targets on a gate-by-gate basis and can be augmented by the NWS forecaster, if needed. However, most of the A2 tapes did not have the bypass map on them since the bypass map information was not written to tape before WSR-88D Build 10. Therefore this process of identifying clutter residue is not strictly accurate and represents the expert's best estimate of where the bypass map appeared to have been applied. For this reason, regions of clutter residue are not currently being evaluated.

Typically, AP clutter is characterized by a near-zero radial velocity, low spectrum width, high texture of the reflectivity field, as well as other characteristics. Further, the expert can examine the scan at the next two elevation angles (1.5 and 2.3 degs) to determine if the radar echo in question has vertical continuity. A lack of vertical continuity does not guarantee that the echo is AP clutter, but does give more information into the decision-making process. Once the expert has determined where the AP clutter is, SOLO is used to draw a polygon around the region and the appropriate value is set within the truth field. Currently, three values (1, 2, or 3) are used to define features within the truth field. Regions of clutter are defined by the value 1; regions of non-clutter echo are defined by the value 3; and regions of clutter residue are defined by the value 2. Example truth fields are shown later in this section of the report. Once truthing is completed, the radar data are input into the APCAT for further processing and statistical evaluation of the fuzzy recognizer algorithm.

Of course, a basic limitation of defining the truth in this manner is the lack of independence between the truth field and the radar fields used as input into the AP Detection Algorithm. A preferable methodology would have an independent source for determination of AP clutter regions (such as satellite data or dense rain gauge networks). Determining the truth in any of these manners is beyond the scope of work for the program and the availability of these data sets is severely limited. However, the data available from NCAR's S-Pol dual-polarimetric radar provides an opportunity to obtain an independent classification of the locations of clutter (i.e., the truth). A few scans of S-Pol data are planned to be put through the AP Detection Algorithm and the results evaluated. This effort is described in more detail in a later section.

In most cases, differentiating between clutter echoes and weather/clear air echoes is relatively easy especially when clutter is severe; however, several issues became apparent during the process. Clearly, human subjectivity effected the final truth fields. The location of the boundary between clutter echo and not-clutter echoes is sometimes difficult to determine precisely, especially in cases of weak or marginal clutter contamination.

In some cases, the AP clutter occurred in a few gates within a region of weather echo. Drawing a polygon around each of these small "speckles" is a difficult task for any human. These situations may have some error in the truth fields.

The radar scanning strategy of the WSR-88D system effects the quality of the statistical results for the AP detection algorithm. The reflectivity data are collected with 1 km gate spacing while radial velocity data are collected with 250 m gate spacing. The reflectivity scan at 0.5 deg is performed first at low pulse repetition frequency (PRF), followed by the radial velocity scan at a higher PRF. The APCAT is used to merge the two scans in range (to 250 m) and in azimuth. Because so-called "indexed beams" are not used, precise positioning of the beams cannot be accomplished scan-to-scan. Typically, the azimuths between the two scans match within a few tenths of a degree. Therefore, temporal and spatial differences exist between the reflectivity and radial velocity data.

The quality of the truth field is adversely effected by the spatial differences of the reflectivity and radial velocity fields since there are 4 radial velocity gates per 1 reflectivity gate. Despite the diligence of the human experts, drawing polygons to enclose AP on a gate-by-gate basis is difficult. Within APCAT, the statistical results are scored on a gate-by-gate basis. While difficult to quantify, the difficulties with the truth field likely tend to degrade the statistical scores.

When clutter echoes are mixed with weather echoes, the data may be biased by the clutter echo but not so much that it is clearly clutter. From the A2 data stream and derived fields, determination of regions of mixed clutter and weather signals is difficult. Clearly, application of the ground clutter filters are needed in situations such as this, but detection of the feature may be difficult. Steiner and Smith (1997) noted that mixed clutter and weather signals are the most challenging for any detection algorithm.

Truthing the data at far ranges is extremely difficult since the reflectivity and derived reflectivity fields are frequently the only fields available beyond 150 km. In good situations, all fields are available to 230 km. However, the Doppler fields usually have significant missing data regions beyond 150 km due to a failure in the WSR-88D velocity recovery algorithms. Also, at far ranges, the values of the texture of the SNR and the spectrum width become less distinct between clutter and weather echoes due to the larger size of each radar resolution volume. Furthermore, the vertical difference of reflectivity is adversely effected as an indicator of AP clutter at long ranges due to the increasing height of the beam. The radar beam may "over-shoot" low altitude weather echoes, such as stratiform rain or winter storms, resulting in a vertical difference of reflectivity field that appears to indicate the presence of AP clutter.

For the work summarized in the FY96 Status Report (dated November 1996 and revised in September 1997), different criteria were used to define regions of truth than are being used now. The primary focus of the FY96 work was the neural network (NN) AP detection algorithm, rather than the fuzzy logic AP detection algorithm. The NN algorithm required very "pure" truth to train the neural network. For this reason, very limited regions were used to define the truth such that only the most definitive characteristics of the feature were included. Using these same truth regions, the preliminary fuzzy logic algorithm was evaluated. The evaluation showed it had similar statistical skill as the NN algorithm and performed quite well. Using very pure truth will produce higher statistical skill scores than if all regions of the scan are truthed because of the natural variability of the radar data that are either totally or partially biased by clutter. Due to the difference in truthing methodology, statistical skill scores of the current AP Detection Algorithm will be lower, but more representative of actual performance, than the scores reported in FY96.

2.7 APCAT Improvements

The AP Clutter Analysis Tool (APCAT) is a software package developed by Joseph VanAndel and others at NCAR that calculates the interest fields used in the AP Detection Algorithm, applies the fuzzy membership functions to the features, outputs the algorithm results, and performs a statistical analysis of results using the truth field described above. Section 2.4 describes this process in detail. The APCAT is under development as needed and has had several major enhancements made this fiscal year. In this section, the improvements made to APCAT are discussed.

The vertical difference calculation was added to compute the difference in reflectivity and radial velocity between the 0.5 and 1.5 deg scans. As recommended by Steiner and Smith (1997), the vertical gradient in reflectivity is useful for detection of AP contamination since AP echoes are generally at maximum intensity near the ground and weaken rapidly with height. In our implementation, the difference in reflectivity is computed rather than the gradient. For the A2 data format, an SNR threshold is applied to the base data, resulting in a considerable number of missing data regions. The number of missing data regions typically increases as the elevation angle increases especially when AP is present. Also, because AP weakens rapidly with height, there are many fewer gates of AP at the 1.5 deg elevation angle than at the 0.5 deg elevation angle. For these reasons, when the reflectivity value is missing at the 1.5 deg scan (usually because it has been thresholded out), the radar noise power is used to compute a reflectivity value at the "missing" gate. The result of using the radar noise power to compute a new reflectivity value is that the vertical difference is exaggerated. Having an unthresholded base data stream will eventually correct this problem. Membership functions (described below) for the vertical difference in reflectivity will likely need some adjustment when an unthresholded data stream is available in the Open Systems Architecture.

A "diagnostic mode" was added to APCAT to show the interest output of each individual membership function. Prior to this, APCAT produced only the final interest field from adding the summed weights of the individual interest fields. This helps determine if the membership functions have been tuned properly for various types of input data, which should improve the recognizer's performance. In addition, APCAT was restructured to compute multiple fuzzy recognizers in parallel. This allows testing various combinations of membership functions and weights at once, rather than requiring the user to run APCAT multiple times.

The SOLO perusal/editing program stores data in DORADE format within disk files called "sweepfiles." A sweepfile holds one elevation angle of radar data. Because SOLO and APCAT are used in conjunction, APCAT uses the DORADE format within sweepfiles, too. Formerly, users had to run SOLO to create new DORADE variables for the fuzzy logic interest fields. This made for an extra step in the processing and was cumbersome to do. Therefore, APCAT was enhanced to streamline sweepfile processing to automatically create the necessary DORADE variables.

Because of the large number of scans (i.e., 60) within the AP evaluation data set, it became necessary to enhance APCAT's batch processing capabilities. Using pre-selected menus, APCAT can automatically process a list of files by merging the reflectivity and velocity scans, creating DORADE variables, computing the vertical difference feature and other selected features, and computing the fuzzy recognizer outputs and doing the statistical scoring. This simplification in procedures significantly improved data processing.

The APCAT's statistical performance scoring software was improved. The values within the interest final output field vary continuously between 0.0 and 1.0. APCAT computes the statistical performance of the detection algorithm using 21 different thresholds between 0.0 and 1.0. If the output value of the detection algorithm is over the specified threshold, it is considered a True Positive. The contingency tables ( Fig. 2.4), containing the counts of True Positives, True Negatives, False Positives, and False Negatives for each threshold value, are output to a file for diagnostic and debugging purposes. Examination of the contingency tables allows comparison of the algorithm output versus the truth values. In addition, the new version of APCAT allows the interactive selection and display of performance scores for each threshold ( Fig. 2.6). Numeric readouts for Probability of Detection (POD), False Alarm Ratio (FAR), Critical Success Index (CSI), and Percent Correct are displayed for the selected threshold. In addition, the point representing the POD and the FAR for the given threshold is highlighted on the left panel of the performance graph. These improvements make it feasible to optimize the membership functions and to determine the best threshold value for a given detection algorithm.

2.8 Optimization of the AP Detection Algorithm

In this section, the methodology for optimizing the AP Detection Algorithm is discussed and casestudies shown that illustrate the performance of the algorithm. Results are summarized.

2.8.1 Methodology

Before they left NCAR, Pratte and Ecoff had written considerable documentation about the work that they had done. They left personal notes and communications, discussed their methodology on the telephone and with electronic mail, and then returned to NCAR for a night-time meeting with the Data Quality group in February 1998.

As shown in Fig. 2.3, Pratte and Ecoff defined four membership functions for the AP Detection Algorithm (APDA). These functions used the TSNR, MVE, SDVE, and MSW fields as input. Because the work of Steiner and Smith (1997) suggested that the vertical gradient of reflectivity is a useful indicator for AP contamination, a variation of this field (called GDZ) was added to the APCAT software. The GDZ field is the vertical difference of reflectivity and is computed by subtracting the reflectivity value at the lower elevation angle from the higher elevation angle.

The optimization of the APDA consists of modifying the membership functions for each input field and then examining the statistical performance via APCAT to see if improvements occur. Histograms are used to look at the distribution of the interest values within each of the individual interest fields. Using the truth field to distinguish between regions of clutter and regions of not-clutter (i.e., precipitation, clear air return, etc.), histograms are made for the interest fields to see the distribution of values for the 60 scans of the data set ( Table 2.5). For clutter regions, the interest fields should have many values near unity. For regions of non-clutter, the interest fields should have many values near zero. Figure 2.7a shows the histogram for the APDA results for both clutter and not-clutter regions when the original membership functions are used. The original membership function does not define regions of clutter Fig. 2.7b) very well since the number of occurrences does not change much as the interest value increases. Regions of not-clutter are defined quite well when an interest threshold of 0.5 is applied since the majority of the area under the curve is below this threshold (Fig. 2.7c). These plots clearly show that the original membership functions will have a large number of missed detections for AP clutter after application of a threshold (typically near 0.5).

The original membership functions ( Fig. 2.3) served as the starting point for the optimization process. Various iterations were performed on the membership functions to achieve optimization. Histograms of the individual interest fields were examined during this phase. For this report, the iterations are summarized into six tests. Two sets of membership functions are used for these six tests and are shown in Figs. 2.8 and 2.9. Table 2.6 describes each test. Briefly, the "Original Test" uses the original membership functions shown in Fig. 2.3 with the weights for all interest fields being unity. The "Test #1" uses the membership functions shown in Fig. 2.8 with the exception that the GDZ field is not used. New functions for the MVE and MSW fields are defined. Weights for all interest fields are unity. The "Test #2" uses all the membership functions shown in Fig. 2.8 and all weights are unity for the interest fields. The "Test #3" uses the membership functions shown in Fig. 2.9. New functions have been defined for the SDVE and GDZ fields and all weights are unity. The "Test #4" is the same as "Test #3" with the exception that the interest field weights are 2 for the MVE and TSNR fields and are 1 for the SDVE, MSW and GDZ fields. The "Test #5" and "Test #6" are duplicates of Test #3 and Test #4 except that the unfiltered radial velocity (VE) has been substituted for MVE and the unfiltered spectrum width (SW) has been substituted for MSW. This was done to examine the performance of the algorithm when the mean filter is not used. If the fields are not filtered, computational speed may be enhanced; however, algorithm performance may suffer.

Using the 60 scans that comprise the data set for evaluation of the algorithm ( Table 2.5), the six tests plus the original membership functions are evaluated. Tables 2.7-2.10 contain the statistical performance indices (POD, FAR, CSI and Percent Correct) for each of the algorithm tests. Statistical performance results from APCAT are also plotted in Figs. 2.10-2.15 and Figs. 2.18-2.19.

The Original Test has its highest CSI score of 0.462 when a threshold of 0.4 is applied. The POD is 0.64, the FAR is 0.098 and the Percent Correct is 0.848 ( Table 2.7 and Fig. 2.10). These scores are lower than reported by Cornelius et al., (1995) but the differences in truthing methodology as discussed in Section 2.6 should account for the lower statistical scores. The highest CSI score for Test #1 is 0.478 at a threshold of 0.5 ( Table 2.8 and Fig. 2.11), a slight improvement in performance over the Original Test. The Test #1 POD is 0.662, the FAR is 0.099, and the Percent Correct is 0.853. Recall that the difference between the Original Test and Test #1 was to modify the membership functions for the MVE and MSW fields ( Table 2.6). Therefore, changing these two membership functions leads to slightly improved performance of the AP Detection Algorithm. For Test #2 ( Table 2.8 and Fig. 2.12), the membership functions from Test #1 are used with the addition of a new input field, GDZ. The membership function for GDZ is not optimized for this test and resulted in a decreased performance for the detection algorithm. The CSI for Test #2 is 0.453 at a threshold of 0.45 while the POD is 0.599, the FAR is 0.089 and the Percent Correct is 0.844. The FAR is slightly improved over the results from the Original Test.

For Test #3, new membership functions are defined for the SDVE and GDZ fields ( Table 2.9 and Fig. 2.13). At a threshold of 0.55, Test #3 has a CSI of 0.475, a POD of 0.639, an FAR of 0.095 and a Percent Correct of 0.848. The weights for all membership functions is unity. Notice that the skill scores from this test are very similar in magnitude to the skill scores for the Original Test. For Test #4, the membership functions are the same as used for Test #3, but the MVE and TSNR fields have a weight of two instead of one. With a threshold of 0.55 ( Table 2.9 and Fig. 2.14), the skill scores are slightly improved over the Original Test with a CSI of 0.492, a POD of 0.663, a FAR of 0.096 and a Percent Correct of 0.852. Figure 2.15 compares the "POD vs False Alarm Ratio" of the Original Test versus Test #4 and shows the slight statistical improvement of Test #4 over the Original Test. The POD has the best improvement when compared to the best score of the Original Test. The most noticeable difference between Test #4 and the Original Test is the increase in the value of the threshold.

Using histograms, the distribution of interest values within the final output fields of the APDA using Test #3 ( Fig. 2.16) and Test #4 ( Fig. 2.17) are compared. In panel (a) of each figure, the interest values for both clutter and not-clutter regions are plotted; in panel (b), the interest values for clutter regions only are plotted; and in panel (c), the interest values for not-clutter regions only are plotted. The y-axis on these plots are not fixed but are dependent on the number of occurrences within each category. Therefore, examination of the y-axis must be done to facilitate the comparison. Test #4 distinguishes clutter (Fig. 2.17b) slightly better (more of the interest values are closer to 1) than Test #3 (Fig. 2.16b), but the difference is small. For regions of not-clutter, Test #4 (Fig. 2.17c) does noticeably better than Test #3 (Fig. 2.16c) because more of the interest values are below 0.5. In other words, Test #4 will distinguish between clutter and not-clutter better than Test #3 will. Using a threshold of 0.5, the Test #4 will perform better than Test #3 for the APDA.

In Mattias Steiner's evaluation of the AP Detection Algorithm performance (see Section 3), he noted that the algorithm may have difficulty completing its calculations in a timely manner when run in a real-time environment. Given the rapid increase in computing power that is occurring today, slow algorithm performance in a real-time environment is not expected to be a problem at the time of implementation. However, to minimize the amount of computations and to assess the resulting algorithm performance, two algorithm tests were devised that did not use the mean filtering. These tests, Test #5 and Test #6, are identical to Test #3 and Test #4 with the exception that the radial velocity (VE) field is used instead of the MVE field and the spectrum width (SW) field is used instead of the MSW field. As can be seen in Table 2.10 and in Fig. 2.18, Test #5 has its highest CSI score of 0.479 with a threshold of 0.6. The POD is 0.606 and the FAR is 0.073. Comparing the results of Test #5 to Test #3 shows that similar results are obtained. At a threshold of 0.55 (Table 2.10 and Fig. 2.19), Test #6 has a CSI score of 0.489, a POD of 0.665 and a FAR of 0.099. Again, these results are similar to those obtained for Test #4. In summary, disabling the mean filter does not appear to greatly affect the performance of the AP Detection Algorithm. However, disabling the mean filter is not recommended if computing power is sufficient. For the SDVE field, a mean value must be calculated for the radial velocity (see Eq. 2.5) field. Therefore, the only computational savings occurs when the spectrum width is not filtered. Disabling the mean filter, will result in noisier output from the AP Detection Algorithm as shown in examples below in Section 2.8.2. Noisy algorithm output may not be desirable for the operational goals of the algorithm.

2.8.2 Detailed Examination of Data from KAMA

In this section, one scan from the 60-scan data set is examined in detail from the Amarillo, TX WSR-88D (KAMA) on May 25, 1994 at 0035 UTC. This scan contains a region of AP echo and a region of intense convective precipitation and allows the AP Detection Algorithm output to be examined for two diverse types of weather events. Further, second trip contamination is contained with the AP echo.

2.8.2.1 A Region of AP Echo

The AP echo is mostly in the northern quadrant of the 0.5 deg elevation angle ( Fig. 2.20). It is a region having maximum reflectivity of 35-40 dBZ, radial velocities near 0 m s -1 , high texture of the SNR field (> 15), values of 0-1 m s -1 of the SDVE field, strongly negative vertical difference of reflectivity (down to -45 dBZ), and low values of MSW (1-2 m s -1 ). The AP echo is occurring as a large, nearly homogeneous region of return. A region of second trip echo is found near the 330 deg radial and is within the AP return. The second trip echo shows best in the Doppler channels which may explain the lack of detection and correction of the echo by the WSR-88D second trip correction algorithms. The second trip echo has higher radial velocity values, higher SDVE values, and higher MSW values than the AP echo, and is quite easy to distinguish visually with these fields.

Interest field output from the Original Test is shown in Fig. 2.21a . The membership functions used for the Original Test are shown in Fig. 2.3. Unthresholded interest output from each of the input fields is shown in Fig. 2.21b-d and 2.21f. The expert truth field is shown in Fig. 2.21e with the regions of AP echo shaded green. Comparing the truth field to the Original Test output shows that the AP Detection Algorithm identifies the AP echo quite well when a threshold of about 0.5 is used (see Fig. 2.24d for the thresholded output). Notice that the Original Test does not identify the second trip as AP clutter for that threshold. Comparing the MVE and SDVE interest fields (Fig. 2.21b and 2.21d, respectively) to the expert truth field shows that the regions of AP are well-defined for a threshold of about 0.5 for both. The MSW interest field (Fig. 2.21f) also defines the AP region quite well when a lower threshold of about 0.4 is used. The MVE, SDVE, and MSW interest fields distinguish the AP echo from the second trip echo quite well and do not falsely identify the second trip echo as AP clutter. The TSNR interest field (Fig. 2.21c) has high values within the AP and second trip echoes and cannot distinguish between the two types of return.

The interest field output from the Test #4 is shown in Fig. 2.22 and uses the membership functions shown in Fig. 2.9. Comparing the Original Test ( Fig. 2.21a ) to the Test #4 (Fig. 2.22a) shows that the Test #4 interest values are higher within the AP echo for all corresponding fields excluding TSNR. Interest values within the second trip echo are higher than for the Original Test but a threshold of about 0.5 will result in nearly the same algorithm performance in this region. Comparing the Doppler interest fields (MVE, SDVE, and MSW) shows that the Test #4 interest fields have higher interest values than the Original Test, in general. The interest field for TSNR is the same for both tests since the membership function is not changed. Similar to the TSNR interest field, the GDZ interest field (Fig. 2.22e) indicates the presence of AP echo but cannot distinguish between the AP and the second trip echoes. Typically, second trip echo diminishes in intensity with height and will likely be a source of error in GDZ field of the AP Detection Algorithm. However, it is unusual to see much second trip echo within the A2 reflectivity data stream (at least in the cases examined here) and should not be a significant source of algorithm error.

Unthresholded, final algorithm interest fields for all the tests (except the Original Test) are shown in Fig. 2.23. For Test #1 (Fig. 2.23a), modifying the membership functions for MVE and MSW has increased the final interest values over those of the Original Test ( Fig. 2.21a). The second trip echo is correctly identified as a non-clutter target. For Test #2, the GDZ interest field is added but the GDZ membership function is not optimal. The most obvious difference between Test #1 and Test #2 is an increase in interest values within the second trip echo as well as a small, general decrease in interest values within the AP echo. Comparing Test #3 to the Original Test shows that the interest values increase everywhere. Recall that Test #3 uses new membership functions for GDZ and SDVE ( Fig. 2.9). For Test #4, the weights for the MVE and TSNR interest fields are 2 rather than 1 in the computation of the summed weights. The effect of increasing the weights is to better distinguish the second trip echo from the AP echo. Typically, the MVE and TSNR fields provide the best distinction between clutter and non-clutter echoes. Comparing Test #3 (Fig. 2.23c) to Test #5 (Fig. 2.23e) and comparing Test #4 (Fig. 2.23d) to Test #6 (Fig. 2.23f) shows that similar results are obtained but the unsmoothed test output has more small-scale variation, as is expected when unfiltered input fields are used.

To better illustrate the differences in algorithm performance with the various tests, the thresholds determined to be optimal in Section 2.8.1 are applied to the final output of the Original Test ( Fig. 2.24c), the Test #3 (Fig. 2.24d) and the Test #4 (Fig. 2.24e). The mean reflectivity (Fig. 2.24a), MVE (Fig. 2.24b), and the expert truth (Fig. 2.24c) fields are shown for ease of reference in this figure. Further, the summarized case studies discussed in Section 2.8.3 use this figure layout. Duplicating the figure layout for the KAMA scan allows easier comparison of this case to the summarized cases. The thresholds applied to the final algorithm output are 0.4 for the Original Test, and 0.55 for Test #3 and Test #4. Further, the statistical scores for the POD, the FAR and the CSI are summarized in Table 2.11 for each of the scans discussed in Sections 2.8.2 and 2.8.3. Examination of the skill scores for each scan shows the variation in algorithm performance that might be expected. As can be seen in Fig. 2.24, the three tests cover the area of AP fairly well and do not detect the second trip echo as AP. The POD scores for the three tests are 0.713, 0.716 and 0.736, respectively, and shows that the three tests detect the AP echoes consistently. Likewise, the FAR scores are similar and have values of 0.098, 0.096 and 0.099, respectively. For this case, all the algorithm tests have similar statistical performance.

2.8.2.2 A Region of Precipitation Echo

For any evaluation, examining the algorithm performance in regions of non-occurrence of the feature of interest (i.e., the "null" event) is important. For this reason, a precipitation echo is selected for a detailed examination of the APDA performance. Using the same scan as examined above, three convective cells are present to the SW through SE of the KAMA radar ( Fig. 2.25a) and have maximum reflectivity values in excess of 55 dBZ. A gust front has been produced as evidenced by the strong positive radial velocities to the south of and within the cells (Fig. 2.25b). The TSNR field (Fig. 2.25c) shows generally low texture of SNR is present with values generally < 15. In regions of strong reflectivity gradients such as at the edges of cells, the TSNR values can exceed 45. The SDVE field (Fig. 2.25d) has values > 1 m s -1 . The vertical difference field (Fig. 2.25e) shows that the precipitation cells have small vertical differences in reflectivity within the heaviest precipitation. Notice that the vertical differences are greatest at the edges of the precipitation echoes where reflectivity at 0.5 deg is < 20 dBZ. The MSW (Fig. 2.25f) has high values that are > 2 m s -1 within the most intense reflectivity regions of the storms.

Final interest output from the Original Test is shown in Fig. 2.26 with the expert truth field. Within the regions of highest reflectivity ( Fig. 2.25a), the output of the AP Detection Algorithm (Fig. 2.26a) shows interest values of < 0.4, typically. If a threshold of 0.4 is used to determine the presence of AP clutter (as suggested in a previous section), the algorithm output suggests that the convective cells have no AP clutter and do not require additional clutter filtering. Small, scattered regions of AP clutter are identified near the westernmost cell (between azimuths 180 deg to 270 deg). The final algorithm output (2.20a) identifies some of the expert-identified regions as AP clutter. In the region to the east and northeast of the strong cells and to the east of the gust front in a region of weak radial flow, interest values are > 0.5 due to the MVE values being near 0 m s -1 (Fig. 2.25b), the low values of the MSW field (Fig. 2.25f), and the low values of the SDVE field (Fig. 2.25d). A small region of AP clutter is near 125 deg azimuth and 100 km range but the majority of the algorithm output in excess of 0.5 interest in the eastern quadrant must be considered a false alarm on the part of the algorithm.

Examination of the output from the Test #4 version of the membership functions ( Fig. 2.27) shows similar results as the Original Test. Interest values are slightly higher in the region of weak flow to the east of the gust front. The interest fields for MVE (Fig. 2.27b), SDVE (Fig. 2.27d), and MSW (Fig. 2.27f) have typically higher values than those of the Original Test, and contribute to the higher interest values within the final output field in this region. The addition of the GDZ interest field (Fig. 2.27e) supports the expert determination that most of the region shown in the figure is not clutter. The GDZ interest values tend to be highest along the edges of precipitation regions and in regions of clear air return near 260 deg azimuth and 60 km range.

Unthresholded, final output from all the algorithm tests are shown in Fig. 2.28. Comparing Test #3 output (Fig. 2.28c) to Test #4 output (Fig. 2.28d), the most noticeable difference is in regions where the MVE field is not near zero velocity. Notice for the region near 150 deg azimuth and 25 km range, the interest values of Test #4 are lower than those of Test #3 by about 0.1. This occurs despite the high contribution from the SDVE and MSW fields. Likewise, the clear air region to the west of the radar (270 deg azimuth and 65 km range) has lower interest values in Test #4. In the weak flow region to the east of the radar, both Test #3 and Test #4 have similar interest values. Looking at the unsmoothed interest fields of Test #5 (Fig. 2.28e) and Test #6 (Fig. 2.28f), shows that similar results are obtained when compared to Test #3 and Test #4 but more spatial variance occurs in the interest field due to not filtering the input fields.

Applying the thresholds of 0.4 to the final output fields of the Original Test ( Fig. 2.29d), and of 0.55 to the final outputs of both Test #3 (Fig. 2.29e) and Test #4 (Fig. 2.29f) allows comparison of the statistical performance for these tests. As noted in the discussion above, the POD, FAR and CSI scores for this scan are very similar for the three tests. Differences seen in the precipitation regions (that would be considered false detections) are small between the three tests. Within the weak flow area to the east of the radar, Test #4 appears to have the fewest number of false detections while the Original Test has the most number of false detections. These visual inspections are somewhat subjective, however.

2.8.3 Summarized Examination of Case Studies

In this section, scans from the various radars are shown to illustrate algorithm performance in a variety of weather situations. In some cases, clear air return is the predominant echo, in others precipitation is predominant, and others have mixed amounts of return. The amount of AP contamination varies with each case from no echoes to an extensive amount of echoes. Likewise, the amount of second trip echo (leading to missing data regions in the radial velocity field) varies from scan to scan. For these cases, plots of the three algorithm tests (with appropriate threshold applied) are shown to give a quick overview of the AP Detection Algorithm performance within each situation. Results from the Original Test, Test #3 and Test #4 are shown along with the reflectivity data, radial velocity data and the expert truth field. Statistical scores for the individual scans are given in Table 2.11 for the POD, FAR and CSI. Further, a ratio is given for the number of gates of the MVE field versus the number of gates of the TNSR field to examine the effect that missing MVE data may have on the algorithm performance. Likewise, a ratio is given for the number of gates having AP clutter versus the number of gates of not-AP clutter (i.e., clear air or precipitation return) as determined from the expert truth field to examine what effect, if any, the particular atmospheric conditions may have on the algorithm performance.

In Table 2.11, the MVE/TSNR ratio is included since the performance of the algorithm is dependent on the number of fields that are input into the final output of the AP Detection Algorithm. Typically, more of the MVE field has missing gates than the TSNR field due to thresholding differences and second trip echo removal. Values of the ratio do not exceed 100% for the cases shown, indicating that there are always more TSNR range gates of data than MVE range gates. When the MVE field is missing from a gate, the MSW and SDVE fields are also missing. When the TSNR field is missing from a gate, the GDZ field is also missing. Of the 14 cases listed in Table 2.11, seven have MVE/TSNR ratio values >80% while seven have ratio values <80%. Looking at the Test #4 results, of the seven cases with ratio values <80%, six have FAR scores >0.10 indicating that the algorithm does not perform well when there are many missing gates from the three Doppler fields. Of the seven cases with ratio values >80%, only four have FAR scores >0.10. Likewise, five of the seven cases with ratio values <80% have POD scores <70% while only four of the seven cases with ratio values >80% have POD scores <70%. Missing data does not explain all of the performance differences between cases, however, it can be an important contributor and must be taken into account. While the number of case studies is small and may not be statistically significant, it seems intuitively obvious that the APDA will perform best when all input fields are available.

The ratio of the number of range gates declared as AP clutter echo to the number of range gates declared to be not-AP clutter echo (i.e., precipitation or clear air echo) is also included in Table 2.11. This AP/not-AP ratio shows that the three cases having the least amount of AP echo are the three cases with the lowest POD scores (<0.35) overall. These cases are discussed in Sections 2.8.3.5, 2.8.3.6 and 2.8.3.12. In these cases, the regions of AP clutter tend to occur in very small patches, rather than in a large region of nearly homogeneous clutter. It may be that the APDA performs best after a minimum amount of area of AP contamination is attained. If true, quantification of the minimum amount of area is not known at this time. The 19 October KLOT case at 0011 UTC (Section 2.8.3.12) has a POD of 0.337 and an MVE/TSNR ratio of 91.1% indicating that missing data are not contributing much to the low performance of the POD. The AP/not-AP ratio is 3.2% which is quite low and shows that the number of gates having AP return is few. Examination of the data from this case shows that the AP is occurring in relatively small patches within a large region of clear air return and may be contributing to the low POD score.

2.8.3.1 KLSX, July 7, 1993 at 0334 UTC

A case of moderate intensity precipitation (mean reflectivity values of about 35 dBZ are indicated in Fig. 2.30a) that has embedded AP return is examined using data from the St. Louis, MO WSR-88D (KLSX) on July 7, 1993 at 0334 UTC. The mean radial velocity (MVE) field (Fig. 2.30b) has values near 0 m s -1 within the region of AP as determined by the human expert (Fig. 2.30c). Thresholded interest output from the three tests shows a generally good detection of the main AP region with small regions of missed detections to the NE of the main body of AP return (i.e., near 60 deg azimuth and 140 km range). Notice also that the 0 m s -1 line that is oriented from west to east through the radar position also shows high interest values, some of which are false detections. Comparing the output from the Original Test (Fig. 2.30d) and from the Test #3 (Fig. 2.30e) to the output from Test #4 (Fig. 2.30f) shows that Test #4 has slightly fewer regions of false detections, especially in the area of the 0 m s -1 line through the radar position. The POD scores for the Original Test, Test #3 and Test #4 are 0.419, 0.435 and 0.460, respectively, showing that Test #4 has the best POD performance by a small amount. The FAR scores are 0.085, 0.059 and 0.056, respectively. The CSI score of the Original Test is 0.228 versus a score of 0.294 for Test #4.

2.8.3.2 KTLX, July 1, 1994 at 1106 UTC

A line of convective cells oriented north-south and having moderate to high intensity in reflectivity ( Fig. 2.31a) is found with clear air return and an isolated, nearly homogeneous region of AP return (Fig. 2.31c) in a case from the Oklahoma City, OK WSR-88D (KTLX) on July 1, 1994 at 1106 UTC. The MVE field (Fig. 2.31b) shows that the AP return has near 0 m s -1 values while the convective cells have regions of positive, negative and near-zero values. Thresholded output from all tests show good detections of the region of AP return. All the algorithm tests have some false detections within the western edges of the precipitation cells, especially near 330 deg azimuth and 65 km range. These false detections are due to contributions from all the individual interest fields (not shown) as derived from the MVE, TSNR, GDZ, SDVE and MSW fields. Further, a small region of high interest values near 350 deg azimuth and 25-50 km in range is created due to the absence of the Doppler fields (see Fig. 2.31b). The Doppler fields were removed due to the presence of second trip echo. When a field or fields are missing, the AP Detection Algorithm will use the remaining interest fields to compute the final interest value. Comparing Test #3 (Fig. 2.31e) to Test #4 (Fig. 2.31f) shows that the Test #4 had slightly better performance with fewer false detections in the precipitation region.

For this scan the POD scores for the Original Test, Test #3 and Test #4 are 0.781, 0.684 and 0.699, respectively, showing that the Original Test has the best POD performance. The FAR scores are 0.073, 0.065 and 0.069, respectively showing that Test #4 has the best FAR performance although the differences in the scores is slight. The CSI scores for the three tests are 0.320, 0.317 and 0.314, respectively, and show that the three versions of the AP Detection Algorithm have similar performance scores for this case.

2.8.3.3 KHGX, October 19, 1994 at 0530 UTC

Weak precipitation echo ( Fig. 2.32a) is embedded within clear air and AP return in a case from the Houston, TX WSR-88D (KHGX) on October 19, 1994 at 0530 UTC. Regions of AP echo are prevalent in the area of data shown. As can be seen in the figure, the maximum range of the first trip has been reached (near 150 km), leading to a discontinuity in the mean radial velocity (Fig. 2.32b) field. As mentioned above, when an interest field is not available for a particular range gate, the weighted sum containing the final output interest value of the AP Detection Algorithm is computed using as few as 1 field. In this example, the near-complete loss of the three Doppler fields (MVE, SDVE, and MSW) at ranges > 150 km have created a discontinuity in the algorithm output that is visible in all three tests (Figs. 2.32d-f). This discontinuity is characterized as an increase in the interest values from about 0.7 to 1.0. The Original Test (Fig. 2.32d) appears to have the poorest performance with many missed detections in the AP regions. The POD scores for the Original Test, Test #3 and Test #4 are 0.450, 0.586, and 0.589, respectively and verifies the visual inspection that the Original Test has more missed detections. However, the Original Test has the lowest FAR score of 0.096 compared to 0.144 and 0.160 for Tests #3 and #4, respectively. The CSI scores for the three tests are 0.402, 0.496 and 0.490, respectively. The addition of the interest field for GDZ (not shown) and the changes to the SDVE membership function are the likely explanations for the improvement in POD performance for Test #3 and Test #4.

2.8.3.4 KDDC, July 13, 1993 at 0655 UTC

A dissipating squall line is oriented SW-NE in a case from the Dodge City, KS WSR-88D (KDDC) on July 13, 1993 at 0655 UTC ( Fig. 2.33). The AP echoes form behind the squall line after the passage of the gust front at the radar. The gust front is located around 150 deg and 40 km at this time. As seen by the expert truth field (Fig. 2.33c) much of the echo is from AP clutter return that is either embedded within the precipitation return or is occurring in nearly homogeneous patches behind the squall line. Performance of the three algorithm tests (Fig. 2.33d-f) show that all algorithms had similar performance in the region of AP echoes and detect the AP reasonably well. They have similar poor performance in the region of near 0 m s -1 mean radial flow (Fig. 2.33b) located to the south and southeast of the radar and along the leading edge of the gust front where small areas of false detections are present. For this scan, the POD scores are 0.740, 0.680 and 0.701 for the Original Test, Test #3 and Test #4, respectively. The Original Test has the highest POD score, although the range of values is not large. FAR scores for the three tests are 0.126, 0.124 and 0.123, respectively, and the CSI scores are 0.394, 0.382 and 0.397, respectively.

2.8.3.5 KLOT, October 19, 1995 at 2331 UTC

A dissipating squall line with a minimal amount of AP echo is shown ( Fig. 2.34) from the Chicago, IL WSR-88D (KLOT) on October 19, 1995 at 2331 UTC. This case has an AP/not-AP ratio of 1.9%. The near-surface wind flow is from the WNW. A gust front has passed the radar location and is currently located SE of the radar near 120 deg azimuth and 30 km range. Except for a few small regions within 50 km of the radar that are AP return (Fig. 2.34c) most of the echo is from precipitation or clear air. Within the squall line area, all three algorithm tests produce false detections near the edges of the precipitation echoes. These false detections are likely the result of the contribution from the TSNR interest field (not shown) and to a lesser extent, the SDVE interest field (not shown). Within 50 km of the radar in the region of near-zero MVE values, the Original Test appears to have the best performance with the fewest number of false detections. The GDZ interest field (not shown) is the source of the decreased performance of the Test #3 and Test #4 in this region. Regions of clear air return generally have elevated GDZ interest values due to the decrease in reflectivity values with height that is typical of this type of return. For the overall scan, the POD scores are 0.209, 0.295, and 0.336 for the Original Test, Test #3 and Test #4, respectively, and shows that Test #4 has outperformed the Original Test by about 13% despite the false detections within the clear air. The FAR scores are 0.095, 0.116 and 0.107, respectively, and show that all tests had similar performance with false detections. The CSI scores are very low and are 0.047, 0.054 and 0.066, respectively. The poor POD performance of the APDA may be explained by the low AP/not-AP ratio and by the low value of the MVE/TSNR ratio (63.3%) that indicates significant amount of missing Doppler channel data.

2.8.3.6 KFTG, September 21, 1995 at 0206 UTC

An example of stratiform precipitation is examined with the Denver, CO WSR-88D (KFTG). For this case on September 21, 1995 at 0206 UTC, the widespread precipitation is occurring within stratiform echo having maximum reflectivity of about 25-30 dBZ ( Fig. 2.35a). Wind flow is from the NE near the surface with a well-defined regions of 0 m s -1 visible in the MVE field (Fig. 2.35b). The expert truth field (Fig. 2.35c) shows clutter residue regions near the mountains, a few, small regions of AP clutter, and mostly the not-clutter, precipitation areas. The AP/not-AP ratio has a value of 0.5% while the MVE/TSNR ratio has a value of 78.3%. For the three algorithm tests, all produce false detections within the 0 m s -1 MVE regions. The three algorithm tests do not perform very well with this case. The interest fields from the MSW and SDVE fields (not shown) are the strongest contributors to poor algorithm performance when coupled with regions of near-zero MVE values. For this case, the SDVE field (not shown) is characterized by having small values over most of the area shown in Fig. 2.35 as does the MSW field (not shown). Low values of SDVE and MSW are used as indicators for the presence of AP return in the APDA. The GDZ interest field (not shown) varies in intensity depending on the vertical structure of the stratiform echo and is not a consistent contributor to the poor algorithm performance. The POD scores for the three tests are 0.350, 0.206 and 0.248 for the Original Test, Test #3 and Test #4, respectively. However, the high POD for the Original Test is a bit misleading since it also has the highest FAR score. The FAR scores are 0.239, 0.151, and 0.087, respectively, and show that high numbers of false detections are occurring with the Original Test. The CSI scores are 0.004, 0.006 and 0.011, respectively.

This case is very illustrative and shows that the AP Detection Algorithm may not perform as well in stratiform precipitation, at least in this case. More stratiform precipitation cases are clearly needed for additional testing and evaluation.

2.8.3.7 KLSX, July 24, 1993 at 1009 UTC

Clear air return and a large, nearly homogeneous region of AP echo is examined in a case from the St. Louis, MO WSR-88D (KSLX) on July 24, 1993 at 1009 UTC ( Fig. 2.36). For this case, the reflectivity values within the AP echo are about 55 dBZ (Fig. 2.36a) to the SW of the radar location. Clear air return is present, as seen in the reflectivity field and in the mean radial velocity field (Fig. 2.36b). Much of the MVE field is removed to the N of the radar due to the presence of second trip echo. The truth field (Fig. 2.36c) shows that the echo to the SW is AP clutter return. The three algorithms produce similar results with good detections of the AP return. In the region of missing MVE values to the N of the radar, the combination of a scattering of high TSNR values (not shown) coupled with the high GDZ values that are characteristic of clear air return has created small regions of false detections for all three algorithm tests. Overall, the three tests have similar POD performance with scores of 0.725, 0.719 and 0.718 for the Original Test, Test #3 and Test #4, respectively. The FAR scores are 0.133, 0.142 and 0.138, respectively, and the CSI scores are 0.519, 0.511 and 0.515, respectively.

2.8.3.8 KLSX, July 11, 1993 at 0131 UTC

Another clear air and AP clutter returns case ( Fig. 2.37) is taken from the KLSX WSR-88D on July 11, 1993 at 0131 UTC. Comments made regarding the performances of the three algorithm tests for the case discussed in Section 2.8.3.7 are also valid for this case. The MVE/TSNR ratio is 76.4% ( Table 2.11) and shows that a considerable amount of the Doppler fields are missing. Except for the region near the radar (< 50 km range) that is clear air return, most of the radar return has been determined to be AP clutter (Fig. 2.37c). For this case, the Original Test has the best performance with the highest POD (0.744) and the lowest FAR (0.164). Test #3 and Test #4 have similar performance with POD scores of 0.658 and 0.666, respectively, and FAR scores of 0.191 and 0.207, respectively. The large number of missing Doppler fields is a likely contributor to the poor performance of the algorithms with respect to the FAR scores.

2.8.3.9 KLOT, October 18, 1995 at 1005 UTC

A region of AP clutter is embedded within clear air return ( Fig. 2.38) for a case selected from the KLOT WSR-88D radar on October 18, 1995 at 1005 UTC. For this case, Test #3 and Test #4 have the highest POD scores of 0.513 and 0.532, respectively. The Original Test has a POD score of 0.453. However, the Original case has the lowest FAR score of 0.065 with Test #3 having a score of 0.090 and with Test #4 having a score of 0.103. Differences in FAR scores are small, however. The CSI scores are 0.401, 0.433 and 0.439 for the Original Test, Test #3 and Test #4, respectively. For this case, the MVE/TSNR ratio is 95.2% indicating that there are few missing gates of data for the Doppler fields.

2.8.3.10 KNQA, July 6, 1997 at 0454 UTC

A case of clear air return with embedded AP contamination in a weak wind flow pattern is examined using data from the Memphis, TN WSR-88D (KNQA) radar on July 6, 1997 at 0454 ( Fig. 2.39). This case was included in the data collection for the test of the Archive 1 Data Acquisition (A1DA) unit described in the FY-97 annual report (Keeler et al., 1998b). As shown by the expert truth field (Fig. 2.39c), AP echo is prevalent with a few areas of clear air return. The three algorithms have fairly good POD scores of 0.689, 0.704, and 0.721 for the Original Test, Test #3 and Test #4, respectively. The FAR scores are rather high with scores of 0.135, 0.241 and 0.250, respectively. And finally, the CSI scores are 0.651, 0.638, and 0.652, respectively. The higher FAR scores for the Test #3 and #4 are likely due to the GDZ interest field (not shown) that has high interest values in the clear air return. The MVE/TSNR ratio is 97.2% and indicates that missing data from the Doppler fields is not contributing significantly to the high FAR scores.

2.8.3.11 KLWX, August 17, 1997 at 0958 UTC

For this case from the Sterling, VA WSR-88D (KLWX) on August 17, 1997 at 0958 UTC, clear air return is found within about 50 km of the radar ( Fig. 2.40) with AP and suspected NP ground clutter returns embedded within clear air return or occurring within isolated, nearly homogeneous regions. The suspected NP returns are identified as AP in the expert truth field. A weak squall line is approaching from the NW ( Fig. 2.41) that has maximum reflectivity values of about 35 dBZ. This squall line is straddling the first trip range of the velocity scan and has regions of missing data within the Doppler fields at farther ranges. Two portions of this scan of data are examined.

For the first example (Fig. 2.40), AP and NP clutter regions are occurring near and within the clear air return. Significant portions of the Doppler fields are missing in the NW quadrant (at ranges < 25 km) due to the removal of second trip echo originating from the approaching squall line. The MVE/TSNR ratio ( Table 2.11) is 78.4% for the entire scan. The missing Doppler fields will mean that the final outputs of the AP Detection Algorithm tests will use only the TSNR and GDZ fields to determine the likelihood of clutter contamination in this area. As is seen in the three algorithm outputs (Fig. 2.40d-f), this region has a significant number of false detections due to the missing Doppler fields.

For the squall line example (Fig. 2.41), the missing regions of the Doppler fields that are created at the limits of the first trip range again means that the final outputs of the algorithm will depend on only the TSNR and GDZ fields. However, for this example, the missing Doppler fields do not contribute to false detections, indicating that the interest fields derived from the TSNR and GDZ fields are able to discriminate precipitation from AP return, for this case. The three algorithms perform very well in the squall line, with Test #3 having the most false detections.

Overall for this scan, the POD scores are 0.581, 0.539 and 0.586 for the Original Test, Test #3 and Test #4, respectively, and indicates that the three tests had similar performances. Likewise, the FAR scores are 0.181, 0.176 and 0.193, respectively, and the CSI scores are 0.380, 0.373 and 0.393, respectively. Likely, the missing regions of the Doppler fields contributed to the reduced performance of the algorithm.

2.8.3.12 KLOT, October 19, 1995 at 0011 UTC

An example of conditions having mostly clear air return with very small patches of AP clutter is shown in this case from the KLOT WSR-88D on October 19, 1995 at 0011 UTC ( Fig. 2.42). Reflectivity values are generally low with maximum values of about 20 dBZ in the clear air (Fig. 2.42a). Some NP clutter return is present near the radar that has higher reflectivity values. The MVE field (Fig. 2.42b) shows that the winds are from the NE, with a 0 m s -1 line running approximately NNW to SSE through the radar location. In addition, some MVE values are reduced near the shoreline of Lake Michigan (near 60 deg azimuth and 50 km range). The expert truth field (Fig. 2.42c) shows that most of the return is from clear air, with a few, scattered regions of AP clutter present. As discussed above, this case has a low MVE/TSNR ratio of 63.3% ( Table 2.11) indicating that significant regions will use only the interest fields from the TSNR and GDZ fields to determine the presence or absence of AP clutter. The POD scores are quite low at 0.314, 0.299, and 0.337 for the Original Test, Test #3 and Test #4, respectively. However, FAR scores are low with values of 0.030, 0.033 and 0.045, respectively. For this case, the AP Detection Algorithm is having trouble finding the AP return but is not falsely detecting return. The small areas containing the AP return (AP/not-AP ratio is 3.2%) may also be a contributor to the poor algorithm performance. CSI scores are 0.109, 0.117, and 0.108, respectively, and reflect the low POD scores.

2.8.3.13 KFTG, January 11, 1995 at 1548 UTC

The last case is from the Denver, CO WSR-88D (KFTG) and occurred on January 11, 1995 at 1548 UTC. This case is an example of NP clutter ( Fig. 2.43) and is included to illustrate that the AP Detection Algorithm is able to detect NP clutter. Clutter problems in and around the Rocky Mountains can be severe as is easily seen in Fig. 2.43. The expert truth field (Fig. 2.43c) has defined all the NP clutter as AP clutter to allow algorithm testing. The MVE/TSNR ratio is 91.5%, indicating few problems with missing Doppler fields. The three tests (Fig. 2.43d-f) show good skill with this case, with POD scores of 0.866, 0.712 and 0.797, respectively, for the Original Test, Test #3 and Test #4. The FAR scores are the lowest for Test #3 (0.025) and Test #4 (0.051) while the Original Test had a score of 0.111. CSI scores for the three tests are 0.856, 0.710 and 0.790, respectively. The APDA performance is quite good for this case.

2.8.4 Conclusions

The statistical results presented in the sections above show that the AP Detection Algorithm has variable skill in detecting AP clutter and performs best when the AP clutter occurs in isolated, nearly homogeneous patches over a substantial area. Missing data in the Doppler fields (MVE, SDVE, and MSW) contribute to poor algorithm performance. Various schemes for an optimized algorithm have been presented and analyzed. The Test #4 membership functions show the best statistical performance when the 60 scans are considered; however, the statistical improvement over the Original Test is not very much. For many of the summarized cases, the statistical scores were within a few percent over the various tests.

The algorithm performs especially well when the AP is occurring in isolated patches that are not contained within precipitation or clear air return. With mixed AP and precipitation or mixed AP and clear air return, the algorithm performs well in those areas where the radial velocity is significantly biased towards 0 m s -1 and less well when the radial velocities are biased low by clutter return but are not 0 m s -1 . Characterization of the type of AP return (isolated versus embedded) or the areal extent of the AP return may help further explain algorithm performance.

Several cases of convective cells having no AP clutter contamination were shown. The APDA did not falsely identify most of the precipitation echo as AP; however, the edges of the echoes were sometimes identified as AP clutter.

The algorithm performs best when all five input interest fields (MVE, TSNR, SDVE, MSW and GDZ) are present. Scans with a significant amount of missing data from the Doppler channels typically did not perform as well as scans that had all the data fields. The Doppler channels boost algorithm performance significantly.

Additional analysis is needed for stratiform precipitation cases. One case was presented in which the AP Detection Algorithm did not perform particularly well. Additional cases are needed to determine if there is a systematic weakness in the APDA membership functions. This case may be an anomaly but additional analysis is needed to ascertain this possibility.

The APDA can also detect NP ground clutter, as shown with a case from Denver, CO. Therefore, if a region of NP clutter is missed on the bypass map for some reason (perhaps a building is constructed after the clutter filter map was made), the APDA can detect this region of clutter and provide additional quality control for removing clutter contamination.

2.9 Work in Progress and Future Work

The work described in this section is in the category of "fine-tuning" the AP Detection Algorithm and will not delay or impede the implementation of the AP Clutter Mitigation Scheme described in Section 1.

2.9.1 Further Analysis and Optimization of the APDA

Much progress has been completed in the optimization of the AP Detection Algorithm. However, a small amount of work still remains to be accomplished. A detailed analysis of the contribution of each individual interest field (i.e., the TSNR, MVE, SDVE, GDZ, and MSW fields) to the algorithm performance must be done and may lead to small changes in the membership functions shown in Fig. 2.9.

Determination of the optimum weights for the membership functions remains to be accomplished. Research by Jeffrey Keeler during his recent sabbatical should allow for determination of the appropriate weights by an objective method. Currently, weights are determined by a subjective, trial-and-error approach. This method is obviously less desirable than having an objective approach to choose the optimum weights.

Determination of the threshold to be applied to the final interest field is needed.

A few additional stratiform precipitation cases are needed for analysis.

The AP Detection Algorithm uses a mean filter over the specified region to compute the derived mean fields (i.e., the mean radial velocity, mean spectrum width). For radar data, the removal of outliers prior to computation of averages typically leads to better algorithm results. This can be accomplished by using a median filter instead of a mean filter. Substitution of a median filter for the current mean filter will be done and the results evaluated during FY-99.

Historically, the TSNR field has been used in the APDA. For the operational implementation of the APDA within the WSR-88D system, the texture of the reflectivity (TDZ) may be a more appropriate field to use because it is within the A2 data stream. Substituting TDZ for TSNR is not expected to adversely impact the statistical performance of the algorithm. However, a quick test of this hypothesis is needed using the 60 scan data set.

2.9.2 Clutter Filter Maps

After the Build 10 was completed, the clutter filter "bypass" maps used by the WSR-88D systems are being written to Exabyte tape. To examine clutter residue effects and to determine its statistical characteristics, knowing where the clutter map has been applied is important information. A process to read the bypass maps from the Memphis KNQA in-house tapes is needed and will be completed in FY-99. Further information regarding the clutter residue characterization work is contained in Section 4.

2.9.3 Use of Polarimetric Data

A fuzzy logic, hydrometeor and non-hydrometeor particle identification (PID) algorithm has been developed at NCAR using the S-Pol dual-polarimetric base data and derived fields (Vivekanandan et al., 1998). With the PID, ground clutter is easily identified using the standard deviation of differential reflectivity ( ), the standard deviation of differential phase ( ) and the correlation coefficient between the horizontal and vertical copolar received signal (). Figure 2.44 shows an example of data observed with S-Pol near Boulder, CO on a clear day, and the resulting classification of the targets. The ground clutter from the surrounding terrain is successfully identified as clutter. Likewise, the clear air return from insects is correctly identified. The AP Detection Algorithm (APDA) can be run on S-Pol data and the results evaluated using the polarimetric classification as an independent truth field.

Using the PID-identified ground clutter removes some of the ambiguity in the APDA results as introduced by the human expert and by the limitations of the WSR-88D data. Within the PID, ground clutter presence is determined from radar fields that are not used in the APDA and will serve as an independent test. Also, the S-Pol radial velocity and reflectivity data are taken simultaneously, using the same radial beam and gate spacing. This is a significant improvement over the WSR-88D data where temporal and spatial differences exist between the radial velocity and reflectivity data. Another benefit of using S-Pol data is that classification is done on a gate-by-gate basis by an objective method. While the human experts were quite diligent during their truthing efforts, drawing polygons containing AP clutter on a gate-by-gate basis is difficult and is made more difficult by the temporal and spatial differences between the reflectivity and radial velocity fields that is inherent to the WSR-88D data. An objectively-determined truth field is a vast improvement over using human experts.

For FY-99, a few scans of S-Pol data will be input into the APDA and the results evaluated.

2.10 References

Cornelius, R., R. Gagnon, and F. Pratte, 1995: Optimization of WSR-88D clutter processing and AP clutter mitigation, Final Report, submitted to the NOAA WSR-88D Operational Support Facility, 182 pp.

Donaldson, R.J., R.M. Dyer, and M.J. Kraus, 1975: An objective evaluator of techniques for predicting severe weather events. Preprints, 9th Conference on Severe Local Storms, Norman, OK, Amer. Meteor. Soc., 321-326.

Ellis, S.M., F. Pratte, and C. Frush, 1999: Compensating Reflectivity for Clutter Filter Bias in the WSR-88D. Preprints, 15th Int'l Conf. Inter. Info. Proc. Sys. (IIPS) for Meteor., Hydro. Ocean., AMS, Dallas, TX, 11-15 Jan. 1999.

Keeler, R.J., F. Pratte, D. Ecoff, J. VanAndel, and D. Ferraro, 1998a: NEXRAD anomalously propagated ground clutter mitigation. Preprints, 14th Int'l Conf. Inter. Info. Proc. Sys. (IIPS) for Meteor., Hydro. Ocean., AMS, Phoenix, AZ, 11-16 Jan. 1998.

Keeler, R.J., F. Pratte, D. Ecoff, J. VanAndel, D. Ferraro, C. Kessinger, and S. Ellis, 1998b: WSR-88D data quality optimization. FY-97 Annual Report , submitted to the NOAA WSR-88D Operational Support Facility.

Keeler, R.J., C. Kessinger, J. VanAndel, and S. Ellis, 1999: Implementation of NEXRAD AP clutter processing. Preprints, 15th Int'l Conf. Inter. Info. Proc. Sys. (IIPS) for Meteor., Hydro. Ocean., AMS, Dallas, TX, 11-15 Jan. 1999.

O'Bannon, T., 1998: The enhanced WSR-88D precipitation processing subsystems. Preprints, 14th Int'l Conf. Inter. Info. Proc. Sys. (IIPS) for Meteor., Hydro. Ocean., AMS, Phoenix, AZ, 11-16 Jan. 1998.

Oye, R., C. Mueller, and S. Smith, 1995: Software for translation, visualization, editing and interpolation. Preprints, 27th Radar Meteor. Conf., AMS, Vail, CO, 9-13 Oct 1995.

Pratte, F., D. Ecoff, J. VanAndel, and R.J. Keeler, 1997: AP clutter mitigation in the WSR-88D. Preprints, 28th Radar Meteor. Conf., AMS, Austin, TX, 7-12 Sep. 1997.

Saffle, R.E., and L.D. Johnson, 1998: NEXRAD product improvement: An overview of the continuing program to evolve the WSR-88D system to an Open Systems Architecture, Preprints, 14th Int'l Conf. Inter. Info. Proc. Sys. (IIPS) for Meteor., Hydro. Ocean., AMS, Phoenix, AZ, 11-16 Jan. 1998.

Smith, J.A., M.L. Baeck, M. Steiner, B. Bauer-Messmer, W. Zhao, and A. Tapia, 1996: Hydrometeorological assessments of the NEXRAD rainfall algorithms. Final Report to the NOAA National Weather Service, Office of Hydrology-Hydrologic Research Laboratory, Silver Spring, Maryland

Steiner, M. and J. Smith, 1997: Anomalous propagation of radar signals - challenges with clutter, 28th AMS Radar Meteoro. Conf., Austin, TX, 501.

Vivekanandan, J., D.S. Zrnic, S. M. Ellis, R. Oye, A.V, Ryzhkov, and J. Straka, 1999: Cloud microphysics retrieval using S-band dual-polarization radar measurements. 15th Int'l Conf. Inter. Info. Proc. Sys. (IIPS) for Meteor., Hydro. Ocean., AMS, Dallas, TX, 11-15 Jan. 1999.


3.0 Princeton University Evaluation of the AP Detection Algorithm

3.1 Introduction

Princeton University has a computing facility that is used to test software for improving data quality of the NEXRAD WSR-88Ds. Mattias Steiner leads this effort. Steiner and Smith (1997) reported on their evaluation of several AP detection algorithms as gleaned from various sources. Briefly, their findings were that 1) the regions surrounding the near 0 m s -1 radial velocity is a good indicator of AP but can include precipitation or clear air echo and must be combined with other fields to be a useful measure of AP, 2) spectrum width is not a particularly useful field for AP detection, 3) the echo top height is not a particularly useful parameter since storm heights must be accurately measured, mixed AP and weather signals are not handled well, and boundary layer features tend to be removed with this criteria, 4) the vertical gradient of reflectivity is a good indicator of AP contamination, and 5) the "texture" of the reflectivity field is a powerful indicator of the presence of AP.

Because of their work, the functional description of the NCAR AP Detection Algorithm (see Appendix A) was sent to Princeton to allow them to code the algorithm within their computing environment, to run their selected AP case studies through the algorithm, and to compare and evaluate the NCAR results to the work of others. The membership functions from Test #2 ( Fig. 2.8) were used in this evaluation.

After coding the algorithm, Steiner evaluated the NCAR AP Detection Algorithm using Archive 2 (A2) base data from three WSR-88D radar volumes: one volume from St. Louis, MO (KLSX) on July 7, 1993 at 0334 UTC and two volumes from Amarillo, TX (KAMA) on May 25, 1994 at 0034 and 0231 UTC. Once the scans were run through the Princeton implementation of the NCAR algorithm, "screen-captures" that illustrated the algorithm results and Universal Format files of the original data were given to NCAR. Steiner wrote a preliminary evaluation of his findings and sent copies to the OSF and to NCAR.

Using the Universal Format data that Princeton provided, the base data are input into the NCAR AP Detection Algorithm and the results are evaluated and compared to the Princeton results. Screen captures of the algorithm output were given to Steiner.

3.2 Analysis Results

The algorithm results from using the KLSX A2 base data as input are used to illustrate the results of the Steiner evaluation shown in Figure 3.1. As can be seen in the figure, this case is a widespread precipitation case with embedded AP contamination (see Fig. 3.2c for the expert determination of the location of AP echo). These types of AP cases are difficult for the algorithms tested in Steiner and Smith (1997) to detect. Using the A2 input data, the various derived fields used as input into the AP Detection Algorithm (see Sections 2.4 and Appendix A for a full discussion of the derived fields) are calculated and are shown in Fig. 3.3. The derived fields include: the "texture" of the signal-to-noise field (TSNR), the mean radial velocity (MVE), the standard deviation of the radial velocity (SDVE), the mean spectrum width (MSW), and the vertical difference of the 1.5 and 0.5 deg elevation angles (GDZ).

The membership functions designated as Test #2 ( Fig. 2.8) are used in the evaluation because they were optimal at the time of the test.

The Princeton evaluation is considered preliminary since their comparison to the NCAR results and correction of possible errors has not occurred as of this writing. Comparison of the interest fields determined from the Princeton AP Detection Algorithm implementation ( Fig. 3.4) and the NCAR implementation ( Fig. 3.5) suggests that the Princeton implementation may have some errors. The interest fields that appear to have the best agreement are those derived from the MVE (Figs. 3.4b and 3.5b), SDVE (Figs. 3.4c and 3.5c), and the MSW (Figs. 3.4d and 3.5d) input fields. Those interest fields that have the largest discrepancies are those derived from the TSNR (Fig. 3.4a and 3.5a) and the GDZ (Figs. 3.4e and 3.5e) fields. The differences are great enough that an error in the Princeton implementation is likely.

For the interest field from TSNR, the Princeton implementation shows much less areal extent for interest values > 0(?); however, the general locations of regions having higher interest are similar. In his preliminary report, Steiner noted that the interest field derived from TSNR showed a lack of skill; however, this may be explained by an error in their implementation of the algorithm. Steiner also noted that a "ring" of high texture of reflectivity frequently appears around echoes. When the Universal Format data files were displayed at NCAR, erroneous values of reflectivity were apparent at the edges of echoes and appeared to be averages of the missing data value and the reflectivity values of the weather and non-weather echoes. At NCAR, these regions of reflectivity were deleted prior to calculation of the SNR and the texture of the SNR.

For the interest field derived from GDZ, discrepancies exist not only in the interest values themselves but in the locations of regions having higher interest. At close range to the radar, the Princeton implementation shows high interest values ( >  0.5) are present while the NCAR implementation, for the same region, shows interest values of 0. Within the region of AP to the NE of the radar, the Princeton implementation shows interest values to be < 0.5 while the NCAR implementation shows interest values near 1. The areal extent of interest values > 0.5 is much greater in the NCAR implementation of the algorithm in this region of AP echo.

For the final interest output, representing the weighted summation of all the individual interest fields, the Princeton implementation shows maximum interest values of 0.5 to about 0.7 within the AP region to the NE of the radar. The NCAR implementation has maximum interest values from 0.5 to 1.0 in the same region. The areal extent is similar for both implementations in this region.

In his preliminary report, Steiner was concerned that the NCAR AP Detection Algorithm may be too slow for real-time operations. This concern will be addressed at a later time, during the planning for the implementation of the AP Clutter Mitigation Scheme (see Section 1).

3.3 Conclusions

Despite these apparent errors in the algorithm implementation, Steiner's preliminary report showed that the NCAR AP Detection Algorithm recognized AP clutter in his test cases much better than any other tested recognition algorithm. In a later conversation (via email), Steiner noted that several of the Princeton implementation errors were fixed and that the results from the NCAR and the Princeton algorithms were in much closer agreement. This positive news adds a degree of urgency to begin implementation in the Open RPG as soon as practical. NCAR continues to assist Steiner, as needed.

3.4 References

Steiner, M. and J. Smith, 1997: Anomalous propagation of radar signals - challenges with clutter, 28th AMS Radar Meteoro. Conf., Austin, TX, 501.

 


4.0 Compensating Reflectivity for Clutter Filter Bias

4.1 Introduction

The presence of anomalous propagation (AP) and normal propagation (NP) ground clutter presents serious problems for radar precipitation estimation. The high reflectivity values of the ground echoes are misinterpreted as rainfall by reflectivity-based estimation algorithms, leading to substantial over-estimates of rainfall amounts.

Ground clutter is characterized by narrow spectrum width and near-zero velocity. Clutter filters are able to cancel AP clutter in most situations by effectively removing the power near zero velocity. When the clutter filters are used on weather echoes with low velocity, a negative reflectivity bias is introduced (Sirmans, 1992; Cornelius et al., 1995). This bias reduces precipitation estimates and adversely effects hydrology estimates and predictions. Correcting for this reflectivity bias is necessary to obtain accurate, unbiased rainfall amounts.

In practice, when the WSR-88D clutter filters are turned on, the only fields available for a reflectivity compensation algorithm are the moment estimates from the filtered time-series data. A compensation algorithm called the "simple Gaussian correction model" (SGCM) has been developed by Frank Pratte at NCAR (Cornelius et al., 1995) as part of the AP Clutter Mitigation Scheme, as described in Section 1 (Keeler et al., 1999). This scheme includes identifying ground clutter with the AP Detection Algorithm and selectively applying ground clutter filters to the effected regions. Reflectivity compensation is an essential part of a comprehensive AP Clutter Mitigation Scheme because cases of mixed clutter and precipitation echoes occur frequently.

Two variations of the SGCM have been tested, both of which use the estimated Doppler spectrum width and radial velocity information to estimate a correction for the biased reflectivity. The methods are tested on WSR-88D in-phase and quadrature (i.e., I and Q) data (Archive 1 time series data) that are processed using autocovariance calculations and clutter filters that emulate the WSR-88D processor, providing both filtered and unfiltered moment data for verification of the SGCM. The filtered and unfiltered data streams are not available simultaneously in the current WSR-88D system. The SGCM techniques are described and some preliminary results presented.

4.2 Two Variations of the Simple Gaussian Correction Model

Two related techniques are tested. Both assume a Gaussian-shaped velocity spectrum to estimate the reflectivity compensation factor. The first technique, developed by Frank Pratte (Cornelius et al., 1995), uses a family of Gaussian spectra, generated with a variety of radial velocity and spectrum width values. The model spectra are filtered for clutter and the reflectivity, radial velocity, and spectrum width are estimated using moment calculations. The reflectivity compensation factor is computed using the estimated filtered and unfiltered reflectivity values. Using these results, a "look-up" table of reflectivity compensation factors is calculated for each filtered radial velocity and spectrum width pair. The second technique uses the filtered spectrum width and radial velocity measurements directly to approximate the velocity spectrum as Gaussian. Next, the Gaussian spectrum is filtered using a clutter filter and the unfiltered and filtered reflectivity are estimated using moment calculations. These moments are then used to estimate the reflectivity compensation factor.

4.2.1 The "Table Look-Up" SGCM

In the "table look-up" SGCM technique, a family of unfiltered Gaussian spectra, , are generated, such that

.  (Eq. 4.1)

In Eq. 4.1, the and define an array of unfiltered input radial velocity and spectrum width values. Next, the corresponding filtered reflectivity ( ), filtered velocity ( ) and filtered spectrum width ( ) are calculated using classical moment calculations, as follows.

 (Eq. 4.2)

,  (Eq. 4.3)

. (Eq. 4.4)

where is a piecewise continuous function designed to emulate the WSR-88D clutter filter response. Figure 4.1 shows a plot representing . Similarly, the unfiltered reflectivity () is estimated as

. (Eq. 4.5)

Finally, the estimated correction factor is computed as the ratio

. (Eq. 4.6)

The reflectivity correction estimation ( ) is computed for each of the filtered radial velocity and spectrum width pairs in the family of spectra described above for each of the WSR-88D clutter filter configurations (high, medium and low suppression). These values are listed in three look-up tables, so that the proper table can be accessed for the filter characteristics being used. The filtered velocity and spectrum width measured by the radar can then be matched to the appropriate filtered velocity and spectrum width pair from the approximations in Eqs. 4.3 and 4.4 to obtain the corresponding reflectivity correction factor, L. The corrected power is computed by multiplying the measured (filtered) power by L.

In some cases, multiple radial velocity/spectrum width pairs exist within the table (with different compensation factors, ) that are close to the measured (filtered) radial velocity and spectrum width. Choosing the pair from the table that best matches the observations is very important to obtain the best results. To find the proper value, the minimum is found of the sum of the squares of the difference between the measured and the look-up table filtered radial velocity and spectrum width values (i.e. find min[(Vmeas - Vfilt)2 + (Wmeas - Wfilt)2], where V is radial velocity, W is spectrum width and the subscripts meas, and filt refer to measured (filtered) quantities and filtered values from the look-up tables, respectively).

4.2.2 The "Direct" SGCM

The "direct" SGCM technique calculates the correction factor directly without the use of a look-up table or family of Gaussian curves. To do this, the unfiltered and unbiased Doppler spectrum is estimated at a particular range gate as a Gaussian distribution using the measured (filtered) velocity and spectrum width. In practice, when the clutter filters are enabled, the only measurements available are the filtered quantities, therefore,

, (Eq. 4.7)

where is radial velocity, is the measured (filtered) mean radial velocity and is the measured (filtered) spectrum width. Using the observed filtered velocity and spectrum width in a Gaussian curve clearly introduces error since the filtered spectrum is not Gaussian. Hopefully, the biases in radial velocity and spectrum width are not too severe to preclude reasonable estimates of the unfiltered spectrum (which is not available operationally with the clutter filters enabled). The power (0 th moment) of the model Gaussian spectra Su for the range gate in question is now approximated using classical moment calculations, i.e.,

.  (Eq. 4.8)

Next, the filtered power is approximated as,

 (Eq. 4.9)

Finally, similar to the look-up technique, the estimated correction factor is computed as the ratio

, (Eq. 4.10)

and multiplied by the measured reflectivity to obtain the compensated reflectivity.

4.3 Results of the SGCM Tests Using Memphis A1 Data

The SGCM reflectivity compensation was tested on both simulated (table look-up only) and real WSR-88D time-series data. The data were processed in a manner that emulates the WSR-88D processing and obtains filtered and unfiltered moment estimates. The resulting filtered radial velocity and spectrum width are then used in the SGCM techniques. The simulation results are described in Cornelius et al., (1995).

For the tests with real WSR-88D data, Archive I (A1) time series data are used that were recorded at the Memphis WSR-88D radar (KNQA) using the NCAR A1 Data Acquisition (A1DA) unit. Autocovariance calculations are performed on the in-phase and quadrature data from the radar receiver to obtain estimates of the first three moments for both the filtered and unfiltered data. Currently, the simulated clutter filter, h(v), ( Fig. 4.1) is used to filter the real data, however, an upgrade is planned to use the 5-pole elliptic WSR-88D filters. Next, the SGCM techniques are used to obtain corrected powers. In this way, the original power, the filtered power and the compensated power can be evaluated. In the absence of clutter, the severity of the bias introduced by the clutter filter can be quantitatively determined and the effectiveness of the SGCM's to remove this bias can be evaluated. The performance of the system can also be evaluated for ground clutter targets.

The results of the real data tests are very preliminary. Figure 4.2 shows the estimated power and velocity for precipitation echoes. These echoes were determined to be mostly precipitation by examining both the Archive 2 data and the raw spectrum data. Both cells in Fig. 4.2 contain near-zero radial velocities. The average spectrum width is about 2 m s -1 . Experiments are run using filter parameters emulating the WSR-88D low (passband edge velocity, Vp=1.1825, stopband edge velocity, Vs=0.7095 m s  -1 ), medium (Vp=1.5625, Vs=0.9375 m s  -1 ), and high (Vp=2.3125, Vs=1.3875 m s  -1 ), suppression clutter filter characteristics (Sirmans 1992).

Figure 4.3 shows the power estimates after (a) filtering alone, (b) after filtering and compensation using the table look-up SGCM and (c) after filtering and compensation using the direct SGCM. The low suppression filter is used with a notch depth of 40 dB. Both the filtering and compensation are applied over the entire domain shown in Figs. 4.2 and 4.3. Figures 4.4 and 4.5 are similar to Fig. 4.3 for the medium and high suppression filters respectively.

Comparing Figs. 4.2a (original power) to Figs. 4.3a, 4.4a, and 4.5a (low, medium and high suppression filtered power) shows significant clutter filter bias for all three clutter suppression levels. The table look-up method (Figs. 4.3b, 4.4b, and 4.5b) reduces the bias more than the direct method (Figs. 4.3c, 4.4c, and 4.5c), but the variance of the compensated reflectivity is higher for the table look-up method. As expected and for both techniques, the correction is much more effective when the low and medium suppression filters are used and is less effective when the high suppression filters are used. Also, the variance of the table look-up corrected power is lower for the low and medium suppression filters. These trends are in agreement with the simulation results obtained by Pratte in Cornelius et al., (1995).

Table 4.1 gives the mean of the ratio of original (linear) power (Po) to filtered power (Pf) and the ratio of original power to both the table look up (Pt) and direct (Pd) compensated power in dB for different velocity ranges using the low suppression filter. Tables 4.2 and 4.3 are similar to Table 4.1 for the medium and high suppression filters, respectively. The averages are calculated using

,  (Eq. 4.11)

where N is the number of gates used and x = f, t, or d (filtered, table look-up compensation, or direct compensation). The first three columns of the tables show data for the stopband, the transition band and the passband for each filter.

Examination of Tables 4.1 - 4.3 shows that the direct SGCM method is not able to recover fully the power for radial velocities within the stopband. The table look-up method does better, however the power is still biased low. The bias is relatively small for the low and medium suppression filter experiments. The high suppression filter results are significantly biased after compensation. Apparently, not enough information remains (at low radial velocity) after high suppression filtering to fully reconstruct the power field. Recall that the precipitation echoes have relatively low spectrum width values (~1.5 - 2.0 m s -1 ), such that these experiments represent a worst case scenario. Many more experiments must be performed to fully understand the performance of the SGCM schemes.

To examine errors introduced by using the biased (filter induced) velocity and spectrum width in the direct SGCM method, the filtered velocity and spectrum width parameters are replaced with the unfiltered values in Eq. 4.7. This is possible because the A1 data set allows both the filtered and unfiltered data sets. Recall that the precipitation echoes being examined are clutter-free. Comparing the SGCM results from using the unfiltered data to the results using filtered data in Eq. 4.7 should indicate the errors due to the biases introduced by the clutter filter. When the unfiltered data are used, the mean ratios of measured to compensated power are reduced to 1 - 2 dB for velocities less than 2 m s -1 for all three suppression levels. This suggests that most of the error in the direct SGCM is due to errors in the radial velocity and spectrum width estimation caused by the application of the clutter filter. The direct SGCM may be sensitive to errors in spectrum width because it appears as a squared quantity in the exponent of the Gaussian spectrum approximation (Eq. 4.7). The validity of the Gaussian approximation and random noise in the data may also contribute to errors.

4.4 Discussion

The table look-up Simple Gaussian Correction Model shows promise in compensating clutter filter reflectivity bias. The uncertainty of the power estimates in clutter mixed with precipitation that has a near-zero radial velocity is still high after compensation (the variance is increased). However, removing the bias should yield significant improvements in the radar-derived precipitation amount estimates. Many more experiments covering a wide variety of weather conditions are needed to fully understand the performance and properties of the SGCM. These tests will be performed.

The WSR-88D 5-pole elliptic filter will be implemented in the next year to more accurately simulate operational conditions. Comparing experimental results using the filter approximation described above to experimental results using the actual elliptic filter response on the input data should prove interesting.

The SGCM reflectivity compensation will be used in conjunction with the AP Detection Algorithm (described in Section 2) and the Precipitation Detection Algorithm (described in Section 5) such that the reflectivity will be compensated only in areas identified as precipitation. The radial velocity and spectrum width information will be used (along with the clutter filter characteristics) to determine if the reflectivity has been biased by the clutter filter and to compensate only the biased areas.

4.5 Clutter Residue Tracking

With the current WSR-88D system, knowing when to disable the clutter filters in situations of AP clutter is a difficult problem because the filters remove the AP clutter, as they are designed to do. Disabling the clutter filters is one simple method to determine if the AP clutter remains, but can lead to undesirable contamination of the radar data. In the AP Clutter Mitigation Scheme, the clutter filters are enabled in a particular region when AP clutter is identified. To know when to disable the clutter filters, the clutter residue will be identified and tracked. However, clutter residue is generally weak and difficult to identify. Furthermore, the clutter filters often completely suppress the AP echoes, leaving no clutter residue. If the clutter residue is not present or cannot be identified, determination of when to disable the clutter filters is impossible using this technique. Alternate techniques will be investigated. With two base data streams (i.e., filtered and unfiltered) in the future Open Systems Architecture, the solution to this problem is much simpler.

4.6 References

Cornelius, R., R. Gagnon, and F. Pratte, 1995: Optimization of WSR-88D clutter processing and AP clutter mitigation, Final Report to the Operational Support Facility, 182 pp.

Keeler, R. J., C. Kessinger, J. VanAndel, and S. Ellis, 1999: Implementation of NEXRAD AP clutter processing. Preprints, 15th Int'l Conf. Inter. Info. Proc. Sys. (IIPS) for Meteor., Hydro. Ocean., AMS, Dallas, TX, 11-15 Jan. 1999.

Sirmans, D., 1992: Clutter filtering in the WSR-88D, NWS/OSF, Norman, OK.

Zrnic, D.S. 1975: Simulation of weatherlike Doppler spectra and signals, J. Appl. Meteor., 14 , 619-620.


5.0 Convective Precipitation Detection Algorithm

5.1 Introduction

The Radar Echo Classifier, discussed in Section 1, will require algorithms to detect those features that are "not clutter" in addition to detecting features that are "clutter." Classification schemes will be necessary to determine where the ground clutter filters should be employed and to determine where the application of the clutter filter has biased the reflectivity values such that compensation must be performed. The Radar Echo Classifier includes fuzzy-logic-based methodology for detecting precipitation echo, clear air echo, birds, chaff, volcanic ash, and other features as needed for radar sites. These detection algorithms will be developed as needed. The highest priority of these additional detection algorithms is to develop an algorithm for detecting precipitation. The first attempts at developing this algorithm for summertime convective precipitation are shown in this section.

5.2 Preliminary Algorithm Description

Using the APCAT fuzzy logic techniques in a similar method as those used to detect AP contamination, a preliminary, convective precipitation detection algorithm has been developed and tested on three scans from the 60-scan dataset. Four of the five derived quantities used in the AP Detection Algorithm are used for this preliminary Convective Precipitation Detection Algorithm, and includes: the mean radial velocity (MVE), the mean spectrum width (MSW), the texture of the SNR (TSNR), the vertical difference in reflectivity (GDZ) and the mean reflectivity value (MDZ). The mean reflectivity value is selected to ensure that clear air echoes are not included in the determination of the presence of precipitation. The initial membership functions for each of these quantities are shown in Fig. 5.1 and are nearly the inverse of the membership functions used to determine AP echo. A second set of membership functions, that gave some improved statistical scores, is shown in Fig. 5.2.

The ground truth field that was used to distinguish ground clutter echo originating from AP from those echoes originating from "non-clutter" echoes (i.e., precipitation echoes, clear air return, and others) is used to statistically score the precipitation detection algorithm results in APCAT for the three scans. Clearly, these truth data do not provide an adequate measure of the algorithm's ability to discern the presence of precipitation. To show this, the original truth data fields have been redone. A comparison of the "old" truth field ( Fig. 5.3a) and the "new" truth field (Fig. 5.3b) is shown for the radar data shown in Fig. 5.4. In this example, the new truth field defines regions that indicate the presence of AP clutter (green areas), clutter residue from NP clutter (white areas), convective precipitation (yellow areas) clear air return (brown areas), and second trip echoes (red areas). Clearly, there is more information in this new truth field that can be used to develop additional detection algorithms, as needed.

5.3 Preliminary Algorithm Results

This preliminary precipitation detection algorithm is designed for convective precipitation, not stratiform precipitation, because that seemed to be the easiest precipitation algorithm for beginning efforts. Distinguishing stratiform precipitation from clear air return may be difficult in some locations. Likewise, distinguishing winter storms from clear air return or from stratiform precipitation may be difficult.

Using the membership functions shown in Figs. 5.1 and 5.2, the base data for the three scans discussed in Sections 2.8.2 (KAMA), 2.8.3.4 (KDDC) and 2.8.3.11 (KLWX) are put through the Convective Precipitation Detection Algorithm using the APCAT software. Each of the three scans contained precipitation echoes as well as AP clutter and clear air return. The results from the Convective Precipitation Detection Algorithm using the initial set of membership functions are shown in Fig. 5.5. Algorithm results using the improved set of membership functions are shown in Fig. 5.6. Comparison of the new truth field ( Fig. 5.3b) to the final output of the first set of membership functions (Fig. 5.5f) to the second set of membership functions (Fig. 5.6f) shows that some improvement has occurred in the detection of precipitation regions for the KAMA scan. The region of clear air return near 270 deg azimuth and 50 km range has decreased in final interest values with the second set of membership functions. In addition, the region of precipitation is more clearly defined with the second set of membership functions. For both sets of functions, the AP region to the north of the radar has final interest values that are <  0.4. Modifying the membership functions for the MDZ, MVE and GDZ fields appears to have improved the algorithm performance most. For this example, there is good distinction between AP clutter and precipitation echoes.

The statistical results from APCAT are shown for the two sets of membership functions while using the original method for defining the truth field ( Fig. 5.7) and while using the new method for defining the truth field ( Fig. 5.8). A threshold of 0.35 is used for the first set of membership functions and a threshold of 0.30 is used for the second set of membership functions since these thresholds produce the best statistical performance in CSI for the algorithms. For the original truth field, the first set of membership functions produces a POD value of 0.924 versus a POD of 0.843 for the second set of membership functions. The first set has an FAR score of 0.482 versus an FAR score of 0.363 for the second set of membership functions. The first set of membership functions will have more true and more false detections of precipitation than the second set when the original truth field is used for scoring.

Examining the detection algorithm results while using the new, improved truth field ( Fig. 5.8) shows that statistical scores are lower compared to using the original truth field. Despite the lower statistical scores, the resulting detections define the precipitation echoes better for the KAMA scan. A threshold of 0.70 is used for both sets of membership functions since the highest CSI values occur with this threshold. Comparing the POD scores of the two sets of membership functions shows that the first set has a higher POD than the second set (0.653 versus 0.529) but the first set has a higher FAR score than the second set (0.194 versus 0.061). The second set of membership function performs better with a CSI score 0.362 versus a CSI score of 0.264 for the second set with this threshold.

Comparing the right panels (the "Classifier Perf. vs Threshold" plots) of Fig. 5.7 to the right panels of Fig. 5.8 shows that using the new truth field results in higher POD and higher FAR scores for both sets of membership functions. The increase in the FAR scores lowers the resultant CSI scores. The higher POD scores are encouraging and show that limiting the truth field to be only precipitation improves the algorithm performance. Clearly, the high FAR scores may be problematic and indicate the need for optimization of the membership functions. Comparing the left panels (the "POD vs False Alarm Rate" plots) of Fig. 5.7 to the left panels of Fig. 5.8 shows that the second set of membership functions (the red curves) has improved its performance with the new truth field since the curve is closer to the upper left corner of the plot. For this type of plot, algorithm performance is better the closer the curve is to the upper left corner. However, looking at the same plots for the first set of membership functions (the black curves) shows that a performance decrease occurs when the new truth field is used.

These preliminary results are generated for only three radar volumes. It is not clear if these statistical results will be maintained as more volumes are included. The membership functions need further refinement. Further, different variables than the ones used need to be examined for possible use. However, the success shown in identifying precipitation by this preliminary research is very encouraging and should lead to the successful generation of a convective precipitation product.

5.4 Future Work

Much work remains to be done before this algorithm can be considered mature. The 60-scan dataset used for the AP Detection Algorithm optimization can be used to develop the precipitation algorithm, but the ground truth must be redone to distinguish the clear air return from precipitation return. In its current configuration, the APCAT has been used with two detection categories defined: one designation for AP clutter and one for non-AP clutter. Expanding the detection categories to include clear air return and birds, among others, should be in place within the software but has not been tested. Testing may indicate the need for additional software modifications for APCAT.

Once these tests have been made, the precipitation detection algorithm must be optimized such that the appropriate variables are included in the algorithm, membership functions must be optimized and appropriate weights assigned to each individual interest field. Developing an objective technique for defining the appropriate weights and demonstrating the expected high performance of that algorithm may provide justification to reconsider the implementation of the Precipitation Pre-processing Subsystem (PPS) (O'Bannon, 1998) in the future.

It may be possible to use output from the polarimetric Precipitation Identification (PID) algorithm (Vivekanandan et al., 1999), described in Section 2.9.3, as ground truth for input into the WSR-88D precipitation detection algorithm in a similar way as is planned to be done for the AP Detection Algorithm. The advantage of using the PID output as ground truth is that it is based on polarimetric data which is independent of the derived fields used in the WSR-88D precipitation algorithm. This would allow an independent measure of the precipitation algorithm performance and allow objective techniques of optimization to be applied to the Precipitation Detection Algorithm.

5.5 References

O'Bannon, T., 1998: The enhanced WSR-88D precipitation processing subsystems. Preprints, 14th Int'l Conf. Inter. Info. Proc. Sys. (IIPS) for Meteor., Hydro. Ocean., AMS, Phoenix, AZ, 11-16 Jan. 1998.

 

Vivekanandan, J., D.S. Zrnic, S. M. Ellis, R. Oye, A.V, Ryzhkov, and J. Straka, 1999: Cloud microphysics retrieval using S-band dual-polarization radar measurements. 15th Int'l Conf. Inter. Info. Proc. Sys. (IIPS) for Meteor., Hydro. Ocean., AMS, Dallas, TX, 11-15 Jan. 1999.


6.0 Memphis Archive 1 Data

6.1 Introduction

With the advent of the Open Systems Radar Data Acquisition (ORDA) system, it will eventually be possible to have a data quality algorithm that uses the spectral (i.e., Archive 1, I/Q, time series) data as input. Current plans for the Open Systems Architecture are to implement spectral domain processing after 2005. Contamination from AP and NP ground clutter, moving targets, birds, sea clutter or other contaminants could be removed before calculation of the moments values. Spectral-level processing should improve the quality of the resulting moments fields when compared to current processing techniques. Cornman et al., (1998) have developed a spectral-level, data quality algorithm that is used for Profiler data and is based on fuzzy logic techniques. An algorithm such as this may eventually be possible to use with the WSR-88D.

To anticipate these future developments of the WSR-88D system, time-series data were collected from the Memphis WSR-88D in July 1997 with the Archive 1 Data Acquisition (A1DA) unit. These time-series data will provide an important means for examining various methods of improving data quality in the spectral domain, in addition to their role in evaluating the Range-Velocity Mitigation work of Frush et al., (1998). Real-time implementation of spectral-level processing algorithms that detect and suppress the AP and NP ground clutter will be realized in the future. In this section, a first attempt is made at using A1 data to do this type of processing.

6.2 Data Analysis and Methodology

Using the A1DA unit, time-series radar data were collected that contained AP clutter events from the Memphis WSR-88D (KNQA) during two weeks in July 1997. This work was described in the FY-97 Annual Report. Several episodes of AP clutter were recorded both with and without accompanying precipitation echoes. To accommodate this data collection, the KNQA clutter filters were turned off at all range gates.

The A1 data are examined in the spectral domain where spectral power density is plotted as a function of Doppler frequency (i.e., radial velocity). This is done with the A1 Tool, a software package developed by NCAR that allows the data to be displayed as in-phase and quadrature (I and Q) traces in the frequency domain or in the time domain. Range gates containing contamination from AP and NP clutter or weather echoes are visually identified by a human "expert," namely Frush, in the frequency domain. The ground clutter is identified by its characteristic low spectrum width and low radial velocity at 0 m s -1 . This task is performed over a small region containing 12 beams and 160 range gates at 250 m gate spacing (40 km total range). Frush looked for the presence of ground clutter contamination and estimated the clutter-to-signal ratio (CSR) for each range gate. Artifacts in the spectra that are characteristic of moving point targets (i.e., birds, insects, airplanes) as well as the ground clutter contribution are visually removed and the residual, mean radial velocity estimated for the remaining "weather" echo that corresponds to the elevated power in the spectra. Finally, he gave an estimate of the level of confidence in the results. The estimate of radial velocity as determined through the expert-examination of the A1 data spectra is compared to the A2 radial velocity estimated by the radar using the current processing system.

For the A2 data, the truth field is determined (by Kessinger) using the methodology discussed in Section 2.6.

6.3 Discussion of Results

Figure 6.1 shows the expert-determined, A2 and A1 truth fields that show the locations of range gates having either AP or NP ground clutter contamination in them. The two truth fields are similar for the most part. However, the A1 truth field suggests less clutter is present in the upper left and left side of the wedge of radar data. Examination of additional beams (not shown) in the A2 data set shows that a transition from positive to negative radial velocity values is occurring as the radar sweeps toward 270 deg azimuth. Therefore, it is likely that some of the A2-determined ground clutter gates are actually unbiased returns that are near 0 m s -1 and are incorrectly identified as clutter. More ground clutter is indicated in the A1 truth field in the lower right part of the data than in the A2 truth field, for reasons discussed in the next paragraph.

Figure 6.2 shows the A2 radial velocity field compared to the A1, expert-derived radial velocity. As Fig. 6.2a shows, this region is characterized by considerable ground clutter contamination since most of the A2 radial velocities are near 0 m s -1 . After removal of the ground clutter and other contaminants such as moving point targets, the A1 radial velocity field reveals the underlying return from atmospheric targets. In addition, the A2 radial velocity gates that appear to have unbiased values, such as in the lower right corner, are actually biased low by a few meters per second when compared to the A1 radial velocity values. Undoubtedly, ground clutter is biasing the A2 radial velocity values in more gates than a first inspection might suggest. Figure 6.3 shows the radial velocity difference field for the A1 radial velocity subtracted from the A2 radial velocity. Note that the values are typically negative, showing that the A2 radial velocity values are less (i.e., biased by clutter) than the A1 radial velocity values.

6.4 Comparison of the AP Detection Algorithm Results

The A2 base data and its features as defined in Section 2 are input into the AP Detection Algorithm (APDA) and the performance assessed using the two truth fields shown in Fig. 6.1. Recall that the two truth fields are derived from the A2 base data (Fig. 6.1a) and from the A1 time series data (Fig. 6.1b). One advantage of using the A1 data to determine the truth (Fig. 6.1b) is that the truthing is done independently from the A2 base data fields that are used in the APDA. The final, thresholded interest fields are shown for the Original Test ( Fig. 6.4a) and for Test #4 (Fig. 6.4b) as described in Section 2.8.1 Thresholds of 0.4 and 0.55 are applied, respectively.

Because of the small number of radar range gates in this sample of data (i.e., 1,920), the statistical scores provided by APCAT are of interest for comparing the different truthing methods; however, the statistical scores are not indicative of the algorithm performance in an operational setting and should be considered in a relative sense rather than in an absolute sense. Using the A2 truth field, the POD scores are 0.939 and 0.944 for the Original Test and Test #4, respectively. Using the A1 truth field, the POD scores are 0.923 and 0.933, respectively. The POD scores using the A1 truth field are slightly lower than the scores from the A2 truth field because the A1 truth field has slightly fewer radar gates of data designated as clutter than the A2 truth field. Using the A2 truth field, FAR scores are 0.525 and 0.562 for the Original Test and Test #4, respectively, while the FAR scores using the A1 truth field are 0.725 and 0.733, respectively. Because the A1 truth field contains more radar gates declared as "not-clutter" than the A2 truth field, the increases in the FAR scores are explained. For the same reason, the CSI scores using the A2 truth field are 0.874 and 0.875 for the Original Test and for Test #4, respectively, while the CSI scores using the A1 truth field are 0.797 and 0.804, respectively.

This example is useful because it shows the differences between using the A2 base data fields to determine the truth versus using the A1 spectral data (integrated by an expert) to determine the truth. The A1 truth field should be closer to defining the actual regions of clutter rather than using the A2 truth field because the A2 truth field is based on moments data rather than spectral data.

6.5 Summary and Future Work

Initial investigations into spectral domain processing for WSR-88D data have been performed. These preliminary results show that spectral data processing should improve the quality of the moment data within the Open Systems Architecture. Investigating automated algorithms that do spectral domain processing is a next step. Cornman et al. (1998) have developed a fuzzy logic, spectral domain processor for Profiler radar systems that may be a good candidate for a first attempt.

For FY-99, a preliminary investigation into using the Cornman algorithm for the WSR-88D may be attempted.

6.6 References

Cornman, L.B., R.K. Goodrich, C.S. Morse and W.L. Ecklund, 1998: A fuzzy-logic method for improved moment estimation from Doppler spectra. Journal of Atmospheric Technology, 15 , 1287-1305.


7.0 Data Quality Instrumentation

7.1 The Archive 1 Data Acquisition (A1DA) Unit

7.1.1 Introduction

The Archive 1 Data Acquisition unit (A1DA) is a tool developed for the OSF by NCAR/ATD/RSF to record, display, and analyze Archive 1 (time-series) data, as well as providing a real-time maintenance display of Archive 2 (wideband base data). The A1DA is attached to the signal processor of the WSR-88D Radar Data Acquisition (RDA) unit. It also attaches to the existing SCSI bus using optically isolated interfaces, providing a low cost, high-bandwidth means of recording the same A2 data stream being recorded by the WSR-88D RDA. The A1DA was built to record:

    1) test cases for AP clutter analysis (allowing processing of both filtered and unfiltered data),

    2) test cases for range/velocity ambiguity mitigation

    3) test cases for alternative processing schemes (e.g. spectral or wavelet processing)

A detailed data analysis capability is provided by the Archive 1 analysis tool. For example, we can view the I and Q data separately or together, plotted as a function of time or together in polar coordinates. Spectrum analysis of the A1 data is also a commonly used process.

The A1DA consists of a Unix workstation attached to a real-time system. The Unix workstation provides display, control, and analysis functions, while the real-time system provides high-speed (8 Mbytes/second) data recording. Figure 7.1 shows the A1DA analyzer architecture.

The real-time system records time series data onto a set of SCSI disks. The Unix workstation reads the time series data directly from the same disks, using a dedicated SCSI interface. Reading a data file directly from a SCSI disk at 5 Mbytes/second is much faster than reading the same file over a 10Mbit/second ethernet. Attaching the real-time computer and the Unix workstation to the same set of SCSI disks allowed us to quickly transfer data between machines without purchasing specialized (and expensive) networking hardware and software.

Figure 7.2 shows how the A1DA is attached to a WSR-88D radar.

7.1.2 A1DA #1 Modifications for Use with S-Pol

The goals for the PRECIP-98 experiment (conducted in Florida) were to collect AP clutter and dual-PRT time-series data. These data are needed to acquire high resolution A1 AP clutter data for input into the AP Clutter Detection Algorithm and to determine how well the dual-PRT waveforms alleviated range/velocity ambiguity problems. To achieve these goals, both the S-Pol radar and the A1DA #1 were modified. Figure 7.3 shows a block diagram for the A1DA as it is attached to the S-Pol radar.

7.1.2.1 S-Pol Modifications

The pulse timing generator for S-Pol was upgraded to allow transmission of dual-PRT waveforms. In addition, a phase shifter was added to S-Pol, to allow transmission of phase-coded waveforms. The S-Pol software was modified to output timeseries data on the DSP "communication ports". A set of "communication port buffer cards" were built to transmit the time-series data from the S-Pol VME chassis to the A1DA VME chassis.

7.1.2.2 A1DA Modifications

A1DA Unit #1 was modified for use with RSF's S-Pol research radar. A modern CPU card was purchased that had 64 megabytes of on-board memory, on-board SCSI interface and a DMA. This card alleviated the VME bus bandwidth problems that had been experienced, since its architecture reduced the number of times data was copied over the VME bus. The "communication port buffer card" and an NCAR VIRAQ DSP card were installed to receive time-series data from S-Pol. The real-time software was modified to retrieve time-series data from the VIRAQ DSP card, rather than from the WSR-88D interface that was used earlier. A GPS clock card was installed for high-resolution date/time stamping of each radar transmit pulse, to aid later recovery, cataloging and analysis of the time-series data.

7.1.2.3 Time Series Data Collection in PRECIP-98

While in Florida, various staff collected ~15 tapes, containing 1120 files, that total 40 Gigabytes of time-series data. The files are characterized as follows:

The large number of constant PRT scans were acquired as baselines to allow comparisons with the other pulsing schemes.

An on-line catalog of these data files has been prepared, and query tools are being built to enable easy location of interesting portions of this dataset.

7.1.2.4 Recommendations and Future Plans

If engineering resources and budgets permit, further modifications may be made to S-Pol and the A1DA#1 to acquire some AP clutter data sets with dual-polarization. This would assist in the evaluation of advanced processing of dual-polarization datasets for A/P mitigation.

During PRECIP-98, some synchronization difficulties were experienced that hampered the ability to obtain "clean" datasets with good time stamps. When additional data are acquired using the S-Pol radar, a modified system architecture should be designed to avoid the synchronization problems.

7.1.3 A1DA #2 Installation at KCRI

This year NCAR worked with the OSF staff to install the A1DA #2 in the KCRI engineering shelter. Given that the A1DA#2 will remain installed in KCRI indefinitely, permanent cables were built and routed through an underground conduit to connect the A1DA#2 with the KCRI radar. The original cables are still available and could be used to attach the A1DA#2 to another WSR-88D radar, if desired.

Once the A1DA#2 was installed, the system was demonstrated to OSF staff and some OSF engineering staff were trained to operate the system. During the summer of 1998, the OSF engineering staff needed to evaluate prototype A/D replacement parts for the WSR-88D. They collected time-series data using the A1DA#2 to verify that the prototype A/D replacement parts had acceptable linearity, no missing codes, acceptable quantization noise, and adequate dynamic range. To facilitate this work, additional software was installed on the A1DA to allow printing Unix Postscript files from the "A1Tool" onto the instrumentation PC's printer, allowing hardcopy output. In addition, sample MATLAB routines were written to allow the OSF engineering staff to read A1DA time-series netCDF files and to plot a histogram of A/D output. These routines gave the OSF staff a focus for developing their own MATLAB analysis routines to process and plot the A/D performance data.

7.1.3.1 Recommendations and Future Plans

NCAR could significantly improve the performance of the A1DA #2 by installing the software on a higher performance Sun workstation that has additional memory. As the Open RDA progresses, NCAR could build a modern A1DA to record and analyze time-series data from the Open RDA signal processor. The Open RDA processor could be modified fairly easily to send time-series over a Raceway interface. Given the availability of PCI cards with Mercury's Raceway interface, a smaller, significantly less expensive instrument could be built on a PC computer. This PC-based instrument would replace the VME rack and the Unix workstation of the present A1DA. This instrument could eventually be transported to remote sites for diagnostic purposes after the Open RDA is deployed in the field.

7.2 KCRI RDA Instrumentation

7.2.1 Introduction

The WSR-88D Data Acquisition Unit (DAU) monitors about 20 analog values (i.e., temperatures and power supply voltages) along with 100 digital fault/status conditions. Although the WSR-88D displays summary information on its console from the DAU, there is presently no mechanism to save the DAU's data or to retrieve old values for later, off-line diagnostic purposes. This RDA Instrumentation system is designed to record the DAU's data stream and to allow later analysis.

7.2.2 Functionality

The "RDA Instrumentation" system is a virtual client/server instrument. It is designed to acquire, store, and report selected environmental and WSR-88D internal parameters for OSF hardware and software revision testing. It uses a non-interfering, optically isolated connection to the KCRI Data Acquisition Unit (DAU), which monitors the KCRI radar. The DAU data acquisition program (written in "C" by NCAR) captures the data from the DAU serial communication lines. This program executes on a 100Mhz Pentium data acquisition computer under the Windows NT operating system. A data logging program (written in Visual Basic by NCAR) reads the captured DAU data and writes it to a database. A separate 100MHz Pentium computer is used for data analysis. Using the NCAR-developed data analysis program (also written in Visual Basic), the user can query the database over the network to obtain monitored fault conditions, temperatures, and power supply voltages. Additional reports can be user-specified and generated using Visual Basic Crystal Reports. The same Visual Basic analysis software could be installed elsewhere on the local network at the OSF, if desired. Figure 7.4 shows the testbed instrumentation architecture.

In 1998, NCAR finished our scheduled work on the Testbed Instrumentation system. The DAU monitor was attached to the new permanent cables in KCRI conduit and OSF staff were trained in its use.

7.2.3 Recommendations and Future Work

NCAR could assist OSF staff in preparing additional summaries/reports from the database when the OSF staff identifies the additional reports they would find useful. Additional Visual Basic modules could be written by NCAR, or a commercial report generator could be used to build more comprehensive reports.


Appendix A

Functional Description of the NCAR AP Detection Algorithm

PROPOSED CLUTTER/WEATHER RECOGNIZER SOFTWARE MODULE for WSR-88D base data and S-Pol base data.
(Revised by Joe VanAndel & Cathy Kessinger, 4/27/98)

The call prototype:

call recognize (Z 0 , Z 1 , Z 2 , V 0 , V 1 , V 2 , W 0 , W 1 , W 2 , length_of_Z, length_of_VW, Zgain, Zoffset, Vgain, Voffset, Wgain, Woffset, size_of_analysis_cell, min_percent_coverage, min_Z, decision_threshold, m_filter, W 0 , M 0 listx, M 0 listy, M 0 npoints, W 1 , M 1 listx, M 1 listy, M 1 npoints, W 2 , M 2 listx, M 2 listy, M 2 npoints, W 3 , M 3 listx, M 3 listy, M 3 npoints, W 4 , M 4 listx, M 4 listy, M 4 npoints, W 5 , M 5 listx, M 5 listy, M 5 npoints, F1, length_of_F)

     

    I*2 Z 0 , Z 1 , Z 2 input three adjacent beam arrays of Z

    I*2 V 0 , V 1 , V 2 input three adjacent beam arrays of V

    I*2 W 0 , W 1 , W 2 input three adjacent beam arrays of W

    I*2 length_of_Z number of gates in Z input arrays (typically 230 1 km gates)

    I*2 length_of_VW number of gates in V and W input arrays (typically 4*230 0.25 km gates)

     

    float Zgain, Zoffset for conversion to dBZ

    float Vgain, Voffset for conversion to m/s

    float Wgain, Woffset for conversion to m/s

WSR-88D base data bytes are coded as offset binary (see Archive Level II Format document)

    I*2 size_of_analysis_cell size of target analysis cell in gates (typically 9 gates for a 2 km segment)

    float min_percent_coverage percent of gates in analysis cell required to have value > Z_threshold

    float min_Z lowest acceptable Z (in dBZ units)

    float decision_threshold decision threshold for output (nominally 0.0)

     

    I*2 m_filter median filter selection (0=nofilter, 1 contig outlier, or 2 contig outliers)

     

    float W 0 The piecewise linear data, membership function 0

    float M 0 listx where W 0 is a weight, M 0 listx, M 0 listy

    float M 0 listy are pairs of npoints in the piecewise linear

    I*2 M 0 npoints approximation.

Likewise, for membership functions 1 ... 5:

    W 1 , M 1 listx, M 1 listy, M 1 npoints,

    W 2 , M 2 listx, M 2 listy, M 2 npoints,

    W 3 , M 3 listx, M 3 listy, M 3 npoints,

    W 4 , M 4 listx, M 4 listy, M 4 npoints,

    W 5 , M 5 listx, M 5 listy, M 5 npoints,

     

    I*2 F1 array of flags containing coded target type (0=no decision,

    -1=weather,1=clutter)

    I*2 length_of_F number of gates in F output array (typically same as length_of_VW)

A "loose" description of the algorithm follows:

    1.0 Given three beams at a time (ZVW 0 , ZVW 1 , ZVW 2 ) do the following to produce recognizer flags array F1 aligned with the center beam. Need an entry point or call for the "first_time", to set up ( or change) lookup tables for evaluating the membership functions M 0 , M 1 , M 2 , M 3 below.

    2.0 Merge Z, V, W arrays to common gate spacing (0.25 km). Merging will be required in azimuth and in range for WSR-88D.

    3.0 For each clutter analysis cell (size_of_analysis_cell), calculate coverage of Zgates, Vgates, and Wgates

    if Z, or V, or W coverage > min_percent_coverage

    calculate MZE

        Eq. A.1

      if MZE > Z_threshold then

       

    calculate statistical features MVE, TZE (or TSNR), MSW, SDVE, starting with the following formulations

       Eq. A.2
        Eq. A.3
        Eq. A.4
        Eq. A.5

      else

       

      F1 = 0 (target unknown).

    The idea here is to skip the evaluation if there is insufficient coverage or insufficient signal.

    Note on computing textures: Texture is only computed between gates in the same beam (radially), not between gates in different beams. Texture of reflectivity is based on 1 km gates, since finer resolution isn't currently available. Texture is computed on 2 gates on either side of the gate of interest, and for 2 beams on either side, i.e.:

      azimuth 120 : XXXXX

      azimuth 121 : XXOXX

      azimuth 122 : XXXXX

      4.0 Calculate the vertical difference of DZ and VE at each gate of data. The value of the field at the selected, lower elevation angle is subtracted from the selected, higher elevation angle. Typically, the field values at the 0.5 degree elevation angle are subtracted from the 1.5 degree elevation angle. The height of the radar observations is not used in the calculations.

          Eq. A.6
          Eq. A.7

      Evaluate individual membership functions and then evaluate output membership function. The discriminator output Y is formed from

         
          Eq. A.8

      where the M 0 , M 1 , M 2 , M 3 and M 4 are applicable piecewise linear membership function lookup tables, and X 0 , X 1 , X 2 , X 3 and X 4 are the features to be identified as MVE, TZE, MSW, SDVE, plus GDZ.

      For our application the weights W 0 = W 1 = W 2 = W 3 = W 4 = 0.25 when both surveillance TZE and Doppler features MVE, MSW, SDVE are present; W 0 = 1 and W 1 = W 2 = W 3 = 0 if only the surveillance feature TZE is available.

        5.0 If output membership function Y < decision_threshold then

        F1 = 3 (weather or "not clutter")

        else F1 = 1 (clutter).

        6.0 Filter (censor outliers in F1) in range according to control parameter m_filter.

        7.0 An alternative coding of recognizer output is the direct output Y recoded in say 3 bits (7 levels plus unknown).


      Copyright (c) National Center for Atmospheric Research (NCAR) 1997 -- All Rights Reserved
      Last modified: 18 September 1999 5:00:00 pm
      Web Page Author: Cathy Kessinger
      NCAR NEXRAD Data Quality Team: Cathy Kessinger, Scott Ellis, Joe Van Andel, Don Ferraro