Table of ContentsCover Page of Document
|3.0 Princeton University Evaluation of the AP Detection Algorithm|
This document summarizes the work done by NCAR during FY-98 for the NEXRAD Data Quality Optimization program. This program has two main tasks: Specification of an AP Clutter Mitigation Scheme and the KCRI Testbed Instrumentation for the WSR-88D at Norman, OK.
An AP Clutter Mitigation Scheme is proposed that will eventually be implemented as part of the Open Systems Architecture. The AP Clutter Mitigation Scheme will use the Radar Echo Classifier to detect the presence of AP ground echoes, precipitation echoes, clutter residue and other radar scatterers as necessary. The output from the AP Detection Algorithm will determine where clutter filters will be enabled or disabled. The first step in the implementation process of the AP Clutter Mitigation Scheme will be to augment the clutter bypass map with the locations of AP echoes and allow the forecaster to manually edit the map, if necessary. The last step in the implementation process will be full automatic control of the ground clutter filtering process.
In this report, the preliminary optimization of the AP Detection Algorithm (APDA) is presented. Various tests of the membership functions and input variables are performed over a data set consisting of 60 individual scans. One test, termed Test #4, slightly outperformed the other tests. Individual cases are analyzed and the algorithm results examined. An independent evaluation of an early version of the APDA shows that the algorithm has significant skill in detecting AP echoes. A preliminary Precipitation Detection Algorithm is presented. Compensation of the reflectivity values within precipitation echo is required for the AP Clutter Mitigation Scheme and preliminary results of this work are presented.
For the instrumentation work, the installation of the Archive 1 Data Acquisition (A1DA) unit on the Norman, OK WSR-88D (KCRI) and the NCAR S-Pol radars is completed. The A1DA allows the collection of time series data. Time series (Archive 1) data will be necessary for the clutter residue characterization work. Also, one example case compares visually edited spectral domain data to the Archive 2 data for the same location and time. In addition, the Testbed Instrumentation system was installed on KCRI and allows the archival and display of various engineering parameters needed to monitor the operating system of the WSR-88D.
This document summarizes the work done by the National Center for Atmospheric Research (NCAR) for the Operational Support Facility (OSF) on the Data Quality Optimization program. The goals of this program are to improve the quality of the base data from the Weather Surveillance Radar-1988 Doppler (WSR-88D) systems with special emphasis on the problem of ground clutter return due to anomalous propagation (AP) conditions and to provide data quality instrumentation to the OSF.
To meet program goals of improving data quality (Saffle, 1997), an "AP Clutter Mitigation Scheme" is outlined in this document and is discussed briefly in Section 1.2 and comprehensively in Sections 2, 3, 4, and 5. The AP Clutter Mitigation Scheme consists of 1) computer algorithms based on "fuzzy logic" (Kosko, 1991) for the classification and detection of various types of radar echoes, 2) an automated system of ground clutter filter specification and control, 3) compensation of reflectivity within regions of precipitation echo that are biased by application of the clutter filter and 4) regions of AP echo that must be "tracked" by the statistical characterization of the clutter residue (or some other method) to allow filters to be enabled when needed and disabled when not. The AP Clutter Mitigation Scheme is a planned enhancement to the WSR-88D Open Systems Architecture beginning with Open Build 2 in the year 2000, as currently scheduled. The Implementation Plan is discussed in Section 1.3.
To meet program goals of providing data quality instrumentation, the Archive 1 Data Acquisition (A1DA) unit was installed on the NCAR S-Pol radar and the WSR-88D at Norman, OK (KCRI). In addition, the Testbed Instrumentation system was installed on KCRI. This work is described in Section 7. The Testbed Instrumentation system acquires, stores and monitors various environmental and internal system parameters to allow for hardware and software revision testing. The A1DA allows collection of time series data from the radar. The S-Pol radar collected time series data near Melbourne, FL during the summer. These data are needed to meet program goals for the Range-Velocity Mitigation program (Frush et al., 1998) also funded by the OSF. For the AP Clutter Mitigation Scheme, some of the Memphis WSR-88D time series data collected with the A1DA during July 1997 are used to do a preliminary analysis of the benefits of spectral domain processing for AP detection. This work is described in Section 6.
Radar return from AP clutter is an unpredictable event that often contaminates precipitation measurements, causing the generation of erroneous, radar-derived rainfall estimates used in hydrology products and confounding operational applications of WSR-88D data. Atmospheric conditions in which AP echoes are observed by radar are caused by time-variable radar refractivity profiles that modify propagation characteristics of the planetary boundary layer. In this section, the AP Clutter Mitigation Scheme is discussed briefly with complete descriptions of completed work in following sections.
A schematic flow chart of the AP Clutter Mitigation Scheme within the Open Systems Architecture is shown in Fig. 1.1. Within the scheme, the "Radar Echo Classifier" and "Reflectivity Compensation" algorithms are run. Output from the AP Detection Algorithm (APDA), one of the programs within the Radar Echo Classifier, is used to determine where clutter filters are necessary. This information is input into an "augmented clutter filter map" that the "Master System Control" uses to specify the clutter filters. Output from the APDA and possibly the "Precipitation Detection Algorithm" will be used by the Precipitation Pre-processing Subsystem (PPS) (O'Bannon, 1997) to ascertain data quality. In later builds of the Open Systems, information from the AP Clutter Mitigation Scheme will be input from the Radar Data Acquisition (RDA) unit to improve data quality.
The "Radar Echo Classifier" is a collection of fuzzy logic algorithms that will classify the type of echo observed by a WSR-88D. Current detection algorithms include the near-final version of the AP Detection Algorithm (Section 2) and a preliminary version of the Precipitation Detection Algorithm (Section 5). Other algorithms may be implemented to detect clear air return, clutter residue, birds, chaff, or other types of echoes. The Radar Echo Classifier is planned to be flexible such that additional detection algorithms can be added as needed. Figure 1.2 is a schematic showing the flow chart for the data input into these algorithms and follows through the final output of the detection algorithms. "Features" of the data are computed over a two dimensional region of the base data stream and assist in the recognition process for all detection algorithms. For instance, the APDA uses five features: the "texture" of the signal-to-ratio (TSNR) field, the mean radial velocity (MVE) field, the standard deviation of the radial velocity (SDVE) field, the vertical difference of the reflectivity (GDZ) and the mean spectrum width (MSW) field. Other algorithms will use different or similar features. A "membership function" is applied to the features for calculation of the "interest field." Interest fields are a measure of the likelihood that the particular feature is a positive indicator for the presence (or absence) of the desired phenomena. After application of the appropriate threshold to the interest field, the final output of the particular detection algorithm is obtained and a radar "product" generated. The Radar Echo Classification algorithms will be straightforward to modify and amenable to real-time operation in the WSR-88D Open Systems environment. Products from these algorithms will be available for use by all algorithms within the Open RPG system.
Mattias Steiner, Princeton University, has done a preliminary evaluation of the NCAR AP Detection Algorithm by using the functional description of the algorithm (Appendix A) to create a working version of the algorithm there. Using test cases that he has evaluated with other AP detection algorithms (Steiner and Smith, 1997), Steiner found that the NCAR algorithm performed with significant skill. The work done at NCAR on this task is described in Section 3. In his preliminary evaluation to the OSF, Steiner noted the following about the NCAR APDA:
Automation of WSR-88D clutter suppression through selective use of clutter filters and point clutter censoring is an ultimate goal of the Data Quality work. Automation will remove the task of clutter filter control for AP suppression from the responsibility of the National Weather Service (NWS) staff and should provide a standard means of clutter filter control for all WSR-88D systems. Automation will be achieved in real-time by using the output from the AP Detection Algorithm to augment the clutter filter map defined for normally-propagating (NP) ground clutter (currently called the "bypass map") such that clutter filters are enabled or disabled where appropriate for each volume coverage pattern (VCP). The AP clutter suppression will be specified on a gate-by-gate basis in this manner. Figure 1.3 shows an example bypass map for the Open Systems Architecture. This automated scheme must track the evolution of AP echo regions and adjust the clutter suppression as necessary. To prevent application of clutter filters within precipitation regions not affected by AP return, the output from precipitation detection algorithms may be used selectively to enable or disable clutter suppression.
As currently envisioned (but not necessarily rejected as an option), clutter filters will not be enabled at all times and at all locations. Rather, clutter filters will always be enabled at locations having NP clutter and will be enabled and disabled as needed for regions of AP clutter. Figure 1.4 shows a schematic of the clutter filter control process as currently envisioned. For each radar volume of data, the AP Detection Algorithm (APDA) is run and checked to see if a new AP product has been produced. If so, the product is sent to the clutter filter map and clutter filters are activated. If there are no new AP clutter products, then the age of old AP products are checked to see if the time has expired. If the time limit has expired (this limit is user-specified), then the locations contained within the AP product are removed from the clutter filter map. Figure 1.5 illustrates a sequence of AP detections and how they are used to enhance the clutter filter map. The AP product for the current radar volume is the darkest shade of red with lighter shades of red denoting increasing time history. Notice that the detections are cumulative and "age off" as the time limit expires. Note that this scheme does result in a one volume delay between detection of the AP and the application of appropriate clutter filters to remove the contamination from AP.
Reflectivity compensation is necessary in any AP mitigation scheme using clutter filtering. Because clutter filters negatively bias those radar reflectivity components that are located near zero radial velocity, these measurements must be compensated whenever clutter filters are applied to suppress either AP or NP clutter. Furthermore, when strong clutter is suppressed, a residue may remain causing a positive bias to the radar reflectivity. Clutter residue removal is expected to be an integral component of the automated processing.
The amount of precipitation bias created by the clutter filter depends on the radar echo velocity and width of an assumed Gaussian-shaped spectrum model as well as the clutter filter response. The filtered radial velocity and spectrum width define the biased zone of reflectivity. The reflectivity correction is then defined from a "lookup" table that has been previously generated from the known filter functions convolved with parameterized Gaussian weather model (Cornelius et al., 1995; Pratte et al., 1997; Ellis et al., 1999). Simulations indicate that a tolerable level of random compensation error (<3dB) can be achieved if the reflectivity loss in stratiform precipitation is no greater than ~10 dB, which corresponds to one of the WSR-88D medium suppression filters. Higher suppression levels yield greater compensation errors that may not be usable in hydrologic applications. Section 4 describes the work done on reflectivity compensation.
Use of a ground clutter filter can leave a "residue" in the filtered base data fields. Understanding the statistical characteristics of this clutter residue is required to facilitate the development of a fuzzy logic algorithm that can detect clutter residue. This work is needed since detection of the presence of AP clutter is difficult after the ground clutter filters are applied. Characterizing clutter residue will require time series data. This work is planned to occur during FY-99.
Examination of data from mountainous regions supports the conjecture that the "Clutter Residue Editing Map" (CREM) improves system performance. A CREM identifies the residual clutter that the filter fails to remove for NP situations. The Terminal Doppler Weather Radar (TDWR) program (Turnbull et al., 1989) has successfully used CREMs to remove clutter residue and to improve system and algorithm performance. Data within the CREM area may be corrected as part of the reflectivity compensation procedure (described below), censored, or flagged for further attention. Any one of these schemes will improve the base data quality and the meteorological products.
A phased, two-stage implementation is currently planned for the AP Clutter Mitigation Scheme within the Open Systems Architecture of the NEXRAD WSR-88D systems. The first implementation will be within the "Open Systems Radar Product Generator (ORPG)" with a later implementation within the "Open Systems Radar Data Acquisition (ORDA)" system. Figure 1.6 shows the WSR-88D user control panel. Initially, the AP Clutter Mitigation Scheme will occur within the Radar Product Generator (RPG). The enhanced clutter filter map will be accessed through the "Clutter Regions" icon shown within the "Applications" panel to the right.
For the initial ORPG work, a phased implementation of the AP Clutter Mitigation Scheme is planned because testing the full AP Clutter Mitigation Scheme (including reflectivity compensation, residue tracking, and clutter filter control procedures) cannot be performed in a timely manner due to funding limitations. An initial set of Configuration Change Requests (CCRs) for these enhancements will be submitted in 1999, with implementation planned for the Open Build 2. However, the implementation may be delayed to future builds if modifications to the Precipitation Pre-processing Subsystem (PPS) algorithm (O'Bannon, 1997) prove to be a major task. The first CCR will specify a new AP recognition product that could be used in all Open RPG algorithms. The second CCR will specify that the PPS algorithm will use the AP product rather than the "tilt test" in providing estimates of rainfall rate and amounts. The tilt test is a confirmed weak point of the PPS. The third CCR will implement the reflectivity compensation algorithm. The fourth CCR will specify that the clutter bypass map will be "enhanced" with the locations of the AP clutter as defined by the AP Detection Algorithm product. Application of appropriate ground clutter filters using the enhanced clutter filter map will be activated manually. For these CCRs, automation of the clutter filter control is not achieved. This task will be specified with later CCRs.
With the inauguration of the Open RDA WSR-88D, additional RDA processing will allow a higher performance AP Clutter Mitigation Scheme. Access to both the filtered and unfiltered base data streams will allow better clutter suppression, filter bias and residue compensation, and tracking of the AP affected areas. Eventually, the ORDA processor system will likely allow Doppler spectral domain processing. The increase in real time computing power will allow combining AP and precipitation recognition schemes with new operational range-velocity ambiguity mitigation techniques being proposed (Frush and Doviak, 1998; Frush and Daughenbaugh, 1999; Sachidananda and Zrnic, 1997; Sachidananda et al., 1997; Sachidananda et al., 1998; Torres, 1998). Polarimetric data may also be available late in that decade allowing even greater recognition and mitigation performance. At this time, submission of CCRs for this type of work is premature.
Turnbull, D., J. McCarthy, J. Evans, and D. Zrnic, 1989: The FAA Terminal Doppler Weather Radar (TDWR) Program. Preprints, 3rd Intl. Conf. on Avia. Wea. Sys., Anaheim, 29 Jan.-3 Feb., Amer. Meteor. Soc., 414-419.
Ground clutter contamination within radar data has two sources: normally-propagated (NP) ground clutter from stationary targets such as buildings, trees, or terrain, and anomalously-propagated (AP) ground clutter that arises from particular atmospheric conditions within the planetary boundary layer that duct the radar beam to the ground. Because NP targets are stationary and "unchanging," they are relatively easy to remove with the use of clutter "bypass" maps that are part of the WSR-88D data quality control system. However, AP clutter can evolve as atmospheric conditions change and is not always present. Recognition of the presence of AP clutter is the responsibility of the National Weather Service (NWS) forecasters, who must then modify the clutter filter maps to specify additional regions of clutter filter application. Automatic detection of the AP ground echoes and the specification of ground clutter filters to remove the AP echoes will augment current WSR-88D capabilities, improve the quality of the base data fields and derived products and relieve the forecasters of this responsibility.
In Section 1, the AP Mitigation Scheme was described briefly. Within this scheme, the first step necessary for improving WSR-88D data quality is to design an algorithm to detect the presence of AP clutter echoes. NCAR has developed a fuzzy logic algorithm to perform this task that is called the Anomalous Propagation Detection Algorithm (APDA). In this section, details of the fuzzy logic algorithm development are given, software modifications are described, the methodology for the analysis of case studies is described, and the near-final optimization of the APDA is presented.
This year's effort was preceded by a period of inactivity on the project after the departure of both Pratte and Ecoff in April 1997. When Kessinger and Ellis began working on the project in late 1997, their first priority was to understand the work done by Pratte and Ecoff so that the algorithm development could continue and be completed.
Initially, Pratte and Ecoff evaluated three techniques for potential usefulness in detecting AP ground clutter contamination (Cornelius et al., 1995; Pratte et al., 1997; Keeler et al., 1998a). These techniques included a neural network (NN) scheme, a fuzzy logic classifier and an empirical, rule-based technique. After extensive testing, the fuzzy logic classifier was selected as the algorithm of choice due to the ease of implementation and the small differences in statistical performance between the fuzzy logic classifier and the neural network classifier. It was at this decision point that Pratte and Ecoff departed NCAR.
During their testing, Pratte and Ecoff selected two sets of WSR-88D scans. These sets of scans were referred to as the "alpha" and "beta" scan sets. The alpha scans were used to train the neural network (NN) AP detection algorithm while the beta scan set was used to evaluate the trained NN algorithm. The fuzzy logic classifier and the empirical scheme were run using these same scan sets. Currently, eleven of the nineteen scans of the alpha and beta scans are included in the dataset used to evaluate the fuzzy logic AP Detection Algorithm (APDA). Because of the addition of the vertical difference of reflectivity feature in the APDA (see Section 2.4), alpha and beta scans were excluded from the current study when the higher elevation angle scan could not be obtained due to not having the A2 tapes on hand. Because Pratte and Ecoff truthed only small regions of each elevation angle, the alpha and beta dataset was re-truthed following the current methodology that is described in Section 2.6. The alpha and beta scans are listed in Table 2.1.
To enable easier selection of additional AP case studies, the WSR-88D data contained on the in-house, Archive 2 (A2) data tapes were examined and cataloged. Table 2.2 (pages one and two) contains the Archive 2 (A2) tape catalog.
Time lapse movies in MPEG format were made of all the A2 tapes to facilitate case study selection. For most, the movies consist of the lowest three elevation angles: 0.5, 1.5, and (typically) 2.3 deg elevation. For the Memphis KNQA data taken during the Archive 1 Data Acquisition (A1DA) effort in July 1997, an additional set of movies were made with only the 0.5 deg elevation angle. All movies have two panels that show the reflectivity and radial velocity data at each elevation angle. A list of dates and time for which movies were made is in Table 2.3. Two example movie frames are shown in Fig. 2.1.
The movies were examined and notes made regarding the weather or AP events that were occurring. Table 2.4 (pages one, two and three) gives a short summary of events noted. As a result of the review process, additional cases were selected and included within the AP data set. A total of 60 radar volumes comprise the dataset used to evaluate the AP Detection Algorithm. Within these scans, a variety of weather conditions existed: from clear air return to individual convective cells to large, organized convective complexes with regions of stratiform precipitation. The amount of AP contamination varied from very minor to major areal extent. The 60 scans of the dataset are summarized in Table 2.5.
As shown in Table 2.5, data from nine radars are used in the evaluation process, covering selected areas from the Rocky Mountains (KFTG) to the Atlantic seaboard (KLWX), and from the Great Lakes (KLOT) to the Gulf Coast of Texas (KHGX). As shown by Smith et al., (1996), this area of the United States has the highest occurrences of AP clutter contamination, especially during the summer months. Eighteen of the scans were individual scans with little time history, while 42 of the scans had extended temporal coverage. Eleven of the scans were from the alpha and beta datasets. The extended temporal coverage was done to examine the feasibility of "tracking" AP clutter for the automated AP Mitigation Scheme (Keeler et al, 1999). An analysis of the evolution of an AP event should assist in this work.
A fuzzy logic classifier (also called recognizer) scheme is a technique for scaling various input fields into a common reference frame such that each input indicates the presence or absence of the event that is being detected. For our purposes, the fuzzy logic classifier scheme is being used to detect the presence of AP ground clutter. The framework of the fuzzy logic AP Detection Algorithm developed at NCAR is shown in Fig. 2.2 and covers the processes from the input derived fields to the final output product of the detection algorithm. In this section, an overview is given of the processes used within the AP Detection Algorithm. Additional details are contained in following sections.
Briefly, a fuzzy logic classifier uses various derived fields (formally known as "features") as input, scales them to a common reference frame by use of a "membership function," and then computes a weighted sum of the resultant "interest" fields. After application of a threshold, the final output product of the detection algorithm is obtained and contains the locations of the AP ground clutter. The purpose of the membership function is to scale all input fields to be between zero and unity. In the interest fields, values that approach unity are indicative of the presence of the quantity and that is desired to be detected and values that approach zero are indicative of the absence of that quantity. For example, AP clutter typically has radial velocity values near 0 m s -1 . Regions of the radial velocity interest field will have values near 1 when the radial velocity field is near 0 m s -1 .
As determined by Pratte and Ecoff, the derived fields that showed the most promise for detecting the presence of AP clutter are the "texture" of the signal-to-ratio (TSNR) field, the mean radial velocity (MVE) field, the mean spectrum width (MSW) field, and the standard deviation of the radial velocity (SDVE) field. The TSNR field is similar to the texture of the reflectivity field but was much easier to compute for these tests. Steiner and Smith (1997) compared various AP detection algorithms and found that the vertical gradient of reflectivity showed promise in being able to detect AP clutter. For this reason, the vertical difference of the reflectivity (GDZ) between the 1.5 and the 0.5 deg elevation angles has been added to the APDA. To simplify the calculation, the difference in height between the two elevation angles is not considered; therefore the resultant field is not truly a spatial gradient, but more of a range-dependent reflectivity difference. The equations used for calculating the input fields are shown in Eqs. 2.1-2.5,
where N is the number of values within a specified region, i is the index of each value, SNR is the signal-to-ratio, DZ is the reflectivity value at a range gate, SW is the spectrum width value at a gate, Vi is the radial velocity value at a gate, and V is the mean radial velocity within the specified region.
Pratte and Ecoff determined an initial set of membership functions to be applied to the MVE, TSNR, MSW and SDVE fields. Those membership functions are shown in Fig. 2.3. Membership functions are constructed to distinguish those radar characteristics typical of "clutter" from those characteristics that are typical of "not clutter" (i.e., echoes from weather, clear air, birds, etc.). To understand the differences in radar characteristics between clutter and not-clutter, histograms of the variables were constructed for both cases. This process is described in more detail in Section 2.5.
Determination of the regions of clutter versus regions of not-clutter is called "truthing" the data set. The results of the truthing process are used to statistically score the performance of the AP Detection Algorithm. Improving the statistical scores of the algorithm should lead to a more optimum algorithm that has good performance in the field. For this reason, the process of determining the truth is very important in this algorithm development. To decide the truth regions, a human expert (Kessinger or Ellis) examined all scans of the radar data set and decided which gates had clutter and which had not-clutter. Polygons are drawn and values set within the expert truth field. This process is described in more detail in Section 2.6.
The statistical scores computed within the NCAR AP Clutter Analysis Tool (APCAT) are the Probability of Detection (POD), the False Alarm Ratio (FAR), the Critical Success Index (CSI) (Donaldson, et al., 1975), and the Percent Correct. They are defined as
where TP is the True Positive count, FN is the False Negative count, FP is the False Positive count and TN is the True Negative count as shown for the contingency table in Fig. 2.4. The results from the AP Detection Algorithm are scored against the truth field as determined by a human expert.
To determine the characteristics of AP ground clutter radar echoes and echoes from targets that are not-clutter, histogram plots are made for the derived variables using the truth field to distinguish between the two types of radar echoes. Figure 2.5 shows examples of histograms of the variables currently used in the AP Detection Algorithm as derived from data taken by the Dodge City, KS (KDDC) WSR-88D radar. Fields shown include the texture of signal-to-noise ratio (TSNR), mean radial velocity (MVE), vertical difference of reflectivity (GDZ), mean spectrum width (MSW), and the standard deviation of the radial velocity (SDVE). Histograms are plotted for regions identified as clutter and for regions identified as not-clutter by the experts.
After construction of the histograms, they are examined to find the maximum differences in the values of the variables that occur within regions of clutter and regions of not-clutter. When differences are large, this indicates a variable that can be used to distinguish clutter from not-clutter. For instance, clutter has high values of TSNR (Fig. 2.5a, page one) due to its high spatial variability while not-clutter has low values of TSNR since it has low variability; likewise, clutter has MVE values near 0 m s -1 Fig. 2.5a while not-clutter has a wide range of MVE values. Similarly, GDZ has a large magnitude in clutter Fig. 2.5c) compared to not-clutter within precipitation echoes. For clear air return, GDZ can also have a large magnitude. The MSW (Fig. 2.5d, page two) and SDVE Fig. 2.5e) fields typically have smaller values in clutter compared to not-clutter. This process is used to define the features used in the AP Detection Algorithm. The histograms also aid in defining the membership functions of the features used in the algorithm.
"Truthing", or manually reviewing and marking the AP and precipitation areas that will be used for comparison with the fuzzy logic AP Detection Algorithm, is an important activity. Comparison with truth as defined by expert humans is crucial in optimizing the fuzzy logic algorithm for its best performance.
To determine locations of AP clutter, the expert begins by examining the various radar fields and their derived products using the NCAR SOLO program (Oye et al., 1995). Only the 0.5 deg elevation angle is truthed but the other elevation angles are examined for additional insight into the case. All regions of the 0.5 deg scan are characterized as either clutter, not-clutter (precipitation, clear air returns, or anything else), or clutter residue. An attempt was made to define regions where clutter filters were applied for stationary targets (i.e., NP clutter) by use of the clutter "bypass" map. The bypass map defines the location of stationary targets on a gate-by-gate basis and can be augmented by the NWS forecaster, if needed. However, most of the A2 tapes did not have the bypass map on them since the bypass map information was not written to tape before WSR-88D Build 10. Therefore this process of identifying clutter residue is not strictly accurate and represents the expert's best estimate of where the bypass map appeared to have been applied. For this reason, regions of clutter residue are not currently being evaluated.
Typically, AP clutter is characterized by a near-zero radial velocity, low spectrum width, high texture of the reflectivity field, as well as other characteristics. Further, the expert can examine the scan at the next two elevation angles (1.5 and 2.3 degs) to determine if the radar echo in question has vertical continuity. A lack of vertical continuity does not guarantee that the echo is AP clutter, but does give more information into the decision-making process. Once the expert has determined where the AP clutter is, SOLO is used to draw a polygon around the region and the appropriate value is set within the truth field. Currently, three values (1, 2, or 3) are used to define features within the truth field. Regions of clutter are defined by the value 1; regions of non-clutter echo are defined by the value 3; and regions of clutter residue are defined by the value 2. Example truth fields are shown later in this section of the report. Once truthing is completed, the radar data are input into the APCAT for further processing and statistical evaluation of the fuzzy recognizer algorithm.
Of course, a basic limitation of defining the truth in this manner is the lack of independence between the truth field and the radar fields used as input into the AP Detection Algorithm. A preferable methodology would have an independent source for determination of AP clutter regions (such as satellite data or dense rain gauge networks). Determining the truth in any of these manners is beyond the scope of work for the program and the availability of these data sets is severely limited. However, the data available from NCAR's S-Pol dual-polarimetric radar provides an opportunity to obtain an independent classification of the locations of clutter (i.e., the truth). A few scans of S-Pol data are planned to be put through the AP Detection Algorithm and the results evaluated. This effort is described in more detail in a later section.
In most cases, differentiating between clutter echoes and weather/clear air echoes is relatively easy especially when clutter is severe; however, several issues became apparent during the process. Clearly, human subjectivity effected the final truth fields. The location of the boundary between clutter echo and not-clutter echoes is sometimes difficult to determine precisely, especially in cases of weak or marginal clutter contamination.
In some cases, the AP clutter occurred in a few gates within a region of weather echo. Drawing a polygon around each of these small "speckles" is a difficult task for any human. These situations may have some error in the truth fields.
The radar scanning strategy of the WSR-88D system effects the quality of the statistical results for the AP detection algorithm. The reflectivity data are collected with 1 km gate spacing while radial velocity data are collected with 250 m gate spacing. The reflectivity scan at 0.5 deg is performed first at low pulse repetition frequency (PRF), followed by the radial velocity scan at a higher PRF. The APCAT is used to merge the two scans in range (to 250 m) and in azimuth. Because so-called "indexed beams" are not used, precise positioning of the beams cannot be accomplished scan-to-scan. Typically, the azimuths between the two scans match within a few tenths of a degree. Therefore, temporal and spatial differences exist between the reflectivity and radial velocity data.
The quality of the truth field is adversely effected by the spatial differences of the reflectivity and radial velocity fields since there are 4 radial velocity gates per 1 reflectivity gate. Despite the diligence of the human experts, drawing polygons to enclose AP on a gate-by-gate basis is difficult. Within APCAT, the statistical results are scored on a gate-by-gate basis. While difficult to quantify, the difficulties with the truth field likely tend to degrade the statistical scores.
When clutter echoes are mixed with weather echoes, the data may be biased by the clutter echo but not so much that it is clearly clutter. From the A2 data stream and derived fields, determination of regions of mixed clutter and weather signals is difficult. Clearly, application of the ground clutter filters are needed in situations such as this, but detection of the feature may be difficult. Steiner and Smith (1997) noted that mixed clutter and weather signals are the most challenging for any detection algorithm.
Truthing the data at far ranges is extremely difficult since the reflectivity and derived reflectivity fields are frequently the only fields available beyond 150 km. In good situations, all fields are available to 230 km. However, the Doppler fields usually have significant missing data regions beyond 150 km due to a failure in the WSR-88D velocity recovery algorithms. Also, at far ranges, the values of the texture of the SNR and the spectrum width become less distinct between clutter and weather echoes due to the larger size of each radar resolution volume. Furthermore, the vertical difference of reflectivity is adversely effected as an indicator of AP clutter at long ranges due to the increasing height of the beam. The radar beam may "over-shoot" low altitude weather echoes, such as stratiform rain or winter storms, resulting in a vertical difference of reflectivity field that appears to indicate the presence of AP clutter.
For the work summarized in the FY96 Status Report (dated November 1996 and revised in September 1997), different criteria were used to define regions of truth than are being used now. The primary focus of the FY96 work was the neural network (NN) AP detection algorithm, rather than the fuzzy logic AP detection algorithm. The NN algorithm required very "pure" truth to train the neural network. For this reason, very limited regions were used to define the truth such that only the most definitive characteristics of the feature were included. Using these same truth regions, the preliminary fuzzy logic algorithm was evaluated. The evaluation showed it had similar statistical skill as the NN algorithm and performed quite well. Using very pure truth will produce higher statistical skill scores than if all regions of the scan are truthed because of the natural variability of the radar data that are either totally or partially biased by clutter. Due to the difference in truthing methodology, statistical skill scores of the current AP Detection Algorithm will be lower, but more representative of actual performance, than the scores reported in FY96.
The AP Clutter Analysis Tool (APCAT) is a software package developed by Joseph VanAndel and others at NCAR that calculates the interest fields used in the AP Detection Algorithm, applies the fuzzy membership functions to the features, outputs the algorithm results, and performs a statistical analysis of results using the truth field described above. Section 2.4 describes this process in detail. The APCAT is under development as needed and has had several major enhancements made this fiscal year. In this section, the improvements made to APCAT are discussed.
The vertical difference calculation was added to compute the difference in reflectivity and radial velocity between the 0.5 and 1.5 deg scans. As recommended by Steiner and Smith (1997), the vertical gradient in reflectivity is useful for detection of AP contamination since AP echoes are generally at maximum intensity near the ground and weaken rapidly with height. In our implementation, the difference in reflectivity is computed rather than the gradient. For the A2 data format, an SNR threshold is applied to the base data, resulting in a considerable number of missing data regions. The number of missing data regions typically increases as the elevation angle increases especially when AP is present. Also, because AP weakens rapidly with height, there are many fewer gates of AP at the 1.5 deg elevation angle than at the 0.5 deg elevation angle. For these reasons, when the reflectivity value is missing at the 1.5 deg scan (usually because it has been thresholded out), the radar noise power is used to compute a reflectivity value at the "missing" gate. The result of using the radar noise power to compute a new reflectivity value is that the vertical difference is exaggerated. Having an unthresholded base data stream will eventually correct this problem. Membership functions (described below) for the vertical difference in reflectivity will likely need some adjustment when an unthresholded data stream is available in the Open Systems Architecture.
A "diagnostic mode" was added to APCAT to show the interest output of each individual membership function. Prior to this, APCAT produced only the final interest field from adding the summed weights of the individual interest fields. This helps determine if the membership functions have been tuned properly for various types of input data, which should improve the recognizer's performance. In addition, APCAT was restructured to compute multiple fuzzy recognizers in parallel. This allows testing various combinations of membership functions and weights at once, rather than requiring the user to run APCAT multiple times.
The SOLO perusal/editing program stores data in DORADE format within disk files called "sweepfiles." A sweepfile holds one elevation angle of radar data. Because SOLO and APCAT are used in conjunction, APCAT uses the DORADE format within sweepfiles, too. Formerly, users had to run SOLO to create new DORADE variables for the fuzzy logic interest fields. This made for an extra step in the processing and was cumbersome to do. Therefore, APCAT was enhanced to streamline sweepfile processing to automatically create the necessary DORADE variables.
Because of the large number of scans (i.e., 60) within the AP evaluation data set, it became necessary to enhance APCAT's batch processing capabilities. Using pre-selected menus, APCAT can automatically process a list of files by merging the reflectivity and velocity scans, creating DORADE variables, computing the vertical difference feature and other selected features, and computing the fuzzy recognizer outputs and doing the statistical scoring. This simplification in procedures significantly improved data processing.
The APCAT's statistical performance scoring software was improved. The values within the interest final output field vary continuously between 0.0 and 1.0. APCAT computes the statistical performance of the detection algorithm using 21 different thresholds between 0.0 and 1.0. If the output value of the detection algorithm is over the specified threshold, it is considered a True Positive. The contingency tables ( Fig. 2.4), containing the counts of True Positives, True Negatives, False Positives, and False Negatives for each threshold value, are output to a file for diagnostic and debugging purposes. Examination of the contingency tables allows comparison of the algorithm output versus the truth values. In addition, the new version of APCAT allows the interactive selection and display of performance scores for each threshold ( Fig. 2.6). Numeric readouts for Probability of Detection (POD), False Alarm Ratio (FAR), Critical Success Index (CSI), and Percent Correct are displayed for the selected threshold. In addition, the point representing the POD and the FAR for the given threshold is highlighted on the left panel of the performance graph. These improvements make it feasible to optimize the membership functions and to determine the best threshold value for a given detection algorithm.
Before they left NCAR, Pratte and Ecoff had written considerable documentation about the work that they had done. They left personal notes and communications, discussed their methodology on the telephone and with electronic mail, and then returned to NCAR for a night-time meeting with the Data Quality group in February 1998.
As shown in Fig. 2.3, Pratte and Ecoff defined four membership functions for the AP Detection Algorithm (APDA). These functions used the TSNR, MVE, SDVE, and MSW fields as input. Because the work of Steiner and Smith (1997) suggested that the vertical gradient of reflectivity is a useful indicator for AP contamination, a variation of this field (called GDZ) was added to the APCAT software. The GDZ field is the vertical difference of reflectivity and is computed by subtracting the reflectivity value at the lower elevation angle from the higher elevation angle.
The optimization of the APDA consists of modifying the membership functions for each input field and then examining the statistical performance via APCAT to see if improvements occur. Histograms are used to look at the distribution of the interest values within each of the individual interest fields. Using the truth field to distinguish between regions of clutter and regions of not-clutter (i.e., precipitation, clear air return, etc.), histograms are made for the interest fields to see the distribution of values for the 60 scans of the data set ( Table 2.5). For clutter regions, the interest fields should have many values near unity. For regions of non-clutter, the interest fields should have many values near zero. Figure 2.7a shows the histogram for the APDA results for both clutter and not-clutter regions when the original membership functions are used. The original membership function does not define regions of clutter Fig. 2.7b) very well since the number of occurrences does not change much as the interest value increases. Regions of not-clutter are defined quite well when an interest threshold of 0.5 is applied since the majority of the area under the curve is below this threshold (Fig. 2.7c). These plots clearly show that the original membership functions will have a large number of missed detections for AP clutter after application of a threshold (typically near 0.5).
The original membership functions ( Fig. 2.3) served as the starting point for the optimization process. Various iterations were performed on the membership functions to achieve optimization. Histograms of the individual interest fields were examined during this phase. For this report, the iterations are summarized into six tests. Two sets of membership functions are used for these six tests and are shown in Figs. 2.8 and 2.9. Table 2.6 describes each test. Briefly, the "Original Test" uses the original membership functions shown in Fig. 2.3 with the weights for all interest fields being unity. The "Test #1" uses the membership functions shown in Fig. 2.8 with the exception that the GDZ field is not used. New functions for the MVE and MSW fields are defined. Weights for all interest fields are unity. The "Test #2" uses all the membership functions shown in Fig. 2.8 and all weights are unity for the interest fields. The "Test #3" uses the membership functions shown in Fig. 2.9. New functions have been defined for the SDVE and GDZ fields and all weights are unity. The "Test #4" is the same as "Test #3" with the exception that the interest field weights are 2 for the MVE and TSNR fields and are 1 for the SDVE, MSW and GDZ fields. The "Test #5" and "Test #6" are duplicates of Test #3 and Test #4 except that the unfiltered radial velocity (VE) has been substituted for MVE and the unfiltered spectrum width (SW) has been substituted for MSW. This was done to examine the performance of the algorithm when the mean filter is not used. If the fields are not filtered, computational speed may be enhanced; however, algorithm performance may suffer.
Using the 60 scans that comprise the data set for evaluation of the algorithm ( Table 2.5), the six tests plus the original membership functions are evaluated. Tables 2.7-2.10 contain the statistical performance indices (POD, FAR, CSI and Percent Correct) for each of the algorithm tests. Statistical performance results from APCAT are also plotted in Figs. 2.10-2.15 and Figs. 2.18-2.19.
The Original Test has its highest CSI score of 0.462 when a threshold of 0.4 is applied. The POD is 0.64, the FAR is 0.098 and the Percent Correct is 0.848 ( Table 2.7 and Fig. 2.10). These scores are lower than reported by Cornelius et al., (1995) but the differences in truthing methodology as discussed in Section 2.6 should account for the lower statistical scores. The highest CSI score for Test #1 is 0.478 at a threshold of 0.5 ( Table 2.8 and Fig. 2.11), a slight improvement in performance over the Original Test. The Test #1 POD is 0.662, the FAR is 0.099, and the Percent Correct is 0.853. Recall that the difference between the Original Test and Test #1 was to modify the membership functions for the MVE and MSW fields ( Table 2.6). Therefore, changing these two membership functions leads to slightly improved performance of the AP Detection Algorithm. For Test #2 ( Table 2.8 and Fig. 2.12), the membership functions from Test #1 are used with the addition of a new input field, GDZ. The membership function for GDZ is not optimized for this test and resulted in a decreased performance for the detection algorithm. The CSI for Test #2 is 0.453 at a threshold of 0.45 while the POD is 0.599, the FAR is 0.089 and the Percent Correct is 0.844. The FAR is slightly improved over the results from the Original Test.
For Test #3, new membership functions are defined for the SDVE and GDZ fields ( Table 2.9 and Fig. 2.13). At a threshold of 0.55, Test #3 has a CSI of 0.475, a POD of 0.639, an FAR of 0.095 and a Percent Correct of 0.848. The weights for all membership functions is unity. Notice that the skill scores from this test are very similar in magnitude to the skill scores for the Original Test. For Test #4, the membership functions are the same as used for Test #3, but the MVE and TSNR fields have a weight of two instead of one. With a threshold of 0.55 ( Table 2.9 and Fig. 2.14), the skill scores are slightly improved over the Original Test with a CSI of 0.492, a POD of 0.663, a FAR of 0.096 and a Percent Correct of 0.852. Figure 2.15 compares the "POD vs False Alarm Ratio" of the Original Test versus Test #4 and shows the slight statistical improvement of Test #4 over the Original Test. The POD has the best improvement when compared to the best score of the Original Test. The most noticeable difference between Test #4 and the Original Test is the increase in the value of the threshold.
Using histograms, the distribution of interest values within the final output fields of the APDA using Test #3 ( Fig. 2.16) and Test #4 ( Fig. 2.17) are compared. In panel (a) of each figure, the interest values for both clutter and not-clutter regions are plotted; in panel (b), the interest values for clutter regions only are plotted; and in panel (c), the interest values for not-clutter regions only are plotted. The y-axis on these plots are not fixed but are dependent on the number of occurrences within each category. Therefore, examination of the y-axis must be done to facilitate the comparison. Test #4 distinguishes clutter (Fig. 2.17b) slightly better (more of the interest values are closer to 1) than Test #3 (Fig. 2.16b), but the difference is small. For regions of not-clutter, Test #4 (Fig. 2.17c) does noticeably better than Test #3 (Fig. 2.16c) because more of the interest values are below 0.5. In other words, Test #4 will distinguish between clutter and not-clutter better than Test #3 will. Using a threshold of 0.5, the Test #4 will perform better than Test #3 for the APDA.
In Mattias Steiner's evaluation of the AP Detection Algorithm performance (see Section 3), he noted that the algorithm may have difficulty completing its calculations in a timely manner when run in a real-time environment. Given the rapid increase in computing power that is occurring today, slow algorithm performance in a real-time environment is not expected to be a problem at the time of implementation. However, to minimize the amount of computations and to assess the resulting algorithm performance, two algorithm tests were devised that did not use the mean filtering. These tests, Test #5 and Test #6, are identical to Test #3 and Test #4 with the exception that the radial velocity (VE) field is used instead of the MVE field and the spectrum width (SW) field is used instead of the MSW field. As can be seen in Table 2.10 and in Fig. 2.18, Test #5 has its highest CSI score of 0.479 with a threshold of 0.6. The POD is 0.606 and the FAR is 0.073. Comparing the results of Test #5 to Test #3 shows that similar results are obtained. At a threshold of 0.55 (Table 2.10 and Fig. 2.19), Test #6 has a CSI score of 0.489, a POD of 0.665 and a FAR of 0.099. Again, these results are similar to those obtained for Test #4. In summary, disabling the mean filter does not appear to greatly affect the performance of the AP Detection Algorithm. However, disabling the mean filter is not recommended if computing power is sufficient. For the SDVE field, a mean value must be calculated for the radial velocity (see Eq. 2.5) field. Therefore, the only computational savings occurs when the spectrum width is not filtered. Disabling the mean filter, will result in noisier output from the AP Detection Algorithm as shown in examples below in Section 2.8.2. Noisy algorithm output may not be desirable for the operational goals of the algorithm.
In this section, one scan from the 60-scan data set is examined in detail from the Amarillo, TX WSR-88D (KAMA) on May 25, 1994 at 0035 UTC. This scan contains a region of AP echo and a region of intense convective precipitation and allows the AP Detection Algorithm output to be examined for two diverse types of weather events. Further, second trip contamination is contained with the AP echo.
The AP echo is mostly in the northern quadrant of the 0.5 deg elevation angle ( Fig. 2.20). It is a region having maximum reflectivity of 35-40 dBZ, radial velocities near 0 m s -1 , high texture of the SNR field (> 15), values of 0-1 m s -1 of the SDVE field, strongly negative vertical difference of reflectivity (down to -45 dBZ), and low values of MSW (1-2 m s -1 ). The AP echo is occurring as a large, nearly homogeneous region of return. A region of second trip echo is found near the 330 deg radial and is within the AP return. The second trip echo shows best in the Doppler channels which may explain the lack of detection and correction of the echo by the WSR-88D second trip correction algorithms. The second trip echo has higher radial velocity values, higher SDVE values, and higher MSW values than the AP echo, and is quite easy to distinguish visually with these fields.
Interest field output from the Original Test is shown in Fig. 2.21a . The membership functions used for the Original Test are shown in Fig. 2.3. Unthresholded interest output from each of the input fields is shown in Fig. 2.21b-d and 2.21f. The expert truth field is shown in Fig. 2.21e with the regions of AP echo shaded green. Comparing the truth field to the Original Test output shows that the AP Detection Algorithm identifies the AP echo quite well when a threshold of about 0.5 is used (see Fig. 2.24d for the thresholded output). Notice that the Original Test does not identify the second trip as AP clutter for that threshold. Comparing the MVE and SDVE interest fields (Fig. 2.21b and 2.21d, respectively) to the expert truth field shows that the regions of AP are well-defined for a threshold of about 0.5 for both. The MSW interest field (Fig. 2.21f) also defines the AP region quite well when a lower threshold of about 0.4 is used. The MVE, SDVE, and MSW interest fields distinguish the AP echo from the second trip echo quite well and do not falsely identify the second trip echo as AP clutter. The TSNR interest field (Fig. 2.21c) has high values within the AP and second trip echoes and cannot distinguish between the two types of return.
The interest field output from the Test #4 is shown in Fig. 2.22 and uses the membership functions shown in Fig. 2.9. Comparing the Original Test ( Fig. 2.21a ) to the Test #4 (Fig. 2.22a) shows that the Test #4 interest values are higher within the AP echo for all corresponding fields excluding TSNR. Interest values within the second trip echo are higher than for the Original Test but a threshold of about 0.5 will result in nearly the same algorithm performance in this region. Comparing the Doppler interest fields (MVE, SDVE, and MSW) shows that the Test #4 interest fields have higher interest values than the Original Test, in general. The interest field for TSNR is the same for both tests since the membership function is not changed. Similar to the TSNR interest field, the GDZ interest field (Fig. 2.22e) indicates the presence of AP echo but cannot distinguish between the AP and the second trip echoes. Typically, second trip echo diminishes in intensity with height and will likely be a source of error in GDZ field of the AP Detection Algorithm. However, it is unusual to see much second trip echo within the A2 reflectivity data stream (at least in the cases examined here) and should not be a significant source of algorithm error.
Unthresholded, final algorithm interest fields for all the tests (except the Original Test) are shown in Fig. 2.23. For Test #1 (Fig. 2.23a), modifying the membership functions for MVE and MSW has increased the final interest values over those of the Original Test ( Fig. 2.21a). The second trip echo is correctly identified as a non-clutter target. For Test #2, the GDZ interest field is added but the GDZ membership function is not optimal. The most obvious difference between Test #1 and Test #2 is an increase in interest values within the second trip echo as well as a small, general decrease in interest values within the AP echo. Comparing Test #3 to the Original Test shows that the interest values increase everywhere. Recall that Test #3 uses new membership functions for GDZ and SDVE ( Fig. 2.9). For Test #4, the weights for the MVE and TSNR interest fields are 2 rather than 1 in the computation of the summed weights. The effect of increasing the weights is to better distinguish the second trip echo from the AP echo. Typically, the MVE and TSNR fields provide the best distinction between clutter and non-clutter echoes. Comparing Test #3 (Fig. 2.23c) to Test #5 (Fig. 2.23e) and comparing Test #4 (Fig. 2.23d) to Test #6 (Fig. 2.23f) shows that similar results are obtained but the unsmoothed test output has more small-scale variation, as is expected when unfiltered input fields are used.
To better illustrate the differences in algorithm performance with the various tests, the thresholds determined to be optimal in Section 2.8.1 are applied to the final output of the Original Test ( Fig. 2.24c), the Test #3 (Fig. 2.24d) and the Test #4 (Fig. 2.24e). The mean reflectivity (Fig. 2.24a), MVE (Fig. 2.24b), and the expert truth (Fig. 2.24c) fields are shown for ease of reference in this figure. Further, the summarized case studies discussed in Section 2.8.3 use this figure layout. Duplicating the figure layout for the KAMA scan allows easier comparison of this case to the summarized cases. The thresholds applied to the final algorithm output are 0.4 for the Original Test, and 0.55 for Test #3 and Test #4. Further, the statistical scores for the POD, the FAR and the CSI are summarized in Table 2.11 for each of the scans discussed in Sections 2.8.2 and 2.8.3. Examination of the skill scores for each scan shows the variation in algorithm performance that might be expected. As can be seen in Fig. 2.24, the three tests cover the area of AP fairly well and do not detect the second trip echo as AP. The POD scores for the three tests are 0.713, 0.716 and 0.736, respectively, and shows that the three tests detect the AP echoes consistently. Likewise, the FAR scores are similar and have values of 0.098, 0.096 and 0.099, respectively. For this case, all the algorithm tests have similar statistical performance.
For any evaluation, examining the algorithm performance in regions of non-occurrence of the feature of interest (i.e., the "null" event) is important. For this reason, a precipitation echo is selected for a detailed examination of the APDA performance. Using the same scan as examined above, three convective cells are present to the SW through SE of the KAMA radar ( Fig. 2.25a) and have maximum reflectivity values in excess of 55 dBZ. A gust front has been produced as evidenced by the strong positive radial velocities to the south of and within the cells (Fig. 2.25b). The TSNR field (Fig. 2.25c) shows generally low texture of SNR is present with values generally < 15. In regions of strong reflectivity gradients such as at the edges of cells, the TSNR values can exceed 45. The SDVE field (Fig. 2.25d) has values > 1 m s -1 . The vertical difference field (Fig. 2.25e) shows that the precipitation cells have small vertical differences in reflectivity within the heaviest precipitation. Notice that the vertical differences are greatest at the edges of the precipitation echoes where reflectivity at 0.5 deg is < 20 dBZ. The MSW (Fig. 2.25f) has high values that are > 2 m s -1 within the most intense reflectivity regions of the storms.
Final interest output from the Original Test is shown in Fig. 2.26 with the expert truth field. Within the regions of highest reflectivity ( Fig. 2.25a), the output of the AP Detection Algorithm (Fig. 2.26a) shows interest values of < 0.4, typically. If a threshold of 0.4 is used to determine the presence of AP clutter (as suggested in a previous section), the algorithm output suggests that the convective cells have no AP clutter and do not require additional clutter filtering. Small, scattered regions of AP clutter are identified near the westernmost cell (between azimuths 180 deg to 270 deg). The final algorithm output (2.20a) identifies some of the expert-identified regions as AP clutter. In the region to the east and northeast of the strong cells and to the east of the gust front in a region of weak radial flow, interest values are > 0.5 due to the MVE values being near 0 m s -1 (Fig. 2.25b), the low values of the MSW field (Fig. 2.25f), and the low values of the SDVE field (Fig. 2.25d). A small region of AP clutter is near 125 deg azimuth and 100 km range but the majority of the algorithm output in excess of 0.5 interest in the eastern quadrant must be considered a false alarm on the part of the algorithm.
Examination of the output from the Test #4 version of the membership functions ( Fig. 2.27) shows similar results as the Original Test. Interest values are slightly higher in the region of weak flow to the east of the gust front. The interest fields for MVE (Fig. 2.27b), SDVE (Fig. 2.27d), and MSW (Fig. 2.27f) have typically higher values than those of the Original Test, and contribute to the higher interest values within the final output field in this region. The addition of the GDZ interest field (Fig. 2.27e) supports the expert determination that most of the region shown in the figure is not clutter. The GDZ interest values tend to be highest along the edges of precipitation regions and in regions of clear air return near 260 deg azimuth and 60 km range.
Unthresholded, final output from all the algorithm tests are shown in Fig. 2.28. Comparing Test #3 output (Fig. 2.28c) to Test #4 output (Fig. 2.28d), the most noticeable difference is in regions where the MVE field is not near zero velocity. Notice for the region near 150 deg azimuth and 25 km range, the interest values of Test #4 are lower than those of Test #3 by about 0.1. This occurs despite the high contribution from the SDVE and MSW fields. Likewise, the clear air region to the west of the radar (270 deg azimuth and 65 km range) has lower interest values in Test #4. In the weak flow region to the east of the radar, both Test #3 and Test #4 have similar interest values. Looking at the unsmoothed interest fields of Test #5 (Fig. 2.28e) and Test #6 (Fig. 2.28f), shows that similar results are obtained when compared to Test #3 and Test #4 but more spatial variance occurs in the interest field due to not filtering the input fields.
Applying the thresholds of 0.4 to the final output fields of the Original Test ( Fig. 2.29d), and of 0.55 to the final outputs of both Test #3 (Fig. 2.29e) and Test #4 (Fig. 2.29f) allows comparison of the statistical performance for these tests. As noted in the discussion above, the POD, FAR and CSI scores for this scan are very similar for the three tests. Differences seen in the precipitation regions (that would be considered false detections) are small between the three tests. Within the weak flow area to the east of the radar, Test #4 appears to have the fewest number of false detections while the Original Test has the most number of false detections. These visual inspections are somewhat subjective, however.
In this section, scans from the various radars are shown to illustrate algorithm performance in a variety of weather situations. In some cases, clear air return is the predominant echo, in others precipitation is predominant, and others have mixed amounts of return. The amount of AP contamination varies with each case from no echoes to an extensive amount of echoes. Likewise, the amount of second trip echo (leading to missing data regions in the radial velocity field) varies from scan to scan. For these cases, plots of the three algorithm tests (with appropriate threshold applied) are shown to give a quick overview of the AP Detection Algorithm performance within each situation. Results from the Original Test, Test #3 and Test #4 are shown along with the reflectivity data, radial velocity data and the expert truth field. Statistical scores for the individual scans are given in Table 2.11 for the POD, FAR and CSI. Further, a ratio is given for the number of gates of the MVE field versus the number of gates of the TNSR field to examine the effect that missing MVE data may have on the algorithm performance. Likewise, a ratio is given for the number of gates having AP clutter versus the number of gates of not-AP clutter (i.e., clear air or precipitation return) as determined from the expert truth field to examine what effect, if any, the particular atmospheric conditions may have on the algorithm performance.
In Table 2.11, the MVE/TSNR ratio is included since the performance of the algorithm is dependent on the number of fields that are input into the final output of the AP Detection Algorithm. Typically, more of the MVE field has missing gates than the TSNR field due to thresholding differences and second trip echo removal. Values of the ratio do not exceed 100% for the cases shown, indicating that there are always more TSNR range gates of data than MVE range gates. When the MVE field is missing from a gate, the MSW and SDVE fields are also missing. When the TSNR field is missing from a gate, the GDZ field is also missing. Of the 14 cases listed in Table 2.11, seven have MVE/TSNR ratio values >80% while seven have ratio values <80%. Looking at the Test #4 results, of the seven cases with ratio values <80%, six have FAR scores >0.10 indicating that the algorithm does not perform well when there are many missing gates from the three Doppler fields. Of the seven cases with ratio values >80%, only four have FAR scores >0.10. Likewise, five of the seven cases with ratio values <80% have POD scores <70% while only four of the seven cases with ratio values >80% have POD scores <70%. Missing data does not explain all of the performance differences between cases, however, it can be an important contributor and must be taken into account. While the number of case studies is small and may not be statistically significant, it seems intuitively obvious that the APDA will perform best when all input fields are available.
The ratio of the number of range gates declared as AP clutter echo to the number of range gates declared to be not-AP clutter echo (i.e., precipitation or clear air echo) is also included in Table 2.11. This AP/not-AP ratio shows that the three cases having the least amount of AP echo are the three cases with the lowest POD scores (<0.35) overall. These cases are discussed in Sections 126.96.36.199, 188.8.131.52 and 184.108.40.206. In these cases, the regions of AP clutter tend to occur in very small patches, rather than in a large region of nearly homogeneous clutter. It may be that the APDA performs best after a minimum amount of area of AP contamination is attained. If true, quantification of the minimum amount of area is not known at this time. The 19 October KLOT case at 0011 UTC (Section 220.127.116.11) has a POD of 0.337 and an MVE/TSNR ratio of 91.1% indicating that missing data are not contributing much to the low performance of the POD. The AP/not-AP ratio is 3.2% which is quite low and shows that the number of gates having AP return is few. Examination of the data from this case shows that the AP is occurring in relatively small patches within a large region of clear air return and may be contributing to the low POD score.
A case of moderate intensity precipitation (mean reflectivity values of about 35 dBZ are indicated in Fig. 2.30a) that has embedded AP return is examined using data from the St. Louis, MO WSR-88D (KLSX) on July 7, 1993 at 0334 UTC. The mean radial velocity (MVE) field (Fig. 2.30b) has values near 0 m s -1 within the region of AP as determined by the human expert (Fig. 2.30c). Thresholded interest output from the three tests shows a generally good detection of the main AP region with small regions of missed detections to the NE of the main body of AP return (i.e., near 60 deg azimuth and 140 km range). Notice also that the 0 m s -1 line that is oriented from west to east through the radar position also shows high interest values, some of which are false detections. Comparing the output from the Original Test (Fig. 2.30d) and from the Test #3 (Fig. 2.30e) to the output from Test #4 (Fig. 2.30f) shows that Test #4 has slightly fewer regions of false detections, especially in the area of the 0 m s -1 line through the radar position. The POD scores for the Original Test, Test #3 and Test #4 are 0.419, 0.435 and 0.460, respectively, showing that Test #4 has the best POD performance by a small amount. The FAR scores are 0.085, 0.059 and 0.056, respectively. The CSI score of the Original Test is 0.228 versus a score of 0.294 for Test #4.
A line of convective cells oriented north-south and having moderate to high intensity in reflectivity ( Fig. 2.31a) is found with clear air return and an isolated, nearly homogeneous region of AP return (Fig. 2.31c) in a case from the Oklahoma City, OK WSR-88D (KTLX) on July 1, 1994 at 1106 UTC. The MVE field (Fig. 2.31b) shows that the AP return has near 0 m s -1 values while the convective cells have regions of positive, negative and near-zero values. Thresholded output from all tests show good detections of the region of AP return. All the algorithm tests have some false detections within the western edges of the precipitation cells, especially near 330 deg azimuth and 65 km range. These false detections are due to contributions from all the individual interest fields (not shown) as derived from the MVE, TSNR, GDZ, SDVE and MSW fields. Further, a small region of high interest values near 350 deg azimuth and 25-50 km in range is created due to the absence of the Doppler fields (see Fig. 2.31b). The Doppler fields were removed due to the presence of second trip echo. When a field or fields are missing, the AP Detection Algorithm will use the remaining interest fields to compute the final interest value. Comparing Test #3 (Fig. 2.31e) to Test #4 (Fig. 2.31f) shows that the Test #4 had slightly better performance with fewer false detections in the precipitation region.
For this scan the POD scores for the Original Test, Test #3 and Test #4 are 0.781, 0.684 and 0.699, respectively, showing that the Original Test has the best POD performance. The FAR scores are 0.073, 0.065 and 0.069, respectively showing that Test #4 has the best FAR performance although the differences in the scores is slight. The CSI scores for the three tests are 0.320, 0.317 and 0.314, respectively, and show that the three versions of the AP Detection Algorithm have similar performance scores for this case.
Weak precipitation echo ( Fig. 2.32a) is embedded within clear air and AP return in a case from the Houston, TX WSR-88D (KHGX) on October 19, 1994 at 0530 UTC. Regions of AP echo are prevalent in the area of data shown. As can be seen in the figure, the maximum range of the first trip has been reached (near 150 km), leading to a discontinuity in the mean radial velocity (Fig. 2.32b) field. As mentioned above, when an interest field is not available for a particular range gate, the weighted sum containing the final output interest value of the AP Detection Algorithm is computed using as few as 1 field. In this example, the near-complete loss of the three Doppler fields (MVE, SDVE, and MSW) at ranges > 150 km have created a discontinuity in the algorithm output that is visible in all three tests (Figs. 2.32d-f). This discontinuity is characterized as an increase in the interest values from about 0.7 to 1.0. The Original Test (Fig. 2.32d) appears to have the poorest performance with many missed detections in the AP regions. The POD scores for the Original Test, Test #3 and Test #4 are 0.450, 0.586, and 0.589, respectively and verifies the visual inspection that the Original Test has more missed detections. However, the Original Test has the lowest FAR score of 0.096 compared to 0.144 and 0.160 for Tests #3 and #4, respectively. The CSI scores for the three tests are 0.402, 0.496 and 0.490, respectively. The addition of the interest field for GDZ (not shown) and the changes to the SDVE membership function are the likely explanations for the improvement in POD performance for Test #3 and Test #4.
A dissipating squall line is oriented SW-NE in a case from the Dodge City, KS WSR-88D (KDDC) on July 13, 1993 at 0655 UTC ( Fig. 2.33). The AP echoes form behind the squall line after the passage of the gust front at the radar. The gust front is located around 150 deg and 40 km at this time. As seen by the expert truth field (Fig. 2.33c) much of the echo is from AP clutter return that is either embedded within the precipitation return or is occurring in nearly homogeneous patches behind the squall line. Performance of the three algorithm tests (Fig. 2.33d-f) show that all algorithms had similar performance in the region of AP echoes and detect the AP reasonably well. They have similar poor performance in the region of near 0 m s -1 mean radial flow (Fig. 2.33b) located to the south and southeast of the radar and along the leading edge of the gust front where small areas of false detections are present. For this scan, the POD scores are 0.740, 0.680 and 0.701 for the Original Test, Test #3 and Test #4, respectively. The Original Test has the highest POD score, although the range of values is not large. FAR scores for the three tests are 0.126, 0.124 and 0.123, respectively, and the CSI scores are 0.394, 0.382 and 0.397, respectively.
A dissipating squall line with a minimal amount of AP echo is shown ( Fig. 2.34) from the Chicago, IL WSR-88D (KLOT) on October 19, 1995 at 2331 UTC. This case has an AP/not-AP ratio of 1.9%. The near-surface wind flow is from the WNW. A gust front has passed the radar location and is currently located SE of the radar near 120 deg azimuth and 30 km range. Except for a few small regions within 50 km of the radar that are AP return (Fig. 2.34c) most of the echo is from precipitation or clear air. Within the squall line area, all three algorithm tests produce false detections near the edges of the precipitation echoes. These false detections are likely the result of the contribution from the TSNR interest field (not shown) and to a lesser extent, the SDVE interest field (not shown). Within 50 km of the radar in the region of near-zero MVE values, the Original Test appears to have the best performance with the fewest number of false detections. The GDZ interest field (not shown) is the source of the decreased performance of the Test #3 and Test #4 in this region. Regions of clear air return generally have elevated GDZ interest values due to the decrease in reflectivity values with height that is typical of this type of return. For the overall scan, the POD scores are 0.209, 0.295, and 0.336 for the Original Test, Test #3 and Test #4, respectively, and shows that Test #4 has outperformed the Original Test by about 13% despite the false detections within the clear air. The FAR scores are 0.095, 0.116 and 0.107, respectively, and show that all tests had similar performance with false detections. The CSI scores are very low and are 0.047, 0.054 and 0.066, respectively. The poor POD performance of the APDA may be explained by the low AP/not-AP ratio and by the low value of the MVE/TSNR ratio (63.3%) that indicates significant amount of missing Doppler channel data.
An example of stratiform precipitation is examined with the Denver, CO WSR-88D (KFTG). For this case on September 21, 1995 at 0206 UTC, the widespread precipitation is occurring within stratiform echo having maximum reflectivity of about 25-30 dBZ ( Fig. 2.35a). Wind flow is from the NE near the surface with a well-defined regions of 0 m s -1 visible in the MVE field (Fig. 2.35b). The expert truth field (Fig. 2.35c) shows clutter residue regions near the mountains, a few, small regions of AP clutter, and mostly the not-clutter, precipitation areas. The AP/not-AP ratio has a value of 0.5% while the MVE/TSNR ratio has a value of 78.3%. For the three algorithm tests, all produce false detections within the 0 m s -1 MVE regions. The three algorithm tests do not perform very well with this case. The interest fields from the MSW and SDVE fields (not shown) are the strongest contributors to poor algorithm performance when coupled with regions of near-zero MVE values. For this case, the SDVE field (not shown) is characterized by having small values over most of the area shown in Fig. 2.35 as does the MSW field (not shown). Low values of SDVE and MSW are used as indicators for the presence of AP return in the APDA. The GDZ interest field (not shown) varies in intensity depending on the vertical structure of the stratiform echo and is not a consistent contributor to the poor algorithm performance. The POD scores for the three tests are 0.350, 0.206 and 0.248 for the Original Test, Test #3 and Test #4, respectively. However, the high POD for the Original Test is a bit misleading since it also has the highest FAR score. The FAR scores are 0.239, 0.151, and 0.087, respectively, and show that high numbers of false detections are occurring with the Original Test. The CSI scores are 0.004, 0.006 and 0.011, respectively.
This case is very illustrative and shows that the AP Detection Algorithm may not perform as well in stratiform precipitation, at least in this case. More stratiform precipitation cases are clearly needed for additional testing and evaluation.
Clear air return and a large, nearly homogeneous region of AP echo is examined in a case from the St. Louis, MO WSR-88D (KSLX) on July 24, 1993 at 1009 UTC ( Fig. 2.36). For this case, the reflectivity values within the AP echo are about 55 dBZ (Fig. 2.36a) to the SW of the radar location. Clear air return is present, as seen in the reflectivity field and in the mean radial velocity field (Fig. 2.36b). Much of the MVE field is removed to the N of the radar due to the presence of second trip echo. The truth field (Fig. 2.36c) shows that the echo to the SW is AP clutter return. The three algorithms produce similar results with good detections of the AP return. In the region of missing MVE values to the N of the radar, the combination of a scattering of high TSNR values (not shown) coupled with the high GDZ values that are characteristic of clear air return has created small regions of false detections for all three algorithm tests. Overall, the three tests have similar POD performance with scores of 0.725, 0.719 and 0.718 for the Original Test, Test #3 and Test #4, respectively. The FAR scores are 0.133, 0.142 and 0.138, respectively, and the CSI scores are 0.519, 0.511 and 0.515, respectively.
Another clear air and AP clutter returns case ( Fig. 2.37) is taken from the KLSX WSR-88D on July 11, 1993 at 0131 UTC. Comments made regarding the performances of the three algorithm tests for the case discussed in Section 18.104.22.168 are also valid for this case. The MVE/TSNR ratio is 76.4% ( Table 2.11) and shows that a considerable amount of the Doppler fields are missing. Except for the region near the radar (< 50 km range) that is clear air return, most of the radar return has been determined to be AP clutter (Fig. 2.37c). For this case, the Original Test has the best performance with the highest POD (0.744) and the lowest FAR (0.164). Test #3 and Test #4 have similar performance with POD scores of 0.658 and 0.666, respectively, and FAR scores of 0.191 and 0.207, respectively. The large number of missing Doppler fields is a likely contributor to the poor performance of the algorithms with respect to the FAR scores.
A region of AP clutter is embedded within clear air return ( Fig. 2.38) for a case selected from the KLOT WSR-88D radar on October 18, 1995 at 1005 UTC. For this case, Test #3 and Test #4 have the highest POD scores of 0.513 and 0.532, respectively. The Original Test has a POD score of 0.453. However, the Original case has the lowest FAR score of 0.065 with Test #3 having a score of 0.090 and with Test #4 having a score of 0.103. Differences in FAR scores are small, however. The CSI scores are 0.401, 0.433 and 0.439 for the Original Test, Test #3 and Test #4, respectively. For this case, the MVE/TSNR ratio is 95.2% indicating that there are few missing gates of data for the Doppler fields.
A case of clear air return with embedded AP contamination in a weak wind flow pattern is examined using data from the Memphis, TN WSR-88D (KNQA) radar on July 6, 1997 at 0454 ( Fig. 2.39). This case was included in the data collection for the test of the Archive 1 Data Acquisition (A1DA) unit described in the FY-97 annual report (Keeler et al., 1998b). As shown by the expert truth field (Fig. 2.39c), AP echo is prevalent with a few areas of clear air return. The three algorithms have fairly good POD scores of 0.689, 0.704, and 0.721 for the Original Test, Test #3 and Test #4, respectively. The FAR scores are rather high with scores of 0.135, 0.241 and 0.250, respectively. And finally, the CSI scores are 0.651, 0.638, and 0.652, respectively. The higher FAR scores for the Test #3 and #4 are likely due to the GDZ interest field (not shown) that has high interest values in the clear air return. The MVE/TSNR ratio is 97.2% and indicates that missing data from the Doppler fields is not contributing significantly to the high FAR scores.
For this case from the Sterling, VA WSR-88D (KLWX) on August 17, 1997 at 0958 UTC, clear air return is found within about 50 km of the radar ( Fig. 2.40) with AP and suspected NP ground clutter returns embedded within clear air return or occurring within isolated, nearly homogeneous regions. The suspected NP returns are identified as AP in the expert truth field. A weak squall line is approaching from the NW ( Fig. 2.41) that has maximum reflectivity values of about 35 dBZ. This squall line is straddling the first trip range of the velocity scan and has regions of missing data within the Doppler fields at farther ranges. Two portions of this scan of data are examined.
For the first example (Fig. 2.40), AP and NP clutter regions are occurring near and within the clear air return. Significant portions of the Doppler fields are missing in the NW quadrant (at ranges < 25 km) due to the removal of second trip echo originating from the approaching squall line. The MVE/TSNR ratio ( Table 2.11) is 78.4% for the entire scan. The missing Doppler fields will mean that the final outputs of the AP Detection Algorithm tests will use only the TSNR and GDZ fields to determine the likelihood of clutter contamination in this area. As is seen in the three algorithm outputs (Fig. 2.40d-f), this region has a significant number of false detections due to the missing Doppler fields.
For the squall line example (Fig. 2.41), the missing regions of the Doppler fields that are created at the limits of the first trip range again means that the final outputs of the algorithm will depend on only the TSNR and GDZ fields. However, for this example, the missing Doppler fields do not contribute to false detections, indicating that the interest fields derived from the TSNR and GDZ fields are able to discriminate precipitation from AP return, for this case. The three algorithms perform very well in the squall line, with Test #3 having the most false detections.
Overall for this scan, the POD scores are 0.581, 0.539 and 0.586 for the Original Test, Test #3 and Test #4, respectively, and indicates that the three tests had similar performances. Likewise, the FAR scores are 0.181, 0.176 and 0.193, respectively, and the CSI scores are 0.380, 0.373 and 0.393, respectively. Likely, the missing regions of the Doppler fields contributed to the reduced performance of the algorithm.
An example of conditions having mostly clear air return with very small patches of AP clutter is shown in this case from the KLOT WSR-88D on October 19, 1995 at 0011 UTC ( Fig. 2.42). Reflectivity values are generally low with maximum values of about 20 dBZ in the clear air (Fig. 2.42a). Some NP clutter return is present near the radar that has higher reflectivity values. The MVE field (Fig. 2.42b) shows that the winds are from the NE, with a 0 m s -1 line running approximately NNW to SSE through the radar location. In addition, some MVE values are reduced near the shoreline of Lake Michigan (near 60 deg azimuth and 50 km range). The expert truth field (Fig. 2.42c) shows that most of the return is from clear air, with a few, scattered regions of AP clutter present. As discussed above, this case has a low MVE/TSNR ratio of 63.3% ( Table 2.11) indicating that significant regions will use only the interest fields from the TSNR and GDZ fields to determine the presence or absence of AP clutter. The POD scores are quite low at 0.314, 0.299, and 0.337 for the Original Test, Test #3 and Test #4, respectively. However, FAR scores are low with values of 0.030, 0.033 and 0.045, respectively. For this case, the AP Detection Algorithm is having trouble finding the AP return but is not falsely detecting return. The small areas containing the AP return (AP/not-AP ratio is 3.2%) may also be a contributor to the poor algorithm performance. CSI scores are 0.109, 0.117, and 0.108, respectively, and reflect the low POD scores.
The last case is from the Denver, CO WSR-88D (KFTG) and occurred on January 11, 1995 at 1548 UTC. This case is an example of NP clutter ( Fig. 2.43) and is included to illustrate that the AP Detection Algorithm is able to detect NP clutter. Clutter problems in and around the Rocky Mountains can be severe as is easily seen in Fig. 2.43. The expert truth field (Fig. 2.43c) has defined all the NP clutter as AP clutter to allow algorithm testing. The MVE/TSNR ratio is 91.5%, indicating few problems with missing Doppler fields. The three tests (Fig. 2.43d-f) show good skill with this case, with POD scores of 0.866, 0.712 and 0.797, respectively, for the Original Test, Test #3 and Test #4. The FAR scores are the lowest for Test #3 (0.025) and Test #4 (0.051) while the Original Test had a score of 0.111. CSI scores for the three tests are 0.856, 0.710 and 0.790, respectively. The APDA performance is quite good for this case.
The statistical results presented in the sections above show that the AP Detection Algorithm has variable skill in detecting AP clutter and performs best when the AP clutter occurs in isolated, nearly homogeneous patches over a substantial area. Missing data in the Doppler fields (MVE, SDVE, and MSW) contribute to poor algorithm performance. Various schemes for an optimized algorithm have been presented and analyzed. The Test #4 membership functions show the best statistical performance when the 60 scans are considered; however, the statistical improvement over the Original Test is not very much. For many of the summarized cases, the statistical scores were within a few percent over the various tests.
The algorithm performs especially well when the AP is occurring in isolated patches that are not contained within precipitation or clear air return. With mixed AP and precipitation or mixed AP and clear air return, the algorithm performs well in those areas where the radial velocity is significantly biased towards 0 m s -1 and less well when the radial velocities are biased low by clutter return but are not 0 m s -1 . Characterization of the type of AP return (isolated versus embedded) or the areal extent of the AP return may help further explain algorithm performance.
Several cases of convective cells having no AP clutter contamination were shown. The APDA did not falsely identify most of the precipitation echo as AP; however, the edges of the echoes were sometimes identified as AP clutter.
The algorithm performs best when all five input interest fields (MVE, TSNR, SDVE, MSW and GDZ) are present. Scans with a significant amount of missing data from the Doppler channels typically did not perform as well as scans that had all the data fields. The Doppler channels boost algorithm performance significantly.
Additional analysis is needed for stratiform precipitation cases. One case was presented in which the AP Detection Algorithm did not perform particularly well. Additional cases are needed to determine if there is a systematic weakness in the APDA membership functions. This case may be an anomaly but additional analysis is needed to ascertain this possibility.
The APDA can also detect NP ground clutter, as shown with a case from Denver, CO. Therefore, if a region of NP clutter is missed on the bypass map for some reason (perhaps a building is constructed after the clutter filter map was made), the APDA can detect this region of clutter and provide additional quality control for removing clutter contamination.
The work described in this section is in the category of "fine-tuning" the AP Detection Algorithm and will not delay or impede the implementation of the AP Clutter Mitigation Scheme described in Section 1.
Much progress has been completed in the optimization of the AP Detection Algorithm. However, a small amount of work still remains to be accomplished. A detailed analysis of the contribution of each individual interest field (i.e., the TSNR, MVE, SDVE, GDZ, and MSW fields) to the algorithm performance must be done and may lead to small changes in the membership functions shown in Fig. 2.9.
Determination of the optimum weights for the membership functions remains to be accomplished. Research by Jeffrey Keeler during his recent sabbatical should allow for determination of the appropriate weights by an objective method. Currently, weights are determined by a subjective, trial-and-error approach. This method is obviously less desirable than having an objective approach to choose the optimum weights.
The AP Detection Algorithm uses a mean filter over the specified region to compute the derived mean fields (i.e., the mean radial velocity, mean spectrum width). For radar data, the removal of outliers prior to computation of averages typically leads to better algorithm results. This can be accomplished by using a median filter instead of a mean filter. Substitution of a median filter for the current mean filter will be done and the results evaluated during FY-99.
Historically, the TSNR field has been used in the APDA. For the operational implementation of the APDA within the WSR-88D system, the texture of the reflectivity (TDZ) may be a more appropriate field to use because it is within the A2 data stream. Substituting TDZ for TSNR is not expected to adversely impact the statistical performance of the algorithm. However, a quick test of this hypothesis is needed using the 60 scan data set.
After the Build 10 was completed, the clutter filter "bypass" maps used by the WSR-88D systems are being written to Exabyte tape. To examine clutter residue effects and to determine its statistical characteristics, knowing where the clutter map has been applied is important information. A process to read the bypass maps from the Memphis KNQA in-house tapes is needed and will be completed in FY-99. Further information regarding the clutter residue characterization work is contained in Section 4.
A fuzzy logic, hydrometeor and non-hydrometeor particle identification (PID) algorithm has been developed at NCAR using the S-Pol dual-polarimetric base data and derived fields (Vivekanandan et al., 1998). With the PID, ground clutter is easily identified using the standard deviation of differential reflectivity ( ), the standard deviation of differential phase ( ) and the correlation coefficient between the horizontal and vertical copolar received signal (). Figure 2.44 shows an example of data observed with S-Pol near Boulder, CO on a clear day, and the resulting classification of the targets. The ground clutter from the surrounding terrain is successfully identified as clutter. Likewise, the clear air return from insects is correctly identified. The AP Detection Algorithm (APDA) can be run on S-Pol data and the results evaluated using the polarimetric classification as an independent truth field.
Using the PID-identified ground clutter removes some of the ambiguity in the APDA results as introduced by the human expert and by the limitations of the WSR-88D data. Within the PID, ground clutter presence is determined from radar fields that are not used in the APDA and will serve as an independent test. Also, the S-Pol radial velocity and reflectivity data are taken simultaneously, using the same radial beam and gate spacing. This is a significant improvement over the WSR-88D data where temporal and spatial differences exist between the radial velocity and reflectivity data. Another benefit of using S-Pol data is that classification is done on a gate-by-gate basis by an objective method. While the human experts were quite diligent during their truthing efforts, drawing polygons containing AP clutter on a gate-by-gate basis is difficult and is made more difficult by the temporal and spatial differences between the reflectivity and radial velocity fields that is inherent to the WSR-88D data. An objectively-determined truth field is a vast improvement over using human experts.
Donaldson, R.J., R.M. Dyer, and M.J. Kraus, 1975: An objective evaluator of techniques for predicting severe weather events. Preprints, 9th Conference on Severe Local Storms, Norman, OK, Amer. Meteor. Soc., 321-326.
Ellis, S.M., F. Pratte, and C. Frush, 1999: Compensating Reflectivity for Clutter Filter Bias in the WSR-88D. Preprints, 15th Int'l Conf. Inter. Info. Proc. Sys. (IIPS) for Meteor., Hydro. Ocean., AMS, Dallas, TX, 11-15 Jan. 1999.
Keeler, R.J., F. Pratte, D. Ecoff, J. VanAndel, and D. Ferraro, 1998a: NEXRAD anomalously propagated ground clutter mitigation. Preprints, 14th Int'l Conf. Inter. Info. Proc. Sys. (IIPS) for Meteor., Hydro. Ocean., AMS, Phoenix, AZ, 11-16 Jan. 1998.
Keeler, R.J., F. Pratte, D. Ecoff, J. VanAndel, D. Ferraro, C. Kessinger, and S. Ellis, 1998b: WSR-88D data quality optimization. FY-97 Annual Report , submitted to the NOAA WSR-88D Operational Support Facility.
Keeler, R.J., C. Kessinger, J. VanAndel, and S. Ellis, 1999: Implementation of NEXRAD AP clutter processing. Preprints, 15th Int'l Conf. Inter. Info. Proc. Sys. (IIPS) for Meteor., Hydro. Ocean., AMS, Dallas, TX, 11-15 Jan. 1999.
Saffle, R.E., and L.D. Johnson, 1998: NEXRAD product improvement: An overview of the continuing program to evolve the WSR-88D system to an Open Systems Architecture, Preprints, 14th Int'l Conf. Inter. Info. Proc. Sys. (IIPS) for Meteor., Hydro. Ocean., AMS, Phoenix, AZ, 11-16 Jan. 1998.
Smith, J.A., M.L. Baeck, M. Steiner, B. Bauer-Messmer, W. Zhao, and A. Tapia, 1996: Hydrometeorological assessments of the NEXRAD rainfall algorithms. Final Report to the NOAA National Weather Service, Office of Hydrology-Hydrologic Research Laboratory, Silver Spring, Maryland
Vivekanandan, J., D.S. Zrnic, S. M. Ellis, R. Oye, A.V, Ryzhkov, and J. Straka, 1999: Cloud microphysics retrieval using S-band dual-polarization radar measurements. 15th Int'l Conf. Inter. Info. Proc. Sys. (IIPS) for Meteor., Hydro. Ocean., AMS, Dallas, TX, 11-15 Jan. 1999.