logo (17413 byte gif)


Bulletin No. 9

STANDARD OUTPUT DATA PRODUCTS
FROM THE NCAR RESEARCH AVIATION FACILITY


  1. Introduction

    This document describes the standard aircraft data output product furnished to each Principal Investigator (PI). Appendix A has a list of names and detailed descriptions for most of the standard measurements made aboard RAF aircraft. Typically, data for a research project will contain a subset of this list plus measurements made by user-supplied and/or experimental instruments. Appendix B describes how RAF makes many of its derived calculations. Appendix C briefly describes the vapor pressure enhancement factor (mentioned in Appendix B). Appendix D gives information on netCDF. Appendix E describes our now-obsolete GENPRO data format. At the end of this document is a list of references.

  2. Brief Description of the Standard Output Products

    The specific content of these products varies as a function of the type of research project flown but typically will contain the following items, if available:

    The Project Documentation Summary contains information to aid the PI in interpreting the production data. It may include, but is not limited to:

    The Final Production Data Set is written to data files in netCDF format (using the Unidata-registered RAF conventions). RAF aircraft gather raw data which are recorded onto removable hard disk drives by an on-board data system (ADS). (See brief description in Section V below.) Following the project's field phase, these raw data are processed using the RAF-developed NIMBUS software (See brief description in Section VI below.) to produce "production" data (with calibrated values in engineering units) having the highest-quality that RAF can provide. These data can be delivered to the user as MSS bitfiles, via the internet, on magnetic tape or on other available media.

    Video Recordings, if obtained during a project from cameras aboard RAF aircraft, presently are in VHS or S-VHS format depending on aircraft.

  3. Data Description

    RAF classifies projects (and the data sets produced from them) as either low-rate turbulence (LRT) or high-rate turbulence (HRT). For both LRT and HRT projects RAF produces an LRT data set (using the RAF-developed NIMBUS software) having one (1) sample-per-second (sps) values centered at mid-second. For HRT projects RAF produces a second data set with mixed output rates: fast instruments at the designated high rate, presently 25 sps, corresponding to a Nyquist frequency of 12.5 Hz; and slower instruments at the designated low rate, presently 1 sps. (As an option, selected raw variables can, instead, be output at their original, sampled rate.) The LRT and HRT data are both suitable for turbulence calculations (the HRT data for flux calculations, as well).

    All data sets provided by RAF contain basic meteorological and aircraft measurements, usually a subset of the variables listed in Appendices A and B. In addition, PIs may get data from PMS cloud microphysics probes, experimental instruments and/or user-supplied instrumentation.

    Data sets collected from different projects can vary in several respects: the sample rates used, the type of wind data obtained, the research instrumentation used, etc. A project investigating fluxes in the boundary layer would require high-rate pressure, temperature, moisture, and three-dimensional wind data. A project requiring less spatial or temporal resolution, such as a study of the flow in and around a developing cumulonimbus, could use low-rate data, but still require measurement of the three-dimensional wind components. An air chemistry sampling project might only require mean horizontal wind data. Thus, depending on the needs of a project, different types of data sets are obtained, and different processing is required. (Please note that HRT processing can have a significant impact on RAF's resources.)

    In addition to the standard variable data set, cloud microphysical data obtained from PMS probes are available. Both one- and two-dimensional probes are flown on RAF aircraft. Multiple probes can be flown at the same time; the total number varies with each aircraft. Information regarding the variables recorded from the PMS probes and the available data output formats can be found in Appendix B. Refer to Bulletin No. 24 for more detailed probe information.

  4. Special Requests for Data

    The standard output product from a project is designed to provide as much readily-usable information as possible to the PI. If the format and content of this standard product are insufficient for a given project, the PI must make a request to RAF for additional or special processing. Such a request may or may not be accommodated, depending upon its impact on RAF's workload. This type of request is handled separately from the request for aircraft support.

    RAF's Project & Data Support Group (PDSG) has an established policy regarding requests for special processing and for changes to standard formats. A request must be reviewed, finalized, and documented no later than two weeks prior to the start of a project. Thus, a lead time of three to four weeks prior to the start of a project is recommended for submitting such a request. Factors considered when reviewing a request for additional data processing include: the amount of new software to be written, the extent of requested changes to existing software, and the impact on computer resources and staff.

    All requests for special processing should initially be made to the RAF's Project Manager. Simple requests, such as including an additional derived variable or recording and processing inputs from user-supplied equipment, can be handled and approved by the Project Manager. More involved requests, such as a significant addition to the standard processing software, will be reviewed by the PDSG.

  5. Brief Description of RAF's Airborne Data System (ADS)

    Data acquisition and display aboard the aircraft are handled by the second-generation aircraft data system (ADS2). Several Distributed Sampling Modules (DSMs) are located at strategic points around the aircraft. Each DSM collects data from the sensors and instruments near its location. The DSMs transmit their data to a central DSM control computer via a single 10 base-T Ethernet.

    The control computer collects data from each of the DSMs and integrates the data into binary records. These records are then written redundantly to large-capacity hard disk drives. The data are also passed to a display computer network via a second Ethernet port. Within the aircraft, the data-collection and display networks are physically isolated from each other.

    Each DSM is capable of sampling up to 64 analog channels. All channels are synchronously sampled at the fixed rate of 10K sps by individual 16-bit sigma-delta analog-to-digital (A/D) converters on each channel. A digital signal-processing (DSP) board controls the analog sampling and collects data from the A/D converters. The DSP then digitally filters and decimates the analog data to generate the desired sample rate and to reduce possible aliasing error from higher-frequency noise. Standard sample rates of 5, 25, 250 and 1000 sps may be selected for each channel. (Other special sample rates may be programmed, if warranted.)

    Other digital and instrument-specific interfaces, such as the PMS probes (1D, 2D and grey-scale), serial and parallel I/O, ARINC, syncros, counters, etc., are provided by VME-Bus-based interface cards.

  6. Brief Description of RAF's NIMBUS software

    NIMBUS is the name given to RAF's data-processing software. Its main features are:

    Note: Low-rate data are computed by simple average of the high-rate data over each elapsed second. Each resulting value will be centered in time at mid-second even though the time stamp will not indicate it as such. For example, all the samples of measurement X at time 10:03:34 will be averaged to one sample whose time really synchronizes to 10:03:34.5, but its time label will remain 10:03:34.)

  7. RAF's Data Archive

    RAF uses the CISL Mass Store System (MSS) as its primary archive. Data from all projects since about 1978 are present. In addition to the final production data, RAF also archives support files and the software used to produce these data sets, as well as images of the raw data. The MSS has proven to be extremely reliable.

    The data acquired from July 1993 to the present exist in netCDF format. These data have been archived on the MSS using a "transparent" format. Since netCDF files are self-documenting, there is only one file per data set. RAF has registered its specific netCDF NIMBUS conventions with Unidata. They can be found at the URL http://www.eol.ucar.edu/raf/Software/netCDF.html.

    The data acquired from 1983 through March 1993 exist in GENPRO-II format. During that time period, RAF's GENPRO-II processor was run on a Cray Research Corporation computer system (Cray-1, Cray YMP). In general, the format for a data set created with this system consisted of a header file written in ASCII (800 bytes per record) and its associated data file written with 32-bit, positive integers. (The header information explains how to decode the data records to recover the measured values in engineering units.) These data were originally archived on the MSS using a COS-blocked format to preserve record and file boundaries.

    All data acquired between 1975 and 1983 exist in GENPRO-I format. During that time period, RAF's GENPRO-I processing software was run on a Control Data Corporation Model 7600 (CDC-7600) computer system. In general, the format for a data set created with this system consisted of a header file written in CDC Display Code (DPC), not ASCII, and its associated data file written with 20-bit binary, positive integers packed into 60-bit words (the word size for a CDC-7600). (The header information explains how to decode the data records to recover the measured values in engineering units.) The data were originally archived on an Ampex Terabit Memory System (TMS-4) using 2-inch-wide video tape. When the TMS-4 was retired, SCD transferred all those data to the MSS preserving the old TBM format. Most of these data now have been copied and converted to COS-blocked files.

  8. Availability of Archived Data

    RAF encourages the open exchange of information among the scientific community. Qualified individuals can acquire RAF data sets for their research from projects which we have archived. We can easily provide data sets from 1983 to the present. (Some may be subject to distribution restrictions.) Older data sets pose more of a challenge, largely due to their obsolete format, but may be available. See the RAF Archived-Data Catalog or contact the RAF Data Manager [via email; phone: (303) 497-1084; fax: (303) 497-1092].

    RAF retired its GENRPO data-processing software in 1993. (Most of our previous data sets were written in GENPRO format. RAF can provide FORTRAN subroutines which will allow a user to read these data into a program the user writes. There is also a translation program g2n which runs under Sun Solaris that can convert GENPRO-II formatted data to netCDF.) Some of the specific variable names for the measurements present in these old data sets may not be found in Appendices A and B, because their names have changed or their instruments have been retired. (A GENPRO header should provide adequate descriptions for most variables.)


Appendix A, Standard Variable List

Appendix B, Brief Description of Variables and Algorithms

  1. Variables Related To Time
  2. Variables Obtained from the Inertial Reference System
  3. Variables Obtained from the Global Positioning System
  4. Combined Output from the Inertial Reference System (IRS) and the Global Positioning System (GPS)
  5. Altitude and Position Variables
  6. Aircraft and Meteorological State Variables
  7. Thermodynamic Variables
  8. Winds
  9. Radiation Variables
  10. Cloud Physics Variables
  11. Cloud Physics Variables from PMS 1-D and 2-D Probes
  12. Particle Measurements
  13. Air Chemistry Measurements
  14. Experimental Variables
  15. Obsolete Variables

Appendix C, Enhancement Factors

Appendix D, netCDF Information

Appendix E, GENPRO Format Description

References


RAF Technical Bulletins | RAF Home Page | EOL Home Page | NCAR Home Page

Last update: Thu Feb 13 09:32:34 MST 2003 (Minor update Jan 24 2007 JAG)