Standard data formats

Standardized formats

Standardized data formats are fundamental to the development ot tools that will address community needs.

Keeping the formats to a limited number of standards simplifies the work of the developers and frees them up to handle more interesting needs such as algorithms and displays.

It is important that, to the extent possible, the data formats be 'self-describing'. The primary candidates are:

Sometimes a binary format is essential - for example, for radar time series data the size is so large that only a binary format makes sense. For this circumstances, the binary format will be properly documented.

CfRadial

The CfRadial format was developed to faithfully record all aspects of radar moments data in radial coordinates, i.e. the native coordinate system for the instrument.

CfRadial is designed to follow the Climate and Forecasting conventions for NetCDF data. This helps to facilitate the use of the data by numerical models, through the use of agreed-upon standard names for field variables. For details see: CF-compliant NetCDF.

CfRadial is hosted on GitHub at http://ncar.github.io/CfRadial/

See http://opensky.library.ucar.edu/collections/OSGC-000-000-021-355 for documentation for CfRadial version 1.3 in PDF form.

See cf-conventions.1.6.pdf for a copy of the latest CF conventions document.

ODIM HDF5 radial data format

The BALTRAD group, and EuMetNet, use the ODIM standard formats for representing both radial and Cartesian data.

The ODIM specification is available as a PDF.

Cf-compliant NetCDF for Cartesian data

For gridded data that is in Cartesian coordinates, the recommended format is CF-compliant NetCDF.

IWRF time series format

The Integrated Weather Radar Format (IWRF) was developed to provide an efficient binary, platform-independend data exchange format for time series data.

The IWRF documentation is available in PDF.

Legacy formats

A number of legacy formats are supported by LROSE, because so much historical data is stored in these formats.

DORADE FORMAT

DORADE was developed in the 1990's, to handle the more complex nature of Doppler radars, especially airborne platforms. DORADE is a binary format composed of distinct header and data blocks, making it extensible.

Many NCAR field projects from 1990 to 2010 stored the raw radial radar data, as well as derived products, in DORADE format.

The NCAR/EOL soloii display is a primary tool for displaying, editing and manipulating DORADE files.

From 2010, DORADE files will gradually be replaced with the new CfRadial format.

The DORADE documentation was updated in 2010.

UNIVERSAL FORMAT

Commonly known as UF, the Universal Format is formally known as the Common Doppler Radar Exchange Format. UF was documented by S.L Barnes in 1980 in the Bulletin of the Americal Meterological Society (BAMS), Vol 61, 1401 - 1404.

UF was primarily designed for access via FORTRAN. A FORTRAN binary block holds the data for each ray (or beam). UF is composed exclusively from ASCII characters and 16-bit integer words. Floating point values are computed by applying scale factors and offsets to the integer values.

UF has some significant limitations. Field names are limited to 2 characters. For example, reflectivity is often referred to as ZH, or DZ, and radial velocity as VR, because the 2-character limitation prevents use of longer names such as DBZ or VEL. Also, the headers are limited in length, so that the metadata associated with more sophisticated radars cannot be accurately represented.

Neverthless, UF is still popular with many researchers, and much analysis code has been written for UF data.

The UF documentation is based on the 1980 BAMS article.

FORAY NetCDF FORMAT

The FORAY project at EOL developed a radar data exchange format based on the Unidata netCDF scientific data format.

FORAY netCDF is based on NetCdf version 3, i.e. it does not support data compression.

The FORAY netCDF sweep file format was the primary format used for the TIMREX field experiment in Taiwan. TIMREX also stored the radial data in DORADE format.

An example ncdump from a FORAY netCDF file is the best way to understand the format.

The FORAY project has now ended. TIMREX will be the only NCAR field project which will use FORAY netCDF files. All subsequent projects will use CfRadial and DORADE.

Data format support

The following table shows the extent to which various formats will be supported:

Radx file format support
Format Read access Write access
CfRadial - NCAR/EOL/UNIDATA Yes Yes
ODIM HDF5 Yes Coming soon
DORADE - NCAR/EOL Yes Yes
UF - Universal Format Yes Yes
Foray-1 netCDF - NCAR/EOL Yes Yes
DOE ARM netcdf - precedes CfRadial Yes No
MDV radial - NCAR/RAL Yes Yes
NEXRAD msg31 level 2 archive Yes Yes
NEXRAD msg1 level 2 archive Yes No
SIGMET - raw format (Vaisala) Yes No
RAPIC - BOM Australia Yes No
HRD - HRD NOAA Yes No
LEOSPHERE LIDAR, ASCII format Yes No
EEC - now supports CfRadial Yes N/A
GAMIC HDF5 Yes No