Official version, 3 February 2006
The MILAGRO team is committed to full sharing of data between all participants during all phases of the field experiment and subsequent research and analysis activities. This document is intended to provide a framework through which this sharing might be efficient and inclusive. The ultimate goal is not only to have unfettered access to data, but to foster collaborations across the various components of MILAGRO that will fully exploit the scientific value of the MILAGRO observations. Given the size and diversity of the MILAGRO effort, it is imperative that scientific courtesy be observed and that investigators be consulted and appropriately acknowledged whenever their data is used.
The status of data sets can be thought of as going through three phases: Field, Research, and Public. Each phase will have its own Data Repository (described below). For MILAGRO, the dates corresponding to these phases are:
During the Field Phase, it is requested, if possible, for investigators to submit their data to the Field Repository within 24 hours of the measurement (from aircraft or ground). These Field Data (raw data, QA/QC level 0) will be used only during the Field Study to assess that the science goals of the mission are being met and to evaluate the forecast models. All data in the Field Repository will be deleted at the end of the mission (i.e., March 30, 2006).
To facilitate scientific analysis of all the data, it is requested that Preliminary Data (QA/QC level 1) be submitted to the Research Repository by September 1, 2006. Final Data must be submitted by March 1, 2007. During the Research Phase, data are only available to MILAGRO participants and the Research Repository will be protected by password. Each data file should include in its header (metadata) a clear indication of whether the data is Preliminary or Final. The ICARTT format also specifies to label each file with a revision number (in the file name as well as the file header).
All data will be made publicly available on March 1, 2008 by removing the password control on the Research Repository.
In order to be sure that data is used and acknowledged fairly and properly, all MILAGRO participants are requested to accept the following responsibilities:
During the Research Phase, all MILAGRO participants will have access to all data.
All data will become public on March 1, 2008. This is consistent with NASA and NSF data policies to make data public 1-2 years after an experiment. Individual MILAGRO component groups (i.e., MCMA, MAX-Mex, MIRAGE, and INTEX-B) may choose to release their data to the public sooner. Such elections to accelerate public release will not affect the protected status of the larger MILAGRO data holdings. Component groups may also elect to share data or collaborate with groups outside the MILAGRO community. Such data sharing with third parties will be arbitrated by leadership within the relevant component group and will respect the protected status of data from the other component groups.
The data to be archived includes:
When any of the MILAGRO data are used in a publication, an acknowledgement statement should be included, recognizing the efforts and funding from the large number of people and agencies involved.
[Please send suggestions to Louisa Emmons(emmons@ucar.edu) - either for an overall MILAGRO statement or for individual missions (MIRAGE, MCMA, etc.), satellite data sets, etc.]
To allow consistency with previous measurement campaigns, two data formats will be used during MILAGRO: (1) ICARTT (modified NASA Ames) and (2) NARSTO Data Exchange Standard (DES). Both of these formats are text (ascii) file formats that are easily produced and used by most investigators. Data that are not suitable for text files (e.g., from LIDAR, AMS, satellites, models, etc.) should be archived in their community standard format (e.g., NetCDF, graphics, etc.). Analysis of the data sets will be facilitated if one format is used by all groups at a given site. All DC8 and C130 investigators will use the ICARTT format.
The Data Catalog (on the NCAR/EOL MILAGRO website) will list all measurements made and have links directly to the data for all phases (Field, Research, Archive).
"Readme" files should be provided for the data catalog, detailing the measurement and analysis techniques.
Combining all measurements from one platform or site on a common time base makes the files much easier to use for analysis. However, this is time-consuming work and someone needs to be identified to create and check them. Merges are valuable to have at all stages (preliminary, as data is revised, final).
The NASA Langley group (Jim Crawford et al.) plans to create merges of the ICARTT-format files that are in the INTEX-B archive (measurements from the DC8, C130).
It seems feasible to convert files between NARSTO DES and ICARTT formats, since these two formats are simply text files of columns of numbers with a header of metadata. NARSTO DES files should have all the information needed for ICARTT, and it may also be possible to convert from ICARTT to NARSTO DES. However, it is not trivial, and we need someone to write programs and do the conversions.
There are a few issues that would make the data formats more compatible, and would improve the formats:
Standard variable names. NARSTO DES uses a set of standard variable names. Users will be asked (but not required) to use these standard names in the ICARTT format. NARSTO will develop a cross-referenced set of shorter standard names for the ICARTT format users, including eliminating special characters from the variable names (commas, semicolons, etc.) to make them easier to parse and use in various software programs.
Units. A set of standard units names will be compiled. Units will be added to the ICARTT format.
Data Managers are needed for each platform or site. They will ensure that if possible, data is submitted to the Field Repository during the field study. After the campaign, they will ensure that all data is submitted to the Research and Final Repositories, according to the deadlines given above.
NASA DC8, J31, B200, IONS: Jim Crawford, NASA Langley
NCAR C130: Louisa Emmons, NCAR
DOE G1: John Hubbe, PNNL
Twin Otter: Bob Yokelson, U.Montana
T0: Jared Morante, MCE
T1: Alex Guenther/Louisa Emmons, NCAR (NSF)
Telma Casto (Mex.)
Jeff Gafney (DOE)
T2: Will Shaw, PNNL