Data Management Services

The DMS Facility provides all facets of scientific data management including data collection, processing, quality assurance/control, archival, long-term data stewardship, and dissemination of information from the activities of National and International field and Research Projects.

The EOL/DMS Facility is composed of associate scientists, software engineers, programmer techs, and student assistants dedicated to providing comprehensive data management services throughout the complete life cycle of a project. 

Collectively, we have over 150 years of data management experience supporting a variety of field and non-field project research on climate, weather, arctic studies, atmospheric chemistry, human health, etc.  In any given year, the DMS Facility may be simultaneously supporting a multitude of projects at various stages in their life cycles from pre-field campaign to final dataset archive and analysis.

What Happens to the Data Collected during Field Campaigns?

Multi-disciplinary data (in a variety of formats) collected as part of field campaigns come from a variety of sources such as in-situ, remote sensing, and model output providers.  Such measurement platforms (or regional analyses) could be aircraft, ships, land, balloon, radars, or satellite based. Depending on the size of a field project, there could be hundreds of datasets collected from both routine operational sources to special project specific research instrumentation and models. 

The DMS provides support in the stewardship of all these data and metadata from the initial collection to the long-term archival. When data/metadata are received by the DMS, they are inventoried, reviewed, documented, and status recorded using the DMS Data Configuration Management System (as part of EMDAC). The DMS performs a series of automatic and manual data/metadata consistency checks (quality assurance) which include format verification, gross limit checks, exact/inexact duplicate records, dataset completeness, visual inspection, followed by the production of statistical summaries and documentation of any problems encountered. The DMS then coordinates closely with the respective data sources to resolve any problem issues and review any re-submitted data/metadata.Once datasets are finalized, the data/metadata are merged and loaded into EMDAC for long-term archival and access by the scientific community (Figure 1). 

Figure 1. DMS Data/Metadata flow from ingest to final archive of various sources and types supporting a field project

In some cases, the DMS will provide additional data processing or form "composite" datasets from various networks [such as surface mesonets and upper air profile (balloon) data]. The "composite" data set involves the collection of all operational/research surface network data from all available sources (i.e., special research observations and existing mesonets in the field project domain), extraction of common standard meteorological parameters, conversion of all data to be a common format, provision of uniform quality control, and generation of final "composite" data sets at various time or spatial resolutions. The major advantage of creating these "composite" data sets is cost efficiency by eliminating the requirements of each investigator to individually re-process and re-format separate network data sets. In some cases data are collected from dozens of different networks. Other value-added integrated datasets an media products (such CDs, DVDs) have been reproduced to support project requirements.

As part of the overall project data archives, the DMS includes other project supporting information such as (1) links to related project web pages and other long-term Data Centers; (2) publication listings/citations; (3) meeting summaries and presentations; (4) participants and mailing lists; (4) complete project documentation and reports; (5) photography; and (6) media and outreach products.

EOL Metadata Database and Cyberinfrastructure (EMDAC)

To improve our data management services, DMS is supporting the development of the EOL Metadata Database and Cyberinfrastructure (EMDAC). EMDAC is a comprehensive metadata database and integrated cyberinfrastructure that will be the hub of all EOL data services. Through EMDAC, DMS will create bridges to multi-agency data portals, creating compatible metadata and data access infrastructure which connects EOL to the common services of today while allowing us to meet future needs through a modular and extensible architecture. For more information about EMDAC, Web Access to archived data, DMS internal tools, Metadata Catalog Export, Browsing and Visualization, satellite and other personalized data services click here.

Long Term Data Management Collaborators

To coordinate data management activities, the DMS works routinely with many International Partners, National Agencies, Research Institutions, Universities, UCAR, and other long term data management collaborators.  Also, DMS personnel participate on various international and project data management committees to ensure that data management practices are consistent and standardized to the extent possible.

 

For more information about EOL Data Management Services, please contact Greg Stossmeister.