Developing and Providing Advanced Services and Tools

EOL's EMDAC (EOL Metadata Database And Cyber infrastructure). Researchers can use this web- and Java-based tool to easily access approximately 30 million data entries from experiments spanning 20 years by searching its contents, browsing quick-look products, and retrieving data files either via the Internet or via an offline media order, and database administrators can maintain the database just as easily.

EOL advances atmospheric science by creating integrated software and data system services that are at the heart of today's state-of-the-art observing systems.EOL is continually improving its data products and services to increase overall functionality, accessibility, reliability and ease of maintenance. These development activities are key contributions to NCAR's Strategic Priority of creating and fielding tools and services in response to the explosive growth of scientific data, and often have broad community impacts when the technologies are shared and distributed to outside users and agencies. EOL's efforts in support of this objective during FY 2006 include integration of a sophisticated suite of proven tools that researchers can use to easily access and analyze massive stores of data; the development of components that allow full internet accessibility of real-time data streams from Field Projects; and the creation of a flexible data acquisition system that can be utilized on a variety of ground-based or airborne platforms.

• FODM Integration into EOL [Highlight]
• Field Programmable Gate Arrays
• Combined Data Acquisition System - NIDAS

FODM Integration into EOL [Highlight]

In FY2006, EOL completed the integration of the Field Operations and Data Management (FODM) group of the UOP Joint Office for Science Support (JOSS) into EOL. FODM brought significant computing infrastructure, both hardware and software, that required integration into EOL’s environment. This was a major undertaking, requiring months of effort due to the long history of service provided by the group and the complicated legacy of data and web systems that had been built over the span of many years.

The first major accomplishment within this project was the creation of EMDAC (EOL Metadata Database And Cyber infrastructure). Researchers can use this web- and Java-based tool to easily access approximately 30 million data entries from experiments spanning 20 years by searching its contents, browsing quick-look products, and retrieving data files either via the Internet or via an offline media order, and database administrators can maintain the database just as easily. The JOSS/FODM metadata database, known as CODIAC, serves as the core of EMDAC. EOL staff developed a generator for THREDDS catalogs and worked with SCD to provide access to EMDAC datasets via the UCAR Community Data Portal (CDP). In 2007 EOL will continue to develop EMDAC, providing a next-generation user interface and enhancing the links between EMDAC and the CDP. We will develop tools to make it easier for EOL facilities to manage their datasets via EMDAC, and provide education on their use. To further improve its efficiency, stability and ease of use, we will incorporate EOL’s Mass Store Retrieval tool into EMDAC.

Another key product integrated into the EOL infrastructure is the highly popular Field Catalog used in over two dozen projects to date. EOL’s Field Catalog was initially developed by JOSS/FODM as a tool for collecting, organizing, and presenting reports and quick-look data products during the field phase of observational experiments. During 2006, Field Catalogs were deployed and supported for T-REX, MILAGRO, CuPIDO, and the summer WRF exercise. In FY 2007 EOL plans to develop a single integrated codebase for the Field Catalog, providing a secure and robust platform for future projects. This will take advantage of the EMDAC infrastructure and will allow products from field experiments to be managed alongside completed datasets. EOL will provide additional ways of monitoring and retrieving products and reports from the Catalog, such as the Internet standard RSS (Really Simple Syndication) and Unidata’s IDV (Integrated Data Viewer).

The FODM-into-EOL integration was funded by NFS.

Field Programmable Gate Arrays

EOL works toward achieving total Internet accessibility of its data streams, both real-time and post project, using standardized cyber infrastructure services. It is important to ensure that these data systems remain current and adopt new technology as we are constantly looking for improved ways to meet the engineering and scientific challenges.

EOL made significant progress in adopting several new technologies, the most most significant of which is in the realm of “software defined radio”, where powerful signal processing can be embedded on Field Programmable Gate Arrays (FPGAs). Unlike custom Integrated Circuits (ICs), the functionality of FPGAs can be reconfigured, allowing such a tool to be utilized across the breadth of NCAR's radars, lidars and profilers. The use of such a generic radar processing platform would help reduce engineering development and maintenance costs.

During FY06, EOL software engineers worked with prototyped and tested FPGA codes on a commercial transceiver card which is the basis for a replacement wind profiler signal processor. EOL brings a software system perspective to FPGA based system design, emphasizing componentization of signal processing codes, reusability, and “toolbox” architectures for the host side support of signal processing components. Our staff also invested significant effort in upgrading the EOL software development technologies, with the migration to a modern build system (SCONS), and adoption of a new and powerful revision control infrastructure (Subversion).

Growth in software defined radio expertise will continue as a high priority for the CDS/SSG during FY07, as FPGA systems become operational in EOL developed radar signal processors. The Virtual Operations Center will require new visualization and data integration techniques. New techniques will be explored for implementing remote system control and interaction, to enhance capabilities such as real-time Internet access to HIAPER instruments.

Combined Data Acquisition System - NIDAS

NCAR In-situ Data Acquisition System (NIDAS) is part of EOL's reorganization efforts to address commonality of software and data systems across our facilities. For the first time, two previously distinct in-situ platforms (the G-V and ISFF) share a data system and the associated software base. The system, which was first deployed during the T-REX project in March of 2006, has been proven to be robust and flexible.

During FY 2006, EOL software engineers built, tested and deployed the first data system that can be used on both airborne and ground based in-situ measurement platforms. The first release of the NCAR In-situ Data Acquisition System (NIDAS) during the 2006 T-Rex project leveraged software development across observing facilities, and was successfully deployed on the NCAR G-V and the ISFF. NIDAS increases the observational capabilities of both these observing platforms and the modern electronics allow for easier maintenance and upgrades. In FY 2007 EOL plans to replace the existing C-130 ADS data system with NIDAS.