ADS Data Catalog (for each project)

1. Raw ADS data archived to the MSS
   • Tape Images
     • One large file per tape
     • Occasional multiple-tape flights
     • After project completion, tapes are sent to ML machine room
     • MIGS jobs with "msimp" verb used to transfer tape image to COS-blocked bitfiles
       • Operators mount tapes when jobs are run
       • Occasional mismounted tape, probably less than 1%, causes rearchival
       • Files larger than about 1.2GB are split into segments
   • Disc File Images
     • Usualy one large file per flight
     • Occasional multiple-file flights
     • After project completion, disc files are archived via NCAR network into transparent bitfiles
   • Stripped categorized data (SDI, PMS-2D, MCR, MASP)
     • Data are stripped at RAF and put into separate files <= 1GB
     • Use MIGS or DCS to transfer files (transparent format) to MSS (can set a write password)

2. Production data sets produced using disc images, tape images or stripped ADS data
   • Bitfiles for processing are transferred from the MSS to Jeffco using C-shell scripts (msget...)
   • NIMBUS produces the netCDF files
   • After QC, netCDF files are archived to the MSS
     • MSS path prior to 2000: /RAF/YYYY/PPP/xRT/fltno.nc

       where:
             YYYY = fiscal year
             PPP = project number
             fltno = flight number

     • MSS path since 2000: /ATD/DATA/yyyy/Project/Platform/type/fltno.yyyymmdd.ShSmSs_EhEmEs.PNI.nc

       where:
             yyyy = calendar year
             Project = project name or acronym
             Platform = example: C-130
             type = high- or low-rate data
             fltno = flight number
             yyyymmdd = flight date
             ShSmSs = flight start time (hhmmss)
             EhEmEs = flight end time (hhmmss)
             PNI.nc = extent (production NIMBUS netCDF)

   • Metadata (description) files also archived to the MSS
   • Support files (e.g., NIMBUS processor) archived to the MSS (large tar files)

3. Data catalog added to RAF web page
   • URL: http://www.eol.ucar.edu/raf/data_catalog.html
   • MSS bitfile log (per project), which includes
     • Project name, fiscal year and number
     • Aircraft ID
     • Principal Investigator(s)
     • RAF Project staff
     • Processor type (NIMBUS or GENPRO)
     • Processed data rate (LRT = 1 sps, HRT = 25 sps)
     • Output file format (netCDF or GENPRO and COS-blocked or transparent)
     • MSS path and bitfile names
     • Flight number
     • Flight date
     • Start and end times
     • netCDF bitfile size
     • Bitfile log example (http://www.eol.ucar.edu/raf/Catalog/taplog.101.html)
   • Example header (per project), which includes
     • Data identifier (RAF contact information) (netCDF only)
     • Date data were processed
     • Project name
     • Aircraft tail number
     • Flight number
     • Flight start date
     • Flight duration (start and end times)
     • Descriptions of measurements made on the project
       • Short name
       • Output data rate (samples per second = sps)
       • Calibration coefficients (netCDF only)
       • Dependencies for derived measurements, e.g., winds (netCDF only)
       • Housekeeping information, like SampledRate, VectorLength (netCDF only)
     • Header example (netCDF or GENPRO)
   • Future enhancement possibilities
     • Scientific objectives
     • Data quality report
     • Condensed flight data summaries
     • Scientist notes
     • Map showing where project's measurement were made
     • Publications resulting from this project
     • "Automatic" data ordering
       (Now available for data sets produced since 2000)
       ATD Mass Store Data-Retrieval Web Page

Last update: Fri Oct 3 14:03:43 MDT 2003