ADS Data Catalog (for each project)

Files from each project are archived to the SCD Mass Store System (MSS) in an unique directory (See description below.)

1. Raw ADS data archived to the MSS
   
   • Tape Images
     
     • One large file per tape
     • Occasional multiple-tape flights
     • After project completion, tapes are sent to ML machine room
     • MIGS jobs with "msimp" verb used to transfer tape image to COS-blocked bitfiles
       
       • Operators mount tapes when jobs are run
       • Occasional mismounted tape, probably less than 1%, causes rearchival
       • Files larger than about 1.2GB are split into segments
   • Disc File Images
     
     • Usualy one large file per flight
     • Occasional multiple-file flights
     • After project completion, disc files are archived via NCAR network into transparent bitfiles
   • Stripped categorized data (SDI, PMS-2D, MCR, MASP)
     
     • Data are stripped at RAF and put into separate files <= 1GB
     • Use MIGS or DCS to transfer files (transparent format) to MSS (can set a write password)
2. Production data sets produced using disc images, tape images or stripped ADS data
   
   • Bitfiles for processing are transferred from the MSS to Jeffco using C-shell scripts (msget...)
   • NIMBUS produces the netCDF files
   • After QC, netCDF files are archived to the MSS
     
     • MSS path prior to 2000: /RAF/YYYY/PPP/xRT/fltno.nc
       

       where:
       
             YYYY = fiscal year
       
             PPP = project number
       
             fltno = flight number

     • MSS path since 2000: /ATD/DATA/yyyy/Project/Platform/type/fltno.yyyymmdd.ShSmSs_EhEmEs.PNI.nc
       

       where:
       
             yyyy = calendar year
       
             Project = project name or acronym
       
             Platform = example: C-130
       
             type = high- or low-rate data
       
             fltno = flight number
       
             yyyymmdd = flight date
       
             ShSmSs = flight start time (hhmmss)
       
             EhEmEs = flight end time (hhmmss)
       
             PNI.nc = extent (production NIMBUS netCDF)

   • Metadata (description) files also archived to the MSS
   • Support files (e.g., NIMBUS processor) archived to the MSS (large tar files)
3. Data catalog added to RAF web page
   
   • URL: http://data.eol.ucar.edu
   • MSS bitfile log (per project), which includes
     
     • Project name, fiscal year and number
     • Aircraft ID
     • Principal Investigator(s)
     • RAF Project staff
     • Processor type (NIMBUS or GENPRO)
     • Processed data rate (LRT = 1 sps, HRT = 25 sps)
     • Output file format (netCDF or GENPRO and COS-blocked or transparent)
     • MSS path and bitfile names
     • Flight number
     • Flight date
     • Start and end times
     • netCDF bitfile size
     • Bitfile log example (/node/9680)
   • Example header (per project), which includes
     
     • Data identifier (RAF contact information) (netCDF only)
     • Date data were processed
     • Project name
     • Aircraft tail number
     • Flight number
     • Flight start date
     • Flight duration (start and end times)
     • Descriptions of measurements made on the project
       
       • Short name
       • Output data rate (samples per second = sps)
       • Calibration coefficients (netCDF only)
       • Dependencies for derived measurements, e.g., winds (netCDF only)
       • Housekeeping information, like SampledRate, VectorLength (netCDF only)
     • Header example (netCDF or GENPRO)
   • Future enhancement possibilities
     
     • Scientific objectives
     • Data quality report
     • Condensed flight data summaries
     • Scientist notes
     • Map showing where project's measurement were made
     • Publications resulting from this project
     • "Automatic" data ordering (EMDAC)

Last update: Fri Oct 3 14:03:43 MDT 2003