PMS-2D Raw Data Format

This document covers Optical Array Probe file format for packaging PMS2D, HVPS, 2DS, Fast2D and other OAP style probes. First is the file header, starting with ADS3 this is a simple XML text header. Second is the record structure, which has not changed. And third is format of the actual 2D buffer, this includes the new 64 bit Fast2DC.

OAP (PMS-2D) File format

Starting in 2007, with the PACDEX project, Optical Array Probe data files will have the following XML based header at the start of the file.  The data records will follow and are the same as they have been all along, with the exception that tas is no longer encoded.

<?xml version="1.0" encoding="ISO-8859-1"?>
<OAP version="1">
 <probe id="C6" type="Fast2DC" resolution="10" nDiodes="64" serialnumber="F2DC002" suffix="_LPC"/>
 <probe id="C4" type="Fast2DC" resolution="25" nDiodes="64" serialnumber="F2DC001" suffix="_RPC"/>
 <probe id="P1" type="TwoDP" resolution="200" nDiodes="32" serialnumber="2DP10" suffix="_RWI"/>

Additional meta-data may present.  Each line is ended with a new-line character ('\n').


PMS-2D Record Formats

The PMS-2D raw data comes in 4 kilo-byte records of 32 bit slices from the probe (1024 slices). The interface card then wraps a record header around that with time stamp and other pertinent information.

The data is stored in big-endian format, so all data will need to be swapped when used on Intel architecture (Linux or MS Windows).


OAP Logical Record Format

/* Values currently in use for the 'id' field. */
#define PMS2D_C1         0x4331		// First PMS-2DC
#define PMS2D_C2         0x4332		// Second PMS-2DC
#define PMS2D_C4         0x4334         // RAF 64 diode 25 um Fast 2DC
#define PMS2D_C6         0x4336         // RAF 64 diode 10 um Fast 2DC
#define PMS2D_G1         0x4731		// First 2D Greyscale; unused to date
#define PMS2D_H1         0x4831		// SPEC HVPS
#define PMS2D_P1         0x5031		// First PMS-2DP
#define PMS2D_P2         0x5032		// Second PMS-2DP

struct P2d_rec {
  short id;                             /* 'P1','C1','P2','C2', H1, etc */
  short hour;
  short minute;
  short second;
  short year;                           /* starting in 2007 w/ PACDEX */
  short month;                          /* starting in 2007 w/ PACDEX */
  short day;                            /* starting in 2007 w/ PACDEX */
  short tas;                            /* true air speed	*/
  short msec;                           /* msec of this record	*/
  short overld;                         /* overload time, msec	*/
  unsigned char data[4096];		/* image buffer		*/
typedef struct P2d_rec P2d_rec;
The id word tells whether it is a 2d-C, 2d-P or an HVPS. The 1 or 2 signifies first or second probe. Hex values are 'C1' = 0x4331, 'P1' = 0x5031 etc.


Time stamp
The hour, minute, second, and msec belong to the LAST slice of the record. This time stamp is stamped the moment the VME interface card is interrupted by the probe when a record is ready for downloading (ADS2) or the after the data has been transfered from the probe to the USB driver (ADS3) in the data acquisition system.


True Air Speed
The true air speed (tas) is packed in the same format as shipped to the probe.  If the XML primary file tag is <PMS2D> then decode tas as follows: tas_float = rec.tas * 125 / 255.  If the tag is <OAP> then there is no decoding, the tas is a truncated integer.


The overld value is the amount of time in milliseconds the probe was shut off while the interface card was unloading data. For ADS2 data (~1996 - ~2005) the overld value belongs to the NEXT logical data record (at least for the VME card used since May 1997, SHEBA).

For ADS2 data, typical values when in heavy cloud should be on the order of 37 milliseconds for 1 probe attached to the VME interface card and 50-60 milliseconds if 2 probes are attached. The probe can typically fill the buffer in as fast as 3 milliseconds.

For ADS3 this value will always be 0 and the overload is word signifies an overload in the data.  A difference between that and the previous timing word will give you the dead time.



PMS-2D Raw Data Format

Data are recorded in 32 bit slices, which are inverted (data are 0's and blanks are 1's). Starts with a "sync" word, 0xff000000, followed by N particle slices, followed by 3 blank slices, followed by a "timing" word.

The manual defines the sync word as 0x55000000, however, in an attempt to move to a 32 bit timing word the sync word changed to 0xff000000. RAF probes also produce 3 blank slices, compared to 1 as defined in the manual. As of DYCOMS-II (mid-2001), RAF has put the sync word back to 0x55000000.
Timing slice
The timing slice begins with 0x55, then the following 24 bits are the number of true air speed clock pulses which have elapsed since the last blank slice was recorded.

From a PMS-2D manual:

 0	0011111111111111111111100
 1	1111111111111111111111111
 2	0011111111111111111111100
 3	1111111111111111111111111
 4	0011111111111111111111100
 5	1111111111111111111111111
 6	0011111111111111111111100
 7	1111111111111111111111111
 8	0011111111101111111111100
 9	0011111111000111111111100
10	0011111110000011111111100
11	0011111100000001111111100
12	0011111000000000111111100
13	0011110000000000011111100
14	1011100000000000001111100
15	1011000000000000000111110
16	0010000000000000000011100
17	0000000000000000000001100
18	1000000000000000000000110
19	0010000000000000000000110
20	0011000000000000000001100
21	0011100000000000000011110
22	1011110000000000000111100
23	1011111000000000001111110
24	0011111100000000011111100
25	0011111110000000111111100
26	0011111111000001111111110
27	1011111111100011111111110
28	0011111111110111111111100
29	1011111111111111111111110
30	0011111111111111111111100
31	1011111111111111111111100
	 ^                    ^^^
	 |                    ||| Sync word
	 | Sync word.         ||
                              || Timing word.
                              | All '1's, end of particle.

A code snippet I use for locating the beggining of a particle:

  if (slice == 0x55000000 && pSlice != 0xffffffff && ppSlice == 0xffffffff)

where slice is the current slice, pSlice is the previous slice, and ppSlice is 2 slices previous.

A code snippet I use for getting the time of a particle:

  frequency = probe.resolution / tas;
  timeWord = slice & 0x00ffffff;
  deltaTime = (unsigned long)((float)timeWord * frequency);
  *** Note: The 24 bit timing word is typically rolled over in about 8 seconds: 
  the deltaTime is unknown if next particle is detected 8 seconds later.



PMS-2D Processed Data Format

The nimbus processor can produce 1D histograms from the 2D-C and 2D-P probe data. These are in the standard PMS-1D format as described in the NCAR-RAF netCDF conventions.




RAF Fast-2D Raw Data Format

Data are recorded in 64 bit slices and inverted: data are 0's and blanks are 1's. A particle, consisting of N data slices, is terminated by the sync/timing word.  Blank slices may appear before or after the particle.

Sync word
The new RAF Fast-2DC probe defines the sync word as 0xAAAAAA0000000000LL. It is in the first 3 bytes of the leading slice.  If the Sync word is 0x5555AA000000000000LL, then data has been lost.  This is the equivalent of the overld value used in the original 2D data.
Timing tag
The time tag takes the last 5 bytes' space in the leading slice. This is a running 12 MHz clock starting from probe power on.

A Fast-2DC particle structure:

 0	11111111111111111111111
 1	01111111111111111111110
 2	11111111111111111111111
 3	01111111111111111111110
 4	11111111111111111111111
 5	01111111111111111111110
 6	11111111111111111111111
 7	01111111111111111111110
 8	11111111111011111111111
 9	01111111110001111111110
10	11111111100000111111111
11	01111111000000011111110
12	11111110000000001111111
13	01111100000000000111110
14	11111000000000000011111
15	01110000000000000001110
16	11100000000000000000111
17	01000000000000000000010
18	10000000000000000000001
19	01100000000000000000000
20	11110000000000000000011
21	01111000000000000000110
22	01111100000000000001110
23	01111110000000000011110
24	01111111000000000111110
25	01111111100000001111110
26	01111111110000011111111
27	11111111111000111111111
28	01111111111101111111110
29	11111111111111111111110
30	11111111111111111111111
31	11111111111111111111110
32      11111111111111111111111 
33      01111111111111111111110
34      01111111111111111111111
35	11111111111111111111111
	                      | Sync/timing word for this particle.

A code snippet I use for locating the beginning of a particle:

  if ((slice & 0xFFFFFF0000000000LL) == 0xAAAAAA0000000000LL)

A code snippet I use for getting the time of a particle:

  timeTag = slice & 0x000000FFFFFFFFFFLL;
  time = timeTag/12000 (in milli-seconds)