Steps In MTP Post-Campaign Data Analysis
MJ Mahoney
Last Revision: February 16, 2012
12. Calculate Retrieval Coefficient Sets
There have been many formats used for retrieval coefficients over
the years, and they were generally different for every platform. In
addition, until recently the horizon observable -- the outside air
temperature -- was assumed to be the same for each frequency. Thus,
assuming 10 elevation angles for making brightness temperature
measurements, there would be 19 observables for a two channel
receiver, and 28 for a three channel receiver. That has all changed.
All instruments/platforms now use the same RC format, and they all
now use a three channel receiver with 30 observables. That is, each
frequency channel measures it's own horizon temperature. Until
recently, a set of RCs would include a configuration file (.cfg)
and an RC file for each flight level for which RCs were
calculated. This has also changed. We now write an RC set as a
single binary file, called a Retrieval Coefficient File or RCF file,
with one configuration record, and an additional record for each
flight level's RCs. This simplifies keeping track of RC sets, and
speeds up data processing. For the time being, we continue to write
RC files in the old ASCII format, and a tool is also available for
converting either direction between the old and new formats.
Figure 1. RCcalc Setup tab.
Retrieval coefficients (RCs) are calculated using the the RCcalc program, the Setup tab of which is shown in
Figure 1. Most of the combo
and text boxes are automatically populated for the default mission.
As mentioned above, both ascii and binary RCs are calculated. By
default, the ascii RCs are written to the RAOB folder for the
particular mission (e.g., the C:\MTP\Data\GH\ATTREX\RAOB\RC\
folder for the Global Hawk ATTREX field campaign). Also
by default, the MTPbin
program looks for the binary RCF files in the default RC folder for
the particular mission (e.g., the C:\MTP\Data\GH\ATTREX\RC\
folder for the Global Hawk ATTREX field campaign) so that's
where RCcalc saves them.
(If MTPbin does not find
the RCF files in the \MISSION\RC\
folder, then it looks for the ascii RCs in the same folder.)
The RAOB2 File combo box
will be populated with all ROAB2
files in the C:\MTP\Data\Platform\MISSION\RAOB\Templates\
folder. In Figure 1 the PKMJ___2011100500.RAOB2 file was
selected from a drop down list. This in turn caused the MISSION_RAOBrangeAll.txt file
to be searched for this template profile. If the sounding is found,
the LR1, LR2 and Zb text boxes are
automatically populated with the information from this file. If the
template sounding sounding cannot be found, you will prompted with
the message shown in Figure 2
warning you to manually enter LR1, LR2 and Zb values.
Figure 2. Missing template
profile warning box.
For years RCs were calculated one set at a time. When dozens of sets
of RCs need to be calculated, this is a very tedious process.
Therefore, the Mode Combo box was create. In Figure 2, the Mode was
set to 'Process Single File', which was the old way of doing
business.
Figure 3. The RCcalc Setup tab with the Mode combo box set to 'Process
Templates'
An important new addition to the calculation of RCs is batch
processing. This is why the \Templates\
sub-folder was created under the \MISSION\ROAB\
folder. As shown in Figure 3,
when the Mode combo box is
set to 'Process Templates' all of the RAOB2 files in the \MISSION\RAOB\Templates\ folder
are used to calculate RC sets. The RC sets are named starting with
the notation in the two digit Region text box ('HA' in Figure
3), and then 'HB', 'HC', to 'HZ', and then 'IA', and so on as
necessary. The naming convention for RC sets is: a one character
aircraft designation, the two letters 'RC', two letters for the Region, a single number for
the RC region, and two number for the month(s) that the RCs
are calculated for. For example, GRCHA2AB.RCF is the Global Hawk (G)
set for Region 'HA', RC set '2' for the months of October ('A" in
hexidecimal) and November ('B' in hexidecimal). The RC set
designation is very useful because when RCs are calculated before a
campaign the RC set is normally '0'. Once the a priori observable
errors have been assessed, they can be modified in the RCS file, and
a new RC set calculated with a different RC region number ('2' in
the example just given).
When RCs are calculated the time consuming step is the forward
calculation relating calculated observables to RAOB temperature
profiles. This calculation does not need to be repeated when the a
priori observable errors are changed. Just check the | Do Regression ONLY | check box,
and new sets of coefficients are calculated in a few minutes,
instead of hours.
Figure 3. Expanded RCcalc
Setup tab.
If the Right Arrow in the
upper right had corner of the Setup
tab is clicked, an expanded view of the RC configuration file is
shown as in Figure 3.
Clicking the Left Arrow
closes the expanded view.
Figure 4. The RCcalc Frequencies/Angles tab.
Included on the RCcalc Frequencies/Angles tab (Figure 4) are the IF Offset and Weight frame
illustrating the IF bandpass shape. This information is loaded from
a file in the C:\MTP\Data\Platform\Mission\Setup\
folder named: Mission_RCS.txt.
RCS stands for Retrieval Coefficient Setup file. It contains all the
information needed to calculate an RC set: flight levels
(altitudes), elevation angles, frequencies, a priori observable
errors, the IF bandpass shape, and the pressure altitude offsets of
the retrieved temperature profile with respect to flight level. This
information used to reside in separate text files, but are now
consolidated into a single file.
For example, the ATTREX_RCS.txt
file contains the following informations.
' This RCS-file follows a few simple but mandatory format
requirements.
' Settings are entered into categories which occupy a single line
and
' must be bracketed by square brackets (e.g. [RC_ALTITUDES]).
Variables for
' which settings are assigned must have exactly the same name in
the program
' in which they are used and be followed by an equal sign (=).
Spaces are
' not allowed. Anything beyond the first space in a line is
ignored.
' Blank lines are also ignored, but are required betweeen
categories.
[GENERAL]
Ceiling=25
'Ceiling
CycleTime=18
'Scan Cycle Time
SU=SU2
'Sensor unit name
Nel=10
'Number of Elevation Angles
Nlo=3
'Number of LO frequencies
Nobs=30
'Number of observables
Nif=11
'Number of IF segments in bandpass model
Nret=33
'Number of retrieval levels
CH1LSBloss=0.0
'CH1 LSB loss
CH2LSBloss=0.0
'CH2 LSB loss
CH3LSBloss=0.0
'CH3 LSB loss
[RC_ALTITUDES]
'Altitudes RCs are calculated at
20.0
19.0
18.0
17.0
16.0
15.0
14.0
13.0
12.0
10.0
08.0
06.0
04.0
02.0
0.01
[ELEVATION_ANGLES]
+70.0
+45.0
+32.0
+20.0
+9.5
+0.0
-10.0
-22.0
-37.0
-70.0
[LO_FREQUENCIES]
55.51
56.65
58.80
[OBSERVABLE_ERRORS]
+0.53 +0.66 +0.82 +0.72 +0.57 +0.43 +0.51 +0.51 +0.52 +0.51
+0.46 +0.41 +0.43 +0.40 +0.46 +0.38 +0.44 +0.42 +0.46 +0.46
+0.60 +0.62 +0.57 +0.58 +0.63 +0.52 +0.65 +0.58 +0.65 +0.63
[IF_BANDPASS]
01 258.696 000.1178
02 266.772 000.6292
03 281.724 000.9175
04 304.027 000.9634
05 333.998 000.9501
06 362.875 000.8599
07 378.052 000.6435
08 386.336 000.2108
09 396.323 000.0344
10 410.308 000.0054
11 427.471 000.0017
01 258.829 000.1094
02 266.785 000.6199
03 281.741 000.9124
04 304.036 000.9659
05 333.960 000.9495
06 362.867 000.8507
07 378.046 000.6340
08 386.294 000.2035
09 395.836 000.0281
10 406.319 000.0008
11 428.636 000.0000
01 258.837 000.1063
02 266.794 000.6073
03 281.783 000.9065
04 304.036 000.9693
05 333.895 000.9406
06 362.859 000.8339
07 378.048 000.6226
08 386.290 000.2000
09 395.830 000.0276
10 406.307 000.0008
11 428.636 000.0000
[ZP_OFFSETS]
-9.5 -8.0 -6.5 -5.1 -4.0 -3.2 -2.5 -2.0 -1.6 -1.3 -1.0
-0.7 -0.5 -0.3 -0.15 0.0 0.15 0.3 0.5 0.7 1.0 1.3
1.6 2.0 2.5 3.2 4.0 5.2 6.9 9.0 11.4 14.5 18.0
[EOF]
In this file for the [IF_BANDPASS] section, the first column is the
IF frequency offset in GHz and the second is the weight assigned to
that frequency channel; separate IF band passes are provided for
each frequency (another recent innovation). The IF Bandpass should
be measured before and after a field campaign to be sure that it is
stable. Measuring and modelling the IF Bandpass is discussed
elsewhere.
Figure 5. The RCcalc Altitudes tab.
The RCcalc Altitudes tab list the altitudes at which RCs should be
calculated, as well as the pressure altitude offsets and a
priori observable errors from the RCS file.
The OBS Files
We conclude this discussion with an explanation of the format of the
ascii OBS files and RC files. The first few records from an OBS file
(EI0_1035.OBS for START-08) is given below. The OBS file contains
the RAOB temperature profile and the calculated brightness
temperature for each elevation angle and frequency of observation.
Comment lines (which begin with an apostrophe) contain
comments of what the different parameters in the file are. In the
example below the parameters are as follows:
Pz the
pressure (corresponding to the pressure altitude for the
calculation)
NTB the number
of TBs calculated for each RAOB (number of frequencies x number of
IF channels x number of elevation angles)
Nret the number of
retrieved temperature levels. Level 16 counting from below to above
the aircraft is always the flight level corresponding to Pz
Pret the
pressures at the Nret retrieval levels
IKT the number
starting at 1 of the RAOB for which the forward calculation is done
Year the year the RAOB
was launched
MM the month the RAOB was
launched
DD the day of
the month that the RAOB was launched
HH the UT hour
that the RAOB was launched
WMOno the WMO number of the RAOB site
WMO4l the WMO (or ICAO) four letter
designation of the RAOB site
Tret
the sampled RAOB sounding at the Pret retrieved altitudes
TBvsEL the (normally 30)
calculated observables for each RAOB
and so on for each RAOB.
'Pz
NTB Nret
250.50
660 33
' Pret -
Nret retrieval pressure altitudes (lowest to highest)
915.21 761.00 628.54 522.39 449.60
402.00 363.79 338.33 319.03
305.14 291.75 278.83 270.49 262.34
256.36 250.50 244.74 239.09
231.72 224.54 214.16 204.27 194.83
182.92 169.05 151.38 133.44
110.43 84.47 60.65
41.59 25.69 15.04
' IKT Year
MM DD HH WMOno WMO41
01 2008 6 23 12 72649 MPX
' Tret (1
to Nret) - Nret RAOB temperatures from ground up
287.69 277.16 270.11 259.90 252.64
247.25 242.02 236.93
233.53 231.01 228.30 226.08 224.92
223.21 221.89 220.56
219.18 217.79 216.05 214.94 212.59
209.78 209.94 215.29
219.42 219.06 218.40 218.34 215.70
219.10 219.94 226.96
234.18
' TBvsEL
((M-1)*Nel + L), M=Channel (1-3), L=elevation angle (1-10)
200.60 205.88 210.01 214.69 216.63
220.56 225.06 229.04
233.06 237.48 215.76 215.88 216.08
216.87 218.39 220.56
222.58 224.42 226.49 229.05 216.58
216.97 217.43 218.41
219.41 220.56 221.64 222.59 223.66
225.02 250.50
' Next
sounding
02 2008 6 24 00 72649 MPX
287.19 276.41 270.71 258.66 253.08
246.05 242.46 236.23
233.14 230.35 227.88 225.53 223.71
221.22 219.32 217.41
217.25 217.25 217.28 217.40 214.29
207.74 208.23 214.39
221.61 220.73 218.17 218.91 213.47
218.23 221.19 226.79
234.37
200.53 205.82 209.96 214.62 216.27
217.41 223.42 227.94
232.29 236.97 215.74 215.81 215.94
216.43 217.14 217.41
220.27 222.64 225.06 227.92 216.26
216.50 216.76 217.15
217.29 217.41 218.93 220.26 221.66
223.32 250.50
03 2008 6 24 12 72649 MPX
The RC Files
The ascii RC file for the previous OBS file contains the
following parameters:
Header
containing the name of the (ascii) RC file, the pressure at
calculated level, the number of RAOBs used in the OBS file, and
the date it was generated
A priori errors the a
priori observable errors (from the RCS file) used in the
regression
AA observables the average values of the
observables calculated for the IKT RAOBs in the OBS file
And then for each of the Nret retrieved levels, starting at the
lowest level, the following parameters
Plevel
the pressure at each retrieval level
RTav2
the average temperature (from the IKT RAOBs) at each level
RMSa2 the
RMS scatter on RTav2
RMSe2 the
estimated error (from the regression using the a priori errors) on
the retrieved temperature at this level
Nobs RCs the (normally
30) calculated retrieval coefficients (RCs)
And so on for all the retrieved pressure levels.
' Header
line with RC set file name, flight level 250.50 hPa = 110.35 km,
Nraob=150
C:\MTP\Data\NGV\START08\RAOB\RC\NRCEI056.1035
250.50 150 Generated: 04-19-2009 09:30:41
' Apriori
observable errors CH1 to Ch3 at Nel elevation angles each
0.85 0.84
0.75 0.68
0.57 0.67
0.67 0.67 0.65
0.80 0.95
0.91 0.89
0.85 0.96
1.15 1.24 1.09
1.02 1.03
0.62 0.58
0.52 0.49
0.58 0.61 0.59
0.57 0.56 0.59
' Archive
average observables for the RC set at Nobs retrieval levels
200.57
205.85 209.97 214.65 216.59 220.55 225.04 229.01 233.03
237.45
215.72 215.84 216.04 216.83 218.35 220.55 222.57 224.40
226.46
229.01 216.53 216.92 217.39 218.37 219.38 220.55 221.63
222.58
223.64 224.99
'
RTav2=Avg T at this level, RMSa2=RMS on RTav2, and RMSe2=expected
error on retrieved T
'
Plevel RTav2 RMSa2 RMSe2
915.21 287.77 1.45 1.44
' Nobs RCs
-0.02772 -0.02620 -0.02940 -0.01877
-0.00167 0.04212
-0.01486 -0.03113 -0.04453 -0.03431
-0.00711 -0.00307
0.00133 0.00358 -0.00433
0.01430 -0.00359 -0.00541
-0.00843 -0.01280 0.01390
0.01201 0.00652 -0.01595
-0.00050 0.05082 0.00013
-0.01506 -0.01864 -0.02259
'Next
pressure level up
761.00 277.12 1.50 1.49
' Nobs RCs
0.00887 0.00183 -0.00686 -0.02729
-0.04568 0.03759
-0.01423 -0.01202 -0.02209 -0.02675
-0.01893 -0.02140
-0.02157 -0.01833 -0.00404 0.01276
-0.00104 -0.00527
-0.00543 -0.00513 -0.03924 -0.03757
-0.03826 -0.01093
0.01878 0.04535 0.01417
-0.00565 -0.01632 -0.01746
.
.
Some RCs removed
.
256.36 221.88 1.00 0.25
0.00080 0.00032 -0.00053 -0.00276
-0.00255 0.08854
0.06427 0.01524 -0.00148 -0.00507
-0.00203 -0.00253
-0.00273 -0.00124 0.00735
0.03005 0.03652 0.02926
0.01657 0.00588 -0.00477
-0.00281 0.00244 0.02765
0.05581 0.10681 0.16326
0.17438 0.14114 0.08365
' Flight
level for this RC set
250.50 220.56 1.39 0.33
-0.00562 -0.00684 -0.01025 -0.01522
-0.00894 0.24575
-0.00590 -0.01604 -0.01589 -0.00924
-0.00907 -0.00979
-0.00934 -0.00290 0.02187
0.08341 0.01395 -0.00103
-0.00627 -0.00709 -0.01246 -0.00523
0.01102 0.08158
0.16140 0.29647 0.15683
0.06513 0.01376 -0.01029
244.74 219.15 1.13 0.24
-0.00477 -0.00410 -0.00289
0.00843 0.06533 0.09625
-0.00430 -0.00924 -0.01173 -0.00908
0.00290 0.00761
0.01428 0.03569 0.05195
0.03267 0.00457 -0.00117
-0.00333 -0.00402 0.05364
0.08062 0.12703 0.19966
0.15970 0.11611 0.05843
0.02111 0.00198 -0.00665
239.09 217.72 1.14 0.44
.
. Some
levels removed
.
The RCF File Data Structure
There are two data structures associated with the binary Retrieval
Coefficient Files (RCF). The first, named RCconfiguration2, provides all the information in
the ascii RCS file, but in a binary data structure. The size of this
data structure is 5000 bytes. The second data structure, which
obviously must be the same size, contains all the parameters
associated with a set of RCs at a particular flight level. It is
name RC_Set_1FL. It
contains the following parameters:
sBP
the flight level pressure altitude (hPa)
sOBrms!(1 To
30) the 1-sigma a
priori observable errors
sOBav!(1 To
30) the Archive
Average observables
sBPrl!(1 To
33)
the pressure at the retrieval levels
sRTav!(1 To
33)
the average T at retrieval levels
sRMSa!(1 To
33) the
variance in T at the retrieval levels
sRMSe!(1 To
33) the
formal error in T at the retrieval levels
Src!(1 To
33, 1 To 30) the RCs themselves. For each of the 33
retrieval levels there are 30 RC coefficients corresponding to
each of the 30 observables, which all influence the retrieve
temperature at each level
Type RCconfiguration2
RCformat As Integer
CreationDateTime As Date
RAOBfilename As String * 80
RCfilename As String * 80
RAOBcount As Integer
LR1 As
Single
'LR above top of RAOB
zLRb As
Single
'LR break altitude
LR2 As
Single
'LR above break altitude
RecordStep As Single 'Record
Step through available RAOBs
RAOBmin As
Single
'Minimum acceptable RAOB altitude
ExcessTamplitude As Single 'Random Excess Noise Level on Ground
Nobs As
Integer
'Number of observables
Nret As
Integer
'Number of retrieval levels
dZ!(1 To
33)
'Retrieval offset levels wrt flight level
NFL As
Integer
'Number of flight levels
Zr!(1 To
20)
'Flight levels (km)
Nlo As
Integer
'Number of LO channels
LO!(1 To
3)
'LO frequencies (GHz)
Nel As
Integer
'Number of elevation angles
El!(1 To
10)
'Scan mirror elevation angles
Nif As
Integer
'Number of IF frequencies
IFoff2!(1 To
12)
'IF frequency offsets (GHz)
IFwt2!(1 To
12)
'Weights assigned to each IF frequency
SURC As String * 4
'SU - to make sure correct IF is used
Spare!(1 To 207)
'
Sensitivity matrix: iRC, NFL, Nlo, Nel, bias and slope
SmatrixN1!(1 To 15, 1 To 3, 1 To 10)
SmatrixN2!(1 To 15, 1 To 3, 1 To 10)
End
Type
Public RCFG2 As RCconfiguration2
' RC set
for one flight level
Type RC_Set_1FL
sBP As
Single
'Flight level pressure altitude (hPa)
sOBrms!(1 To
30)
'1-sigma apriori observable errors
sOBav!(1 To
30)
'Archive Average observables
sBPrl!(1 To
33)
'Pressure at retrieval levels
sRTav!(1 To
33)
'Average T at retrieval levels
sRMSa!(1 To
33)
'Variance in T at retrieval levels
sRMSe!(1 To
33)
'Formal error in T at retrieval levels
Src!(1 To 33, 1 To 30) '33 retrieval
levels, 30 observables
Spare!(1 To 67)
End
Type
Public RCFL As RC_Set_1FL
Previous | Next | Index |