Bruce L. Gary

Abbreviations used in this web page

    Zg = geometric altitude
    Zp = pressure altitude
    Zh = geopotential altitude (appearing in RAOB reports)
    RAOB = radiosonde
    ZGnav = ER-2 navigation system's GPS-based version of Zg
    ZPnav = ER-2 navigation system's version of Zp
    ZPnavCOR = correction to ZPnav to arrive at an approximation of true Zp
    OAT = outside air temperature (at a location on ER-2's flight path)
    OATnav = ER-2 navigation system's version of OAT (i.e., static air temperature, or total air temperature corrected for Mach heating)
    OATraob = RAOB-based OAT
    OATnavCOR = correction to OATnav to arrive at an approxiamtion of OATraob


This web page was created for the purpose of sharing my calibration of the ER-2's navigation system pressure altitude, ZPnav, and outside air temperature, OATnav, during the CAMEX4 mission.  It appears that during this mission there was a leak in the line that delivers ambient air pressure to a transducer inside the pressurized cabin.  The magnitude of the required correction to ZPnav is approximately 400 meters at cruise altitude, and decreases with altitude.  The approximate temperature correction is -1.8 K.  All corrections have been achieved by comparing ER-2 navigation system data to nearby radiosondes.

The existence of a "pressure line leak" is merely a theory, but it is supported by two facts:  the ZPnav error increases rapidly with altitude, and 2) the ZPnav error exhibits a discontinuity at 28000 feet, which is the altitude determining whether cabin pressure is regulated to have a higher value than ambient (for all higher altitudes).  A similar pressure line leak once existed in the MMS system, and it exhibited similar error behaviors to the ER-2 navigation system's errors.

The existence of the ZPnav errors was discovered almost by accident.  In order to refine the Microwave Temperature Profiler's system gain calibration, good quality outside air temperatures are employed.  Radiosondes (RAOBs) are the preferred source for these temperatures, but since the ER-2 only occasionally flies near RAOB sites I have become accustomed to using an in situ instrument's OAT data for this purpose.  The Meteorology Measurement System, MMS, is ideal for this purpose since it's calibration is stable.  I customarily calibrate MMS temperature data by comparing it to RAOBs, and during the past several missions the MMS has required a correction of -0.8 K (although recently the correction has become -0.3 K).  When MMS is not aboard the ER-2, I use the ER-2's navigation system's temperatures, which require the same calibration against RAOBs.  Historically, the required correction to OATnav (OATnavCOR) is less stable than OATmmsCOR; OATnavCOR varies with altitude, mission, etc.  For the past several missions when I determined OATnavCOR it has been approximately -0.8 K.  It was a surprise, therefore, when my initial estimate for OATnavCOR for CAMEX4 was -4.3 K.  Another flight gave -1.6 K.  I noticed that the individual RAOB comparisons leading to these OATnavCOR estimates were correlated to the lapse rate at flight level.  That led to the hypothesis that the altitudes I was using, also from the navigation system, were in error.  Indeed, when I compared the difference between ZPnav and ZGnav (geometric altitude) with the corresponding parameter derived from a nearby RAOB, there was supportive evidence that ZPnav was in error.

Method for Determining Navigation System Pressure Altitude Correction

The calibration of the navigation system's pressure altitude and OAT require the use of RAOB data interpolated in time (and sometimes location) for times that the ER-2 flew near a RAOB site.  The RAOB data includes pressure and geopotential altitude, Zh.  As Dr. M. J. Mahoney has so ably pointed out to me, the RAOB's Zh is not exactly the same as geometric altitude, Zg; slight differences are present due to an assumed constant value for the force of gravity (for all altitudes and for all stations) when the RAOB raw data are analyzed using the hydrostatic equation.  He explains the subtleties of geopotential, geometric and pressure altitude at the following web site:  Geopotential Altitude Explained.  I have analyzed Mahoney's exact results for converting from geopotential to geometric altitude, and find that the following equation can be used for all altitudes and latitudes with an RMS accuracy of 1 meter (throughout the altitude range 5 to 22 km):

  Zg [km] = ( 1 + 0.002644 cos( 2*LAT ) ) x Zh [km]  +  ( 1 + 0.0089 cos( 2*LAT ) ) x Zh [km] x Zh [km] /6245 [km]          (1)

        where Zg = geometric altitude [km]
                  Zh = geopotential altitude [km], and
                  LAT = latitude

Figure 1Comparison of corrections to Zh for obtaining Zg using equation (1) versus using the exact equation.

The procedure for calibrating ZPnav requires, as a first step, that RAOB data for Zh be converted to Zg.  A plot can then be made of the difference between Zg and Zp, Zg-Zp, which varies with altitude (because any given temperature profile will differ from the standard atmosphere's temperature profile).  For example, RAOB-based Zg-Zp for the Key West, FL RAOB site on 2001 September 9, 12Z and September 10, 0Z (two RAOB soundings that straddle the times for an ER-2 flight), are shown in the next figure (blue lines) plotted versus pressure altitude.

Figure 2.  Difference between geometric and pressure altitude over Key West, FL during the time of the ER010909 flight.  Blue traces are from the EYW RAOB site (010909, 12Z and 010910, 0Z).  Red trace is from the ER-2 navigation system (ascent is the lower trace).  (Note:  This data includes a correction of RAOB Zh to Zg.)

The ER-2 navigation system includes a GPS-based geometric altitude, ZGnav, as well as a pressure-based pressure altitude, ZPnav, so the difference parameter, ZGnav - ZPnav can be determined from the navigation data (in the figures I employ a shortened notation:  ZGnav becomes Zg, and ZPnav becomes Zp, and ZGnav-ZPnav becomes Zg-Zp).  In the above figure there are several clues indicating the presenceof a leak in that cabin portion of the pressure tube connecting the outside probe to the pressure transducer (inside the pressurized cabin).

First, notice the discontinuities at about 8.7 km.  This is the altitude above which cabin pressure is held constant (at a pressure corresponding to 28,000 feet, or 8.5 km).  Below this altitude cabin pressureis allowed to follow the outside (ambient) pressure.  It would be a remarkable coincidence if cabin pressure had nothing to do with the navigation's "ZGnav - ZPnav" parameter.  One of these components, either ZGnav or ZPnav, is in error since the difference parameter differs from the RAOB's difference parameter.  The change in error at about 8.7 km implicates ZPnav instead of ZGnav.

Second, the parameter "ZGnav-ZPnav" departs by greater amounts at higher altitudes.  This can be accounted for by the fact that the pressure difference where the putative leak exists increases with altitude above 8.7 km.  At 8.7 km (and below) there is no pressure difference to cause air to leak into the tube from the cabin, whereas at higher altitudes the leak is driven by a pressure difference corresponding to 330 mb (8.7 km) and the outside air pressure (for example, 55 mb at 20 km).

Thirdly, the discrepancy between "ZGnav-ZPnav" and the RAOB-based "Zg-Zp" seems to grow during the flight.  This is consistent with a slow leak of air from the cabin into the tube aheadof the pressure transducer.

The next figure shows the difference between the navigation's Zg-Zp and that derived from the RAOBs.  The sign is such as to represent a requried correction to ZPnav to achieve agreement with the RAOB's Zp (assuming ZGanv to be correct).

Figure 3.   Required correction to ZPnav that would produce agreement between "ZGnav-ZPnav" and "Zg-Zp" from the EYW RAOBs, assuming ZGnav is correct. The ascending and descending portions of flight are labelled ASC and DSC.

This figure can be thought of as ZPnavCOR, or a correction that must be applied to ZPnav to correct for the pressure line leak.

Figure 4Example of an empirical fit to the previous figure's data.  The fit is meant to represent the portion of data above 9 km, where most science data was taken.

This figure shows that an empirical fit can be made to the ZPnavCOR trace for this flight.  The equation is:

    ZPnavCOR [meters] = -20 + 25 * (utsec - utsec_takeoff)/3600 + 150 * (ZPnav/20000)2 + 120 * (ZPnav/20000)10                           Eqn 1

with a residual error after applying this correction of 14 meters (above 9.5 km).

The temporal term is linear with time, starting with the takeoff time (utsec_takeoff); a squared-term and a tenth-order term complete the equation.  If this equation is to be used for other flights it must be validated for at least another flight that stayed close to a RAOB site.  The next figures show that this is possible.

Figure 5.   Difference between geometric and pressure altitude over an area between Key West and Miami, FL during the time of the ER010919 flight.  The black dotted traces are from the EYW RAOB site (010919, 12Z and 010920, 0Z), and the blue line traces are from MFL (Miami and Fort Lauderdale).  Red trace is from the ER-2 navigation system (ascent is the lower trace).  (Note:  This data includes a correction of RAOB Zh to Zg.)

This figure exhibits the same shape as in Fig. 2, implying that the same error source existed during the 10-day period encompased by these two flights.

Figure 6Example of an empirical fit to the previous figure's data.  The fit is meant to represent the portion of data above 9 km, where most science data was taken.

Again, the ZPnav error apepars to have the same shape versus altitude as in the first flight.  The ER010919 flight was 20% longer than the ER010909 flight, making it a better case for evaluating the time-dependent term.  For this longer flight I have determined the following equation for ZPnavCOR:

    ZPnavCOR [meters] = -20 + 10 * (utsec - utsec_takeoff)/3600 + 120 * (ZPnav/20000)2 + 140 * (ZPnav/20000)10                          Eqn 2

The resultant error after imposing this correction is 17 meters (above 9.5 km).

A weighted average ZPnavCOR equation is:

    ZPnavCOR [meters] = -20 + 17 * (utsec - utsec_takeoff)/(3600 sec) + 135 * (ZPnav/20 km)2 + 130 * (ZPnav/20 km)10                Eqn 3

and I tentatively suggest that it be used for the entire CAMEX4 mission to correct ER-2 navigation system pressure altitudes.

When Did ZPnav Lose Calibration?

Is the calibration error just determined for the navigation system's pressure altitude unique to the CAMEX4 mission?  For example, did it exist during the SOLVE mission?

Figure 7. Required correction to ZPnav for a SOLVE mission flight, ER000111 (red squares) and plots of CAMEX4 ZPnavCOR fits (green dotted and blue dashed lines). The SOLVE data consist of 7 discreet estimates because the ER-2 flew past 4 RAOB sites, then returned on an aborted attempt to fly from Bangor, Maine to Sweden.

The required correction to ZPnav for ER000111, one of the first SOLVE mission flights made 1.5 years before the CAMEX4, appears to be idential to the corrections requried for CAMEX4 flight ER010909.  This suggests that the calibration problem existed at the outset of the SOLVE mission, and continued throughout the CAMEX4 mission.

Method for Determining Navigation Temperature Correction

RAOBs will be used to determine a correction to navigation temperatures, OATnav.  This requires the use of corrected pressure altitudes, so the results of the previous section will be adopted for this purpose.  I will adopt equation 3 for use with all CAMEX4 ER-2 flights in subsequent analyses.

The procedure for determining OATnavCOR, the correction to the ER-2 navigation system's OAT, is to perform a spatial and temporal interpolation at the ER-2's flight altitude to arrive at a RAOB-based true air temperature, which is to be compared with OATnav.  This procedure is done for all occasions that the ER-2 passes within 150 km of a RAOB site.  For ER010909 and ER010919 there were 10 and 12 such comparisons, respectively.  For example:

Figure 8Example of spatial and temporal interpolation of RAOB profiles at ER-2 altitude to arrive at RAOB-based "true" OAT for comparison with OATnav.  The thick black horizontal line is the ER-2's pressure altitude (after correction using the ZPnavCOR equation).  The red trace can be disregarded, as it is a provisional MTP retrieved temperature profile (prior to refining instrument gain which will be done after calibrating OATnav).  The dotted green trace is RAOB EYW (Key West) for 12Z on 010909; the solid green trace is for the same RAOB site at 0Z, 010910. The dotted blue trace is RAOB MIA (Miami) for 12Z on 010909; the solid blue trace is for the same RAOB site at 0Z, 010910.  The green and blue arrows are a temporal interpolation of the EYW and MIA profiles at the ER-2's altitude (using a 47% interpolation, i.e., 63.5 ks is 47% of the way between 43.2 ks and 86.4 ks).  The yellow oval is a spatial interpolation between the EYW and MIA sites (based on a 40% interpolation, corresponding to the ER-2's ground location being slightly closer to the EYW site), and the yellow oval thus represents a spatial and temporal interpolation of RAOB temperature information at the ER-2's altitude.

An analysis similar to the one depicted in the above figure was performed for 28 comparison events (for the flights ER010903, ER010909, 010919 AND 010926), and each spatial/temporal RAOB interpolation was compared with OATnav.  Three comparisons were rejected because they were associated with large spatial temperature gradients. The final comparison is shown in the following figure.

Figure 9.  OATNavCOR for three flights.

The average value for OATnavCOR using all data points in Fig. 9 is:

    OATnavCOR = -1.60 +/- 0.36 (SE) K  (N=31)

In other words, OATnav is too warm by about 1.6 K.  This correction is about 0.8 K greater than on previous missions (-0.8 K), and may be related to an incorrect conversion of total air temperature (which includes "Mach heating") to static air temperature.  This conversion may be in error because the Mach number was affected by the pressure leak that led to an incorrect pressure altitude.

Alternative Data Weighting Philosophy

What could be the meaning of the flight-to-flight changes in ZPnavCOR and OATnavCOR?  Either 1) ZPnavCOR and OATnavCOR change from flight to flight (they could change together, since the Mach heating correction to Ttot for getting Tsat depends on a correct pressure altitude), or 2) when the ER-2 flew in the vicinity of EYW on 20001.09.09 the atmosphere was in an expansion phase at ER-2 flight levels, causing OATs to be colder than the average of the pre- and post-flight RAOBs (also leading to a different function for Zg-Zp versus Zp).

Concerning this second hypothesis, for ER010909 the 12-hr difference in temperature at a given ER-2 altitude at EYW (and MFL, Miami/Fort Lauderdale) is about 4 K, typically, so the "active atmosphere" theory may have merit.  Also, at TBW and EYW (on 2001.09.09) there are several ~4 K oscillations with altitude, having an altitude period of about 1 km, in the lower stratosphere.  This could explain the fact that for ER010909 the OATnavCOR from TBW (Tampa Bay) is ~-1.4 whereas the EYW and MFL OATnavCORs are ~-4 K (as seen in the attached graph).  This constitutes an argument for viewing a group of OATnavCOR values associated with one RAOB site as just one data point.  Under this way of thinking it is important to use as many flights, or RAOB sites, as possible, and isn't important to re-do the same RAOB site over and over on the same flight, even though the flight track goes back and forth past the same RAOB site.

What should Fig. 9 look like if it were modified so that only one data point was used to represent the ER-2's many passages by a single RAOB site?  The following figure shows this, and also includes new data for the ferry flight from California to Florida at the begining of the CAMEX4 mission (010815).

Figure 10.  OATNavCOR for five flights, using only one data point per RAOB site, per flight, except that data are accepted where the same RAOB site is encountered at altitudes that differ by more than 1.5 km.  Where several passes of the same RAOB site occur at essentially the same altitude the several OATnavCOR values have been averaged and displayed as one data point.

Using only one data point to represent multiple passes by a RAOB site leads to an average OATnavCOR = -1.51 +/- 0.36 K, N = 22.  The distribution of data points with respect to their average value is approximately Gaussian, which favors this modified approach of treating multiple RAOB passes.  The average OATnavCOR for CAMEX4 is 0.7 K more negative than for previous missions, where -0.8 K was typical.  It is my subjective opinion that this is a more conservative interpretation of the data, and I recommend adopting it for use with CAMEX4 data.


Before using the navigation system's pressure altitudes or outside air temperatures for CAMEX4 science purposes it is advisable to correct them for calibration errors described in this web page.  I recommend that pressure altitudes be increased using the following equation:

    ZPnavCOR [meters] = -20 + 17 * (utsec - utsec_takeoff)/(3600 sec) + 135 * (ZPnav/20 km)2 + 130 * (ZPnav/20 km)10

and that outsdie air temperatures be adjusted using the following offset:

    OATnavCOR [K] = -1.5 +/- 0.4 K


This site opened:  September 28, 2001 Last Update:  March 5, 2002