An Analysis of Methods of Linearization for the IRAC Arrays (12/4/00)




I. Parameterizations

For all of these methods, it is assumed that we want a conversion that will allow us to convert from non-linear DN S to linear DN S'. The calibration data is supplied as a set of images with DN S made by observing a light source of fixed intensity but with differing exposure times t.

Method A. Lincal

Lincal as currently programmed for IRAC utilizes a functional form of

S'=S (C + A/Sqrt(B-S))

where A, B, and C are parameters which are fit via an interative chi^2 minimization routine.

Method B. Rochester "Rate" Method

In this approach a function is also fit with S as the independent varable. Specifically, the input data is converted to a rate in DN/s by dividing by t, and the following is fit:

S/t = A + B S

This has the considerable advantage that no iterative fitting is required, it is merely a direct linear fit. The resulting inversion is

S'=S (1+(A/B) S)

The ratio (A/B) is the non-linearity coefficient, and S(A/B) is the fractional deviation from linearity.

Method C. Quadratic Fit

Here a function is fit with t as the independent variable of the form

S = A t + B t^2

This assumes zero dark current or pedestal. It is desirable to reparameterize this as

S/t = A + B t

In which case this also becomes a simple linear fit. The solution is then the inversion of the quadratic equation for t. Since the linear DN S' = A t, we have:

S' = A (-A+sqrt((A^2)-(4*B*S)))/(2 * B))

II. Implementation

Due to previous problems (primarily suspect repeatablility) with the IRAC CPT data a new dataset was delivered to the SSC from Rochester. This consisted of 44 images of a test lamp with increasing exposure times. The data was taken under labaratory conditions and is extremely stable. It does not use the flight electronics, nor is it a flight array.

The individual images were converted to 32-bit format, and the needed keywords for LINCAL inserted. LINCAL was then used to process the input images. The input namelist was adjusted to match the input data since the input data has a different total data range than the flight arrays (due to a different gain).

The data was converted to a FITS datacube and imported into the Mac Igor data analysis package. Software was written to implement methods B and C and analyze the results of both these and the LINCAL results.

The rate method (B) was implemented by going through each X,Y position in the datacube and slicing out the given pixel stack. The saturation point was identified by locating the datapoint at which the curve S=f(t) turns over and decreases. A linear function was then fit to data points between 10% full-well and the datapoint 1 time-unit below saturation. This is similar to the method used by LINCAL, albeit less robust. The coefficients derived from this linear fit, with S/t as the dependent variable and S as the independent variable, are the values of A and B. This is repeated for all 65536 pixels in the array.

The same approach is used to derive the quadratic fits (C). The only difference was that the fit used t as the independent variable.

III. Results

Fig. 1 - Raw counts vs. exposure time for pixel (128,128).

Figure 1 illustrates a typical pixel for this InSb array. Full-well capacity is near 16500 DN. The array is fairly linear - non-linearity is typically around 10-15% near full-well.


Lincal A Lincal B Lincal C
Saturation Linear Slope
Fig. 2 - Parameters derived from LINCAL



Figure 2 illustrates the parameters derived via LINCAL. The slope image is an estimate of the linear coefficient, and should be an estimate of the lamp intensity convolved with the flat-field.

Rate A Rate B
Rate Non-Linearity (B/A) Rate Saturation
Fig. 3 - Parameters derived using Rochester "Rate" Method


Figure 3 illustrates the parameters derived using the Rochester "Rate" Method. The A parameter is the linear term, and is equal to the lamp illumination pattern multiplied by the flat-field. The dimensionless ratio (B/A) is the non-linearity coefficient. We see that it is roughly the same over the entire array. The saturation image has quantized values because the method used to derive the saturation level always selects a given integer exposure time, but this is different for different parts of the array due to the non-even illumination.

Quadratic A Quadratic B
Fig. 4 - Parameters derived using quadratic fit


Figure 4 illustrates the parameters derived from the quadratic fit. The A parameter is once again the linear rate as it is for the "rate" method. The saturation level is the same for this approach as for the "rate" method since they were coded the same way. There is no simple expression for the conversion to linear DN S' (as there is for the rate method) since it is a complex function of A, B, and S.

The performance requirement for LINCAL states that it must be capable of linearizing every pixel to 1% over the operable range of the detector. The best way of looking at this is to compute the percent error relative to a perfectly linear response for all the pixels for all of the input images. However, this is 44x65536 data points, which overwhelms most graphical representations. Instead I have chosen a 25x25 grid of data points evenly distributed over the array. This should be a sufficient number of datapoints as to be statistically meaningful. For LINCAL, this is just a matter of specifying these pixels in the "DmPixels" file, the relative error is already generated as part of it's diagnostics output. In order to check this calculation in LINCAL, I independently computed these values. For each of the test pixel locations, the pixels were sliced from the datacube and the LINCAL A, B, and C parameters used to derive the linear solution. A line was fit to this, and the resulting residuals stored. As shown in Figure 5, the two agree very closely - the difference is probably in the details of the linear fit I am reperforming. The observed distribution is similar to what has been observed previously for the flight array CPT data. The curve slopes upwards at low DN, with the best solution near 75% full well. The solution extends for at least some pixels to nearly the full-well capacity.

Fig. 5 - Percent relative error vs input observed DN. Green is LINCAL diagnostic output, blue is recomputed from raw data and LINCAL linearity solution.


Fig. 6 - Percent relative error vs input observed DN. Blue is the LINCAL solution, red is the rate method solution.


Figure 6 compares the relative errors for the "rate" method and LINCAL. It is aparrent that the rate method outperforms LINCAL below about 60% full-well. The rate method begins to break down above 90% full-well, while LINCAL can still operate in this data range. In Figure 7, the mean percent error is computed for all the pixels in the array, for every input image. Also plotted are the mean scatter about the mean error for every pixel. The rate method falls within the 1% requirement from around 4% to 85% full-well. I am not sure right now what causes the deviation at small DN, it is probably due to small residual dark and bias offsets.

Fig. 7 - Mean percent relative error and scatter for all pixels. Blue is the LINCAL solution, red is the rate method solution. The filled symbols are the mean error, and the open are mean scatter.


Fig. 8 - Percent relative error vs input observed DN. Blue is the LINCAL solution, red is the quadratic solution.


Figure 8 presents the same type of data as Figure 6, only using the quadratic solution instead of the rate method. Rather than beginning to slope away from zero error at 75% full-well, the quadratic solution remains very accurate up to about 95% full-well. Figure 9 is similar to Figure 7. It compares the rate method and the quadratic solution. The scatter for both solutions are nearly identical. The two solutions have nearly identical mean errors up until 60% full well. Above that the quadratic solution remains more accurate.

Fig. 9 - Mean percent relative error and scatter for all pixels. Black is the quadratic solution, red is the rate method solution. The filled symbols are the mean error, and the open are mean scatter.


IV. Conclusions

For the InSb arrays, fitting the data with a quadratic function of t and inverting this provides the best possible solution, and which will meet the instrument performance requirement. The "rate" method also appears very workable - however it will result in a lower upper limit to the usable DN range, and hence will sacrifice a small amount of dynamic range. LINCAL is providing a solution that deviates too significantly from a linear solution, and which is noisier. Some of this might be alleviated by changing LINCAL to not fit the lowest part of the DN range, as was done for the other two methods (currently done, see this report. In any case, more work will have to done to characterize the linearity below a few % full-well capacity.

Addendum I - Constraints of Fits & Bad Pixel Detection

Rate Method:

The ratio of (b/a) must be negative, so that the non-linearity decreases the observed DN. There are 13 pixels where this is not true.

(B/A) > 0

Quadratic:

S=A t + B t^2

Here we have two constraints. The first is that the second derivative,

S" ~ B

should be negative, in order to ensure the proper curvature of the function S. This ensures that the non-linearity decreases the observed DN. The first derivative

S' = A + 2 B t

must be positive in order to ensure that flux increases with time (a condition enforced by the cutoff used for saturation). This constrains A to be positive. Furthermore, the saturation cutoff forces

t < -(A/2B)

which chooses only the +(radical) solution (since it is +radical/2B and B is negative). The above also gives

S < - A^2 / (4 B)

We therefore always have the expression under the radical

A^2 - 4 B S > 0

Every pixel meets the positivity constraint on A. There are 10 pixels with B positive. These are shown in the figure below. However, the curvature does not actually invalidate the solution. If we are willing to accept both curvatures (as there are obviously a handful that have this), then we need only require

A^2 - 4 B S > 0

for any possible value of S. S cannot exceed the saturation value. With that in mind (and setting a saturation value of 40,000, which no pixel ever exceeds), then we find that the above constraint is satisfied by every pixel.

Pixels with B > 0 Pixel 32,234

Generalized Bad Pixel Detection

Constraints on the above parameters can be used to pick out bad pixels and note them in the cmask. The most obvious constraints are on A. Excessively large values of A indicate an anomalously sensitive (hot) pixel. Very low or negative values of A (as indicated from the above) are "dead" or pathological.

Sample Raw Data A > 3xmean(A) A < 0.33xmean(A)