General Notes

It is too hard to navigate the archive directory tree. IRAC (the IST, anyway) has never actually used the flight archive structure until now. Having each DCE at the bottom of an individual directory tree that is 10 levels deep makes analysis of the calibration threads almost impossible. Furthermore, for each DCE processed there are typically 130-140 associated files! We are going to have to have tools to condense this information further. Specifically, the final product generator will have to do this for the GOs, but there will be serious problems during IOC if the instrument teams are expected to look at data in this fashion while troubleshooting the pipeline. They are used to putting their hands directly on the data and manipulating it. I do not think anyone will be very satisfied if they are entirely limited to tweaking our pipeline and having to run it in order to do anything to the data. As a simple start, I suggest something like a "gather" script that can recursively parse down the archive directory tree, find all the files matching a given wildcard, and then deliver them to some other location, i.e. we need a tool to flatten the directory structure for easy access. I also suggest that it is likely that a lot of what the IT does during IOC will not be so much them tweaking our modules and rerunning the pipeline so much as it will be them reprocessing the data themselves and then telling us how to change the pipeline. (author's note - such a script is currently under development)

On a related note, there is no way to adequately trace the pedigree of the input files, at least not easily. In many cases while looking at the thread outputs I was able to trace problems back to the input calibration files, but from there was stuck since there was no traceability to the original data that went into them. Presumably this will change when the calibration server is in place. I think this underscores that the number of files (namelists, calibration images, etc.) that need to be juggled to make everything work has become so large that it is neccessary to fully automate the process. Up until this point we have been able to get away with Russ adapting the scripts to run primarily by hand. We have reached the point that this is breaking down.

Science Thread

I went through the intermediate products, at each stage differencing the stages to see what changes had been made by each module. In most cases everything was as expected, and I have only noted here things that did not work or were not completely clear. They are listed as follows, broken doen by module:

• TRANHEAD

  Why are TRACFRA1 and TRACFRA2 written to the header when no means or standard deviation are reported?

  I assume that the format for the comment lines "/The XXX Keyword" will be replaced by the units, or will just be a copy of what FOS write. Is this because we are using test data and not real data right now?. For example, currently we have

  AVDETVE =                 6138 /The AVDETVE Keyword 
  

  which I assume will become

  AVDETVE =                 6138 / Volts
  

  with appropriate units conversion.

• IMFLIPROT

  The headers contain the following:

  IFRFLIP =                    1 / Image flipped about vertical axis
  IFRROTAT=                  180 / Image rotation, in degrees clockwise
  

  This is misleading. The operation that occurred is just a simple vertical transposition. Technically what is written here is correct, but it didn't actually require two operations to do it.

• IRACEBWC

  Does this write any keywords? How about writing the coefficients to a COMMENT line? It would be easy to encode the tables used as a comment block:

  COMMENT = 'Electronic Bandwidth Coefficients'
  COMMENT = ' Readout   0       1       2       3'
  COMMENT = ' Kup       0       0       0       0'  
  COMMENT = ' Kup_col   0       0       0       0'
  COMMENT = ' Kdown     1.58e-3 2.17e-3 2.33e-3 1.64e-3'
  COMMENT = ' Kdown_col 1.2e-5  2.1e-5  2.4e-5  1.3e-5'
  

  On the same note, it would be very useful if each module wrote it's name and version number to a comment line when it runs, and if these comment lines are kept in sequential order in the header they will make a simple processing history for the file. There are less than a dozen of these. This will make it easy to tell, for any given image, what has been done to it. For example, muxbleedcorr writes no header entries. It would be impossible to tell, if someone just picked up the file by itself, that the muxbleed correction had been appllied. There seems to currently be no standardization of the modules in regards to this behaviour. Some of them write this information, others do not.

• DARKSUB

  Something is wrong. After the science BCD is flipped, the electronics glow in the lower right corner should be in the upper right. When I difference darksub.fits and iracebwc.fits I should see what darksub thought the dark current image was. The result has a glow in the bottom right, as if the dark image was never flipped before subtraction. Frankly, it doesn't even look very much like the dark.

  Difference of DARKSUB and IRACEBWC (left) vs. DARKCAL (right). The dark itself looks pretty good.

  Also the cmask "darkcal_cmask.fits" is the bad pixel mask for channel 1, but the accompanying dark (and the science data) is from channel 4. "Pixelmask_for_raw.fits" is the correct mask, as is the one for lincal. The latter was also flipped - is lincal.fits flipped? I can't tell by looking at it. All of the masks need updating, in any case, since the current ones are a year out of date. (author's note - new pmasks have been delivered to /ssc/testdata/irac/masks)

• DARKDRIFT

  If I difference darkdrift.fits and darksub.fits the result should be the offsets determined by darkdrift. The resulting image has a mean value of 1.247e-4, for a total offset in the entire image of 8.2 DN. I computed the offset through 2 independent pieces of code, so it is real. I would expect it to be exactly zero. Is this commensurate with roundoff error in the pixels?

  Here is the output from IRAF and from the IPAC imstats routine:

  sscsci4% imstats temp10
  |   MEAN   |  MINIMUM  |  MAXIMUM  | STD.DEV. |  N    |  IMAGE
   0.00012474     -14.430      12.230     9.5298   65536 temp10
  cl> imstat temp10.fits
  #     MIDPT      MEAN    STDDEV       MIN       MAX
       0.8181  1.247E-4      9.53    -14.43     12.23
  

  In the grand scheme of things it is irrelevant, since this is many thousands of times smaller than the residual error we expect due to drift in the darks. I would be interested to know where this came from, anyway.

• FLATAP

  Here we run into problems. Mechanically looks ok, but the flat (flatfield.fits) is terrible. I think the problem in the flat is due to point source residuals, probably from a focus test, contaminating the darks used to process the tcal images, which in turn are used for the baseline by which to adjust the Fixsen skyflats (see how complicated the chain of logic in the calibration is?). Both the cmask and the flat are unflipped (which is how I assume they should be).

  Derived flatfield (left) and the original fixsen flat it started with (right). It's pretty noisy, and has latent image residuals in it from the focus test.

  FLATAP produces no header information, beyond saying that it ran. Also, as a general comment, filenames like 'lincal.fits' are not unique identifiers and impart no information. If there is a unique identifier in flatfield.fits, I can't find it. I would suggest, for example, that every file have a serial number, either in the file name or in the header, which enables it to be traced with relative ease to a series of events in the cal server. (author's note - I think the new DPID covers exactly this)

  An extremely useful alternative might be a perl "report" script which gathers up all the information on a given DCE and reports the pedigree of the files at each step in a clear and simple manner. For example, it might give output like:

  DCE #123423
  Program: SWIRE
  Instrument: IRAC
  Channel: 2
  Exp Time: 12 seconds
  RA: 12h35m34s DEC: 45d34m12s
  Date Taken: 2001-335T12:00:00.000
  Archive Path: 
  
  Dark Image Serial#: A23FC12
  Archive Path:
  Input Files: List files here
  
  Flat Image Serial#:
  Skyflat Serial#:
  Skyflat TCAL#:
  Skyflat Dark#:
  Current TCAL#:
  Current Dark#:
  

  This kind of reporting software is used by STScI for condensing the information on various instruments. See, for example, the STISINFO task in IRAF.

  There appears to be a flipping problem again, as well. When I divide flatap.fits by darkdrift.fits I should get back the flat that was divided into darkdrift.fits. When I do this I recover the flatfield, unflipped. Flatap.fits and darkdrift.fits, however, are flipped, and thus the wrong (unflipped) flat was divided into the data. Given all of this, would it not simplify the logic to do everything in the original array coordinates, and then do IMFLIPROT at the very end?

  Unflipped image (top left), darkdrift (top right), flatap (bottom left), and the quotient of darkdrift and flatap (bottom right). Note that the result is the same as the unflipped fixsen flat above, when it should have been flipped vertically.

  
  

• SNESTIMATOR

  The SNESTIMATOR output looks ok. I note that the IRAC model estimator was used. If so, then the software must be given valid values for the read noise. I note that this module receives this as a command line parameter. In that case I assume that the calibration server will presumably be used to look up the appropriate value. The reason is that the model assumes a simple number for the read noise, but the use of Fowler sampling means that the efffective readnoise is reduced by roughly the sqrt of the number of samples. For the allowed values of fowler number and frame time in the AOT the measured read noises are:

   Time   0.4  2    12   30   100   200
   Ch 1   28.3 15.3 12.3 10.8 10.8  8.2
   Ch 2   34.9 18.7 13.9 13.3 13.3  8.7
   Ch 3   26.3 15.0 14.8 14.5 14.5 13.0
   Ch 4   29.0 16.6 12.6 10.8 10.8  9.9
   

FLATFIELD Thread

There are 130+ files for 1 DCE!!!

What is "flattrack_prior_state.fits"? It doesn't look like anything I can recognize.

In the /scr/S5.0D/0001/0000/IRAC/0000257024/4/0001/0011/01/1/ subdirectory the file "flatfield.fits" looks terrible. It is erroneous, with step functions in the image. This can be traced back to FOWLINEARIZE. FOWLINEARIZE in turn failed because the lincal input was bad. I can't tell where the lincal data came from. It appears to ultimately be CPT data - it would have been better to have used CTA data. I also think this may be an older version of lincal as it does not have the model type written to the header, and is version 1.X and not 2.X. FOWLINEARIZE probably choked when it got data from an incompatible version of LINCAL. The other cal files look correct. This underscores the fact that we need the cal server and the executor so that we can keep track of these files more precisely.

Ater DARKSUB (left), and after FOWLINEARIZE (middle). The resulting flat (right) is ruined because FOWLINEARIZE got an incompatible input data set. Things look pretty good up until then.

LINCAL Thread

The LINCAL thread also did not work. This was a little surprising since it was a data set that I have run through lincal manually with good results.

Some sample input data for LINCAL, in order of increasing exposure time.

	LINCAL output. Roughly half the pixels have no solution.
	It should have looked like this (sscsci4:/local/s2/irac/sf2_linearity/chan4/ds/ch4_out.fits).

 No. of pixels for which model could not be computed:  5
   2: insufficient dynamic range
   1: insufficient number of usable points
   0: nonlinear system was singular
   2: masked in PMask
 No. of pixels with Sat < max usable DNobs:   36486
 No. of pixels with rejected saturated data:  21305

Pipeline namelist:

> !more lincal.nl
 &lincin
   minimg = 5,
   dynhi = -9,
   dynlo = 99999,
   bmax = 500000.0,
   maxdmp = 5, dmpneg = t,
   ntable = 2000,
   ngpmax = 20, nsearch = 2,
   sigmax = 10000.0,
   slopemin = 0.4,
   satstats = t,
   dnsat = 25000,
 &end
 

 > !more ch4.nl
 &lincin
   devfrac = 0.5,
   minimg = 3,
   dynhi = -9,
   dynlo = 99999,
   bmax = 500000.0,
   maxdmp = 5, dmpneg = t,
   ntable = 2000,
   ngpmax = 20, nsearch = 2,
   sigmax = 10000.0,
   slopemin = 0.3,
   satstats = t,
   dnsat = 20000,
   poscon = f,
   mindn = 2000,
 &end
 

> !more ch1.nl
 &lincin
   devfrac = 0.5,
   minimg = 3,
   dynhi = -9,
   dynlo = 99999,
   bmax = 500000.0,
   maxdmp = 5, dmpneg = t,
   ntable = 2000,
   ngpmax = 20, nsearch = 2,
   sigmax = 10000.0,
   slopemin = 0.3,
   satstats = t,
   dnsat = 10000,
   poscon = f,
   mindn = 100,
   zsig = 3,
 &end

> !more ch2.nl
 &lincin
   devfrac = 0.5,
   minimg = 3,
   dynhi = -9,
   dynlo = 99999,
   bmax = 500000.0,
   maxdmp = 5, dmpneg = t,
   ntable = 2000,
   ngpmax = 20, nsearch = 2,
   sigmax = 10000.0,
   slopemin = 0.5,
   satstats = t,
   dnsat = 20000,
   poscon = f,
   mindn = 2000,
 &end
 
 > !more ch3.nl
 &lincin
   devfrac = 0.5,
   minimg = 3,
   dynhi = -9,
   dynlo = 99999,
   bmax = 500000.0,
   maxdmp = 5, dmpneg = t,
   ntable = 2000,
   ngpmax = 20, nsearch = 2,
   sigmax = 10000.0,
   slopemin = 0.2,
   satstats = t,
   dnsat = 20000,
   poscon = f,
   mindn = 1000,
 &end

Pipeline input illumination pattern (left; used for saturation rejection) and Jason's input file (right; made from smoothed input data). The two are substantially the same.