EDGES data processing


The first step of the data processing is to download the data and origanize the files. The SSC pipeline provides us with a number of files for each pointing:

bcd - The "basic calibrated data" which is mostly unadulterated from the telescope, it has been calibrated to MJy/sr but that is about it

cbcd - The "corrected basic calibrated data" has been processed by the automatic artifact correction routines from the SSC. These corrections include a column pulldown correction (new) and a bias correction.

bimsk - The "image mask" files include masks which identify problem pixels such as cosmic rays or saturated pixels.

bunc - The uncertainty image.

A BCD from NGC5005.

Two stars cause column pulldown and there is noticeable flat fielding errors.

A CBCD from NGC5005.

The pulldown effect has been corrected for.

A median-pulldown-first frame-buildup corrected BCD from NGC5005.

The final input to MOPEX.


There are three major corrections which are done for the cbcd files:

Median image correction: When IRAC EDGES_processing/images a bright source there is occasionally serious ghosting due to stuck pixels. It takes time for these pixels to become unstuck, so we make a flat-field by taking a median down a 3-D data cube constructed by stacking the pointings on top of one another and taking the median down the z-axis.

The "First-Frame Effect": There is a relation between delay time between taking EDGES_processing/images and the bias of that image. This is found by plotting each cbcds sky (by using the IDL task sky, which is similar to the IRAF task sky) vs. delay time from the previous exposure and fitting a logarithmic function (of the form y=c1*alog(x)+c2) to those data.

The "Buildup Effect": As IRAC takes EDGES_processing/images the bias increases. This is found by plotting each cbcds sky vs the frame number. As frames are taken the bias level increases roughly exponentially and this must be corrected for (this also corrects much of the tilted-plane sky that is seen in final mosaics where the buildup is not corrected for).

The median image for NGC5005

The buildup effect in NGC5005

The first-frame effect in NGC5005

Previously, SSC did not include a pulldown correction in their cbcd data, and I developed code to fix it because the available contributed software was not very good. However, the bias in the cbcds for NGC3998 failed so I had to use the bcd data, and I applied my pulldown correction to those data.

"Column Pulldown" occurs whenever there is a bright star (generally with central pixel > 10 MJy/sr). This artifact creates a bias in the column the star appears in. In the cold mission this was a constant bias, but with the warm mission it appears to be linear. This makes calculating and correction for this bias much more complicated.


MOPEX

The next step is to feed the data to MOPEX, the software which creates the mosaics. MOPEX creates a "fiducial image frame" which is an astrometry solution and mosaic dimensions which will fit every pointing into the mosaic. It then drizzles each frame onto the mosaic, while correcting for things like cosmic rays and other outliers.

1.) The first script is the FIF script. The FIF describes the grid that the cbcds will be projected onto. For example, it contains how many pixels the mosaic will be, its center in RA and DEC, rotation, pixel scale, etc. Since the AORs have two passes (and sometimes four) a single FIF file must be created that each pass can use. Thus, the cbcd list that is input to the FIF creation tool must contain all cbcds that will appear in the final image.

2.) The next script calls MOPEXs overlap function. Overlap refines the pointing for each frame by minimizing the differences between overlapping pixels. From the MOPEX documentation:

Overlap finds the additive offsets for each frame so that the pixel differences in the overlap regions (after interpolation to a common grid) are minimized. The metric that is minimized is the sum of the uncertainty-weighted squared differences between overlapping pixels in all pairs of interpolated input EDGES_processing/images, however, this does not determine the overall level that the frames are corrected to.

3.) The final script that MOPEX runs is the mosaic script. This takes the final cbcds, with their overlap corrections and projects them onto the mosaic image. It also performs outlier rejection and removes pixels which have been flagged by the pipeline for various reasons (if it is an outlier, or the chip detected a problem, or if the pixel has saturation issues).

When MOPEX is done there are mosaics for each pass. A major artifact to overcome is a plane-gradient which is found in some of the mosaics. This is corrected by fitting a tilted plane to the sky of the image. To find the sky individual boxes in the mosaics were defined by hand, and the median of those regions is used as the sky for the central pixel of that region. Then, a plane-gradient is fit with mpfit2dfun, in IDL. This gradient is subtracted from each pass. Finally, each pass's mosaic is averaged.


PSF

The SSC provides us with the PRF, or probability response function. This is a model of how IRAC sees point sources and can be used to generate the PSF, which is what is seen on the image. The PSF is generated by treating it like a single frame and feeding it to MOPEX. MOPEX will give the PSF the correct plate-scale and rotation for a particular mosaic. Also, passes oftentimes have different rotation angles so the PSF is generated for each pass and can then be combined to create the PSF of a mosaic with multiple passes. It is especially important to have an accurate PSF with IRAC because the IRAC PSF is not symmetric.


SATURATION

Due to the lack of HDR mode in EDGES a number of galaxies in our survey suffer from saturation in the cores of the galaxy. I've solved this by downloading legacy data of the cores through a modified pipeline. These data are mostly from the cold mission and do not suffer from the warm mission problems so column pulldown correction, first-frame effect, builup correction, and median subtraction are ommited from the pipeline. They are matched with our mosaics by using the same FIF.tbl as our data. To find when pixels are saturated I downloaded data for galaxies which were so saturated that NaNs appeared in their cores. I then compared the cores of these galaxies to cores of archival data and found that at 30 MJy/sr our data significantly underestimate the correct flux (found from the archival data). So, every galaxy with a core with pixels > 30 MJy/sr has had those pixels replaced with archival data.