CCD Calibration Files and Image Processing

The first thing to be done for reducing these echelle spectra is to process the raw CCD images.

To do this, we first need to generate the necessary CCD calibration files: namely a combined bias frame for bias subtraction and a combined flat lamp frame for flat fielding.

Once these calibration files are generated, we can then apply them to the rest of the frames taken during a night.

CCD Calibration Files

The first step is to generate the bias and flat field calibration files. The functions for these steps are hosted in the ccd_calibrations module. There are individual functions to generate each of the necessary calibration FITS files, and a “wrapper” function (build_calibrations) that is called in the main reduction script to run this entire step.

The options for this step in the reduction are defined in the calibration section of the main config YAML file, which is described in full here.

Combining the bias frames

The combined bias frame (which we call the “super bias”) is generated by median combining all bias frames taken during a night. The frames used are chosen based on the imagetyp keyword in the raw FITS image file headers, and is defined in the config file. For our default usage with the Tull coudé spectrograph this is “zero”.

We also calculate the error of the super bias for error propagation in the later processing of science images. This is done by adding in quadrature the photon noise and the detector read noise: \(\sigma_B^2 = B + \sigma_{\rm read}^2\), where \(B\) is the super bias flux image and \(\sigma_{\rm read}^2\) is the read noise.

Note

A note on units: the units of the bias flux and read noise must be the same. In our case, the raw FITS images are in units of ADU, but the read noise from the header is in electrons. As written in this pipeline, the read noise is converted to ADU using the header gain value so all resulting flux and error images are in ADU.

Super bias FITS file structure

The super bias is saved in a FITS file called “super_bias.fits”. It has three extensions:

• Index 0: The primary extension that only contains header information.

• Index 1: Extension named “bias flux” that contains the super bias flux, with the same shape as the detector images.

• Index 2: Extension named “bias error” that contains the super bias errors, with the same shape as the detector images.

The primary extension includes a few header keywords with information about the super bias generation:

Keyword	Description
NBIAS	the number of bias images used in creating the super bias.
RDNOISE	the read noise value, in the same units as the images, used in creating the super bias error.
HISTORY	multiple entries on what is done in these steps, and when they were executed.

These steps for generating the super bias calibration file are in the build_super_bias function.

Diagnostic plot: An image of the super bias in log scale with the median value written to the title is output in the cals subdirectory. It is generated with cal_image_2d_plot.

Combining the flat lamp frames

Similarly to the super bias frame, the flat field calibration frame is generated by median combining all flat lamp frames taken during a night. The frames used are again chosen based on the imagetyp keyword in the raw FITS file headers and is defined in the config file. For our default usage with the Tull coudé spectrograph this is “flat”.

There is also an option in the config file to define header OBJECT keyword values to not include. For example, on some nights we take a flat lamp frame to make sure the integration time results in the right amount of flux, with the OBJECT name “FF integration time test”. Such a frame should not be included in the creation of the flat field calibration file and this option lets us ensure it is kept separate.

The super bias is then subtracted from the median combined flat field flux, and the error in the flat field is calculated as: \(\sigma_F^2 = F + \sigma_{\rm read}^2 + \sigma_B^2\) to propagate the read noise and error on the super bias.

We then rescale and renormalize the flat field flux to have values between 0 and 1 based on its minimum and maximum. The flat field error image is transformed as well.

Flat field FITS file structure

The flat field is saved in a FITS file called “flat_field.fits”. It has three extensions:

• Index 0: The primary extension that only contains header information.

• Index 1: Extension named “flat flux” that contains the scaled and normalized flat field values with the same shape as the detector images.

• Index 2: Extension named “flat error” that contains the scaled and normalized flat field errors with the same shape as the detector images.

The primary extension includes a few header keywords with information about the flat field generation:

Keyword	Description
NFLATS	the number of flat lamp images used in creating the flat field.
RDNOISE	the read noise value, in the same units as the images, used in creating the flat field error.
HISTORY	multiple entries on what is done in these steps, and when they were executed.

These steps for generating the flat field calibration file are in the build_flat_field function.

Diagnostic plot: An image of the flat field in log scale is output in the cals subdirectory. It is generated with cal_image_2d_plot.

Creating a bad pixel mask

With the super bias and flat field calibration files generated, we can create a bad pixel mask of the detector to exclude from subsequent processing and final 1D spectrum extraction.

We do this based on thresholding the super bias and flat field image values. The constraints on identifying a bad pixel are defined in the config file. Currently, there is a single type of constraint available for each of the calibration images:

• For the super bias, we define a bad pixel (in this case a hot pixel) as anything above a certain percentile of the super bias flux image. (The default is 99.9%)

• For the flat field, we define a bad pixel as anything with a very low response below a certain value. (The default is \(10^{-4}\))

Here is an example of a bad pixel mask for the Tull coudé spectrograph. The super bias is plotted in log flux in the background, but there isn’t much to see since the hot pixels are underneath the bad pixel points. There are a few bad columns of the CCD, and the overscan region is seen as the block of “bad pixels” at the right end.

Bad pixel mask FITS file structure

The bad pixel mask is saved as an image of 0s and 1s, where 0s denote the bad pixels. It has the same shape as the detector images. The bad pixel mask is included as the primary extension’s data. There are a couple of HISTORY entries added with the bias and flat field thresholds for defining bad pixels.

These steps for generating the bad pixel mask file are in the make_bad_pixel_mask function.

Diagnostic plot: An image of the super bias with the bad pixels overplotted as points is output in the cals subdirectory. It is generated with cal_image_2d_plot.

CCD Image Processing

Now that the calibration files have been generated, we can process the rest of the science images from the night of observations. We need to bias correct and flat field each of the science images (including arc lamp and on-sky observations), and also remove cosmic ray contamination from on-sky observations. These steps are included in the image_processing module, and options are defined in the image_proess section of the main config YAML file, which is described in full here

For on-sky observations, we first remove cosmic ray contamination using the AstroSCRAPPY package. This package is based on the L.A.Cosmic algorithm – see the documentation for more information. In the astroscrappy.detect_cosmics routine we mostly use the default parameters except for sigclip which is set to 5 and the number of iterations which is set in the config file. There is also a config option to skip cosmic ray subtraction altogether.

After the potential cosmic ray subtraciton all science images are bias subtracted and flat fielded, following: \(( I - B ) / F\) where \(I\) is the science flux image, \(B\) is the super bias flux, and \(F\) is the flat field response values. All error images, including the photon noise and read noise on the science image, are propagated to produce an error image for the processed frame.

We then read in the bad pixel mask and replace all bad pixels with nans, which are handled throughout the rest of the pipeline.

Warning

The image processing module does not currently perform scattered/background light subtraction. It is done in a sense during the extraction step when the background is subtracted from the pixel slices, but a uniform fitting of the background across the detector should be performed. This is an area of active development and will be included in a future version. It would also require the re-ordering of steps in the main reduction run script, because the orders would need to be traced to mask them from fitting the bcakground.

Additional config file options:

The main config file has additional options to include/exclude files from processing:

• An option to set the header imagetyp keyword used for the arc lamp spectra (in our case with Tull, this is “comp”)

• An option to list header object keyword values that are valid for arc lamps (in our case, this excludes arc lamp frames taken for spectrograph focusing)

• An option to list header object keyword values that are invalid for science frames (in our case, this excludes “test” frames)

Processed image FITS file structure

The processed images are saved to new FITS files in the object_files subdirectory of the reduction directory. Their file names follow the format: “tullcoude_YYYYMMDDTHHMMSS.fits”, where YYYYMMDDTHHMMSS is the file token used to uniquely define each observation. This new file name is added as a keyword to the header of the output file’s primary extension.

The file has two extensions:

• Index 0: The primary extension with the header copied from the raw output CCD image file and data containing the processed image.

• Index 1: Extension named “error” with the processed error image as its data entry.

There are also HISTORY entries added nothing which steps are done (bias subtraction, flat fielding, cosmic ray subtraction) and when they were performed.

All of the above steps are executed with the build_images function.

Note

The image processing step is not particularly modular: each of the steps is written directly into the “wrapper” function in the image_processing module.

This is from the relatively simple nature of the current image processing (for example, the lack of a scattered light subtraction). This will be an active area of development in the future, and some of these steps may be replaced with functionality from the CCDPROC package for uniformity with community standards.