Site Map

The Imaging Pipeline

Introduction and Resources

The Early Data Release (EDR) paper is the fundamental resource for understanding the processing and data products from the SDSS, describing the pipelines and contents of generated data products. Successive data release papers: DR1, DR2, DR3, DR4, DR5, DR6 and DR7, describe changes between data releases. The technical summary provides more general information on the survey. The SDSS Project Book, written for a funding proposal early in the survey, is the most exhaustive description of the survey but was last updated in 1997.

The algorithms page includes links to pages describing algorithms used by the data reduction pipelines, and the SAS datamodel has a table of the most commonly useful files in the SAS. Truly gory details can be found on Robert Lupton's page of photo documents, including the draft photometric reduction paper as well as a treasure trove of photometric data reduction wisdom.

The remainder of this page starts with a brief overview of imaging data processing, followed by sections that describe the steps in data processing in detail. In addition to the more detailed description, each detail section privedes references to papers that give additional details, and a table of the files associated with that step of the pipeline that can be found in the SAS. These tables include links to descriptions of the formats of those files and templates that can be used to generate SAS URLs for those files. The templates are in "C printf" format, and can be used in C, bash, Python, and many other languages to automatically generate URLs.

Most of the catalog data (but not the images themselves) have been loaded into the Catalog Archive Server (CAS) database. Users are often better off obtaining SDSS-III data through a carefully constructed CAS query than they are downloading the data files from the SAS. Simple queries can be used to select just the objects and parameters of interest, while more complex queries can be used to do complex calculations on many objects, thereby avoiding the need to download the data on them at all.

Overview

Notes

Imaging Observing

Imager

The imager collected survey data from the imager in drift scan (also called TDI) mode. We use the term "run" to designate a single scan along a great circle. The data acquisition system system divided the data from each CCD into frames, stored as FITS files; the observatory sent data to be processed as a collection of FITS files, each corresponding to an arc along the great circle 1361 rows (539 arcseconds) long from a single CCD. In addition to the images themselves, the data acquisition system also generated a variety of metadata and other engineering files, which included pixel statistics and a catalog of bright stars for each field.

SAS files generated in imager data collection
File Type in/out Description URL format format parameters
idReport out records runs collected on a night groups/boss/photo/redux/%d/%d/logs/idReport-%05d.par rerun, run, mjd
idR out raw imaging frames groups/boss/photo/data/%d/fields/%d/idR-%06d-%s%d-%04d.fit.Z run, camcol, run, filter, camcol, field
idGang out assorted data gathered by the data acquisition system groups/boss/photo/data/%d/gangs/idGang-%06d-%s%d-%04d.fit run, run, r(ow) or c(ol), row number or column number, frame
idFrameLog out camera position information for each frame groups/boss/photo/redux/%d/%d/logs/idFrameLog-%06d-%d.par rerun, run, run, crate
opCamera out imaging camera geometry groups/boss/photo/redux/%d/%d/logs/opCamera-%05d.par rerun, run, mjd (of generation)
opConfig out CCD readout configurations groups/boss/photo/redux/%d/%d/logs/opConfig-%05d.par rerun, run, mjd (of generation)
opECalib out CCD electronic calibrations groups/boss/photo/redux/%d/%d/logs/opECalib-%05d.par rerun, run, mjd (of generation)

Infrared All-Sky Camera

Clouds can be clearly seen at night in a 10 micron all-sky camera. We used two such cameras at different times in the survey to monitor the sky for cloud (and therefore non-photometric conditions).

SAS files generated by the infrared all sky camera or scanner
File Type in/out Description URL format format parameters
cR out A single image from the infrared camera apo_staging/ircam/%05d/cR%06d.fit mjd, UTC time (HHMMSS)
irsc out A log of statistics on ircam frames apo_staging/ircam/%05d/irsc.log mjd

Other metadata

The observatory produces an assortment of engineering and other metadata.

Other SAS files generated in observing
File Type in/out Description URL format format parameters
night log out A prose account of the night apo_staging/astrolog/%05d/manualLog-full.txt mjd
idWeather out A log of weather data groups/boss/photo/redux/%d/%d/logs/idWeather-%05d.par rerun, run, mjd

Imaging Data Reduction

Serial Stamp Collecting Pipeline (SSC)

The SSC repackages some of the data produced by the data acquisition system and produces postage stamps of bright reference stars.

SAS files read by the SSC pipeline
File Type in/out Description URL format format parameters
scPlan in the SSC processing plan groups/boss/photo/redux/%d/%d/ssc/scPlan.par run, rerun
idReport in records runs collected on a night groups/boss/photo/redux/%d/%d/logs/idReport-%05d.par rerun, run, mjd
opCamera in imaging camera geometry groups/boss/photo/redux/%d/%d/logs/opCamera-%05d.par rerun, run, mjd (of generation)
opConfig in CCD readout configurations groups/boss/photo/redux/%d/%d/logs/opConfig-%05d.par rerun, run, mjd (of generation)
opECalib in CCD electronic calibrations groups/boss/photo/redux/%d/%d/logs/opECalib-%05d.par rerun, run, mjd (of generation)
opBC in CCD bad pixel file groups/boss/photo/redux/%d/%d/log/opBC-%d.par rerun, run, mjdbp (see fpPlan)
idR in raw imaging frames groups/boss/photo/data/%d/fields/%d/idR-%06d-%s%d-%04d.fit.Z run, camcol, run, filter, camcol, field
idGang in assorted data gathered by the data acquisition system groups/boss/photo/data/%d/gangs/idGang-%06d-%s%d-%04d.fit run, run, r(ow) or c(ol), row number or column number, frame
koCat in the catalog of known bright stars groups/boss/photo/redux/%d/%d/ssc/koCat-%06d.fit run, rerun, run

Postage Stamp Pipeline (PSP)

References: 2002AJ....123..485S, section 4.3

The PSP measures the bias and global sky level, and fits a model for the point spread function for each field.

SAS files used or generated by the PSP pipeline
File Type in/out Description URL format format parameters
psParam in PSP tunable parameters groups/boss/photo/redux/%d/%d/photo/psParam.par run, rerun
psPlan in the PSP processing plan groups/boss/photo/redux/%d/%d/photo/psPlan.par run, rerun
opCamera in imaging camera geometry groups/boss/photo/redux/%d/%d/logs/opCamera-%05d.par rerun, run, mjd (of generation)
opConfig in CCD readout configurations groups/boss/photo/redux/%d/%d/logs/opConfig-%05d.par rerun, run, mjd (of generation)
opECalib in CCD electronic calibrations groups/boss/photo/redux/%d/%d/logs/opECalib-%05d.par rerun, run, mjd (of generation)
opBC in CCD bad pixel file groups/boss/photo/redux/%d/%d/log/opBC-%d.par rerun, run, mjdbp (see fpPlan)
idB in bias vector groups/boss/photo/redux/%d/%d/photo/calib/idB-%06d-%c%d.fit rerun, run, calibration run, filter, camcol
psField out the initial photometric calibration and point spread function fit by field (see the extracting PSF images page) groups/boss/photo/redux/%d/%d/objcs/%d/psField-%06d-%d-%04d.fit rerun, run, camcol, run, camcol, field

Astrometric Pipeline (astrom)

References: 2003AJ....125.1559P

The astrometric pipeline calculates the astrometric solution. The astrometry and astrometry QA (UPDATE LINK) web pages provide more information on the algorithms used and the proper interpretation of the output.

SAS files used or generated by the astrom pipeline
File Type in/out Description URL format format parameters
asParam in astrom tunable parameters groups/boss/photo/redux/%d/%d/astrom/asParam.par rerun, run
asPlan in the astrom processing plan groups/boss/photo/redux/%d/%d/astrom/asPlan.par rerun, run
opCamera in imaging camera geometry groups/boss/photo/redux/%d/%d/logs/opCamera-%05d.par rerun, run, mjd (of generation)
opConfig in CCD readout configurations groups/boss/photo/redux/%d/%d/logs/opConfig-%05d.par rerun, run, mjd (of generation)
idReport in records runs collected on a night groups/boss/photo/redux/%d/%d/logs/idReport-%05d.par rerun, run, mjd
idWeather in A log of weather data groups/boss/photo/redux/%d/%d/logs/idWeather-%05d.par rerun, run, mjd
asTrans out tranformation coefficients from row, column to great circle coordinates groups/boss/photo/redux/%d/%d/astrom/asTrans-%06d.fit rerun, run, run

Frames Pipeline (frames)

References: 2002AJ....123..485S, section 4.4

The frames pipeline applies flat field and bias corrections to each frame, and interpolates values for pixels in bad columns and bleed trails and those corrupted by cosmic rays. In previous data releases, this pipeline would produce a "corrected frame" or fpC file. These files are no longer produced. Instead, a frame file is produced by a different method, though it contains substantially the same information. The frames pipeline described here also detects bright objects and estimates the varying sky background, detects fainter objects, and measures a variety of properties for each object. Several algorithms pages describe details of the processing performed by frames.

SAS files used or generated by the frames pipeline
File Type in/out Description URL format format parameters
fpParam in frames tunable parameters groups/boss/photo/redux/%d/%d/photo/fpParam.par rerun, run
fpPlan in frames processing plan groups/boss/photo/redux/%d/%d/photo/fpPlan.par rerun, run
idReport in records runs collected on a night groups/boss/photo/redux/%d/%d/logs/idReport-%05d.par rerun, run, mjd
koCat in the catalog of known bright stars groups/boss/photo/redux/%d/%d/ssc/koCat-%06d.fit run, rerun, run
opConfig in CCD readout configurations groups/boss/photo/redux/%d/%d/logs/opConfig-%05d.par rerun, run, mjd (of generation)
opECalib in CCD electronic calibrations groups/boss/photo/redux/%d/%d/logs/opECalib-%05d.par rerun, run, mjd (of generation)
opBC in CCD bad pixel file groups/boss/photo/redux/%d/%d/log/opBC-%d.par rerun, run, mjdbp (see fpPlan)
idB in bias vector groups/boss/photo/redux/%d/%d/photo/calib/idB-%06d-%c%d.fit rerun, run, calibration run, filter, camcol
psField in the initial photometric calibration and point spread function fit by field (see the extracting PSF images page) groups/boss/photo/redux/%d/%d/objcs/%d/psField-%06d-%d-%04d.fit rerun, run, camcol, run, camcol, field
asTrans in tranformation coefficients from row, column to great circle coordinates groups/boss/photo/redux/%d/%d/astrom/asTrans-%06d.fit rerun, run, run
fpFieldStat out field statistics groups/boss/photo/redux/%d/%d/objcs/%d/fpFieldStat-%06d-%d-%04d.fit rerun, run, camcol, run, camcol, field
fpObjc out the (uncalibrated) object catalog groups/boss/photo/redux/%d/%d/objcs/%d/fpObjc-%06d-%d-%04d.fit rerun, run, camcol, run, camcol, field
fpBIN out a 4x4 binned version of the corrected image after masking of objects and subtraction of sky, an image of the estimate of the sky, an image of the uncertainty in the sky estimate, and a table of bright star wings groups/boss/photo/redux/%d/%d/objcs/%d/fpBIN-%06d-%c%d-%04d.fit.gz rerun, run, camcol, run, filter, camcol, field
fpM out the frame masks (read using readAtlasImages) groups/boss/photo/redux/%d/%d/objcs/%d/fpM-%06d-%c%d-%04d.fit.gz rerun, run, camcol, run, filter, camcol, field
fpAtlas out the atlas images for all objects detected (read using readAtlasImages) groups/boss/photo/redux/%d/%d/objcs/%d/fpAtlas-%06d-%d-%04d.fit rerun, run, camcol, run, camcol, field

Resolve Multiple Detections

A given area on the sky may be observed by multiple runs. The final calibrated catalogs contain a field declaring whether the the specific observation of the object is "primary," and the canonical measurement of the object, or "secondary," a duplicate. These catalogs are generated twice, once based on the single run in isolation by exportChunk, and again in the resolve step. The catalogs produced by exportChunk declare an object primary or secondary based on position alone, introducing the possibility of inconsistencies due to errors in position; it is possible, for example, that two separate observation of an object be declared primary if the position measured for the object is slightly different in each detection.

The catalog generated by the resolve step, on the other hand, uses an operational database to search for other detections, and guarantees that one and only one detection be declared primary.

See the EDR paper, 2002AJ....123..485S, section 4.7 for details on how objects are resolved.

The catalogs produced by exportChunk can be found in the run directory of each run:

SAS files used or generated by the resolve stage
File Type in/out Description URL format format parameters
reObjGlobal out Global resolve status for every object in a field groups/boss/resolve/%s/%d/%d/resolve/%d/reObjGlobal-%06d-%d-%04d.fit resolve rerun, rerun, run, camcol, run, camcol, field
thingList out The full list of catalog entries for the resolved survey groups/boss/resolve/%s/thingList.fits resolve rerun
thingIndex out The full list of unique primary objects, gives position of an object in the thingList groups/boss/resolve/%s/thingIndex.fits resolve rerun
window_flist out The full list of fields used to determine the window function groups/boss/resolve/%s/window_flist.fits resolve rerun
window_unified out The full list of polygons determining the primary area of the window function groups/boss/resolve/%s/window_unified.fits resolve rerun
window_blist out The full list of balkans determining the primary area of the window function groups/boss/resolve/%s/window_blist.fits resolve rerun
window_bcaps out The full list of caps determining the primary area of the window function groups/boss/resolve/%s/window_bcaps.fits resolve rerun
window_bindx out Matches balkans to fields in the window function, sorted by balkans groups/boss/resolve/%s/window_bindx.fits resolve rerun
window_findx out Matches balkans to fields in the window function, sorted by sorted by fields groups/boss/resolve/%s/window_findx.fits resolve rerun

Photometric Recalibration (Ubercal)

References: 2008ApJ...674.1217P

The "ubercal" recalibration recalibrates the survey by simultaneously solving for the calibration parameters and relative stellar fluxes using overlapping observations. The algorithm used decouples the problem of relative calibrations from that of absolute calibrations; the absolute calibration is reduced to determining a few numbers for the entire survey. See the photometric calibration algorithms page.

SAS files generated in photometric recalibration
File Type in/out Description URL format format parameters
calibPhotomGlobal out Ubercal results (nanomaggies/count) and associated flat fields groups/boss/calib/%s/%d/%d/nfcalib/calibPhotomGlobal-%06d-%d.fits calibration rerun, rerun, run, run, camcol
calib out Uberal quality assurance plots groups/boss/calib/%s/plots/calib-%06d-%c-%s.png calibration rerun, run, filter, type (flat, hist, run)

Photometric Catalogs

The final photometric catalogs contain the raw data that will be loaded into the CAS database.

SAS files produced by the final catalog
File Type in/out Description URL format format parameters
photoObj out Full, calibrated outputs of the imager photometric pipeline groups/boss/photoObj/%d/%d/%d/photoObj-%06d-%d-%04d.fits rerun, run, camcol, run, camcol, field
photoField out Summary outputs of the properties of every field groups/boss/photoObj/%d/%d/photoField-%06d-%d.fits rerun, run, run, camcol
photoRunAll out Summary information of the properties of every photometric run groups/boss/photoObj/photoRunAll-%s.fits release (dr8)