Spectroscopic Data Processing
Introduction and Resources
The Early Data Release (EDR) paper is the fundamental resource for understanding the processing and data products from the SDSS, describing the pipelines and contents of generated data products. Successive data release papers: DR1, DR2, DR3, DR4, DR5, DR6 and DR7, describe changes between data releases. The technical summary provides more general information on the survey. The SDSS Project Book, written for a funding proposal early in the survey, is the most exhaustive description of the survey but was last updated in 1997. The SDSS publications web page (UPDATE LINK?) includes a list of additional SDSS technical papers.
The algorithms page includes links to pages describing algorithms used by the data reduction pipelines, and the SAS datamodel has a table of the most commonly useful files in the SAS.
The remainder of this page starts with a brief overview of spectroscopic data processing, followed by sections that describe the steps in data processing in detail. In addition to the more detailed description, each detail section privedes references to papers that give additional details, and a table of the files associated with that step of the pipeline that can be found in the SAS. These tables include links to descriptions of the formats of those files and templates that can be used to generate SAS URLs for those files. The templates are in "C printf" format, and can be used in C, bash, Python, and many other languages to automatically generate URLs.
Most of the catalog data (but not the images themselves) have been loaded into the Catalog Archive Server (CAS) database. Users are often better off obtaining SDSS data through a carefully constructed CAS query than they are downloading the data files from the SAS. Simple queries can be used to select just the objects and parameters of interest, while more complex queries can be used to do complex calculations on many objects, thereby avoiding the need to download the data on them at all.
Overview
- Spectroscopic Observing
The spectrographs mounted on the primary 2.5m telescope collected spectra from each plate. There were two spectrographs, each of which collected data from 320 fibers. Each spectrograph had two CCDs, so the instrument produced a total of four images for each exposure. - Spectroscopic Data Reduction
The spectroscopic pipelines extracted one dimensional spectra from the raw exposures produced by the spectrographs, calibrated them in wavelength and flux, measured features in these spectra, measured redshifts from these features, and classified the objects as galaxies, stars, or quasars.
Notes
- In the tables below, add 'http://data.sdss3.org/sas/dr8/' to all 'URL format' values to get the full URL.
- Data reduction could occur multiple times for both images and spectra. Each time we repeated a data reduction, we labeled the output from that reduction process with a distinct rerun number.
- The target selection algorithms page describes how the pipeline performs target selection, including selection of Main Galaxy Sample, Luminous Red Galaxies (LRG), Quasars, Stars, ROSAT All-Sky Survey sources, Serendipity, and SEGUE targets. The target selection pipeline extracts calibrated catalogs of objects and corresponding field by field statistics from the operations database.
- The tiling algorithms page describes the process by which the spectroscopic plates are designed and placed relative to each other.
Spectroscopic Observing
Plate Plugging (plug)
When the observatory is ready to observe a plate, the observatory staff plugs optical fibers into the holes drilled into the plates, and maps which fiber correponds to which hole (and therefor which object) by shining light through each fiber. This data is incorporated into one of the HDUs of the spPlate file described below.
Data Collection
Observers mount cartridges containing the drilled, plugged plates on the telescope, and collected exposures on each plate until it reached a threshold estimated signal to noise and at least three exposures had been collected.
| File Type | in/out | Description | URL format | format parameters |
|---|---|---|---|---|
| sdReport | out | records exposures collected on a night | Not public | |
| sdR | out | raw spectroscopic data frames | common/sdss-spectro/raw/%d/sdR-%c%d-%08d.fit.gz | mjd, CCD (r or b), camera (1 or 2), exposure id |
Spectroscopic Data Reduction
Two-dimensional Pipeline (spectro2d, also called idlspec2d)
References: 2002AJ....123..485S, section 4.10.1
The spectro2d pipeline reads science and calibration exposures from the spectrographs, reduces and calibrates the science exposures, extracts the one dimensional spectra from the two dimensional exposures, stacks multiple exposures into combined spectra, and produces corresponding masks and noise estimates.
| File Type | in/out | Description | URL format | format parameters |
|---|---|---|---|---|
| spPlan2d | in | the spectro2d processing plan | common/sdss-spectro/redux/%d/%04d/spPlan2d-%04d-%d.par | rerun, plate, plate, mjd |
| spPlancomb | in | the processing plan for combining spectra | common/sdss-spectro/redux/%d/%04d/spPlancomd-%04d-%d.par | rerun, plate, plate, mjd |
| plPlugMapM | in | records which fiber corresponds to which hole in a plate (and therefore objects, and what coordinates on the sky) | Not public | |
| sdReport | in | records exposures collected on a night | Not public | |
| sdR | out | raw spectroscopic data frames | common/sdss-spectro/raw/%d/sdR-%c%d-%08d.fit.gz | mjd, CCD (r or b), camera (1 or 2), exposure id |
| spCFrame | out | calibrated spectra for a single CCD and exposure | common/sdss-spectro/redux/%d/%04d/spCFrame-%c%d-%08d.par | rerun, plate, CCD (r or b), camera, exposure id |
| spPlate | out | the 640 combined flux- and wavelength-calibrated spectra over all exposures (potentially spanning multiple nights) for a given mapped plate | common/sdss-spectro/redux/%d/%04d/spPlate-%04d-%d.par | rerun, plate, plate, mjd |
One-dimensional Pipeline
The idlspec2d product incorporates the Princeton-1D pipeline (sometimes called specBS) which produces the following files:
| File Type | in/out | Description | URL format | format parameters |
|---|---|---|---|---|
| spZline | out | emission line fits | common/sdss-spectro/redux/%d/%04d/spZline-%04d-%d.fits | rerun, plate, plate, mjd |
| spZall | out | all spectroscopic classifications and redshifts | common/sdss-spectro/redux/%d/%04d/spZall-%04d-%d.fits | rerun, plate, plate, mjd |
| spZbest | out | spectroscopic classifications and redshifts | common/sdss-spectro/redux/%d/%04d/spZbest-%04d-%d.fits | rerun, plate, plate, mjd |
Stellar Parameters Pipeline (sspp)
References: 2008AJ....136.2022L, 2008AJ....136.2050L, 2008AJ....136.2070A
The SEGUE stellar parameters pipeline produces a number of files, stored together:
| File Type | in/out | Description | URL format | format parameters |
|---|---|---|---|---|
| ssppOut | out | SSPP stellar parameters ([Fe/H], log g, etc.) | groups/segue2/sspp/%d/%04d/output/param/ssppOut-%04d-%5d.fit | rerun, plate, plate, mjd |
| ssppOut_lineindex | out | SSPP line indices | groups/segue2/sspp/%d/%04d/output/param/ssppOut-%04d-%5d.lineindex.fit | rerun, plate, plate, mjd |