Output Products

pyKOSMOS++ generates a standardized set of output products organized in a structured directory hierarchy. This page documents the format and content of all output files.

Directory Structure

The pipeline creates the following output directory structure:

output_dir/
├── calibrations/
│   ├── master_bias.fits
│   ├── master_flat.fits
│   └── bad_pixel_mask.fits
├── reduced_2d/
│   ├── science_001_2d.fits
│   ├── science_002_2d.fits
│   └── ...
├── spectra_1d/
│   ├── science_001_trace1.fits
│   ├── science_001_trace2.fits
│   ├── science_002_trace1.fits
│   └── ...
├── wavelength_solutions/
│   ├── wavelength_solution_arc001.fits
│   └── ...
├── quality_reports/
│   ├── science_001_quality.yaml
│   ├── science_002_quality.yaml
│   └── summary_report.txt
├── diagnostic_plots/
│   ├── wavelength_solution.png
│   ├── science_001_2d.png
│   ├── science_001_trace1_profile.png
│   └── ...
└── logs/
    └── reduction_log.txt

Calibration Products

master_bias.fits

Combined master bias frame.

Format: FITS image (Primary HDU)

Header Keywords:

NAXIS   = 2
NAXIS1  = 2048          / Spectral axis
NAXIS2  = 515           / Spatial axis
BUNIT   = 'ADU'

# Calibration metadata
NCOMBINE= 5             / Number of bias frames combined
COMBMETH= 'median'      / Combination method
BIASLVL = 389.2         / Median bias level (ADU)
BIASSTD = 3.5           / Bias standard deviation (ADU)

# Provenance
PIPELINE= 'pyKOSMOS++ v0.1.0'
DATE    = '2025-01-15T10:23:45'
PROCSTEP= 'BIAS_COMBINE'

Data: 2D float32 array, bias level in ADU

Usage:

from astropy.io import fits
with fits.open('master_bias.fits') as hdul:
    bias_data = hdul[0].data
    bias_level = hdul[0].header['BIASLVL']

master_flat.fits

Normalized master flat field.

Format: FITS image (Primary HDU)

Header Keywords:

NAXIS   = 2
BUNIT   = 'normalized'

NCOMBINE= 3             / Number of flat frames combined
COMBMETH= 'median'
NORMLVL = 1.0           / Normalization level
BADPIXFR= 0.0023        / Bad pixel fraction

PIPELINE= 'pyKOSMOS++ v0.1.0'
PROCSTEP= 'FLAT_COMBINE'

Data: 2D float32 array, normalized response (median = 1.0)

Bad Pixel Mask: Pixels with response <0.5 or >1.5 are flagged

bad_pixel_mask.fits

Boolean mask of bad pixels.

Format: FITS image (Primary HDU)

Data: 2D boolean array (0=good, 1=bad)

Bad pixel criteria:

Flat response <0.5 or >1.5 (vignetting or hot pixels)
Saturated pixels in calibration frames
Cosmic ray hits in master flat

Reduced 2D Spectra

science_NNN_2d.fits

Calibrated 2D spectrum with trace information.

Format: Multi-extension FITS

Extension 0 (Primary): Calibrated 2D spectrum

NAXIS   = 2
BUNIT   = 'ADU'

# Original frame metadata
OBJECT  = 'NGC1234'
EXPTIME = 1200.0        / seconds
AIRMASS = 1.15
DATEOBS = '2024-01-15T05:30:00'

# Calibration applied
BIASFILE= 'master_bias.fits'
FLATFILE= 'master_flat.fits'
BIASCORR= T
FLATCORR= T
COSMCORR= T             / Cosmic ray cleaning applied

PIPELINE= 'pyKOSMOS++ v0.1.0'
PROCSTEP= '2D_CALIBRATION'

Extension 1 (VARIANCE): Variance array

Propagated from read noise and Poisson statistics
Same shape as primary data

Extension 2 (MASK): Pixel mask

0 = good pixel
1 = bad pixel from flat
2 = cosmic ray detected
4 = saturation
8 = other (can combine bits)

Extension 3 (TRACES): Binary table of detected traces

Columns:

TRACE_ID (int): Trace identifier
SPATIAL_POS (float array): Spatial center positions [pixels]
SPECTRAL_PIX (float array): Spectral pixel coordinates
SNR_EST (float): Estimated signal-to-noise ratio

Usage:

with fits.open('science_001_2d.fits') as hdul:
    data = hdul[0].data
    variance = hdul[1].data
    mask = hdul[2].data
    traces = hdul[3].data

Extracted 1D Spectra

science_NNN_traceM.fits

Extracted and wavelength-calibrated 1D spectrum.

Format: Multi-extension FITS (compatible with Astropy Spectrum1D)

Extension 0 (Primary): Header only

# Source information
OBJECT  = 'NGC1234'
TRACEID = 1
EXPTIME = 1200.0
AIRMASS = 1.15

# Extraction metadata
EXTMETH = 'optimal'     / Extraction method
APWIDTH = 10            / Aperture width (pixels)
SKYBUFFE= 30            / Sky buffer distance (pixels)

# Quality metrics
MED_SNR = 12.5          / Median signal-to-noise ratio
WLRMS   = 0.08          / Wavelength RMS residual (Angstroms)
GRADE   = 'Good'        / Quality grade

PIPELINE= 'pyKOSMOS++ v0.1.0'
PROCSTEP= '1D_EXTRACTION'

Extension 1 (FLUX): 1D flux array

NAXIS: 1
NAXIS1: Number of wavelength points
BUNIT: ‘ADU’ or ‘erg/s/cm^2/Angstrom’ (if flux calibrated)

Extension 2 (WAVELENGTH): Wavelength array

BUNIT: ‘Angstrom’
WAVEMIN, WAVEMAX: Wavelength range

Extension 3 (UNCERTAINTY): 1D uncertainty array

Propagated from variance in 2D spectrum
Same units as flux

Extension 4 (MASK): 1D mask array

0 = good pixel
Nonzero = various flags (cosmic ray, saturation, etc.)

Usage:

from specutils import Spectrum1D
from astropy import units as u

# Load as Spectrum1D
spectrum = Spectrum1D.read('science_001_trace1.fits')

# Access data
flux = spectrum.flux            # with units
wavelength = spectrum.spectral_axis
uncertainty = spectrum.uncertainty

# Or use FITS directly
with fits.open('science_001_trace1.fits') as hdul:
    flux = hdul['FLUX'].data
    wavelength = hdul['WAVELENGTH'].data
    uncertainty = hdul['UNCERTAINTY'].data

Wavelength Solutions

wavelength_solution_arcNNN.fits

Wavelength calibration solution for arc frame.

Format: FITS binary table (Primary HDU)

Columns:

PIXEL (float): Pixel position
WAVELENGTH (float): Wavelength in Angstroms
RESIDUAL (float): Fit residual in Angstroms
INTENSITY (float): Line intensity
USED (bool): Line included in fit (not clipped)

Header Keywords:

LAMPTYPE= 'HeNeAr'     / Arc lamp type
POLYORD = 5            / Polynomial order
NLINES  = 45           / Total lines identified
NUSED   = 42           / Lines used in fit
RMS     = 0.078        / RMS residual (Angstroms)
WAVEMIN = 3650.0       / Minimum wavelength (Angstroms)
WAVEMAX = 7200.0       / Maximum wavelength (Angstroms)

# Polynomial coefficients
COEFF0  = 3645.123
COEFF1  = 1.0234
COEFF2  = -0.000123
...

Usage:

with fits.open('wavelength_solution_arc001.fits') as hdul:
    table = hdul[1].data
    header = hdul[0].header

    # Extract coefficients
    order = header['POLYORD']
    coeffs = [header[f'COEFF{i}'] for i in range(order+1)]

    # Reconstruct solution
    from numpy.polynomial.chebyshev import chebval
    wavelength = chebval(pixel, coeffs)

Quality Reports

science_NNN_quality.yaml

Detailed quality metrics for each science frame.

Format: YAML

Structure:

# Source metadata
source_file: science_001.fits
object_name: NGC1234
exposure_time: 1200.0
observation_date: '2024-01-15T05:30:00'

# Calibration quality
calibration:
  bias_level: 389.2
  bias_stdev: 3.5
  flat_bad_pixel_fraction: 0.0023
  cosmic_ray_fraction: 0.0087

# Wavelength solution
wavelength:
  polynomial_order: 5
  rms_residual: 0.078
  n_lines_identified: 45
  n_lines_used: 42
  wavelength_range: [3650.0, 7200.0]

# Extraction metrics (per trace)
traces:
  - trace_id: 1
    median_snr: 12.5
    peak_snr: 45.2
    spatial_center: 257.5
    profile_consistency: 0.92
    flux_conservation: 0.98

  - trace_id: 2
    median_snr: 8.3
    ...

# Overall assessment
overall_grade: Good
quality_flags: []
warnings:
  - 'Trace 2 has lower SNR (8.3 < 10.0 recommended)'

# Processing metadata
pipeline_version: '0.1.0'
processing_date: '2025-01-15T10:45:00'
processing_time_seconds: 125.3

Usage:

import yaml
with open('science_001_quality.yaml') as f:
    quality = yaml.safe_load(f)

print(f"Overall grade: {quality['overall_grade']}")
for trace in quality['traces']:
    print(f"Trace {trace['trace_id']}: SNR = {trace['median_snr']:.1f}")

summary_report.txt

Human-readable summary of all processed frames.

Format: Plain text

Example:

pyKOSMOS++ Pipeline Summary Report
==================================

Processing Date: 2025-01-15 10:45:00
Pipeline Version: 0.1.0

Input Directory: data/2024-01-15
Output Directory: reduced/2024-01-15

Calibration Summary:
-------------------
Master Bias:  5 frames combined, level=389.2±3.5 ADU
Master Flat:  3 frames combined, bad pixels=0.23%
Arc Lamp:     HeNeAr, 45 lines identified, RMS=0.078 Å

Science Frames Processed: 10
----------------------------

Frame                    Traces  Grade      Median SNR  Wavelength RMS
science_001.fits         2       Good       12.5        0.078 Å
science_002.fits         2       Excellent  23.1        0.065 Å
science_003.fits         1       Fair       7.2         0.092 Å
...

Overall Statistics:
------------------
Total spectra extracted: 15
Mean SNR: 14.8
Grade distribution:
  Excellent: 3 (20%)
  Good: 10 (67%)
  Fair: 2 (13%)
  Poor: 0 (0%)

Processing Time: 2.3 minutes (13.8 seconds/frame)

Diagnostic Plots

wavelength_solution.png

Wavelength calibration diagnostic.

Contents:

Top panel: Wavelength vs pixel with Chebyshev fit
Bottom panel: Fit residuals with RMS threshold lines

Format: PNG, 14x8 inches, 300 DPI

science_NNN_2d.png

Calibrated 2D spectrum with detected traces.

Contents:

2D spectrum with log-scale colormap
Overlaid trace positions (red lines)
Wavelength and spatial axes labeled

Format: PNG, 14x6 inches, 300 DPI

science_NNN_traceM_profile.png

Spatial profile fit diagnostic.

Contents:

Top panel: Data vs fitted profile (Gaussian/Moffat)
Bottom panel: Fit residuals

Format: PNG, 10x6 inches, 300 DPI

science_NNN_traceM_1d.png

Extracted 1D spectrum.

Contents:

Flux vs wavelength
Major spectral features annotated (if known)

Format: PNG, 14x4 inches, 300 DPI

Logs

reduction_log.txt

Detailed processing log.

Format: Plain text with timestamps

Example:

2025-01-15 10:30:00 INFO: pyKOSMOS++ v0.1.0 starting
2025-01-15 10:30:00 INFO: Input directory: data/2024-01-15
2025-01-15 10:30:01 INFO: Discovered 5 bias, 3 flat, 1 arc, 10 science frames
2025-01-15 10:30:02 INFO: Creating master bias from 5 frames
2025-01-15 10:30:05 INFO: Master bias: level=389.2±3.5 ADU
2025-01-15 10:30:05 INFO: Creating master flat from 3 frames
2025-01-15 10:30:12 INFO: Master flat: bad pixels=0.23%
2025-01-15 10:30:13 INFO: Fitting wavelength solution
2025-01-15 10:30:15 INFO: Wavelength solution: order=5, RMS=0.078 Å, 45 lines
2025-01-15 10:30:16 INFO: Processing science_001.fits
2025-01-15 10:30:18 INFO:   Detected 2 traces
2025-01-15 10:30:25 INFO:   Trace 1: SNR=12.5, extracted
2025-01-15 10:30:30 INFO:   Trace 2: SNR=8.3, extracted
2025-01-15 10:30:30 INFO:   Overall grade: Good
...
2025-01-15 10:32:45 INFO: Pipeline completed successfully
2025-01-15 10:32:45 INFO: Total time: 2.75 minutes

Reading Output Products

Python Example

Complete workflow to read and analyze outputs:

from pathlib import Path
from astropy.io import fits
from specutils import Spectrum1D
import yaml
import matplotlib.pyplot as plt

output_dir = Path('reduced/2024-01-15')

# Read quality report
with open(output_dir / 'quality_reports/science_001_quality.yaml') as f:
    quality = yaml.safe_load(f)

print(f"Overall grade: {quality['overall_grade']}")
print(f"Median SNR: {quality['traces'][0]['median_snr']:.1f}")

# Load 1D spectrum
spectrum_file = output_dir / 'spectra_1d/science_001_trace1.fits'
spectrum = Spectrum1D.read(spectrum_file)

# Plot spectrum
plt.figure(figsize=(12, 4))
plt.plot(spectrum.spectral_axis, spectrum.flux)
plt.xlabel(f'Wavelength ({spectrum.spectral_axis.unit})')
plt.ylabel(f'Flux ({spectrum.flux.unit})')
plt.title(f"{quality['object_name']} - Grade: {quality['overall_grade']}")
plt.show()

# Load 2D spectrum
with fits.open(output_dir / 'reduced_2d/science_001_2d.fits') as hdul:
    data_2d = hdul[0].data
    variance = hdul[1].data
    mask = hdul[2].data
    traces = hdul[3].data

    print(f"Detected {len(traces)} traces")
    for trace in traces:
        print(f"  Trace {trace['TRACE_ID']}: SNR={trace['SNR_EST']:.1f}")

Output Products

Directory Structure

Calibration Products

master_bias.fits

master_flat.fits

bad_pixel_mask.fits

Reduced 2D Spectra

science_NNN_2d.fits

Extracted 1D Spectra

science_NNN_traceM.fits

Wavelength Solutions

wavelength_solution_arcNNN.fits

Quality Reports

science_NNN_quality.yaml

summary_report.txt

Diagnostic Plots

wavelength_solution.png

science_NNN_2d.png

science_NNN_traceM_profile.png

science_NNN_traceM_1d.png

Logs

reduction_log.txt

Reading Output Products

Python Example

See Also