Troubleshooting

Common issues and solutions for pyKOSMOS++ pipeline problems.

Installation Issues

ImportError: No module named ‘pykosmospp’

Problem: Package not installed or not in Python path.

Solution:

# Verify installation
pip list | grep pykosm

# If not found, install
pip install pykosmospp

# Or for development
pip install -e .

ModuleNotFoundError: No module named ‘astropy’

Problem: Missing dependencies.

Solution:

# Install all dependencies
pip install -e .

# Or manually
pip install astropy scipy numpy matplotlib specutils pyyaml

Version conflicts with existing packages

Problem: Dependency version incompatibility.

Solution:

# Create isolated environment
conda create -n pykosmospp python=3.10
conda activate pykosmospp
pip install pykosmospp

# Or with venv
python -m venv venv_pykosmospp
source venv_pykosmospp/bin/activate  # Linux/Mac
pip install pykosmospp

Data Loading Issues

“No FITS files found”

Problem: Pipeline cannot locate input files.

Diagnosis:

# Check directory structure
ls -R data/

# Should show:
# data/
#   biases/
#     *.fits
#   flats/
#     *.fits
#   arcs/
#     *.fits
#   science/
#     *.fits

Solutions:

Wrong directory structure:

# Create correct structure
mkdir -p data/{biases,flats,arcs,science}

# Move files
mv bias*.fits data/biases/
mv flat*.fits data/flats/

Wrong file extensions:

# Rename .fit to .fits
rename 's/\.fit$/.fits/' *.fit

# Or lowercase .FITS
rename 'y/A-Z/a-z/' *.FITS

Files in wrong subdirectory:

Check FITS headers to verify frame type:

from astropy.io import fits
with fits.open('mystery_file.fits') as hdul:
    print(hdul[0].header['IMAGETYP'])

“IMAGETYP keyword not found”

Problem: FITS headers missing required keywords.

Solution:

Add missing keywords manually:

from astropy.io import fits

# Read file
with fits.open('file.fits', mode='update') as hdul:
    header = hdul[0].header

    # Add IMAGETYP
    header['IMAGETYP'] = 'Bias'  # or 'Flat', 'Arc', 'Object'
    header['OBJECT'] = 'bias'
    header['EXPTIME'] = 0.0

Or use FITS editor:

fitsheader -u IMAGETYP 'Bias' bias_001.fits

Calibration Errors

“Insufficient calibration frames”

Problem: Not enough bias/flat/arc frames.

Requirements:

Minimum 3 bias frames
Minimum 3 flat frames
Minimum 1 arc frame

Solution:

# Check frame counts
ls data/biases/*.fits | wc -l
ls data/flats/*.fits | wc -l
ls data/arcs/*.fits | wc -l

# If insufficient, acquire more frames or adjust minimum in config:

# config.yaml
calibration:
  min_frames:
    bias: 2   # Reduce minimum (not recommended)
    flat: 2

“Bias level variation too high”

Problem: Bias frames have inconsistent levels.

Diagnosis:

from pykosmospp.models import BiasFrame
import numpy as np

levels = []
for f in bias_files:
    bias = BiasFrame.from_fits(f)
    levels.append(np.median(bias.data.data))

print(f"Bias levels: {levels}")
print(f"Std dev: {np.std(levels):.2f} ADU")

Solutions:

Temperature variation: Ensure CCD temperature stable during calibrations
Different readout modes: Check all bias frames use same binning/readout

Outlier frames: Remove problematic bias with different level:

# Remove outlier
rm data/biases/bias_003.fits  # If level differs by >20 ADU

“Flat field saturation”

Problem: Flat frames over-exposed.

Diagnosis:

from astropy.io import fits

with fits.open('flat_001.fits') as hdul:
    data = hdul[0].data
    saturate = hdul[0].header['SATURATE']
    n_saturated = np.sum(data >= saturate * 0.9)

    print(f"Saturated pixels: {n_saturated} ({100*n_saturated/data.size:.2f}%)")

Solution:

Re-observe flats with shorter exposure time
Target median counts 10k-50k ADU (well below saturation)
Or remove saturated frames from calibration set

Wavelength Calibration Issues

“Wavelength solution failed to converge”

Problem: Cannot fit wavelength solution to arc lines.

Common Causes:

Too few lines identified: Need ≥20 lines

# Check line detection
from pykosmospp.wavelength.identify import detect_arc_lines
import numpy as np

arc_spectrum = np.median(arc_frame.data.data, axis=0)
lines = detect_arc_lines(arc_spectrum, detection_threshold=5.0)
print(f"Detected {len(lines)} lines")

# If <20 lines:
# - Lower detection_threshold to 3.0
# - Check arc exposure time (may be too short)
# - Verify correct lamp type selected

Wrong arc lamp type:

# Try different lamp types
kosmos-reduce wavelength --input-arc arc.fits --lamp-type HeNeAr --plot
kosmos-reduce wavelength --input-arc arc.fits --lamp-type Ar --plot
kosmos-reduce wavelength --input-arc arc.fits --lamp-type Kr --plot

Arc lamp under-exposed:
- Arc lines should have SNR >10
- Increase exposure time for next observation
- Can try spatial binning to boost SNR

“Wavelength RMS too high”

Problem: Wavelength fit residuals exceed threshold.

Diagnosis:

# Generate diagnostic plot
kosmos-reduce wavelength --input-arc arc.fits \\
                         --lamp-type HeNeAr \\
                         --plot

# Check wavelength_fit.pdf for:
# - Systematic residuals (wrong polynomial order)
# - Outlier lines (misidentifications)
# - Overall RMS value

Solutions:

Adjust polynomial order:

# config.yaml
wavelength:
  max_order: 5  # Try lower order if overfitting
  # or
  max_order: 9  # Try higher order if underfitting

Tighten sigma-clipping:

wavelength:
  sigma_clip: 2.5  # More aggressive outlier rejection

Use more arc lines:
- Increase arc exposure time for better S/N
- Expand wavelength_range to detect more lines

Trace Detection Issues

“No traces detected”

Problem: Pipeline finds no spectral traces.

Diagnosis:

# Visualize 2D spectrum
from astropy.io import fits
import matplotlib.pyplot as plt

with fits.open('reduced_2d/science_001_2d.fits') as hdul:
    data = hdul[0].data

plt.imshow(data, cmap='viridis', origin='lower',
          vmin=np.percentile(data, 1),
          vmax=np.percentile(data, 99))
plt.colorbar()
plt.xlabel('Spectral Direction')
plt.ylabel('Spatial Direction')
plt.show()

# Visual inspection:
# - Is there a visible horizontal trace?
# - Is the spectrum too faint?
# - Is the slit illuminated?

Solutions:

Lower detection threshold:

# config.yaml
trace_detection:
  min_snr: 2.0  # Default is 3.0

Adjust expected FWHM:

trace_detection:
  expected_fwhm: 5.0  # Try wider if seeing poor
  # or
  expected_fwhm: 3.0  # Try narrower if sharp

Enable spatial binning:

binning:
  spatial:
    enabled: true
    factor: 2

Use interactive mode:

kosmos-reduce --input-dir data/ --output-dir reduced/ --mode interactive

Manually select trace position in GUI.

“Too many false trace detections”

Problem: Pipeline detects spurious traces.

Solution:

Increase detection threshold and minimum separation:

# config.yaml
trace_detection:
  min_snr: 4.0        # Stricter threshold
  min_separation: 30  # Wider spacing

Extraction Issues

“Negative flux values in extracted spectrum”

Problem: Sky over-subtracted or poor profile fit.

Diagnosis:

Check sky estimation:

from pykosmospp.extraction.sky import estimate_sky_background

sky_2d = estimate_sky_background(data, trace, sky_buffer=30)

# Visualize
plt.plot(np.median(sky_2d, axis=0))
plt.title('Estimated Sky Background')
plt.show()

Solutions:

Increase sky buffer:

extraction:
  sky_buffer: 50  # Move sky regions away from trace wings

Use boxcar extraction:

extraction:
  method: boxcar  # Simpler, less sensitive to profile errors

“Low flux conservation”

Problem: Extracted 1D flux doesn’t match 2D aperture sum.

Solution:

Increase aperture width to capture all flux:

extraction:
  aperture_width: 15  # Default is 10

Quality Issues

“Low SNR warning”

Problem: Signal-to-noise ratio below threshold.

This is not an error, just a quality flag. Options:

Accept lower quality: Spectrum still usable for some science
Re-observe with longer integration:
- SNR improves as √(exposure time)
- To double SNR, need 4× exposure time

Spatial binning:

binning:
  spatial:
    enabled: true
    factor: 2  # Increases SNR by √2

“Poor wavelength calibration grade”

Problem: Wavelength RMS above threshold.

Impact:

Redshift measurements less accurate
Line identifications may be uncertain

Solutions:

See “Wavelength Calibration Issues” above.

Performance Issues

“Pipeline runs very slowly”

Expected Performance: ~15 seconds per science frame on modern hardware.

Optimizations:

Use batch mode:

kosmos-reduce --mode batch  # Faster than interactive

Limit trace extraction:

kosmos-reduce --max-traces 2  # Process only first 2 traces

Disable diagnostic plots:

(Feature not yet implemented, but in roadmap)
Use SSD for I/O:
- Move data to fast storage
- Output to SSD, copy to network storage later

“Out of memory errors”

Problem: Large FITS files exhaust RAM.

Solutions:

Process in batches:

# Process 5 frames at a time
kosmos-reduce --input-dir data/subset1/ --output-dir reduced/
kosmos-reduce --input-dir data/subset2/ --output-dir reduced/

Close Python sessions:

Large arrays may not be garbage collected immediately
Increase system RAM or swap:

Minimum 8 GB RAM recommended for KOSMOS data

Debugging Tips

Enable Verbose Logging

kosmos-reduce --input-dir data/ --output-dir reduced/ --verbose

Logs show detailed processing steps and intermediate values.

Check Intermediate Products

# Verify calibrations created
ls -lh reduced/calibrations/

# Check if 2D spectra processed
ls reduced/reduced_2d/

# Verify 1D extraction
ls reduced/spectra_1d/

Use Python Interactively

# Load modules and test step-by-step
from pykosmospp.pipeline import PipelineRunner
from pathlib import Path

runner = PipelineRunner(
    input_dir=Path('data'),
    output_dir=Path('reduced')
)

# Run individual stages
# (requires modifying PipelineRunner to expose internal methods)

Examine FITS Headers

# List all header keywords
fitsheader science_001.fits

# Check specific keyword
fitsheader -k IMAGETYP -k EXPTIME science_001.fits

Validate FITS Files

# Check FITS integrity
fitsverify science_001.fits

Getting Help

GitHub Issues: https://github.com/gkhullar/pykosmospp/issues

When reporting issues, include:

Pipeline version:

kosmos-reduce --version
# or
pip show pykosmospp

Full error message:

Copy complete traceback from terminal

Input file info:

ls -lh data/
fitsheader science_001.fits | head -20

Configuration file:

Attach your custom config.yaml (if used)
Log file:

Attach reduction_log.txt from output directory

Email: gk@astro.washington.edu

For sensitive data or private discussions.

Documentation: https://pykosmospp.readthedocs.io

Search docs for specific error messages or concepts.

Troubleshooting

Installation Issues

ImportError: No module named ‘pykosmospp’

ModuleNotFoundError: No module named ‘astropy’

Version conflicts with existing packages

Data Loading Issues

“No FITS files found”

“IMAGETYP keyword not found”

Calibration Errors

“Insufficient calibration frames”

“Bias level variation too high”

“Flat field saturation”

Wavelength Calibration Issues

“Wavelength solution failed to converge”

“Wavelength RMS too high”

Trace Detection Issues

“No traces detected”

“Too many false trace detections”

Extraction Issues

“Negative flux values in extracted spectrum”

“Low flux conservation”

Quality Issues

“Low SNR warning”

“Poor wavelength calibration grade”

Performance Issues

“Pipeline runs very slowly”

“Out of memory errors”

Debugging Tips

Enable Verbose Logging

Check Intermediate Products

Use Python Interactively

Examine FITS Headers

Validate FITS Files

Getting Help

See Also