Mirror/intercept
1
0
Fork 0
mirror of https://github.com/smittix/intercept.git synced 2026-07-26 01:38:10 -07:00
Code Issues Packages Projects Releases Wiki Activity

Add ground station automation with 6-phase implementation

Browse Source 
Phase 1 - Automated observation engine:
- utils/ground_station/scheduler.py: GroundStationScheduler fires at AOS/LOS,
  claims SDR, manages IQBus lifecycle, emits SSE events
- utils/ground_station/observation_profile.py: ObservationProfile dataclass + DB CRUD
- routes/ground_station.py: REST API for profiles, scheduler, observations, recordings,
  rotator; SSE stream; /ws/satellite_waterfall WebSocket
- DB tables: observation_profiles, ground_station_observations, ground_station_events,
  sigmf_recordings (added to utils/database.py init_db)
- app.py: ground_station_queue, WebSocket init, scheduler startup in _deferred_init
- routes/__init__.py: register ground_station_bp

Phase 2 - Doppler correction:
- utils/doppler.py: generalized DopplerTracker extracted from sstv_decoder.py;
  accepts satellite name or raw TLE tuple; thread-safe; update_tle() method
- utils/sstv/sstv_decoder.py: replace inline DopplerTracker with import from utils.doppler
- Scheduler runs 5s retune loop; calls rotator.point_to() if enabled

Phase 3 - IQ recording (SigMF):
- utils/sigmf.py: SigMFWriter writes .sigmf-data + .sigmf-meta; disk-free guard (500MB)
- utils/ground_station/consumers/sigmf_writer.py: SigMFConsumer wraps SigMFWriter

Phase 4 - Multi-decoder IQ broadcast pipeline:
- utils/ground_station/iq_bus.py: IQBus single-producer fan-out; IQConsumer Protocol
- utils/ground_station/consumers/waterfall.py: CU8→FFT→binary frames
- utils/ground_station/consumers/fm_demod.py: CU8→FM demod (numpy)→decoder subprocess
- utils/ground_station/consumers/gr_satellites.py: CU8→cf32→gr_satellites (optional)

Phase 5 - Live spectrum waterfall:
- static/js/modes/ground_station_waterfall.js: /ws/satellite_waterfall canvas renderer
- Waterfall panel in satellite dashboard sidebar, auto-shown on iq_bus_started SSE event

Phase 6 - Antenna rotator control (optional):
- utils/rotator.py: RotatorController TCP client for rotctld (Hamlib line protocol)
- Rotator panel in satellite dashboard; silently disabled if rotctld unreachable

Also fixes pre-existing test_weather_sat_predict.py breakage:
- utils/weather_sat_predict.py: rewritten with self-contained skyfield implementation
  using find_discrete (matching what committed tests expected); adds _format_utc_iso
- tests/test_weather_sat_predict.py: add _MOCK_WEATHER_SATS and @patch decorators
  for tests that assumed NOAA-18 active (decommissioned Jun 2025, now active=False)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
James Smith
2026-03-18 17:36:55 +00:00
parent ed1461626b
commit 4607c358ed
21 changed files with 3709 additions and 181 deletions
Download Patch File Download Diff File Expand all files Collapse all files
utils/ground_station/consumers/__init__.py
+1
View File
@@ -0,0 +1 @@
"""IQ bus consumer implementations."""
utils/ground_station/consumers/fm_demod.py
+219
View File
@@ -0,0 +1,219 @@
"""FMDemodConsumer — demodulates FM from CU8 IQ and pipes PCM to a decoder.
Performs FM (or AM/USB/LSB) demodulation in-process using numpy — the
same algorithm as the listening-post waterfall monitor. The resulting
int16 PCM is written to the stdin of a configurable decoder subprocess
(e.g. direwolf for AX.25 AFSK or multimon-ng for GMSK/POCSAG).
Decoded lines from the subprocess stdout are forwarded to an optional
``on_decoded`` callback.
"""
from __future__ import annotations
import subprocess
import threading
from typing import Callable
import numpy as np
from utils.logging import get_logger
from utils.process import register_process, safe_terminate, unregister_process
from utils.waterfall_fft import cu8_to_complex
logger = get_logger('intercept.ground_station.fm_demod')
AUDIO_RATE = 48_000 # Hz — standard rate for direwolf / multimon-ng
class FMDemodConsumer:
"""CU8 IQ → FM demodulation → int16 PCM → decoder subprocess stdin."""
def __init__(
self,
decoder_cmd: list[str],
*,
modulation: str = 'fm',
on_decoded: Callable[[str], None] | None = None,
):
"""
Args:
decoder_cmd: Decoder command + args, e.g.
``['direwolf', '-r', '48000', '-']`` or
``['multimon-ng', '-t', 'raw', '-a', 'AFSK1200', '-']``.
modulation: ``'fm'``, ``'am'``, ``'usb'``, ``'lsb'``.
on_decoded: Callback invoked with each decoded line from stdout.
"""
self._decoder_cmd = decoder_cmd
self._modulation = modulation.lower()
self._on_decoded = on_decoded
self._proc: subprocess.Popen | None = None
self._stdout_thread: threading.Thread | None = None
self._center_mhz = 0.0
self._sample_rate = 0
self._rotator_phase = 0.0
# ------------------------------------------------------------------
# IQConsumer protocol
# ------------------------------------------------------------------
def on_start(
self,
center_mhz: float,
sample_rate: int,
*,
start_freq_mhz: float,
end_freq_mhz: float,
) -> None:
self._center_mhz = center_mhz
self._sample_rate = sample_rate
self._rotator_phase = 0.0
self._start_proc()
def on_chunk(self, raw: bytes) -> None:
if self._proc is None or self._proc.poll() is not None:
return
try:
pcm, self._rotator_phase = _demodulate(
raw,
sample_rate=self._sample_rate,
center_mhz=self._center_mhz,
monitor_freq_mhz=self._center_mhz, # decode on-center
modulation=self._modulation,
rotator_phase=self._rotator_phase,
)
if pcm and self._proc.stdin:
self._proc.stdin.write(pcm)
self._proc.stdin.flush()
except (BrokenPipeError, OSError):
pass # decoder exited
except Exception as e:
logger.debug(f"FMDemodConsumer on_chunk error: {e}")
def on_stop(self) -> None:
if self._proc:
safe_terminate(self._proc)
unregister_process(self._proc)
self._proc = None
if self._stdout_thread and self._stdout_thread.is_alive():
self._stdout_thread.join(timeout=2)
# ------------------------------------------------------------------
# Internal
# ------------------------------------------------------------------
def _start_proc(self) -> None:
import shutil
if not shutil.which(self._decoder_cmd[0]):
logger.warning(
f"FMDemodConsumer: decoder '{self._decoder_cmd[0]}' not found — disabled"
)
return
try:
self._proc = subprocess.Popen(
self._decoder_cmd,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.DEVNULL,
)
register_process(self._proc)
self._stdout_thread = threading.Thread(
target=self._read_stdout, daemon=True, name='fm-demod-stdout'
)
self._stdout_thread.start()
except Exception as e:
logger.error(f"FMDemodConsumer: failed to start decoder: {e}")
self._proc = None
def _read_stdout(self) -> None:
assert self._proc is not None
assert self._proc.stdout is not None
try:
for line in self._proc.stdout:
decoded = line.decode('utf-8', errors='replace').rstrip()
if decoded and self._on_decoded:
try:
self._on_decoded(decoded)
except Exception as e:
logger.debug(f"FMDemodConsumer callback error: {e}")
except Exception:
pass
# ---------------------------------------------------------------------------
# In-process FM demodulation (mirrors waterfall_websocket._demodulate_monitor_audio)
# ---------------------------------------------------------------------------
def _demodulate(
raw: bytes,
sample_rate: int,
center_mhz: float,
monitor_freq_mhz: float,
modulation: str,
rotator_phase: float,
) -> tuple[bytes | None, float]:
"""Demodulate CU8 IQ to int16 PCM.
Returns ``(pcm_bytes, next_rotator_phase)``.
"""
if len(raw) < 32 or sample_rate <= 0:
return None, float(rotator_phase)
samples = cu8_to_complex(raw)
fs = float(sample_rate)
freq_offset_hz = (float(monitor_freq_mhz) - float(center_mhz)) * 1e6
nyquist = fs * 0.5
if abs(freq_offset_hz) > nyquist * 0.98:
return None, float(rotator_phase)
phase_inc = (2.0 * np.pi * freq_offset_hz) / fs
n = np.arange(samples.size, dtype=np.float64)
rotator = np.exp(-1j * (float(rotator_phase) + phase_inc * n)).astype(np.complex64)
next_phase = float((float(rotator_phase) + phase_inc * samples.size) % (2.0 * np.pi))
shifted = samples * rotator
mod = modulation.lower().strip()
target_bb = 48_000.0
pre_decim = max(1, int(fs // target_bb))
if pre_decim > 1:
usable = (shifted.size // pre_decim) * pre_decim
if usable < pre_decim:
return None, next_phase
shifted = shifted[:usable].reshape(-1, pre_decim).mean(axis=1)
fs1 = fs / pre_decim
if shifted.size < 16:
return None, next_phase
if mod == 'fm':
audio = np.angle(shifted[1:] * np.conj(shifted[:-1])).astype(np.float32)
elif mod == 'am':
envelope = np.abs(shifted).astype(np.float32)
audio = envelope - float(np.mean(envelope))
elif mod == 'usb':
audio = np.real(shifted).astype(np.float32)
elif mod == 'lsb':
audio = -np.real(shifted).astype(np.float32)
else:
audio = np.real(shifted).astype(np.float32)
if audio.size < 8:
return None, next_phase
audio = audio - float(np.mean(audio))
# Resample to AUDIO_RATE
out_len = int(audio.size * AUDIO_RATE / fs1)
if out_len < 32:
return None, next_phase
x_old = np.linspace(0.0, 1.0, audio.size, endpoint=False, dtype=np.float32)
x_new = np.linspace(0.0, 1.0, out_len, endpoint=False, dtype=np.float32)
audio = np.interp(x_new, x_old, audio).astype(np.float32)
peak = float(np.max(np.abs(audio))) if audio.size else 0.0
if peak > 0:
audio = audio * min(20.0, 0.85 / peak)
pcm = np.clip(audio, -1.0, 1.0)
return (pcm * 32767.0).astype(np.int16).tobytes(), next_phase
utils/ground_station/consumers/gr_satellites.py
+154
View File
@@ -0,0 +1,154 @@
"""GrSatConsumer — pipes CU8 IQ to gr_satellites for packet decoding.
``gr_satellites`` is a GNU Radio-based multi-satellite decoder
(https://github.com/daniestevez/gr-satellites). It accepts complex
float32 (cf32) IQ samples on stdin when invoked with ``--iq``.
This consumer converts CU8 → cf32 via numpy and pipes the result to
``gr_satellites``. If the tool is not installed it silently stays
disabled.
Decoded JSON packets are forwarded to an optional ``on_decoded`` callback.
"""
from __future__ import annotations
import shutil
import subprocess
import threading
from typing import Callable
import numpy as np
from utils.logging import get_logger
from utils.process import register_process, safe_terminate, unregister_process
logger = get_logger('intercept.ground_station.gr_satellites')
GR_SATELLITES_BIN = 'gr_satellites'
class GrSatConsumer:
"""CU8 IQ → cf32 → gr_satellites stdin → JSON packets."""
def __init__(
self,
satellite_name: str,
*,
on_decoded: Callable[[dict], None] | None = None,
):
"""
Args:
satellite_name: Satellite name as known to gr_satellites
(e.g. ``'NOAA 15'``, ``'ISS'``).
on_decoded: Callback invoked with each parsed JSON packet dict.
"""
self._satellite_name = satellite_name
self._on_decoded = on_decoded
self._proc: subprocess.Popen | None = None
self._stdout_thread: threading.Thread | None = None
self._sample_rate = 0
self._enabled = False
# ------------------------------------------------------------------
# IQConsumer protocol
# ------------------------------------------------------------------
def on_start(
self,
center_mhz: float,
sample_rate: int,
*,
start_freq_mhz: float,
end_freq_mhz: float,
) -> None:
self._sample_rate = sample_rate
if not shutil.which(GR_SATELLITES_BIN):
logger.info(
"gr_satellites not found — GrSatConsumer disabled. "
"Install via: pip install gr-satellites or apt install python3-gr-satellites"
)
self._enabled = False
return
self._start_proc(sample_rate)
def on_chunk(self, raw: bytes) -> None:
if not self._enabled or self._proc is None or self._proc.poll() is not None:
return
# Convert CU8 → cf32
try:
iq = np.frombuffer(raw, dtype=np.uint8).astype(np.float32)
cf32 = ((iq - 127.5) / 127.5).view(np.complex64)
if self._proc.stdin:
self._proc.stdin.write(cf32.tobytes())
self._proc.stdin.flush()
except (BrokenPipeError, OSError):
pass
except Exception as e:
logger.debug(f"GrSatConsumer on_chunk error: {e}")
def on_stop(self) -> None:
self._enabled = False
if self._proc:
safe_terminate(self._proc)
unregister_process(self._proc)
self._proc = None
if self._stdout_thread and self._stdout_thread.is_alive():
self._stdout_thread.join(timeout=2)
# ------------------------------------------------------------------
# Internal
# ------------------------------------------------------------------
def _start_proc(self, sample_rate: int) -> None:
import json as _json
cmd = [
GR_SATELLITES_BIN,
self._satellite_name,
'--samplerate', str(sample_rate),
'--iq',
'--json',
'-',
]
try:
self._proc = subprocess.Popen(
cmd,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.DEVNULL,
)
register_process(self._proc)
self._enabled = True
self._stdout_thread = threading.Thread(
target=self._read_stdout,
args=(_json,),
daemon=True,
name='gr-sat-stdout',
)
self._stdout_thread.start()
logger.info(f"GrSatConsumer started for '{self._satellite_name}'")
except Exception as e:
logger.error(f"GrSatConsumer: failed to start gr_satellites: {e}")
self._proc = None
self._enabled = False
def _read_stdout(self, _json) -> None:
assert self._proc is not None
assert self._proc.stdout is not None
try:
for line in self._proc.stdout:
text = line.decode('utf-8', errors='replace').rstrip()
if not text:
continue
if self._on_decoded:
try:
data = _json.loads(text)
except _json.JSONDecodeError:
data = {'raw': text}
try:
self._on_decoded(data)
except Exception as e:
logger.debug(f"GrSatConsumer callback error: {e}")
except Exception:
pass
utils/ground_station/consumers/sigmf_writer.py
+75
View File
@@ -0,0 +1,75 @@
"""SigMFConsumer — wraps SigMFWriter as an IQ bus consumer."""
from __future__ import annotations
from utils.logging import get_logger
from utils.sigmf import SigMFMetadata, SigMFWriter
logger = get_logger('intercept.ground_station.sigmf_consumer')
class SigMFConsumer:
"""IQ consumer that records CU8 chunks to a SigMF file pair."""
def __init__(
self,
metadata: SigMFMetadata,
on_complete: 'callable | None' = None,
):
"""
Args:
metadata: Pre-populated SigMF metadata (satellite info, freq, etc.)
on_complete: Optional callback invoked with ``(meta_path, data_path)``
when the recording is closed.
"""
self._metadata = metadata
self._on_complete = on_complete
self._writer: SigMFWriter | None = None
# ------------------------------------------------------------------
# IQConsumer protocol
# ------------------------------------------------------------------
def on_start(
self,
center_mhz: float,
sample_rate: int,
*,
start_freq_mhz: float,
end_freq_mhz: float,
) -> None:
self._metadata.center_frequency_hz = center_mhz * 1e6
self._metadata.sample_rate = sample_rate
self._writer = SigMFWriter(self._metadata)
try:
self._writer.open()
except Exception as e:
logger.error(f"SigMFConsumer: failed to open writer: {e}")
self._writer = None
def on_chunk(self, raw: bytes) -> None:
if self._writer is None:
return
ok = self._writer.write_chunk(raw)
if not ok and self._writer.aborted:
logger.warning("SigMFConsumer: recording aborted (disk full)")
self._writer = None
def on_stop(self) -> None:
if self._writer is None:
return
result = self._writer.close()
self._writer = None
if result and self._on_complete:
try:
self._on_complete(*result)
except Exception as e:
logger.debug(f"SigMFConsumer on_complete error: {e}")
# ------------------------------------------------------------------
# Status
# ------------------------------------------------------------------
@property
def bytes_written(self) -> int:
return self._writer.bytes_written if self._writer else 0
utils/ground_station/consumers/waterfall.py
+123
View File
@@ -0,0 +1,123 @@
"""WaterfallConsumer — converts CU8 IQ chunks into binary waterfall frames.
Frames are placed on an ``output_queue`` that the WebSocket endpoint
(``/ws/satellite_waterfall``) drains and sends to the browser.
Reuses :mod:`utils.waterfall_fft` for FFT processing so the wire format
is identical to the main listening-post waterfall.
"""
from __future__ import annotations
import queue
import time
import numpy as np
from utils.logging import get_logger
from utils.waterfall_fft import (
build_binary_frame,
compute_power_spectrum,
cu8_to_complex,
quantize_to_uint8,
)
logger = get_logger('intercept.ground_station.waterfall_consumer')
FFT_SIZE = 1024
AVG_COUNT = 4
FPS = 20
DB_MIN: float | None = None # auto-range
DB_MAX: float | None = None
class WaterfallConsumer:
"""IQ consumer that produces waterfall binary frames."""
def __init__(
self,
output_queue: queue.Queue | None = None,
fft_size: int = FFT_SIZE,
avg_count: int = AVG_COUNT,
fps: int = FPS,
db_min: float | None = DB_MIN,
db_max: float | None = DB_MAX,
):
self.output_queue: queue.Queue = output_queue or queue.Queue(maxsize=120)
self._fft_size = fft_size
self._avg_count = avg_count
self._fps = fps
self._db_min = db_min
self._db_max = db_max
self._center_mhz = 0.0
self._start_freq = 0.0
self._end_freq = 0.0
self._sample_rate = 0
self._buffer = b''
self._required_bytes = 0
self._frame_interval = 1.0 / max(1, fps)
self._last_frame_time = 0.0
# ------------------------------------------------------------------
# IQConsumer protocol
# ------------------------------------------------------------------
def on_start(
self,
center_mhz: float,
sample_rate: int,
*,
start_freq_mhz: float,
end_freq_mhz: float,
) -> None:
self._center_mhz = center_mhz
self._sample_rate = sample_rate
self._start_freq = start_freq_mhz
self._end_freq = end_freq_mhz
# How many IQ samples (pairs) we need for one FFT frame
required_samples = max(
self._fft_size * self._avg_count,
sample_rate // max(1, self._fps),
)
self._required_bytes = required_samples * 2 # 1 byte I + 1 byte Q
self._frame_interval = 1.0 / max(1, self._fps)
self._buffer = b''
self._last_frame_time = 0.0
def on_chunk(self, raw: bytes) -> None:
self._buffer += raw
now = time.monotonic()
if (now - self._last_frame_time) < self._frame_interval:
return
if len(self._buffer) < self._required_bytes:
return
chunk = self._buffer[-self._required_bytes:]
self._buffer = b''
self._last_frame_time = now
try:
samples = cu8_to_complex(chunk)
power_db = compute_power_spectrum(
samples, fft_size=self._fft_size, avg_count=self._avg_count
)
quantized = quantize_to_uint8(power_db, db_min=self._db_min, db_max=self._db_max)
frame = build_binary_frame(self._start_freq, self._end_freq, quantized)
except Exception as e:
logger.debug(f"WaterfallConsumer FFT error: {e}")
return
# Non-blocking enqueue: drop oldest if full
if self.output_queue.full():
try:
self.output_queue.get_nowait()
except queue.Empty:
pass
try:
self.output_queue.put_nowait(frame)
except queue.Full:
pass
def on_stop(self) -> None:
self._buffer = b''