Mirror/intercept
1
0
Fork 0
mirror of https://github.com/smittix/intercept.git synced 2026-05-23 00:04:50 -07:00
Code Issues Packages Projects Releases Wiki Activity

fixing bands and how the gsm scanner loops with tshark

Browse Source
This commit is contained in:
Marc
2026-02-06 08:27:25 -06:00
parent 8e9588c4ff
commit e8a9afa221
6 changed files with 815 additions and 606 deletions
Download Patch File Download Diff File Expand all files Collapse all files

224
GSM_SPY_DEVICE_SELECTION_IMPLEMENTATION.md
View File

@@ -1,224 +0,0 @@
# GSM Spy SDR Device Selection Implementation
## Summary
Successfully implemented dynamic SDR device detection, selection, and management for the GSM Spy feature, following the same pattern used in the Aircraft/ADS-B implementation.
## Changes Made
### Frontend Changes (`templates/gsm_spy_dashboard.html`)
#### 1. Dynamic Device Selector
- **Changed**: Device dropdown from hardcoded options to dynamic detection
- **Location**: Line ~1155 (Signal Source Panel)
- **Before**: Static options (Device 0, Device 1, etc.)
- **After**: Dynamic population with "Detecting devices..." placeholder
#### 2. Device Detection on Page Load
- **Added**: `initDeviceSelector()` function
- **Location**: ~Line 1395
- **Functionality**:
- Fetches available SDR devices from `/devices` endpoint
- Populates dropdown with detected devices
- Shows device name, type (RTL-SDR, HackRF, etc.), and serial number
- Handles errors gracefully with user-friendly messages
- Logs detection results to console
#### 3. Scanner Controls Update
- **Modified**: `startScanner()` function (~Line 1410)
- **Changes**:
- Made async for better error handling
- Reads device index from `deviceSelect` dropdown
- Disables device and region selectors during active scanning
- Enhanced error handling with device conflict detection
- Shows user-friendly alerts for device busy errors
#### 4. Stop Scanner Enhancements
- **Modified**: `stopScanner()` function (~Line 1494)
- **Changes**:
- Re-enables device and region selectors after stopping
- Maintains UI consistency
#### 5. Region Selector Sync
- **Modified**: `selectRegion()` function (~Line 1882)
- **Changes**:
- Capitalizes region name to match backend API expectations
- Syncs region button selection with dropdown
#### 6. Removed Redundant Controls
- **Removed**: `scannerDevice` dropdown from bottom controls bar
- **Reason**: Consolidated to single device selector in left sidebar
### Backend Changes (`routes/gsm_spy.py`)
#### 1. Enhanced Error Response
- **Modified**: `/start` endpoint device claiming logic (~Line 115)
- **Changes**:
- Added `error_type: 'DEVICE_BUSY'` to 409 conflict responses
- Enables frontend to distinguish device conflicts from other errors
- Allows for targeted user-friendly error messages
#### 2. Existing Device Management (Verified)
- **Confirmed**: Device claiming/releasing already implemented
- `claim_sdr_device()` called at line 115
- `release_sdr_device()` called at line 289
- Device index stored in `gsm_spy_active_device`
- Region stored in `gsm_spy_region`
#### 3. Status Endpoint (Verified)
- **Confirmed**: `/status` endpoint returns device info
- Returns `device` (active device index)
- Returns `region` (selected region)
- Returns all necessary status information
## Features Implemented
### ✅ Device Detection
- Dynamically detects all available SDR devices on page load
- Supports all 5 SDR types: RTL-SDR, HackRF, LimeSDR, Airspy, SDRPlay
- Shows device name, type, and serial number in dropdown
### ✅ Device Registry Integration
- Properly claims devices before starting scanner
- Releases devices when stopping scanner
- Prevents conflicts with other INTERCEPT modes
### ✅ UI State Management
- Disables device selector during active scanning
- Re-enables selector after stopping
- Provides clear visual feedback to user
### ✅ Error Handling
- User-friendly error messages for device conflicts
- Graceful handling of "no devices detected" scenario
- Clear console logging for debugging
### ✅ Validation
- Uses existing `validate_device_index()` function (already in code)
- Validates region against `REGIONAL_BANDS` dictionary
- Checks for already running scanner
## Architecture Pattern
The implementation follows the same pattern as Aircraft/ADS-B:
1. **Device Detection**: `/devices` endpoint (shared across all modes)
2. **Device Claiming**: `claim_sdr_device()` before starting
3. **Device Releasing**: `release_sdr_device()` on stop
4. **UI Consistency**: Dynamic dropdown, disabled during operation
5. **Error Handling**: Clear user messages, console logging
## Testing Recommendations
### 1. Device Detection
```bash
# Start application
sudo -E venv/bin/python intercept.py
# Open GSM Spy dashboard in browser
# Open DevTools console
# Should see: "[GSM SPY] Detected X SDR device(s)"
# Verify dropdown shows detected devices
```
### 2. Device Claiming
```bash
# Start GSM scanner on device 0
# Try to start another mode (e.g., ADS-B) on device 0
# Should see conflict error message
# Stop GSM scanner
# Now ADS-B should be able to claim device 0
```
### 3. Multiple Devices
```bash
# Connect multiple SDR devices
# Open GSM Spy dashboard
# Verify all devices appear in dropdown
# Select different devices and verify they work independently
```
### 4. UI State
```bash
# Start GSM scanner
# Verify device selector is disabled
# Verify region selector is disabled
# Stop scanner
# Verify both selectors are re-enabled
```
### 5. Error Scenarios
```bash
# Disconnect SDR device
# Try to start scanner
# Should see graceful error message
# Reconnect device
# Refresh page - device should be detected
```
## Known Limitations
1. **gr-gsm Hardware Support**: The `gr-gsm` tools may have limited support for non-RTL-SDR devices. This implementation handles device selection properly, but `gr-gsm` itself may only work with RTL-SDR.
2. **Command Builder Integration**: Full SDRFactory integration (using device-specific command builders) would require adding GSM-specific methods to command builders in `utils/sdr/`. This is a future enhancement.
3. **Remote Device Support**: Unlike ADS-B which supports remote dump1090 connections, GSM Spy currently only supports local SDR devices.
## Future Enhancements
### 1. SDRFactory Integration
```python
# In start_scanner():
from utils.sdr import SDRFactory
devices = SDRFactory.detect_devices()
sdr_device = next((d for d in devices if d.index == device_index), None)
builder = SDRFactory.get_builder(sdr_device.sdr_type)
cmd = builder.build_gsm_scanner_command(device=sdr_device, bands=REGIONAL_BANDS[region])
```
Note: This requires adding `build_gsm_scanner_command()` method to command builders.
### 2. Device-Specific Tuning
- Different gain settings per SDR type
- Frequency correction (PPM) based on device calibration
- Sample rate optimization per hardware
### 3. Multi-Device Monitoring
- Simultaneously monitor multiple towers on different devices
- Parallel scanning across multiple frequency bands
## Compatibility
- **Frontend**: Modern browsers with ES6+ support (async/await)
- **Backend**: Python 3.8+
- **SDR Hardware**: RTL-SDR, HackRF, LimeSDR, Airspy, SDRPlay
- **gr-gsm**: Requires gr-gsm toolkit installed
## Files Modified
1. `/opt/intercept/templates/gsm_spy_dashboard.html` - Frontend UI and JavaScript
2. `/opt/intercept/routes/gsm_spy.py` - Backend route handlers
## Files Referenced (Not Modified)
1. `/opt/intercept/routes/adsb.py` - Reference implementation
2. `/opt/intercept/utils/sdr/detection.py` - Device detection
3. `/opt/intercept/utils/sdr/__init__.py` - SDRFactory
4. `/opt/intercept/utils/validation.py` - Input validation
5. `/opt/intercept/app.py` - Device registry functions
## Verification
All changes have been implemented according to the plan. The implementation:
- ✅ Follows existing INTERCEPT patterns
- ✅ Maintains UI consistency across modes
- ✅ Includes proper error handling
- ✅ Uses centralized validation
- ✅ Integrates with device registry
- ✅ Provides clear user feedback
## Implementation Date
2026-02-06

289
GSM_SPY_ZOMBIE_PROCESS_FIX.md
View File

@@ -1,289 +0,0 @@
# GSM Spy Zombie Process Fix
## Issue Description
When starting GSM Spy, `grgsm_scanner` and `grgsm_livemon` processes were becoming zombies (defunct processes):
```
root 12488 5.1 0.0 0 0 pts/2 Z+ 14:29 0:01 [grgsm_scanner] <defunct>
```
## Root Cause
**Zombie processes** occur when a child process terminates but the parent process doesn't call `wait()` or `waitpid()` to collect the exit status. The process remains in the process table as a zombie until the parent reaps it.
In the GSM Spy implementation, there were three issues:
### Issue 1: scanner_thread not reaping grgsm_scanner process
- The `scanner_thread` function reads from `grgsm_scanner` stdout
- When the process terminates (either normally or due to error), the thread exits
- But it never calls `process.wait()` to reap the child process
- Result: zombie `grgsm_scanner` process
### Issue 2: monitor_thread not reaping tshark process
- The `monitor_thread` function reads from `tshark` stdout
- Same problem as Issue 1
- Result: zombie `tshark` process
### Issue 3: grgsm_livemon process not tracked at all
- When starting monitoring, two processes are created:
1. `grgsm_livemon` - captures GSM traffic and feeds it to tshark
2. `tshark` - filters and parses GSM data
- Only `tshark` was being tracked in `gsm_spy_monitor_process`
- `grgsm_livemon` was started but never stored or cleaned up
- Result: zombie `grgsm_livemon` process
## Solution
### Fix 1: Reap processes in scanner_thread
**File**: `/opt/intercept/routes/gsm_spy.py`
**Function**: `scanner_thread()` (line ~1026)
**Changes**:
```python
finally:
# Reap the process to prevent zombie
try:
if process.poll() is None:
# Process still running, terminate it
process.terminate()
process.wait(timeout=5)
else:
# Process already terminated, just collect exit status
process.wait()
logger.info(f"Scanner process terminated with exit code {process.returncode}")
except Exception as e:
logger.error(f"Error cleaning up scanner process: {e}")
try:
process.kill()
process.wait()
except Exception:
pass
logger.info("Scanner thread terminated")
```
**How it works**:
1. Check if process is still running with `poll()`
2. If running, terminate gracefully with `terminate()` then `wait()`
3. If already terminated, just call `wait()` to collect exit status
4. If anything fails, try `kill()` then `wait()`
5. This ensures the child process is always reaped
### Fix 2: Reap processes in monitor_thread
**File**: `/opt/intercept/routes/gsm_spy.py`
**Function**: `monitor_thread()` (line ~1089)
**Changes**: Same cleanup logic as Fix 1, applied to the monitor thread.
### Fix 3: Track and clean up grgsm_livemon process
#### 3a. Add global variable for grgsm_livemon
**File**: `/opt/intercept/app.py` (line ~185)
**Changes**:
```python
# GSM Spy
gsm_spy_process = None
gsm_spy_livemon_process = None # For grgsm_livemon process
gsm_spy_monitor_process = None # For tshark monitoring process
```
#### 3b. Update global declarations
**File**: `/opt/intercept/app.py` (line ~677)
**Changes**:
```python
global gsm_spy_process, gsm_spy_livemon_process, gsm_spy_monitor_process
```
#### 3c. Clean up grgsm_livemon in reset function
**File**: `/opt/intercept/app.py` (line ~755)
**Changes**:
```python
if gsm_spy_livemon_process:
try:
safe_terminate(gsm_spy_livemon_process, 'grgsm_livemon')
killed.append('grgsm_livemon')
except Exception:
pass
gsm_spy_livemon_process = None
```
#### 3d. Store grgsm_livemon process when starting
**File**: `/opt/intercept/routes/gsm_spy.py`
**Changes in `/monitor` endpoint** (line ~212):
```python
app_module.gsm_spy_livemon_process = grgsm_proc
app_module.gsm_spy_monitor_process = tshark_proc
```
**Changes in `auto_start_monitor()` function** (line ~997):
```python
app_module.gsm_spy_livemon_process = grgsm_proc
app_module.gsm_spy_monitor_process = tshark_proc
```
#### 3e. Stop grgsm_livemon when stopping scanner
**File**: `/opt/intercept/routes/gsm_spy.py` (line ~254)
**Changes**:
```python
if app_module.gsm_spy_livemon_process:
try:
app_module.gsm_spy_livemon_process.terminate()
app_module.gsm_spy_livemon_process.wait(timeout=5)
killed.append('livemon')
except Exception:
try:
app_module.gsm_spy_livemon_process.kill()
except Exception:
pass
app_module.gsm_spy_livemon_process = None
```
## Files Modified
1. `/opt/intercept/routes/gsm_spy.py`
- `scanner_thread()` - Added process reaping in finally block
- `monitor_thread()` - Added process reaping in finally block
- `/monitor` endpoint - Store grgsm_livemon process
- `auto_start_monitor()` - Store grgsm_livemon process
- `/stop` endpoint - Clean up grgsm_livemon process
2. `/opt/intercept/app.py`
- Added `gsm_spy_livemon_process` global variable
- Updated global declarations in `reset_decoder_processes()`
- Added cleanup for `gsm_spy_livemon_process`
## Testing
### Before Fix
```bash
# Start GSM Spy
# Check processes
ps aux | grep grgsm
# You would see:
root 12488 0.0 0.0 0 0 pts/2 Z+ 14:29 0:00 [grgsm_scanner] <defunct>
root 12489 0.0 0.0 0 0 pts/2 Z+ 14:29 0:00 [grgsm_livemon] <defunct>
```
### After Fix
```bash
# Start GSM Spy
# Check processes
ps aux | grep grgsm
# Active processes (no zombies):
root 12488 1.2 0.5 12345 5678 pts/2 S+ 14:29 0:01 grgsm_scanner -d 0 --freq-range...
root 12489 0.8 0.4 10234 4567 pts/2 S+ 14:29 0:01 grgsm_livemon -a 123 -d 0
# Stop GSM Spy
# Check processes
ps aux | grep grgsm
# No processes (all cleaned up properly)
```
### Verification Commands
1. **Check for zombie processes**:
```bash
ps aux | grep defunct
# Should return nothing after fix
```
2. **Monitor process lifecycle**:
```bash
# In one terminal, watch processes
watch -n 1 'ps aux | grep grgsm'
# In another terminal, start/stop GSM Spy
# Verify:
# - Processes start properly (S or R state, not Z)
# - Processes disappear when stopped (not left as zombies)
```
3. **Check process tree**:
```bash
pstree -p | grep grgsm
# Should show proper parent-child relationships
# No defunct/zombie entries
```
## Process Lifecycle
### Normal Operation
1. **Scanner Start**:
- `grgsm_scanner` spawned → stored in `gsm_spy_process`
- `scanner_thread` reads output
- Process running normally
2. **Monitor Start** (auto or manual):
- `grgsm_livemon` spawned → stored in `gsm_spy_livemon_process`
- `tshark` spawned → stored in `gsm_spy_monitor_process`
- `monitor_thread` reads tshark output
- Both processes running normally
3. **Stop**:
- All three processes terminated gracefully
- `wait()` called on each to collect exit status
- Process handles set to None
- No zombies remain
### Error Handling
1. **Process crashes during operation**:
- Thread's stdout loop exits
- `finally` block executes
- `process.wait()` collects exit status
- No zombie created
2. **Process hangs**:
- `terminate()` called
- `wait(timeout=5)` gives 5 seconds to exit
- If timeout, `kill()` is called
- `wait()` collects exit status
3. **Exception during cleanup**:
- Fallback to `kill()` + `wait()`
- Ensures zombie is always prevented
## Best Practices Applied
1. **Always reap child processes**: Call `wait()` or `waitpid()` after child process terminates
2. **Use process.poll()**: Check if process is still running before terminating
3. **Graceful shutdown**: Try `terminate()` before `kill()`
4. **Timeout handling**: Use `wait(timeout=N)` to prevent hanging
5. **Error recovery**: Multiple fallback levels in try/except blocks
6. **Track all processes**: Store handles for all spawned processes, not just the primary one
7. **Cleanup in finally**: Ensures cleanup happens even if exceptions occur
## Related Issues
This fix prevents:
- Zombie processes accumulating over time
- Process table filling up
- System resource leaks
- Confusing process listings for users
## Implementation Date
2026-02-06
## Additional Notes
- The fix follows the same patterns used in other INTERCEPT decoders
- Compatible with existing SDR device selection implementation
- No breaking changes to API or user interface
- Applies to both manual monitoring and auto-monitoring

435
routes/gsm_spy.py
View File

@@ -6,6 +6,7 @@ import json
import logging
import queue
import re
import select
import subprocess
import threading
import time
@@ -33,7 +34,8 @@ REGIONAL_BANDS = {
'PCS1900': {'start': 1930e6, 'end': 1990e6, 'arfcn_start': 512, 'arfcn_end': 810}
},
'Europe': {
'EGSM900': {'start': 925e6, 'end': 960e6, 'arfcn_start': 0, 'arfcn_end': 124}
'EGSM900': {'start': 925e6, 'end': 960e6, 'arfcn_start': 0, 'arfcn_end': 124},
'DCS1800': {'start': 1805e6, 'end': 1880e6, 'arfcn_start': 512, 'arfcn_end': 885}
},
'Asia': {
'EGSM900': {'start': 925e6, 'end': 960e6, 'arfcn_start': 0, 'arfcn_end': 124},
@@ -47,6 +49,9 @@ gsm_connected = False
gsm_towers_found = 0
gsm_devices_tracked = 0
# Geocoding worker state
_geocoding_worker_thread = None
# ============================================
# API Usage Tracking Helper Functions
@@ -82,6 +87,100 @@ def can_use_api():
return current_usage < config.GSM_API_DAILY_LIMIT
# ============================================
# Background Geocoding Worker
# ============================================
def start_geocoding_worker():
"""Start background thread for async geocoding."""
global _geocoding_worker_thread
if _geocoding_worker_thread is None or not _geocoding_worker_thread.is_alive():
_geocoding_worker_thread = threading.Thread(
target=geocoding_worker,
daemon=True,
name='gsm-geocoding-worker'
)
_geocoding_worker_thread.start()
logger.info("Started geocoding worker thread")
def geocoding_worker():
"""Worker thread processes pending geocoding requests."""
from utils.gsm_geocoding import lookup_cell_from_api, get_geocoding_queue
geocoding_queue = get_geocoding_queue()
while True:
try:
# Wait for pending tower with timeout
tower_data = geocoding_queue.get(timeout=5)
# Check rate limit
if not can_use_api():
current_usage = get_api_usage_today()
logger.warning(f"OpenCellID API rate limit reached ({current_usage}/{config.GSM_API_DAILY_LIMIT})")
geocoding_queue.task_done()
continue
# Call API
mcc = tower_data.get('mcc')
mnc = tower_data.get('mnc')
lac = tower_data.get('lac')
cid = tower_data.get('cid')
logger.debug(f"Geocoding tower via API: MCC={mcc} MNC={mnc} LAC={lac} CID={cid}")
coords = lookup_cell_from_api(mcc, mnc, lac, cid)
if coords:
# Update tower data with coordinates
tower_data['lat'] = coords['lat']
tower_data['lon'] = coords['lon']
tower_data['source'] = 'api'
tower_data['status'] = 'resolved'
tower_data['type'] = 'tower_update'
# Add optional fields if available
if coords.get('azimuth') is not None:
tower_data['azimuth'] = coords['azimuth']
if coords.get('range_meters') is not None:
tower_data['range_meters'] = coords['range_meters']
if coords.get('operator'):
tower_data['operator'] = coords['operator']
if coords.get('radio'):
tower_data['radio'] = coords['radio']
# Update DataStore
key = f"{mcc}_{mnc}_{lac}_{cid}"
app_module.gsm_spy_towers[key] = tower_data
# Send update to SSE stream
try:
app_module.gsm_spy_queue.put_nowait(tower_data)
logger.info(f"Resolved coordinates for tower: MCC={mcc} MNC={mnc} LAC={lac} CID={cid}")
except queue.Full:
logger.warning("SSE queue full, dropping tower update")
# Increment API usage counter
usage_count = increment_api_usage()
logger.info(f"OpenCellID API call #{usage_count} today")
else:
logger.warning(f"Could not resolve coordinates for tower: MCC={mcc} MNC={mnc} LAC={lac} CID={cid}")
geocoding_queue.task_done()
# Rate limiting between API calls (be nice to OpenCellID)
time.sleep(1)
except queue.Empty:
# No pending towers, continue waiting
continue
except Exception as e:
logger.error(f"Geocoding worker error: {e}", exc_info=True)
time.sleep(1)
def arfcn_to_frequency(arfcn):
"""Convert ARFCN to downlink frequency in Hz.
@@ -163,22 +262,18 @@ def start_scanner():
logger.info(f"Starting GSM scanner: {' '.join(cmd)}")
process = subprocess.Popen(
cmd,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
universal_newlines=True,
bufsize=1
)
app_module.gsm_spy_process = process
# Set a flag to indicate scanner should run
app_module.gsm_spy_active_device = device_index
app_module.gsm_spy_region = region
app_module.gsm_spy_process = True # Use as flag initially
# Start output parsing thread
# Start geocoding worker (if not already running)
start_geocoding_worker()
# Start scanning thread (will run grgsm_scanner in a loop)
scanner_thread_obj = threading.Thread(
target=scanner_thread,
args=(process,),
args=(cmd, device_index),
daemon=True
)
scanner_thread_obj.start()
@@ -242,14 +337,18 @@ def start_monitor():
logger.info(f"Starting GSM monitor: {' '.join(grgsm_cmd)} | {' '.join(tshark_cmd)}")
# Start grgsm_livemon
# Start grgsm_livemon (outputs to UDP port 4729 by default)
grgsm_proc = subprocess.Popen(
grgsm_cmd,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE
)
logger.info(f"Started grgsm_livemon (PID: {grgsm_proc.pid})")
# Start tshark
# Give grgsm_livemon time to initialize and start sending UDP packets
time.sleep(2)
# Start tshark (captures from loopback interface where UDP packets arrive)
tshark_proc = subprocess.Popen(
tshark_cmd,
stdout=subprocess.PIPE,
@@ -257,6 +356,7 @@ def start_monitor():
universal_newlines=True,
bufsize=1
)
logger.info(f"Started tshark (PID: {tshark_proc.pid})")
app_module.gsm_spy_livemon_process = grgsm_proc
app_module.gsm_spy_monitor_process = tshark_proc
@@ -291,17 +391,10 @@ def stop_scanner():
with app_module.gsm_spy_lock:
killed = []
# Stop scanner (now just a flag, thread will see it and exit)
if app_module.gsm_spy_process:
try:
app_module.gsm_spy_process.terminate()
app_module.gsm_spy_process.wait(timeout=5)
killed.append('scanner')
except Exception:
try:
app_module.gsm_spy_process.kill()
except Exception:
pass
app_module.gsm_spy_process = None
killed.append('scanner')
if app_module.gsm_spy_livemon_process:
try:
@@ -917,33 +1010,45 @@ def traffic_correlation():
# ============================================
def parse_grgsm_scanner_output(line: str) -> dict[str, Any] | None:
"""Parse grgsm_scanner output line."""
"""Parse grgsm_scanner output line.
Actual output format is a table:
ARFCN | Freq (MHz) | CID | LAC | MCC | MNC | Power (dB)
--------------------------------------------------------------------
23 | 940.6 | 31245 | 1234 | 214 | 01 | -48
"""
try:
# Example output: "ARFCN: 123, Freq: 935.2MHz, CID: 1234, LAC: 567, MCC: 310, MNC: 260, PWR: -85dBm"
# This is a placeholder - actual format depends on grgsm_scanner output
# Skip progress, header, and separator lines
if 'Scanning:' in line or 'ARFCN' in line or '---' in line or 'Found' in line:
return None
# Simple regex patterns
arfcn_match = re.search(r'ARFCN[:\s]+(\d+)', line)
freq_match = re.search(r'Freq[:\s]+([\d.]+)', line)
cid_match = re.search(r'CID[:\s]+(\d+)', line)
lac_match = re.search(r'LAC[:\s]+(\d+)', line)
mcc_match = re.search(r'MCC[:\s]+(\d+)', line)
mnc_match = re.search(r'MNC[:\s]+(\d+)', line)
pwr_match = re.search(r'PWR[:\s]+([-\d.]+)', line)
# Parse table row: " 23 | 940.6 | 31245 | 1234 | 214 | 01 | -48"
# Split by pipe and clean whitespace
parts = [p.strip() for p in line.split('|')]
if arfcn_match:
data = {
'type': 'tower',
'arfcn': int(arfcn_match.group(1)),
'frequency': float(freq_match.group(1)) if freq_match else None,
'cid': int(cid_match.group(1)) if cid_match else None,
'lac': int(lac_match.group(1)) if lac_match else None,
'mcc': int(mcc_match.group(1)) if mcc_match else None,
'mnc': int(mnc_match.group(1)) if mnc_match else None,
'signal_strength': float(pwr_match.group(1)) if pwr_match else None,
'timestamp': datetime.now().isoformat()
}
return data
if len(parts) >= 7:
arfcn = parts[0]
freq = parts[1]
cid = parts[2]
lac = parts[3]
mcc = parts[4]
mnc = parts[5]
power = parts[6]
# Validate that we have numeric data (not header line)
if arfcn.isdigit():
data = {
'type': 'tower',
'arfcn': int(arfcn),
'frequency': float(freq),
'cid': int(cid),
'lac': int(lac),
'mcc': int(mcc),
'mnc': int(mnc),
'signal_strength': float(power),
'timestamp': datetime.now().isoformat()
}
return data
except Exception as e:
logger.debug(f"Failed to parse scanner line: {line} - {e}")
@@ -1025,14 +1130,18 @@ def auto_start_monitor(tower_data):
logger.info(f"Starting auto-monitor: {' '.join(grgsm_cmd)} | {' '.join(tshark_cmd)}")
# Start grgsm_livemon (we don't capture its output)
# Start grgsm_livemon (outputs to UDP port 4729 by default)
grgsm_proc = subprocess.Popen(
grgsm_cmd,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE
)
logger.info(f"Started grgsm_livemon for auto-monitor (PID: {grgsm_proc.pid})")
# Start tshark
# Give grgsm_livemon time to initialize and start sending UDP packets
time.sleep(2)
# Start tshark (captures from loopback interface where UDP packets arrive)
tshark_proc = subprocess.Popen(
tshark_cmd,
stdout=subprocess.PIPE,
@@ -1040,6 +1149,7 @@ def auto_start_monitor(tower_data):
universal_newlines=True,
bufsize=1
)
logger.info(f"Started tshark for auto-monitor (PID: {tshark_proc.pid})")
app_module.gsm_spy_livemon_process = grgsm_proc
app_module.gsm_spy_monitor_process = tshark_proc
@@ -1069,66 +1179,192 @@ def auto_start_monitor(tower_data):
logger.error(f"Error in auto-monitoring: {e}")
def scanner_thread(process):
"""Thread to read grgsm_scanner output."""
def scanner_thread(cmd, device_index):
"""Thread to continuously run grgsm_scanner in a loop with non-blocking I/O.
grgsm_scanner scans once and exits, so we loop it to provide
continuous updates to the dashboard.
"""
global gsm_towers_found
strongest_tower = None
auto_monitor_triggered = False
auto_monitor_triggered = False # Moved outside loop - persists across scans
scan_count = 0
process = None
try:
for line in process.stdout:
if not line:
continue
while app_module.gsm_spy_process: # Flag check
scan_count += 1
logger.info(f"Starting GSM scan #{scan_count}")
parsed = parse_grgsm_scanner_output(line)
if parsed:
# Store in DataStore
key = f"{parsed.get('mcc')}_{parsed.get('mnc')}_{parsed.get('lac')}_{parsed.get('cid')}"
app_module.gsm_spy_towers[key] = parsed
try:
# Start scanner process
process = subprocess.Popen(
cmd,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
universal_newlines=True,
bufsize=1
)
# Track strongest tower for auto-monitoring
signal_strength = parsed.get('signal_strength', -999)
if strongest_tower is None or signal_strength > strongest_tower.get('signal_strength', -999):
strongest_tower = parsed
# Non-blocking stderr reader
def read_stderr():
try:
for line in process.stderr:
if line:
logger.debug(f"grgsm_scanner: {line.strip()}")
except Exception as e:
logger.error(f"stderr read error: {e}")
# Queue for SSE stream
try:
app_module.gsm_spy_queue.put_nowait(parsed)
except queue.Full:
pass
stderr_thread = threading.Thread(target=read_stderr, daemon=True)
stderr_thread.start()
gsm_towers_found += 1
# Non-blocking stdout reader with timeout
last_output = time.time()
scan_timeout = 120 # 2 minute maximum per scan
# Auto-monitor strongest tower after finding 3+ towers
if gsm_towers_found >= 3 and not auto_monitor_triggered and strongest_tower:
auto_monitor_triggered = True
threading.Thread(
target=auto_start_monitor,
args=(strongest_tower,),
daemon=True
).start()
while app_module.gsm_spy_process:
# Check if process died
if process.poll() is not None:
logger.info(f"Scanner exited (code: {process.returncode})")
break
# Check for output with 1-second timeout
ready, _, _ = select.select([process.stdout], [], [], 1.0)
if ready:
line = process.stdout.readline()
if not line:
break # EOF
last_output = time.time()
parsed = parse_grgsm_scanner_output(line)
if parsed:
# Enrich with coordinates
from utils.gsm_geocoding import enrich_tower_data
enriched = enrich_tower_data(parsed)
# Store in DataStore
key = f"{enriched['mcc']}_{enriched['mnc']}_{enriched['lac']}_{enriched['cid']}"
app_module.gsm_spy_towers[key] = enriched
# Track strongest tower
signal = enriched.get('signal_strength', -999)
if strongest_tower is None or signal > strongest_tower.get('signal_strength', -999):
strongest_tower = enriched
# Queue for SSE
try:
app_module.gsm_spy_queue.put_nowait(enriched)
except queue.Full:
logger.warning("Queue full, dropping tower update")
# Thread-safe counter update
with app_module.gsm_spy_lock:
gsm_towers_found += 1
current_count = gsm_towers_found
# Auto-monitor strongest tower (once per session)
if current_count >= 3 and not auto_monitor_triggered and strongest_tower:
auto_monitor_triggered = True
logger.info("Auto-starting monitor on strongest tower")
threading.Thread(
target=auto_start_monitor,
args=(strongest_tower,),
daemon=True
).start()
else:
# No output, check timeout
if time.time() - last_output > scan_timeout:
logger.warning(f"Scan timeout after {scan_timeout}s")
break
# Clean up process with timeout
if process.poll() is None:
logger.info("Terminating scanner process")
process.terminate()
try:
process.wait(timeout=5)
except subprocess.TimeoutExpired:
logger.warning("Process didn't terminate, killing")
process.kill()
process.wait()
else:
process.wait() # Reap zombie
logger.info(f"Scan #{scan_count} complete")
except Exception as e:
logger.error(f"Scanner scan error: {e}", exc_info=True)
if process and process.poll() is None:
try:
process.terminate()
process.wait(timeout=2)
except Exception:
try:
process.kill()
except Exception:
pass
# Check if should continue
if not app_module.gsm_spy_process:
break
# Wait between scans with responsive flag checking
logger.info("Waiting 5 seconds before next scan")
for i in range(5):
if not app_module.gsm_spy_process:
break
time.sleep(1)
except Exception as e:
logger.error(f"Scanner thread error: {e}")
logger.error(f"Scanner thread fatal error: {e}", exc_info=True)
finally:
# Reap the process to prevent zombie (don't terminate, just wait)
try:
process.wait()
logger.info(f"Scanner process exited with code {process.returncode}")
except Exception as e:
logger.error(f"Error waiting for scanner process: {e}")
# Always cleanup
if process and process.poll() is None:
try:
process.terminate()
process.wait(timeout=5)
except Exception:
try:
process.kill()
process.wait()
except Exception:
pass
logger.info("Scanner thread terminated")
# Reset global state
with app_module.gsm_spy_lock:
app_module.gsm_spy_process = None
if app_module.gsm_spy_active_device is not None:
from app import release_sdr_device
release_sdr_device(app_module.gsm_spy_active_device)
app_module.gsm_spy_active_device = None
def monitor_thread(process):
"""Thread to read grgsm_livemon | tshark output."""
"""Thread to read tshark output with non-blocking I/O and timeouts."""
global gsm_devices_tracked
try:
for line in process.stdout:
while app_module.gsm_spy_monitor_process:
# Check if process died
if process.poll() is not None:
logger.info(f"Monitor process exited (code: {process.returncode})")
break
# Non-blocking read with timeout
ready, _, _ = select.select([process.stdout], [], [], 1.0)
if not ready:
continue # Timeout, check flag again
line = process.stdout.readline()
if not line:
continue
break # EOF
parsed = parse_tshark_output(line)
if parsed:
@@ -1218,15 +1454,28 @@ def monitor_thread(process):
except Exception as e:
logger.error(f"Error storing device data: {e}")
gsm_devices_tracked += 1
# Thread-safe counter
with app_module.gsm_spy_lock:
gsm_devices_tracked += 1
except Exception as e:
logger.error(f"Monitor thread error: {e}")
logger.error(f"Monitor thread error: {e}", exc_info=True)
finally:
# Reap the process to prevent zombie (don't terminate, just wait)
# Reap process with timeout
try:
process.wait()
if process.poll() is None:
process.terminate()
try:
process.wait(timeout=5)
except subprocess.TimeoutExpired:
logger.warning("Monitor process didn't terminate, killing")
process.kill()
process.wait()
else:
process.wait()
logger.info(f"Monitor process exited with code {process.returncode}")
except Exception as e:
logger.error(f"Error waiting for monitor process: {e}")
logger.error(f"Error reaping monitor process: {e}")
logger.info("Monitor thread terminated")

12
templates/gsm_spy_dashboard.html
View File

@@ -1548,6 +1548,10 @@
if (data.type === 'tower') {
updateTower(data);
} else if (data.type === 'tower_update') {
// Background geocoding resolved coordinates for a tower
console.log(`Tower coordinates resolved via API: MCC=${data.mcc} MNC=${data.mnc} LAC=${data.lac} CID=${data.cid}`);
updateTower(data);
} else if (data.type === 'device') {
updateDevice(data);
} else if (data.type === 'rogue_alert') {
@@ -1576,6 +1580,14 @@
const key = `${data.mcc}-${data.mnc}-${data.lac}-${data.cid}`;
towers[key] = data;
// Validate coordinates before creating map marker
if (!data.lat || !data.lon || isNaN(parseFloat(data.lat)) || isNaN(parseFloat(data.lon))) {
console.log(`Tower ${data.cid} pending geocoding (status: ${data.status || 'unknown'})`);
// Update towers list but skip map marker
updateTowersList();
return;
}
// Create or update marker
if (!towerMarkers[key]) {
// Create new marker

261
test_gsm_spy_fixes.sh Executable file
View File

@@ -0,0 +1,261 @@
#!/bin/bash
# GSM Spy System - Verification Test Script
# Tests the 4 critical fixes: geocoding, pipeline, scanner loop, process management
set -e
echo "=========================================="
echo "GSM Spy System - Verification Tests"
echo "=========================================="
echo ""
# Colors
GREEN='\033[0;32m'
RED='\033[0;31m'
YELLOW='\033[1;33m'
NC='\033[0m' # No Color
# Test results
TESTS_PASSED=0
TESTS_FAILED=0
function pass_test() {
echo -e "${GREEN}✓ PASS:${NC} $1"
((TESTS_PASSED++))
}
function fail_test() {
echo -e "${RED}✗ FAIL:${NC} $1"
((TESTS_FAILED++))
}
function info() {
echo -e "${YELLOW}ℹ INFO:${NC} $1"
}
# Test 1: Check that geocoding module exists
echo "Test 1: Geocoding Module"
echo "-------------------------"
if [ -f "utils/gsm_geocoding.py" ]; then
pass_test "Geocoding module exists"
# Check for key functions
if grep -q "def enrich_tower_data" utils/gsm_geocoding.py; then
pass_test "enrich_tower_data() function present"
else
fail_test "enrich_tower_data() function missing"
fi
if grep -q "def lookup_cell_coordinates" utils/gsm_geocoding.py; then
pass_test "lookup_cell_coordinates() function present"
else
fail_test "lookup_cell_coordinates() function missing"
fi
else
fail_test "Geocoding module missing"
fi
echo ""
# Test 2: Check scanner thread improvements
echo "Test 2: Scanner Thread Non-Blocking I/O"
echo "---------------------------------------"
if grep -q "import select" routes/gsm_spy.py; then
pass_test "select module imported"
else
fail_test "select module not imported"
fi
if grep -q "select.select.*process.stdout" routes/gsm_spy.py; then
pass_test "Non-blocking I/O with select.select() implemented"
else
fail_test "select.select() not found in scanner thread"
fi
if grep -q "scan_timeout = 120" routes/gsm_spy.py; then
pass_test "Scan timeout configured"
else
fail_test "Scan timeout not configured"
fi
if grep -q "with app_module.gsm_spy_lock:" routes/gsm_spy.py; then
pass_test "Thread-safe counter updates implemented"
else
fail_test "Thread-safe counter updates missing"
fi
echo ""
# Test 3: Check geocoding worker
echo "Test 3: Background Geocoding Worker"
echo "-----------------------------------"
if grep -q "def start_geocoding_worker" routes/gsm_spy.py; then
pass_test "start_geocoding_worker() function exists"
else
fail_test "start_geocoding_worker() function missing"
fi
if grep -q "def geocoding_worker" routes/gsm_spy.py; then
pass_test "geocoding_worker() function exists"
else
fail_test "geocoding_worker() function missing"
fi
if grep -q "start_geocoding_worker()" routes/gsm_spy.py; then
pass_test "Geocoding worker is started in start_scanner()"
else
fail_test "Geocoding worker not started in start_scanner()"
fi
echo ""
# Test 4: Check enrichment integration
echo "Test 4: Tower Data Enrichment"
echo "-----------------------------"
if grep -q "from utils.gsm_geocoding import enrich_tower_data" routes/gsm_spy.py; then
pass_test "enrich_tower_data imported in scanner thread"
else
fail_test "enrich_tower_data not imported"
fi
if grep -q "enriched = enrich_tower_data(parsed)" routes/gsm_spy.py; then
pass_test "Tower data enrichment called in scanner"
else
fail_test "Tower data enrichment not called"
fi
echo ""
# Test 5: Check monitor pipeline fixes
echo "Test 5: Monitor Pipeline Connection"
echo "-----------------------------------"
if grep -q "Give grgsm_livemon time to initialize" routes/gsm_spy.py; then
pass_test "Pipeline initialization delay comment present"
else
fail_test "Pipeline initialization delay comment missing"
fi
if grep -A 5 "Start grgsm_livemon" routes/gsm_spy.py | grep -q "time.sleep(2)"; then
pass_test "2-second delay between grgsm_livemon and tshark"
else
fail_test "Initialization delay not implemented"
fi
if grep -q "Started grgsm_livemon (PID:" routes/gsm_spy.py; then
pass_test "Process verification logging added"
else
fail_test "Process verification logging missing"
fi
echo ""
# Test 6: Check monitor thread improvements
echo "Test 6: Monitor Thread Non-Blocking I/O"
echo "---------------------------------------"
if grep -q "def monitor_thread(process):" routes/gsm_spy.py; then
pass_test "monitor_thread() function exists"
if grep -A 20 "def monitor_thread(process):" routes/gsm_spy.py | grep -q "select.select.*process.stdout"; then
pass_test "Monitor thread uses non-blocking I/O"
else
fail_test "Monitor thread doesn't use select.select()"
fi
else
fail_test "monitor_thread() function missing"
fi
echo ""
# Test 7: Check frontend coordinate validation
echo "Test 7: Frontend Coordinate Validation"
echo "--------------------------------------"
if grep -q "Validate coordinates before creating map marker" templates/gsm_spy_dashboard.html; then
pass_test "Coordinate validation comment present"
else
fail_test "Coordinate validation comment missing"
fi
if grep -q "isNaN(parseFloat(data.lat))" templates/gsm_spy_dashboard.html; then
pass_test "Coordinate validation checks implemented"
else
fail_test "Coordinate validation checks missing"
fi
if grep -q "tower_update" templates/gsm_spy_dashboard.html; then
pass_test "tower_update message handler added"
else
fail_test "tower_update message handler missing"
fi
echo ""
# Test 8: Check process cleanup improvements
echo "Test 8: Process Cleanup & Zombie Prevention"
echo "-------------------------------------------"
if grep -q "process.terminate()" routes/gsm_spy.py; then
pass_test "Process termination implemented"
else
fail_test "Process termination missing"
fi
if grep -q "subprocess.TimeoutExpired" routes/gsm_spy.py; then
pass_test "Timeout handling for process termination"
else
fail_test "Timeout handling missing"
fi
if grep -q "process.kill()" routes/gsm_spy.py; then
pass_test "Force kill fallback implemented"
else
fail_test "Force kill fallback missing"
fi
echo ""
# Test 9: Python syntax check
echo "Test 9: Python Syntax Validation"
echo "--------------------------------"
if python3 -m py_compile routes/gsm_spy.py 2>/dev/null; then
pass_test "routes/gsm_spy.py has valid syntax"
else
fail_test "routes/gsm_spy.py has syntax errors"
fi
if python3 -m py_compile utils/gsm_geocoding.py 2>/dev/null; then
pass_test "utils/gsm_geocoding.py has valid syntax"
else
fail_test "utils/gsm_geocoding.py has syntax errors"
fi
echo ""
# Test 10: Check auto-monitor persistence
echo "Test 10: Auto-Monitor Flag Persistence"
echo "--------------------------------------"
if grep -q "auto_monitor_triggered = False.*# Moved outside loop" routes/gsm_spy.py; then
pass_test "auto_monitor_triggered flag moved outside loop"
else
fail_test "auto_monitor_triggered flag not properly placed"
fi
if grep -q "if current_count >= 3 and not auto_monitor_triggered" routes/gsm_spy.py; then
pass_test "Auto-monitor only triggers once per session"
else
fail_test "Auto-monitor trigger condition incorrect"
fi
echo ""
# Summary
echo "=========================================="
echo "Test Summary"
echo "=========================================="
echo -e "Tests passed: ${GREEN}${TESTS_PASSED}${NC}"
echo -e "Tests failed: ${RED}${TESTS_FAILED}${NC}"
echo ""
if [ $TESTS_FAILED -eq 0 ]; then
echo -e "${GREEN}All tests passed! ✓${NC}"
echo ""
echo "Next steps:"
echo "1. Start INTERCEPT: sudo -E venv/bin/python intercept.py"
echo "2. Navigate to GSM Spy dashboard in browser"
echo "3. Click 'Start Scanner' to test tower detection with geocoding"
echo "4. Verify towers appear on map with coordinates"
echo "5. Check that auto-monitor starts after 3+ towers found"
echo "6. Test Stop button for responsive shutdown (< 2 seconds)"
exit 0
else
echo -e "${RED}Some tests failed. Please review the output above.${NC}"
exit 1
fi

200
utils/gsm_geocoding.py Normal file
View File

@@ -0,0 +1,200 @@
"""GSM Cell Tower Geocoding Service.
Provides hybrid cache-first geocoding with async API fallback for cell towers.
"""
from __future__ import annotations
import logging
import queue
from typing import Any
import requests
import config
from utils.database import get_db
logger = logging.getLogger('intercept.gsm_geocoding')
# Queue for pending geocoding requests
_geocoding_queue = queue.Queue(maxsize=100)
def lookup_cell_coordinates(mcc: int, mnc: int, lac: int, cid: int) -> dict[str, Any] | None:
"""
Lookup cell tower coordinates with cache-first strategy.
Strategy:
1. Check gsm_cells table (cache) - fast synchronous lookup
2. If not found, return None (caller decides whether to use API)
Args:
mcc: Mobile Country Code
mnc: Mobile Network Code
lac: Location Area Code
cid: Cell ID
Returns:
dict with keys: lat, lon, source='cache', azimuth (optional),
range_meters (optional), operator (optional), radio (optional)
Returns None if not found in cache.
"""
try:
with get_db() as conn:
result = conn.execute('''
SELECT lat, lon, azimuth, range_meters, operator, radio
FROM gsm_cells
WHERE mcc = ? AND mnc = ? AND lac = ? AND cid = ?
''', (mcc, mnc, lac, cid)).fetchone()
if result:
return {
'lat': result['lat'],
'lon': result['lon'],
'source': 'cache',
'azimuth': result['azimuth'],
'range_meters': result['range_meters'],
'operator': result['operator'],
'radio': result['radio']
}
return None
except Exception as e:
logger.error(f"Error looking up coordinates from cache: {e}")
return None
def lookup_cell_from_api(mcc: int, mnc: int, lac: int, cid: int) -> dict[str, Any] | None:
"""
Lookup cell tower from OpenCellID API and cache result.
Args:
mcc: Mobile Country Code
mnc: Mobile Network Code
lac: Location Area Code
cid: Cell ID
Returns:
dict with keys: lat, lon, source='api', azimuth (optional),
range_meters (optional), operator (optional), radio (optional)
Returns None if API call fails or cell not found.
"""
try:
api_url = config.GSM_OPENCELLID_API_URL
params = {
'key': config.GSM_OPENCELLID_API_KEY,
'mcc': mcc,
'mnc': mnc,
'lac': lac,
'cellid': cid,
'format': 'json'
}
response = requests.get(api_url, params=params, timeout=10)
if response.status_code == 200:
cell_data = response.json()
# Cache the result
with get_db() as conn:
conn.execute('''
INSERT OR REPLACE INTO gsm_cells
(mcc, mnc, lac, cid, lat, lon, azimuth, range_meters, samples, radio, operator, last_verified)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, CURRENT_TIMESTAMP)
''', (
mcc, mnc, lac, cid,
cell_data.get('lat'),
cell_data.get('lon'),
cell_data.get('azimuth'),
cell_data.get('range'),
cell_data.get('samples'),
cell_data.get('radio'),
cell_data.get('operator')
))
conn.commit()
logger.info(f"Cached cell tower from API: MCC={mcc} MNC={mnc} LAC={lac} CID={cid}")
return {
'lat': cell_data.get('lat'),
'lon': cell_data.get('lon'),
'source': 'api',
'azimuth': cell_data.get('azimuth'),
'range_meters': cell_data.get('range'),
'operator': cell_data.get('operator'),
'radio': cell_data.get('radio')
}
else:
logger.warning(f"OpenCellID API returned {response.status_code} for MCC={mcc} MNC={mnc} LAC={lac} CID={cid}")
return None
except Exception as e:
logger.error(f"Error calling OpenCellID API: {e}")
return None
def enrich_tower_data(tower_data: dict[str, Any]) -> dict[str, Any]:
"""
Enrich tower data with coordinates using cache-first strategy.
If coordinates found in cache, adds them immediately.
If not found, marks as 'pending' and queues for background API lookup.
Args:
tower_data: Dictionary with keys mcc, mnc, lac, cid (and other tower data)
Returns:
Enriched tower_data dict with added fields:
- lat, lon (if found in cache)
- status='pending' (if needs API lookup)
- source='cache' (if from cache)
"""
mcc = tower_data.get('mcc')
mnc = tower_data.get('mnc')
lac = tower_data.get('lac')
cid = tower_data.get('cid')
# Validate required fields
if not all([mcc is not None, mnc is not None, lac is not None, cid is not None]):
logger.warning(f"Tower data missing required fields: {tower_data}")
return tower_data
# Try cache lookup
coords = lookup_cell_coordinates(mcc, mnc, lac, cid)
if coords:
# Found in cache - add coordinates immediately
tower_data['lat'] = coords['lat']
tower_data['lon'] = coords['lon']
tower_data['source'] = 'cache'
# Add optional fields if available
if coords.get('azimuth') is not None:
tower_data['azimuth'] = coords['azimuth']
if coords.get('range_meters') is not None:
tower_data['range_meters'] = coords['range_meters']
if coords.get('operator'):
tower_data['operator'] = coords['operator']
if coords.get('radio'):
tower_data['radio'] = coords['radio']
logger.debug(f"Cache hit for tower: MCC={mcc} MNC={mnc} LAC={lac} CID={cid}")
else:
# Not in cache - mark as pending and queue for API lookup
tower_data['status'] = 'pending'
tower_data['source'] = 'unknown'
# Queue for background geocoding (non-blocking)
try:
_geocoding_queue.put_nowait(tower_data.copy())
logger.debug(f"Queued tower for geocoding: MCC={mcc} MNC={mnc} LAC={lac} CID={cid}")
except queue.Full:
logger.warning("Geocoding queue full, dropping tower")
return tower_data
def get_geocoding_queue() -> queue.Queue:
"""Get the geocoding queue for the background worker."""
return _geocoding_queue