mirror of
https://github.com/smittix/intercept.git
synced 2026-05-23 00:04:50 -07:00
fixing bands and how the gsm scanner loops with tshark
This commit is contained in:
Marc
@@ -1,224 +0,0 @@
|
|||||||
# GSM Spy SDR Device Selection Implementation
|
|
||||||
|
|
||||||
## Summary
|
|
||||||
|
|
||||||
Successfully implemented dynamic SDR device detection, selection, and management for the GSM Spy feature, following the same pattern used in the Aircraft/ADS-B implementation.
|
|
||||||
|
|
||||||
## Changes Made
|
|
||||||
|
|
||||||
### Frontend Changes (`templates/gsm_spy_dashboard.html`)
|
|
||||||
|
|
||||||
#### 1. Dynamic Device Selector
|
|
||||||
- **Changed**: Device dropdown from hardcoded options to dynamic detection
|
|
||||||
- **Location**: Line ~1155 (Signal Source Panel)
|
|
||||||
- **Before**: Static options (Device 0, Device 1, etc.)
|
|
||||||
- **After**: Dynamic population with "Detecting devices..." placeholder
|
|
||||||
|
|
||||||
#### 2. Device Detection on Page Load
|
|
||||||
- **Added**: `initDeviceSelector()` function
|
|
||||||
- **Location**: ~Line 1395
|
|
||||||
- **Functionality**:
|
|
||||||
- Fetches available SDR devices from `/devices` endpoint
|
|
||||||
- Populates dropdown with detected devices
|
|
||||||
- Shows device name, type (RTL-SDR, HackRF, etc.), and serial number
|
|
||||||
- Handles errors gracefully with user-friendly messages
|
|
||||||
- Logs detection results to console
|
|
||||||
|
|
||||||
#### 3. Scanner Controls Update
|
|
||||||
- **Modified**: `startScanner()` function (~Line 1410)
|
|
||||||
- **Changes**:
|
|
||||||
- Made async for better error handling
|
|
||||||
- Reads device index from `deviceSelect` dropdown
|
|
||||||
- Disables device and region selectors during active scanning
|
|
||||||
- Enhanced error handling with device conflict detection
|
|
||||||
- Shows user-friendly alerts for device busy errors
|
|
||||||
|
|
||||||
#### 4. Stop Scanner Enhancements
|
|
||||||
- **Modified**: `stopScanner()` function (~Line 1494)
|
|
||||||
- **Changes**:
|
|
||||||
- Re-enables device and region selectors after stopping
|
|
||||||
- Maintains UI consistency
|
|
||||||
|
|
||||||
#### 5. Region Selector Sync
|
|
||||||
- **Modified**: `selectRegion()` function (~Line 1882)
|
|
||||||
- **Changes**:
|
|
||||||
- Capitalizes region name to match backend API expectations
|
|
||||||
- Syncs region button selection with dropdown
|
|
||||||
|
|
||||||
#### 6. Removed Redundant Controls
|
|
||||||
- **Removed**: `scannerDevice` dropdown from bottom controls bar
|
|
||||||
- **Reason**: Consolidated to single device selector in left sidebar
|
|
||||||
|
|
||||||
### Backend Changes (`routes/gsm_spy.py`)
|
|
||||||
|
|
||||||
#### 1. Enhanced Error Response
|
|
||||||
- **Modified**: `/start` endpoint device claiming logic (~Line 115)
|
|
||||||
- **Changes**:
|
|
||||||
- Added `error_type: 'DEVICE_BUSY'` to 409 conflict responses
|
|
||||||
- Enables frontend to distinguish device conflicts from other errors
|
|
||||||
- Allows for targeted user-friendly error messages
|
|
||||||
|
|
||||||
#### 2. Existing Device Management (Verified)
|
|
||||||
- **Confirmed**: Device claiming/releasing already implemented
|
|
||||||
- `claim_sdr_device()` called at line 115
|
|
||||||
- `release_sdr_device()` called at line 289
|
|
||||||
- Device index stored in `gsm_spy_active_device`
|
|
||||||
- Region stored in `gsm_spy_region`
|
|
||||||
|
|
||||||
#### 3. Status Endpoint (Verified)
|
|
||||||
- **Confirmed**: `/status` endpoint returns device info
|
|
||||||
- Returns `device` (active device index)
|
|
||||||
- Returns `region` (selected region)
|
|
||||||
- Returns all necessary status information
|
|
||||||
|
|
||||||
## Features Implemented
|
|
||||||
|
|
||||||
### ✅ Device Detection
|
|
||||||
- Dynamically detects all available SDR devices on page load
|
|
||||||
- Supports all 5 SDR types: RTL-SDR, HackRF, LimeSDR, Airspy, SDRPlay
|
|
||||||
- Shows device name, type, and serial number in dropdown
|
|
||||||
|
|
||||||
### ✅ Device Registry Integration
|
|
||||||
- Properly claims devices before starting scanner
|
|
||||||
- Releases devices when stopping scanner
|
|
||||||
- Prevents conflicts with other INTERCEPT modes
|
|
||||||
|
|
||||||
### ✅ UI State Management
|
|
||||||
- Disables device selector during active scanning
|
|
||||||
- Re-enables selector after stopping
|
|
||||||
- Provides clear visual feedback to user
|
|
||||||
|
|
||||||
### ✅ Error Handling
|
|
||||||
- User-friendly error messages for device conflicts
|
|
||||||
- Graceful handling of "no devices detected" scenario
|
|
||||||
- Clear console logging for debugging
|
|
||||||
|
|
||||||
### ✅ Validation
|
|
||||||
- Uses existing `validate_device_index()` function (already in code)
|
|
||||||
- Validates region against `REGIONAL_BANDS` dictionary
|
|
||||||
- Checks for already running scanner
|
|
||||||
|
|
||||||
## Architecture Pattern
|
|
||||||
|
|
||||||
The implementation follows the same pattern as Aircraft/ADS-B:
|
|
||||||
|
|
||||||
1. **Device Detection**: `/devices` endpoint (shared across all modes)
|
|
||||||
2. **Device Claiming**: `claim_sdr_device()` before starting
|
|
||||||
3. **Device Releasing**: `release_sdr_device()` on stop
|
|
||||||
4. **UI Consistency**: Dynamic dropdown, disabled during operation
|
|
||||||
5. **Error Handling**: Clear user messages, console logging
|
|
||||||
|
|
||||||
## Testing Recommendations
|
|
||||||
|
|
||||||
### 1. Device Detection
|
|
||||||
```bash
|
|
||||||
# Start application
|
|
||||||
sudo -E venv/bin/python intercept.py
|
|
||||||
|
|
||||||
# Open GSM Spy dashboard in browser
|
|
||||||
# Open DevTools console
|
|
||||||
# Should see: "[GSM SPY] Detected X SDR device(s)"
|
|
||||||
# Verify dropdown shows detected devices
|
|
||||||
```
|
|
||||||
|
|
||||||
### 2. Device Claiming
|
|
||||||
```bash
|
|
||||||
# Start GSM scanner on device 0
|
|
||||||
# Try to start another mode (e.g., ADS-B) on device 0
|
|
||||||
# Should see conflict error message
|
|
||||||
# Stop GSM scanner
|
|
||||||
# Now ADS-B should be able to claim device 0
|
|
||||||
```
|
|
||||||
|
|
||||||
### 3. Multiple Devices
|
|
||||||
```bash
|
|
||||||
# Connect multiple SDR devices
|
|
||||||
# Open GSM Spy dashboard
|
|
||||||
# Verify all devices appear in dropdown
|
|
||||||
# Select different devices and verify they work independently
|
|
||||||
```
|
|
||||||
|
|
||||||
### 4. UI State
|
|
||||||
```bash
|
|
||||||
# Start GSM scanner
|
|
||||||
# Verify device selector is disabled
|
|
||||||
# Verify region selector is disabled
|
|
||||||
# Stop scanner
|
|
||||||
# Verify both selectors are re-enabled
|
|
||||||
```
|
|
||||||
|
|
||||||
### 5. Error Scenarios
|
|
||||||
```bash
|
|
||||||
# Disconnect SDR device
|
|
||||||
# Try to start scanner
|
|
||||||
# Should see graceful error message
|
|
||||||
# Reconnect device
|
|
||||||
# Refresh page - device should be detected
|
|
||||||
```
|
|
||||||
|
|
||||||
## Known Limitations
|
|
||||||
|
|
||||||
1. **gr-gsm Hardware Support**: The `gr-gsm` tools may have limited support for non-RTL-SDR devices. This implementation handles device selection properly, but `gr-gsm` itself may only work with RTL-SDR.
|
|
||||||
|
|
||||||
2. **Command Builder Integration**: Full SDRFactory integration (using device-specific command builders) would require adding GSM-specific methods to command builders in `utils/sdr/`. This is a future enhancement.
|
|
||||||
|
|
||||||
3. **Remote Device Support**: Unlike ADS-B which supports remote dump1090 connections, GSM Spy currently only supports local SDR devices.
|
|
||||||
|
|
||||||
## Future Enhancements
|
|
||||||
|
|
||||||
### 1. SDRFactory Integration
|
|
||||||
```python
|
|
||||||
# In start_scanner():
|
|
||||||
from utils.sdr import SDRFactory
|
|
||||||
|
|
||||||
devices = SDRFactory.detect_devices()
|
|
||||||
sdr_device = next((d for d in devices if d.index == device_index), None)
|
|
||||||
|
|
||||||
builder = SDRFactory.get_builder(sdr_device.sdr_type)
|
|
||||||
cmd = builder.build_gsm_scanner_command(device=sdr_device, bands=REGIONAL_BANDS[region])
|
|
||||||
```
|
|
||||||
|
|
||||||
Note: This requires adding `build_gsm_scanner_command()` method to command builders.
|
|
||||||
|
|
||||||
### 2. Device-Specific Tuning
|
|
||||||
- Different gain settings per SDR type
|
|
||||||
- Frequency correction (PPM) based on device calibration
|
|
||||||
- Sample rate optimization per hardware
|
|
||||||
|
|
||||||
### 3. Multi-Device Monitoring
|
|
||||||
- Simultaneously monitor multiple towers on different devices
|
|
||||||
- Parallel scanning across multiple frequency bands
|
|
||||||
|
|
||||||
## Compatibility
|
|
||||||
|
|
||||||
- **Frontend**: Modern browsers with ES6+ support (async/await)
|
|
||||||
- **Backend**: Python 3.8+
|
|
||||||
- **SDR Hardware**: RTL-SDR, HackRF, LimeSDR, Airspy, SDRPlay
|
|
||||||
- **gr-gsm**: Requires gr-gsm toolkit installed
|
|
||||||
|
|
||||||
## Files Modified
|
|
||||||
|
|
||||||
1. `/opt/intercept/templates/gsm_spy_dashboard.html` - Frontend UI and JavaScript
|
|
||||||
2. `/opt/intercept/routes/gsm_spy.py` - Backend route handlers
|
|
||||||
|
|
||||||
## Files Referenced (Not Modified)
|
|
||||||
|
|
||||||
1. `/opt/intercept/routes/adsb.py` - Reference implementation
|
|
||||||
2. `/opt/intercept/utils/sdr/detection.py` - Device detection
|
|
||||||
3. `/opt/intercept/utils/sdr/__init__.py` - SDRFactory
|
|
||||||
4. `/opt/intercept/utils/validation.py` - Input validation
|
|
||||||
5. `/opt/intercept/app.py` - Device registry functions
|
|
||||||
|
|
||||||
## Verification
|
|
||||||
|
|
||||||
All changes have been implemented according to the plan. The implementation:
|
|
||||||
- ✅ Follows existing INTERCEPT patterns
|
|
||||||
- ✅ Maintains UI consistency across modes
|
|
||||||
- ✅ Includes proper error handling
|
|
||||||
- ✅ Uses centralized validation
|
|
||||||
- ✅ Integrates with device registry
|
|
||||||
- ✅ Provides clear user feedback
|
|
||||||
|
|
||||||
## Implementation Date
|
|
||||||
|
|
||||||
2026-02-06
|
|
||||||
@@ -1,289 +0,0 @@
|
|||||||
# GSM Spy Zombie Process Fix
|
|
||||||
|
|
||||||
## Issue Description
|
|
||||||
|
|
||||||
When starting GSM Spy, `grgsm_scanner` and `grgsm_livemon` processes were becoming zombies (defunct processes):
|
|
||||||
|
|
||||||
```
|
|
||||||
root 12488 5.1 0.0 0 0 pts/2 Z+ 14:29 0:01 [grgsm_scanner] <defunct>
|
|
||||||
```
|
|
||||||
|
|
||||||
## Root Cause
|
|
||||||
|
|
||||||
**Zombie processes** occur when a child process terminates but the parent process doesn't call `wait()` or `waitpid()` to collect the exit status. The process remains in the process table as a zombie until the parent reaps it.
|
|
||||||
|
|
||||||
In the GSM Spy implementation, there were three issues:
|
|
||||||
|
|
||||||
### Issue 1: scanner_thread not reaping grgsm_scanner process
|
|
||||||
- The `scanner_thread` function reads from `grgsm_scanner` stdout
|
|
||||||
- When the process terminates (either normally or due to error), the thread exits
|
|
||||||
- But it never calls `process.wait()` to reap the child process
|
|
||||||
- Result: zombie `grgsm_scanner` process
|
|
||||||
|
|
||||||
### Issue 2: monitor_thread not reaping tshark process
|
|
||||||
- The `monitor_thread` function reads from `tshark` stdout
|
|
||||||
- Same problem as Issue 1
|
|
||||||
- Result: zombie `tshark` process
|
|
||||||
|
|
||||||
### Issue 3: grgsm_livemon process not tracked at all
|
|
||||||
- When starting monitoring, two processes are created:
|
|
||||||
1. `grgsm_livemon` - captures GSM traffic and feeds it to tshark
|
|
||||||
2. `tshark` - filters and parses GSM data
|
|
||||||
- Only `tshark` was being tracked in `gsm_spy_monitor_process`
|
|
||||||
- `grgsm_livemon` was started but never stored or cleaned up
|
|
||||||
- Result: zombie `grgsm_livemon` process
|
|
||||||
|
|
||||||
## Solution
|
|
||||||
|
|
||||||
### Fix 1: Reap processes in scanner_thread
|
|
||||||
|
|
||||||
**File**: `/opt/intercept/routes/gsm_spy.py`
|
|
||||||
**Function**: `scanner_thread()` (line ~1026)
|
|
||||||
|
|
||||||
**Changes**:
|
|
||||||
```python
|
|
||||||
finally:
|
|
||||||
# Reap the process to prevent zombie
|
|
||||||
try:
|
|
||||||
if process.poll() is None:
|
|
||||||
# Process still running, terminate it
|
|
||||||
process.terminate()
|
|
||||||
process.wait(timeout=5)
|
|
||||||
else:
|
|
||||||
# Process already terminated, just collect exit status
|
|
||||||
process.wait()
|
|
||||||
logger.info(f"Scanner process terminated with exit code {process.returncode}")
|
|
||||||
except Exception as e:
|
|
||||||
logger.error(f"Error cleaning up scanner process: {e}")
|
|
||||||
try:
|
|
||||||
process.kill()
|
|
||||||
process.wait()
|
|
||||||
except Exception:
|
|
||||||
pass
|
|
||||||
logger.info("Scanner thread terminated")
|
|
||||||
```
|
|
||||||
|
|
||||||
**How it works**:
|
|
||||||
1. Check if process is still running with `poll()`
|
|
||||||
2. If running, terminate gracefully with `terminate()` then `wait()`
|
|
||||||
3. If already terminated, just call `wait()` to collect exit status
|
|
||||||
4. If anything fails, try `kill()` then `wait()`
|
|
||||||
5. This ensures the child process is always reaped
|
|
||||||
|
|
||||||
### Fix 2: Reap processes in monitor_thread
|
|
||||||
|
|
||||||
**File**: `/opt/intercept/routes/gsm_spy.py`
|
|
||||||
**Function**: `monitor_thread()` (line ~1089)
|
|
||||||
|
|
||||||
**Changes**: Same cleanup logic as Fix 1, applied to the monitor thread.
|
|
||||||
|
|
||||||
### Fix 3: Track and clean up grgsm_livemon process
|
|
||||||
|
|
||||||
#### 3a. Add global variable for grgsm_livemon
|
|
||||||
|
|
||||||
**File**: `/opt/intercept/app.py` (line ~185)
|
|
||||||
|
|
||||||
**Changes**:
|
|
||||||
```python
|
|
||||||
# GSM Spy
|
|
||||||
gsm_spy_process = None
|
|
||||||
gsm_spy_livemon_process = None # For grgsm_livemon process
|
|
||||||
gsm_spy_monitor_process = None # For tshark monitoring process
|
|
||||||
```
|
|
||||||
|
|
||||||
#### 3b. Update global declarations
|
|
||||||
|
|
||||||
**File**: `/opt/intercept/app.py` (line ~677)
|
|
||||||
|
|
||||||
**Changes**:
|
|
||||||
```python
|
|
||||||
global gsm_spy_process, gsm_spy_livemon_process, gsm_spy_monitor_process
|
|
||||||
```
|
|
||||||
|
|
||||||
#### 3c. Clean up grgsm_livemon in reset function
|
|
||||||
|
|
||||||
**File**: `/opt/intercept/app.py` (line ~755)
|
|
||||||
|
|
||||||
**Changes**:
|
|
||||||
```python
|
|
||||||
if gsm_spy_livemon_process:
|
|
||||||
try:
|
|
||||||
safe_terminate(gsm_spy_livemon_process, 'grgsm_livemon')
|
|
||||||
killed.append('grgsm_livemon')
|
|
||||||
except Exception:
|
|
||||||
pass
|
|
||||||
gsm_spy_livemon_process = None
|
|
||||||
```
|
|
||||||
|
|
||||||
#### 3d. Store grgsm_livemon process when starting
|
|
||||||
|
|
||||||
**File**: `/opt/intercept/routes/gsm_spy.py`
|
|
||||||
|
|
||||||
**Changes in `/monitor` endpoint** (line ~212):
|
|
||||||
```python
|
|
||||||
app_module.gsm_spy_livemon_process = grgsm_proc
|
|
||||||
app_module.gsm_spy_monitor_process = tshark_proc
|
|
||||||
```
|
|
||||||
|
|
||||||
**Changes in `auto_start_monitor()` function** (line ~997):
|
|
||||||
```python
|
|
||||||
app_module.gsm_spy_livemon_process = grgsm_proc
|
|
||||||
app_module.gsm_spy_monitor_process = tshark_proc
|
|
||||||
```
|
|
||||||
|
|
||||||
#### 3e. Stop grgsm_livemon when stopping scanner
|
|
||||||
|
|
||||||
**File**: `/opt/intercept/routes/gsm_spy.py` (line ~254)
|
|
||||||
|
|
||||||
**Changes**:
|
|
||||||
```python
|
|
||||||
if app_module.gsm_spy_livemon_process:
|
|
||||||
try:
|
|
||||||
app_module.gsm_spy_livemon_process.terminate()
|
|
||||||
app_module.gsm_spy_livemon_process.wait(timeout=5)
|
|
||||||
killed.append('livemon')
|
|
||||||
except Exception:
|
|
||||||
try:
|
|
||||||
app_module.gsm_spy_livemon_process.kill()
|
|
||||||
except Exception:
|
|
||||||
pass
|
|
||||||
app_module.gsm_spy_livemon_process = None
|
|
||||||
```
|
|
||||||
|
|
||||||
## Files Modified
|
|
||||||
|
|
||||||
1. `/opt/intercept/routes/gsm_spy.py`
|
|
||||||
- `scanner_thread()` - Added process reaping in finally block
|
|
||||||
- `monitor_thread()` - Added process reaping in finally block
|
|
||||||
- `/monitor` endpoint - Store grgsm_livemon process
|
|
||||||
- `auto_start_monitor()` - Store grgsm_livemon process
|
|
||||||
- `/stop` endpoint - Clean up grgsm_livemon process
|
|
||||||
|
|
||||||
2. `/opt/intercept/app.py`
|
|
||||||
- Added `gsm_spy_livemon_process` global variable
|
|
||||||
- Updated global declarations in `reset_decoder_processes()`
|
|
||||||
- Added cleanup for `gsm_spy_livemon_process`
|
|
||||||
|
|
||||||
## Testing
|
|
||||||
|
|
||||||
### Before Fix
|
|
||||||
```bash
|
|
||||||
# Start GSM Spy
|
|
||||||
# Check processes
|
|
||||||
ps aux | grep grgsm
|
|
||||||
|
|
||||||
# You would see:
|
|
||||||
root 12488 0.0 0.0 0 0 pts/2 Z+ 14:29 0:00 [grgsm_scanner] <defunct>
|
|
||||||
root 12489 0.0 0.0 0 0 pts/2 Z+ 14:29 0:00 [grgsm_livemon] <defunct>
|
|
||||||
```
|
|
||||||
|
|
||||||
### After Fix
|
|
||||||
```bash
|
|
||||||
# Start GSM Spy
|
|
||||||
# Check processes
|
|
||||||
ps aux | grep grgsm
|
|
||||||
|
|
||||||
# Active processes (no zombies):
|
|
||||||
root 12488 1.2 0.5 12345 5678 pts/2 S+ 14:29 0:01 grgsm_scanner -d 0 --freq-range...
|
|
||||||
root 12489 0.8 0.4 10234 4567 pts/2 S+ 14:29 0:01 grgsm_livemon -a 123 -d 0
|
|
||||||
|
|
||||||
# Stop GSM Spy
|
|
||||||
# Check processes
|
|
||||||
ps aux | grep grgsm
|
|
||||||
|
|
||||||
# No processes (all cleaned up properly)
|
|
||||||
```
|
|
||||||
|
|
||||||
### Verification Commands
|
|
||||||
|
|
||||||
1. **Check for zombie processes**:
|
|
||||||
```bash
|
|
||||||
ps aux | grep defunct
|
|
||||||
# Should return nothing after fix
|
|
||||||
```
|
|
||||||
|
|
||||||
2. **Monitor process lifecycle**:
|
|
||||||
```bash
|
|
||||||
# In one terminal, watch processes
|
|
||||||
watch -n 1 'ps aux | grep grgsm'
|
|
||||||
|
|
||||||
# In another terminal, start/stop GSM Spy
|
|
||||||
# Verify:
|
|
||||||
# - Processes start properly (S or R state, not Z)
|
|
||||||
# - Processes disappear when stopped (not left as zombies)
|
|
||||||
```
|
|
||||||
|
|
||||||
3. **Check process tree**:
|
|
||||||
```bash
|
|
||||||
pstree -p | grep grgsm
|
|
||||||
# Should show proper parent-child relationships
|
|
||||||
# No defunct/zombie entries
|
|
||||||
```
|
|
||||||
|
|
||||||
## Process Lifecycle
|
|
||||||
|
|
||||||
### Normal Operation
|
|
||||||
|
|
||||||
1. **Scanner Start**:
|
|
||||||
- `grgsm_scanner` spawned → stored in `gsm_spy_process`
|
|
||||||
- `scanner_thread` reads output
|
|
||||||
- Process running normally
|
|
||||||
|
|
||||||
2. **Monitor Start** (auto or manual):
|
|
||||||
- `grgsm_livemon` spawned → stored in `gsm_spy_livemon_process`
|
|
||||||
- `tshark` spawned → stored in `gsm_spy_monitor_process`
|
|
||||||
- `monitor_thread` reads tshark output
|
|
||||||
- Both processes running normally
|
|
||||||
|
|
||||||
3. **Stop**:
|
|
||||||
- All three processes terminated gracefully
|
|
||||||
- `wait()` called on each to collect exit status
|
|
||||||
- Process handles set to None
|
|
||||||
- No zombies remain
|
|
||||||
|
|
||||||
### Error Handling
|
|
||||||
|
|
||||||
1. **Process crashes during operation**:
|
|
||||||
- Thread's stdout loop exits
|
|
||||||
- `finally` block executes
|
|
||||||
- `process.wait()` collects exit status
|
|
||||||
- No zombie created
|
|
||||||
|
|
||||||
2. **Process hangs**:
|
|
||||||
- `terminate()` called
|
|
||||||
- `wait(timeout=5)` gives 5 seconds to exit
|
|
||||||
- If timeout, `kill()` is called
|
|
||||||
- `wait()` collects exit status
|
|
||||||
|
|
||||||
3. **Exception during cleanup**:
|
|
||||||
- Fallback to `kill()` + `wait()`
|
|
||||||
- Ensures zombie is always prevented
|
|
||||||
|
|
||||||
## Best Practices Applied
|
|
||||||
|
|
||||||
1. **Always reap child processes**: Call `wait()` or `waitpid()` after child process terminates
|
|
||||||
2. **Use process.poll()**: Check if process is still running before terminating
|
|
||||||
3. **Graceful shutdown**: Try `terminate()` before `kill()`
|
|
||||||
4. **Timeout handling**: Use `wait(timeout=N)` to prevent hanging
|
|
||||||
5. **Error recovery**: Multiple fallback levels in try/except blocks
|
|
||||||
6. **Track all processes**: Store handles for all spawned processes, not just the primary one
|
|
||||||
7. **Cleanup in finally**: Ensures cleanup happens even if exceptions occur
|
|
||||||
|
|
||||||
## Related Issues
|
|
||||||
|
|
||||||
This fix prevents:
|
|
||||||
- Zombie processes accumulating over time
|
|
||||||
- Process table filling up
|
|
||||||
- System resource leaks
|
|
||||||
- Confusing process listings for users
|
|
||||||
|
|
||||||
## Implementation Date
|
|
||||||
|
|
||||||
2026-02-06
|
|
||||||
|
|
||||||
## Additional Notes
|
|
||||||
|
|
||||||
- The fix follows the same patterns used in other INTERCEPT decoders
|
|
||||||
- Compatible with existing SDR device selection implementation
|
|
||||||
- No breaking changes to API or user interface
|
|
||||||
- Applies to both manual monitoring and auto-monitoring
|
|
||||||
@@ -6,6 +6,7 @@ import json
|
|||||||
import logging
|
import logging
|
||||||
import queue
|
import queue
|
||||||
import re
|
import re
|
||||||
|
import select
|
||||||
import subprocess
|
import subprocess
|
||||||
import threading
|
import threading
|
||||||
import time
|
import time
|
||||||
@@ -33,7 +34,8 @@ REGIONAL_BANDS = {
|
|||||||
'PCS1900': {'start': 1930e6, 'end': 1990e6, 'arfcn_start': 512, 'arfcn_end': 810}
|
'PCS1900': {'start': 1930e6, 'end': 1990e6, 'arfcn_start': 512, 'arfcn_end': 810}
|
||||||
},
|
},
|
||||||
'Europe': {
|
'Europe': {
|
||||||
'EGSM900': {'start': 925e6, 'end': 960e6, 'arfcn_start': 0, 'arfcn_end': 124}
|
'EGSM900': {'start': 925e6, 'end': 960e6, 'arfcn_start': 0, 'arfcn_end': 124},
|
||||||
|
'DCS1800': {'start': 1805e6, 'end': 1880e6, 'arfcn_start': 512, 'arfcn_end': 885}
|
||||||
},
|
},
|
||||||
'Asia': {
|
'Asia': {
|
||||||
'EGSM900': {'start': 925e6, 'end': 960e6, 'arfcn_start': 0, 'arfcn_end': 124},
|
'EGSM900': {'start': 925e6, 'end': 960e6, 'arfcn_start': 0, 'arfcn_end': 124},
|
||||||
@@ -47,6 +49,9 @@ gsm_connected = False
|
|||||||
gsm_towers_found = 0
|
gsm_towers_found = 0
|
||||||
gsm_devices_tracked = 0
|
gsm_devices_tracked = 0
|
||||||
|
|
||||||
|
# Geocoding worker state
|
||||||
|
_geocoding_worker_thread = None
|
||||||
|
|
||||||
|
|
||||||
# ============================================
|
# ============================================
|
||||||
# API Usage Tracking Helper Functions
|
# API Usage Tracking Helper Functions
|
||||||
@@ -82,6 +87,100 @@ def can_use_api():
|
|||||||
return current_usage < config.GSM_API_DAILY_LIMIT
|
return current_usage < config.GSM_API_DAILY_LIMIT
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================
|
||||||
|
# Background Geocoding Worker
|
||||||
|
# ============================================
|
||||||
|
|
||||||
|
def start_geocoding_worker():
|
||||||
|
"""Start background thread for async geocoding."""
|
||||||
|
global _geocoding_worker_thread
|
||||||
|
if _geocoding_worker_thread is None or not _geocoding_worker_thread.is_alive():
|
||||||
|
_geocoding_worker_thread = threading.Thread(
|
||||||
|
target=geocoding_worker,
|
||||||
|
daemon=True,
|
||||||
|
name='gsm-geocoding-worker'
|
||||||
|
)
|
||||||
|
_geocoding_worker_thread.start()
|
||||||
|
logger.info("Started geocoding worker thread")
|
||||||
|
|
||||||
|
|
||||||
|
def geocoding_worker():
|
||||||
|
"""Worker thread processes pending geocoding requests."""
|
||||||
|
from utils.gsm_geocoding import lookup_cell_from_api, get_geocoding_queue
|
||||||
|
|
||||||
|
geocoding_queue = get_geocoding_queue()
|
||||||
|
|
||||||
|
while True:
|
||||||
|
try:
|
||||||
|
# Wait for pending tower with timeout
|
||||||
|
tower_data = geocoding_queue.get(timeout=5)
|
||||||
|
|
||||||
|
# Check rate limit
|
||||||
|
if not can_use_api():
|
||||||
|
current_usage = get_api_usage_today()
|
||||||
|
logger.warning(f"OpenCellID API rate limit reached ({current_usage}/{config.GSM_API_DAILY_LIMIT})")
|
||||||
|
geocoding_queue.task_done()
|
||||||
|
continue
|
||||||
|
|
||||||
|
# Call API
|
||||||
|
mcc = tower_data.get('mcc')
|
||||||
|
mnc = tower_data.get('mnc')
|
||||||
|
lac = tower_data.get('lac')
|
||||||
|
cid = tower_data.get('cid')
|
||||||
|
|
||||||
|
logger.debug(f"Geocoding tower via API: MCC={mcc} MNC={mnc} LAC={lac} CID={cid}")
|
||||||
|
|
||||||
|
coords = lookup_cell_from_api(mcc, mnc, lac, cid)
|
||||||
|
|
||||||
|
if coords:
|
||||||
|
# Update tower data with coordinates
|
||||||
|
tower_data['lat'] = coords['lat']
|
||||||
|
tower_data['lon'] = coords['lon']
|
||||||
|
tower_data['source'] = 'api'
|
||||||
|
tower_data['status'] = 'resolved'
|
||||||
|
tower_data['type'] = 'tower_update'
|
||||||
|
|
||||||
|
# Add optional fields if available
|
||||||
|
if coords.get('azimuth') is not None:
|
||||||
|
tower_data['azimuth'] = coords['azimuth']
|
||||||
|
if coords.get('range_meters') is not None:
|
||||||
|
tower_data['range_meters'] = coords['range_meters']
|
||||||
|
if coords.get('operator'):
|
||||||
|
tower_data['operator'] = coords['operator']
|
||||||
|
if coords.get('radio'):
|
||||||
|
tower_data['radio'] = coords['radio']
|
||||||
|
|
||||||
|
# Update DataStore
|
||||||
|
key = f"{mcc}_{mnc}_{lac}_{cid}"
|
||||||
|
app_module.gsm_spy_towers[key] = tower_data
|
||||||
|
|
||||||
|
# Send update to SSE stream
|
||||||
|
try:
|
||||||
|
app_module.gsm_spy_queue.put_nowait(tower_data)
|
||||||
|
logger.info(f"Resolved coordinates for tower: MCC={mcc} MNC={mnc} LAC={lac} CID={cid}")
|
||||||
|
except queue.Full:
|
||||||
|
logger.warning("SSE queue full, dropping tower update")
|
||||||
|
|
||||||
|
# Increment API usage counter
|
||||||
|
usage_count = increment_api_usage()
|
||||||
|
logger.info(f"OpenCellID API call #{usage_count} today")
|
||||||
|
|
||||||
|
else:
|
||||||
|
logger.warning(f"Could not resolve coordinates for tower: MCC={mcc} MNC={mnc} LAC={lac} CID={cid}")
|
||||||
|
|
||||||
|
geocoding_queue.task_done()
|
||||||
|
|
||||||
|
# Rate limiting between API calls (be nice to OpenCellID)
|
||||||
|
time.sleep(1)
|
||||||
|
|
||||||
|
except queue.Empty:
|
||||||
|
# No pending towers, continue waiting
|
||||||
|
continue
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Geocoding worker error: {e}", exc_info=True)
|
||||||
|
time.sleep(1)
|
||||||
|
|
||||||
|
|
||||||
def arfcn_to_frequency(arfcn):
|
def arfcn_to_frequency(arfcn):
|
||||||
"""Convert ARFCN to downlink frequency in Hz.
|
"""Convert ARFCN to downlink frequency in Hz.
|
||||||
|
|
||||||
@@ -163,22 +262,18 @@ def start_scanner():
|
|||||||
|
|
||||||
logger.info(f"Starting GSM scanner: {' '.join(cmd)}")
|
logger.info(f"Starting GSM scanner: {' '.join(cmd)}")
|
||||||
|
|
||||||
process = subprocess.Popen(
|
# Set a flag to indicate scanner should run
|
||||||
cmd,
|
|
||||||
stdout=subprocess.PIPE,
|
|
||||||
stderr=subprocess.PIPE,
|
|
||||||
universal_newlines=True,
|
|
||||||
bufsize=1
|
|
||||||
)
|
|
||||||
|
|
||||||
app_module.gsm_spy_process = process
|
|
||||||
app_module.gsm_spy_active_device = device_index
|
app_module.gsm_spy_active_device = device_index
|
||||||
app_module.gsm_spy_region = region
|
app_module.gsm_spy_region = region
|
||||||
|
app_module.gsm_spy_process = True # Use as flag initially
|
||||||
|
|
||||||
# Start output parsing thread
|
# Start geocoding worker (if not already running)
|
||||||
|
start_geocoding_worker()
|
||||||
|
|
||||||
|
# Start scanning thread (will run grgsm_scanner in a loop)
|
||||||
scanner_thread_obj = threading.Thread(
|
scanner_thread_obj = threading.Thread(
|
||||||
target=scanner_thread,
|
target=scanner_thread,
|
||||||
args=(process,),
|
args=(cmd, device_index),
|
||||||
daemon=True
|
daemon=True
|
||||||
)
|
)
|
||||||
scanner_thread_obj.start()
|
scanner_thread_obj.start()
|
||||||
@@ -242,14 +337,18 @@ def start_monitor():
|
|||||||
|
|
||||||
logger.info(f"Starting GSM monitor: {' '.join(grgsm_cmd)} | {' '.join(tshark_cmd)}")
|
logger.info(f"Starting GSM monitor: {' '.join(grgsm_cmd)} | {' '.join(tshark_cmd)}")
|
||||||
|
|
||||||
# Start grgsm_livemon
|
# Start grgsm_livemon (outputs to UDP port 4729 by default)
|
||||||
grgsm_proc = subprocess.Popen(
|
grgsm_proc = subprocess.Popen(
|
||||||
grgsm_cmd,
|
grgsm_cmd,
|
||||||
stdout=subprocess.PIPE,
|
stdout=subprocess.PIPE,
|
||||||
stderr=subprocess.PIPE
|
stderr=subprocess.PIPE
|
||||||
)
|
)
|
||||||
|
logger.info(f"Started grgsm_livemon (PID: {grgsm_proc.pid})")
|
||||||
|
|
||||||
# Start tshark
|
# Give grgsm_livemon time to initialize and start sending UDP packets
|
||||||
|
time.sleep(2)
|
||||||
|
|
||||||
|
# Start tshark (captures from loopback interface where UDP packets arrive)
|
||||||
tshark_proc = subprocess.Popen(
|
tshark_proc = subprocess.Popen(
|
||||||
tshark_cmd,
|
tshark_cmd,
|
||||||
stdout=subprocess.PIPE,
|
stdout=subprocess.PIPE,
|
||||||
@@ -257,6 +356,7 @@ def start_monitor():
|
|||||||
universal_newlines=True,
|
universal_newlines=True,
|
||||||
bufsize=1
|
bufsize=1
|
||||||
)
|
)
|
||||||
|
logger.info(f"Started tshark (PID: {tshark_proc.pid})")
|
||||||
|
|
||||||
app_module.gsm_spy_livemon_process = grgsm_proc
|
app_module.gsm_spy_livemon_process = grgsm_proc
|
||||||
app_module.gsm_spy_monitor_process = tshark_proc
|
app_module.gsm_spy_monitor_process = tshark_proc
|
||||||
@@ -291,17 +391,10 @@ def stop_scanner():
|
|||||||
with app_module.gsm_spy_lock:
|
with app_module.gsm_spy_lock:
|
||||||
killed = []
|
killed = []
|
||||||
|
|
||||||
|
# Stop scanner (now just a flag, thread will see it and exit)
|
||||||
if app_module.gsm_spy_process:
|
if app_module.gsm_spy_process:
|
||||||
try:
|
|
||||||
app_module.gsm_spy_process.terminate()
|
|
||||||
app_module.gsm_spy_process.wait(timeout=5)
|
|
||||||
killed.append('scanner')
|
|
||||||
except Exception:
|
|
||||||
try:
|
|
||||||
app_module.gsm_spy_process.kill()
|
|
||||||
except Exception:
|
|
||||||
pass
|
|
||||||
app_module.gsm_spy_process = None
|
app_module.gsm_spy_process = None
|
||||||
|
killed.append('scanner')
|
||||||
|
|
||||||
if app_module.gsm_spy_livemon_process:
|
if app_module.gsm_spy_livemon_process:
|
||||||
try:
|
try:
|
||||||
@@ -917,33 +1010,45 @@ def traffic_correlation():
|
|||||||
# ============================================
|
# ============================================
|
||||||
|
|
||||||
def parse_grgsm_scanner_output(line: str) -> dict[str, Any] | None:
|
def parse_grgsm_scanner_output(line: str) -> dict[str, Any] | None:
|
||||||
"""Parse grgsm_scanner output line."""
|
"""Parse grgsm_scanner output line.
|
||||||
|
|
||||||
|
Actual output format is a table:
|
||||||
|
ARFCN | Freq (MHz) | CID | LAC | MCC | MNC | Power (dB)
|
||||||
|
--------------------------------------------------------------------
|
||||||
|
23 | 940.6 | 31245 | 1234 | 214 | 01 | -48
|
||||||
|
"""
|
||||||
try:
|
try:
|
||||||
# Example output: "ARFCN: 123, Freq: 935.2MHz, CID: 1234, LAC: 567, MCC: 310, MNC: 260, PWR: -85dBm"
|
# Skip progress, header, and separator lines
|
||||||
# This is a placeholder - actual format depends on grgsm_scanner output
|
if 'Scanning:' in line or 'ARFCN' in line or '---' in line or 'Found' in line:
|
||||||
|
return None
|
||||||
|
|
||||||
# Simple regex patterns
|
# Parse table row: " 23 | 940.6 | 31245 | 1234 | 214 | 01 | -48"
|
||||||
arfcn_match = re.search(r'ARFCN[:\s]+(\d+)', line)
|
# Split by pipe and clean whitespace
|
||||||
freq_match = re.search(r'Freq[:\s]+([\d.]+)', line)
|
parts = [p.strip() for p in line.split('|')]
|
||||||
cid_match = re.search(r'CID[:\s]+(\d+)', line)
|
|
||||||
lac_match = re.search(r'LAC[:\s]+(\d+)', line)
|
|
||||||
mcc_match = re.search(r'MCC[:\s]+(\d+)', line)
|
|
||||||
mnc_match = re.search(r'MNC[:\s]+(\d+)', line)
|
|
||||||
pwr_match = re.search(r'PWR[:\s]+([-\d.]+)', line)
|
|
||||||
|
|
||||||
if arfcn_match:
|
if len(parts) >= 7:
|
||||||
data = {
|
arfcn = parts[0]
|
||||||
'type': 'tower',
|
freq = parts[1]
|
||||||
'arfcn': int(arfcn_match.group(1)),
|
cid = parts[2]
|
||||||
'frequency': float(freq_match.group(1)) if freq_match else None,
|
lac = parts[3]
|
||||||
'cid': int(cid_match.group(1)) if cid_match else None,
|
mcc = parts[4]
|
||||||
'lac': int(lac_match.group(1)) if lac_match else None,
|
mnc = parts[5]
|
||||||
'mcc': int(mcc_match.group(1)) if mcc_match else None,
|
power = parts[6]
|
||||||
'mnc': int(mnc_match.group(1)) if mnc_match else None,
|
|
||||||
'signal_strength': float(pwr_match.group(1)) if pwr_match else None,
|
# Validate that we have numeric data (not header line)
|
||||||
'timestamp': datetime.now().isoformat()
|
if arfcn.isdigit():
|
||||||
}
|
data = {
|
||||||
return data
|
'type': 'tower',
|
||||||
|
'arfcn': int(arfcn),
|
||||||
|
'frequency': float(freq),
|
||||||
|
'cid': int(cid),
|
||||||
|
'lac': int(lac),
|
||||||
|
'mcc': int(mcc),
|
||||||
|
'mnc': int(mnc),
|
||||||
|
'signal_strength': float(power),
|
||||||
|
'timestamp': datetime.now().isoformat()
|
||||||
|
}
|
||||||
|
return data
|
||||||
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.debug(f"Failed to parse scanner line: {line} - {e}")
|
logger.debug(f"Failed to parse scanner line: {line} - {e}")
|
||||||
@@ -1025,14 +1130,18 @@ def auto_start_monitor(tower_data):
|
|||||||
|
|
||||||
logger.info(f"Starting auto-monitor: {' '.join(grgsm_cmd)} | {' '.join(tshark_cmd)}")
|
logger.info(f"Starting auto-monitor: {' '.join(grgsm_cmd)} | {' '.join(tshark_cmd)}")
|
||||||
|
|
||||||
# Start grgsm_livemon (we don't capture its output)
|
# Start grgsm_livemon (outputs to UDP port 4729 by default)
|
||||||
grgsm_proc = subprocess.Popen(
|
grgsm_proc = subprocess.Popen(
|
||||||
grgsm_cmd,
|
grgsm_cmd,
|
||||||
stdout=subprocess.PIPE,
|
stdout=subprocess.PIPE,
|
||||||
stderr=subprocess.PIPE
|
stderr=subprocess.PIPE
|
||||||
)
|
)
|
||||||
|
logger.info(f"Started grgsm_livemon for auto-monitor (PID: {grgsm_proc.pid})")
|
||||||
|
|
||||||
# Start tshark
|
# Give grgsm_livemon time to initialize and start sending UDP packets
|
||||||
|
time.sleep(2)
|
||||||
|
|
||||||
|
# Start tshark (captures from loopback interface where UDP packets arrive)
|
||||||
tshark_proc = subprocess.Popen(
|
tshark_proc = subprocess.Popen(
|
||||||
tshark_cmd,
|
tshark_cmd,
|
||||||
stdout=subprocess.PIPE,
|
stdout=subprocess.PIPE,
|
||||||
@@ -1040,6 +1149,7 @@ def auto_start_monitor(tower_data):
|
|||||||
universal_newlines=True,
|
universal_newlines=True,
|
||||||
bufsize=1
|
bufsize=1
|
||||||
)
|
)
|
||||||
|
logger.info(f"Started tshark for auto-monitor (PID: {tshark_proc.pid})")
|
||||||
|
|
||||||
app_module.gsm_spy_livemon_process = grgsm_proc
|
app_module.gsm_spy_livemon_process = grgsm_proc
|
||||||
app_module.gsm_spy_monitor_process = tshark_proc
|
app_module.gsm_spy_monitor_process = tshark_proc
|
||||||
@@ -1069,66 +1179,192 @@ def auto_start_monitor(tower_data):
|
|||||||
logger.error(f"Error in auto-monitoring: {e}")
|
logger.error(f"Error in auto-monitoring: {e}")
|
||||||
|
|
||||||
|
|
||||||
def scanner_thread(process):
|
def scanner_thread(cmd, device_index):
|
||||||
"""Thread to read grgsm_scanner output."""
|
"""Thread to continuously run grgsm_scanner in a loop with non-blocking I/O.
|
||||||
|
|
||||||
|
grgsm_scanner scans once and exits, so we loop it to provide
|
||||||
|
continuous updates to the dashboard.
|
||||||
|
"""
|
||||||
global gsm_towers_found
|
global gsm_towers_found
|
||||||
|
|
||||||
strongest_tower = None
|
strongest_tower = None
|
||||||
auto_monitor_triggered = False
|
auto_monitor_triggered = False # Moved outside loop - persists across scans
|
||||||
|
scan_count = 0
|
||||||
|
process = None
|
||||||
|
|
||||||
try:
|
try:
|
||||||
for line in process.stdout:
|
while app_module.gsm_spy_process: # Flag check
|
||||||
if not line:
|
scan_count += 1
|
||||||
continue
|
logger.info(f"Starting GSM scan #{scan_count}")
|
||||||
|
|
||||||
parsed = parse_grgsm_scanner_output(line)
|
try:
|
||||||
if parsed:
|
# Start scanner process
|
||||||
# Store in DataStore
|
process = subprocess.Popen(
|
||||||
key = f"{parsed.get('mcc')}_{parsed.get('mnc')}_{parsed.get('lac')}_{parsed.get('cid')}"
|
cmd,
|
||||||
app_module.gsm_spy_towers[key] = parsed
|
stdout=subprocess.PIPE,
|
||||||
|
stderr=subprocess.PIPE,
|
||||||
|
universal_newlines=True,
|
||||||
|
bufsize=1
|
||||||
|
)
|
||||||
|
|
||||||
# Track strongest tower for auto-monitoring
|
# Non-blocking stderr reader
|
||||||
signal_strength = parsed.get('signal_strength', -999)
|
def read_stderr():
|
||||||
if strongest_tower is None or signal_strength > strongest_tower.get('signal_strength', -999):
|
try:
|
||||||
strongest_tower = parsed
|
for line in process.stderr:
|
||||||
|
if line:
|
||||||
|
logger.debug(f"grgsm_scanner: {line.strip()}")
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"stderr read error: {e}")
|
||||||
|
|
||||||
# Queue for SSE stream
|
stderr_thread = threading.Thread(target=read_stderr, daemon=True)
|
||||||
try:
|
stderr_thread.start()
|
||||||
app_module.gsm_spy_queue.put_nowait(parsed)
|
|
||||||
except queue.Full:
|
|
||||||
pass
|
|
||||||
|
|
||||||
gsm_towers_found += 1
|
# Non-blocking stdout reader with timeout
|
||||||
|
last_output = time.time()
|
||||||
|
scan_timeout = 120 # 2 minute maximum per scan
|
||||||
|
|
||||||
# Auto-monitor strongest tower after finding 3+ towers
|
while app_module.gsm_spy_process:
|
||||||
if gsm_towers_found >= 3 and not auto_monitor_triggered and strongest_tower:
|
# Check if process died
|
||||||
auto_monitor_triggered = True
|
if process.poll() is not None:
|
||||||
threading.Thread(
|
logger.info(f"Scanner exited (code: {process.returncode})")
|
||||||
target=auto_start_monitor,
|
break
|
||||||
args=(strongest_tower,),
|
|
||||||
daemon=True
|
# Check for output with 1-second timeout
|
||||||
).start()
|
ready, _, _ = select.select([process.stdout], [], [], 1.0)
|
||||||
|
|
||||||
|
if ready:
|
||||||
|
line = process.stdout.readline()
|
||||||
|
if not line:
|
||||||
|
break # EOF
|
||||||
|
|
||||||
|
last_output = time.time()
|
||||||
|
|
||||||
|
parsed = parse_grgsm_scanner_output(line)
|
||||||
|
if parsed:
|
||||||
|
# Enrich with coordinates
|
||||||
|
from utils.gsm_geocoding import enrich_tower_data
|
||||||
|
enriched = enrich_tower_data(parsed)
|
||||||
|
|
||||||
|
# Store in DataStore
|
||||||
|
key = f"{enriched['mcc']}_{enriched['mnc']}_{enriched['lac']}_{enriched['cid']}"
|
||||||
|
app_module.gsm_spy_towers[key] = enriched
|
||||||
|
|
||||||
|
# Track strongest tower
|
||||||
|
signal = enriched.get('signal_strength', -999)
|
||||||
|
if strongest_tower is None or signal > strongest_tower.get('signal_strength', -999):
|
||||||
|
strongest_tower = enriched
|
||||||
|
|
||||||
|
# Queue for SSE
|
||||||
|
try:
|
||||||
|
app_module.gsm_spy_queue.put_nowait(enriched)
|
||||||
|
except queue.Full:
|
||||||
|
logger.warning("Queue full, dropping tower update")
|
||||||
|
|
||||||
|
# Thread-safe counter update
|
||||||
|
with app_module.gsm_spy_lock:
|
||||||
|
gsm_towers_found += 1
|
||||||
|
current_count = gsm_towers_found
|
||||||
|
|
||||||
|
# Auto-monitor strongest tower (once per session)
|
||||||
|
if current_count >= 3 and not auto_monitor_triggered and strongest_tower:
|
||||||
|
auto_monitor_triggered = True
|
||||||
|
logger.info("Auto-starting monitor on strongest tower")
|
||||||
|
threading.Thread(
|
||||||
|
target=auto_start_monitor,
|
||||||
|
args=(strongest_tower,),
|
||||||
|
daemon=True
|
||||||
|
).start()
|
||||||
|
else:
|
||||||
|
# No output, check timeout
|
||||||
|
if time.time() - last_output > scan_timeout:
|
||||||
|
logger.warning(f"Scan timeout after {scan_timeout}s")
|
||||||
|
break
|
||||||
|
|
||||||
|
# Clean up process with timeout
|
||||||
|
if process.poll() is None:
|
||||||
|
logger.info("Terminating scanner process")
|
||||||
|
process.terminate()
|
||||||
|
try:
|
||||||
|
process.wait(timeout=5)
|
||||||
|
except subprocess.TimeoutExpired:
|
||||||
|
logger.warning("Process didn't terminate, killing")
|
||||||
|
process.kill()
|
||||||
|
process.wait()
|
||||||
|
else:
|
||||||
|
process.wait() # Reap zombie
|
||||||
|
|
||||||
|
logger.info(f"Scan #{scan_count} complete")
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Scanner scan error: {e}", exc_info=True)
|
||||||
|
if process and process.poll() is None:
|
||||||
|
try:
|
||||||
|
process.terminate()
|
||||||
|
process.wait(timeout=2)
|
||||||
|
except Exception:
|
||||||
|
try:
|
||||||
|
process.kill()
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
# Check if should continue
|
||||||
|
if not app_module.gsm_spy_process:
|
||||||
|
break
|
||||||
|
|
||||||
|
# Wait between scans with responsive flag checking
|
||||||
|
logger.info("Waiting 5 seconds before next scan")
|
||||||
|
for i in range(5):
|
||||||
|
if not app_module.gsm_spy_process:
|
||||||
|
break
|
||||||
|
time.sleep(1)
|
||||||
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.error(f"Scanner thread error: {e}")
|
logger.error(f"Scanner thread fatal error: {e}", exc_info=True)
|
||||||
|
|
||||||
finally:
|
finally:
|
||||||
# Reap the process to prevent zombie (don't terminate, just wait)
|
# Always cleanup
|
||||||
try:
|
if process and process.poll() is None:
|
||||||
process.wait()
|
try:
|
||||||
logger.info(f"Scanner process exited with code {process.returncode}")
|
process.terminate()
|
||||||
except Exception as e:
|
process.wait(timeout=5)
|
||||||
logger.error(f"Error waiting for scanner process: {e}")
|
except Exception:
|
||||||
|
try:
|
||||||
|
process.kill()
|
||||||
|
process.wait()
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
logger.info("Scanner thread terminated")
|
logger.info("Scanner thread terminated")
|
||||||
|
|
||||||
|
# Reset global state
|
||||||
|
with app_module.gsm_spy_lock:
|
||||||
|
app_module.gsm_spy_process = None
|
||||||
|
if app_module.gsm_spy_active_device is not None:
|
||||||
|
from app import release_sdr_device
|
||||||
|
release_sdr_device(app_module.gsm_spy_active_device)
|
||||||
|
app_module.gsm_spy_active_device = None
|
||||||
|
|
||||||
|
|
||||||
def monitor_thread(process):
|
def monitor_thread(process):
|
||||||
"""Thread to read grgsm_livemon | tshark output."""
|
"""Thread to read tshark output with non-blocking I/O and timeouts."""
|
||||||
global gsm_devices_tracked
|
global gsm_devices_tracked
|
||||||
|
|
||||||
try:
|
try:
|
||||||
for line in process.stdout:
|
while app_module.gsm_spy_monitor_process:
|
||||||
|
# Check if process died
|
||||||
|
if process.poll() is not None:
|
||||||
|
logger.info(f"Monitor process exited (code: {process.returncode})")
|
||||||
|
break
|
||||||
|
|
||||||
|
# Non-blocking read with timeout
|
||||||
|
ready, _, _ = select.select([process.stdout], [], [], 1.0)
|
||||||
|
|
||||||
|
if not ready:
|
||||||
|
continue # Timeout, check flag again
|
||||||
|
|
||||||
|
line = process.stdout.readline()
|
||||||
if not line:
|
if not line:
|
||||||
continue
|
break # EOF
|
||||||
|
|
||||||
parsed = parse_tshark_output(line)
|
parsed = parse_tshark_output(line)
|
||||||
if parsed:
|
if parsed:
|
||||||
@@ -1218,15 +1454,28 @@ def monitor_thread(process):
|
|||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.error(f"Error storing device data: {e}")
|
logger.error(f"Error storing device data: {e}")
|
||||||
|
|
||||||
gsm_devices_tracked += 1
|
# Thread-safe counter
|
||||||
|
with app_module.gsm_spy_lock:
|
||||||
|
gsm_devices_tracked += 1
|
||||||
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.error(f"Monitor thread error: {e}")
|
logger.error(f"Monitor thread error: {e}", exc_info=True)
|
||||||
|
|
||||||
finally:
|
finally:
|
||||||
# Reap the process to prevent zombie (don't terminate, just wait)
|
# Reap process with timeout
|
||||||
try:
|
try:
|
||||||
process.wait()
|
if process.poll() is None:
|
||||||
|
process.terminate()
|
||||||
|
try:
|
||||||
|
process.wait(timeout=5)
|
||||||
|
except subprocess.TimeoutExpired:
|
||||||
|
logger.warning("Monitor process didn't terminate, killing")
|
||||||
|
process.kill()
|
||||||
|
process.wait()
|
||||||
|
else:
|
||||||
|
process.wait()
|
||||||
logger.info(f"Monitor process exited with code {process.returncode}")
|
logger.info(f"Monitor process exited with code {process.returncode}")
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.error(f"Error waiting for monitor process: {e}")
|
logger.error(f"Error reaping monitor process: {e}")
|
||||||
|
|
||||||
logger.info("Monitor thread terminated")
|
logger.info("Monitor thread terminated")
|
||||||
|
|||||||
@@ -1548,6 +1548,10 @@
|
|||||||
|
|
||||||
if (data.type === 'tower') {
|
if (data.type === 'tower') {
|
||||||
updateTower(data);
|
updateTower(data);
|
||||||
|
} else if (data.type === 'tower_update') {
|
||||||
|
// Background geocoding resolved coordinates for a tower
|
||||||
|
console.log(`Tower coordinates resolved via API: MCC=${data.mcc} MNC=${data.mnc} LAC=${data.lac} CID=${data.cid}`);
|
||||||
|
updateTower(data);
|
||||||
} else if (data.type === 'device') {
|
} else if (data.type === 'device') {
|
||||||
updateDevice(data);
|
updateDevice(data);
|
||||||
} else if (data.type === 'rogue_alert') {
|
} else if (data.type === 'rogue_alert') {
|
||||||
@@ -1576,6 +1580,14 @@
|
|||||||
const key = `${data.mcc}-${data.mnc}-${data.lac}-${data.cid}`;
|
const key = `${data.mcc}-${data.mnc}-${data.lac}-${data.cid}`;
|
||||||
towers[key] = data;
|
towers[key] = data;
|
||||||
|
|
||||||
|
// Validate coordinates before creating map marker
|
||||||
|
if (!data.lat || !data.lon || isNaN(parseFloat(data.lat)) || isNaN(parseFloat(data.lon))) {
|
||||||
|
console.log(`Tower ${data.cid} pending geocoding (status: ${data.status || 'unknown'})`);
|
||||||
|
// Update towers list but skip map marker
|
||||||
|
updateTowersList();
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
// Create or update marker
|
// Create or update marker
|
||||||
if (!towerMarkers[key]) {
|
if (!towerMarkers[key]) {
|
||||||
// Create new marker
|
// Create new marker
|
||||||
|
|||||||
261
test_gsm_spy_fixes.sh
Executable file
261
test_gsm_spy_fixes.sh
Executable file
@@ -0,0 +1,261 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# GSM Spy System - Verification Test Script
|
||||||
|
# Tests the 4 critical fixes: geocoding, pipeline, scanner loop, process management
|
||||||
|
|
||||||
|
set -e
|
||||||
|
|
||||||
|
echo "=========================================="
|
||||||
|
echo "GSM Spy System - Verification Tests"
|
||||||
|
echo "=========================================="
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Colors
|
||||||
|
GREEN='\033[0;32m'
|
||||||
|
RED='\033[0;31m'
|
||||||
|
YELLOW='\033[1;33m'
|
||||||
|
NC='\033[0m' # No Color
|
||||||
|
|
||||||
|
# Test results
|
||||||
|
TESTS_PASSED=0
|
||||||
|
TESTS_FAILED=0
|
||||||
|
|
||||||
|
function pass_test() {
|
||||||
|
echo -e "${GREEN}✓ PASS:${NC} $1"
|
||||||
|
((TESTS_PASSED++))
|
||||||
|
}
|
||||||
|
|
||||||
|
function fail_test() {
|
||||||
|
echo -e "${RED}✗ FAIL:${NC} $1"
|
||||||
|
((TESTS_FAILED++))
|
||||||
|
}
|
||||||
|
|
||||||
|
function info() {
|
||||||
|
echo -e "${YELLOW}ℹ INFO:${NC} $1"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Test 1: Check that geocoding module exists
|
||||||
|
echo "Test 1: Geocoding Module"
|
||||||
|
echo "-------------------------"
|
||||||
|
if [ -f "utils/gsm_geocoding.py" ]; then
|
||||||
|
pass_test "Geocoding module exists"
|
||||||
|
|
||||||
|
# Check for key functions
|
||||||
|
if grep -q "def enrich_tower_data" utils/gsm_geocoding.py; then
|
||||||
|
pass_test "enrich_tower_data() function present"
|
||||||
|
else
|
||||||
|
fail_test "enrich_tower_data() function missing"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if grep -q "def lookup_cell_coordinates" utils/gsm_geocoding.py; then
|
||||||
|
pass_test "lookup_cell_coordinates() function present"
|
||||||
|
else
|
||||||
|
fail_test "lookup_cell_coordinates() function missing"
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
fail_test "Geocoding module missing"
|
||||||
|
fi
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Test 2: Check scanner thread improvements
|
||||||
|
echo "Test 2: Scanner Thread Non-Blocking I/O"
|
||||||
|
echo "---------------------------------------"
|
||||||
|
if grep -q "import select" routes/gsm_spy.py; then
|
||||||
|
pass_test "select module imported"
|
||||||
|
else
|
||||||
|
fail_test "select module not imported"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if grep -q "select.select.*process.stdout" routes/gsm_spy.py; then
|
||||||
|
pass_test "Non-blocking I/O with select.select() implemented"
|
||||||
|
else
|
||||||
|
fail_test "select.select() not found in scanner thread"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if grep -q "scan_timeout = 120" routes/gsm_spy.py; then
|
||||||
|
pass_test "Scan timeout configured"
|
||||||
|
else
|
||||||
|
fail_test "Scan timeout not configured"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if grep -q "with app_module.gsm_spy_lock:" routes/gsm_spy.py; then
|
||||||
|
pass_test "Thread-safe counter updates implemented"
|
||||||
|
else
|
||||||
|
fail_test "Thread-safe counter updates missing"
|
||||||
|
fi
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Test 3: Check geocoding worker
|
||||||
|
echo "Test 3: Background Geocoding Worker"
|
||||||
|
echo "-----------------------------------"
|
||||||
|
if grep -q "def start_geocoding_worker" routes/gsm_spy.py; then
|
||||||
|
pass_test "start_geocoding_worker() function exists"
|
||||||
|
else
|
||||||
|
fail_test "start_geocoding_worker() function missing"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if grep -q "def geocoding_worker" routes/gsm_spy.py; then
|
||||||
|
pass_test "geocoding_worker() function exists"
|
||||||
|
else
|
||||||
|
fail_test "geocoding_worker() function missing"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if grep -q "start_geocoding_worker()" routes/gsm_spy.py; then
|
||||||
|
pass_test "Geocoding worker is started in start_scanner()"
|
||||||
|
else
|
||||||
|
fail_test "Geocoding worker not started in start_scanner()"
|
||||||
|
fi
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Test 4: Check enrichment integration
|
||||||
|
echo "Test 4: Tower Data Enrichment"
|
||||||
|
echo "-----------------------------"
|
||||||
|
if grep -q "from utils.gsm_geocoding import enrich_tower_data" routes/gsm_spy.py; then
|
||||||
|
pass_test "enrich_tower_data imported in scanner thread"
|
||||||
|
else
|
||||||
|
fail_test "enrich_tower_data not imported"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if grep -q "enriched = enrich_tower_data(parsed)" routes/gsm_spy.py; then
|
||||||
|
pass_test "Tower data enrichment called in scanner"
|
||||||
|
else
|
||||||
|
fail_test "Tower data enrichment not called"
|
||||||
|
fi
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Test 5: Check monitor pipeline fixes
|
||||||
|
echo "Test 5: Monitor Pipeline Connection"
|
||||||
|
echo "-----------------------------------"
|
||||||
|
if grep -q "Give grgsm_livemon time to initialize" routes/gsm_spy.py; then
|
||||||
|
pass_test "Pipeline initialization delay comment present"
|
||||||
|
else
|
||||||
|
fail_test "Pipeline initialization delay comment missing"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if grep -A 5 "Start grgsm_livemon" routes/gsm_spy.py | grep -q "time.sleep(2)"; then
|
||||||
|
pass_test "2-second delay between grgsm_livemon and tshark"
|
||||||
|
else
|
||||||
|
fail_test "Initialization delay not implemented"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if grep -q "Started grgsm_livemon (PID:" routes/gsm_spy.py; then
|
||||||
|
pass_test "Process verification logging added"
|
||||||
|
else
|
||||||
|
fail_test "Process verification logging missing"
|
||||||
|
fi
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Test 6: Check monitor thread improvements
|
||||||
|
echo "Test 6: Monitor Thread Non-Blocking I/O"
|
||||||
|
echo "---------------------------------------"
|
||||||
|
if grep -q "def monitor_thread(process):" routes/gsm_spy.py; then
|
||||||
|
pass_test "monitor_thread() function exists"
|
||||||
|
|
||||||
|
if grep -A 20 "def monitor_thread(process):" routes/gsm_spy.py | grep -q "select.select.*process.stdout"; then
|
||||||
|
pass_test "Monitor thread uses non-blocking I/O"
|
||||||
|
else
|
||||||
|
fail_test "Monitor thread doesn't use select.select()"
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
fail_test "monitor_thread() function missing"
|
||||||
|
fi
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Test 7: Check frontend coordinate validation
|
||||||
|
echo "Test 7: Frontend Coordinate Validation"
|
||||||
|
echo "--------------------------------------"
|
||||||
|
if grep -q "Validate coordinates before creating map marker" templates/gsm_spy_dashboard.html; then
|
||||||
|
pass_test "Coordinate validation comment present"
|
||||||
|
else
|
||||||
|
fail_test "Coordinate validation comment missing"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if grep -q "isNaN(parseFloat(data.lat))" templates/gsm_spy_dashboard.html; then
|
||||||
|
pass_test "Coordinate validation checks implemented"
|
||||||
|
else
|
||||||
|
fail_test "Coordinate validation checks missing"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if grep -q "tower_update" templates/gsm_spy_dashboard.html; then
|
||||||
|
pass_test "tower_update message handler added"
|
||||||
|
else
|
||||||
|
fail_test "tower_update message handler missing"
|
||||||
|
fi
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Test 8: Check process cleanup improvements
|
||||||
|
echo "Test 8: Process Cleanup & Zombie Prevention"
|
||||||
|
echo "-------------------------------------------"
|
||||||
|
if grep -q "process.terminate()" routes/gsm_spy.py; then
|
||||||
|
pass_test "Process termination implemented"
|
||||||
|
else
|
||||||
|
fail_test "Process termination missing"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if grep -q "subprocess.TimeoutExpired" routes/gsm_spy.py; then
|
||||||
|
pass_test "Timeout handling for process termination"
|
||||||
|
else
|
||||||
|
fail_test "Timeout handling missing"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if grep -q "process.kill()" routes/gsm_spy.py; then
|
||||||
|
pass_test "Force kill fallback implemented"
|
||||||
|
else
|
||||||
|
fail_test "Force kill fallback missing"
|
||||||
|
fi
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Test 9: Python syntax check
|
||||||
|
echo "Test 9: Python Syntax Validation"
|
||||||
|
echo "--------------------------------"
|
||||||
|
if python3 -m py_compile routes/gsm_spy.py 2>/dev/null; then
|
||||||
|
pass_test "routes/gsm_spy.py has valid syntax"
|
||||||
|
else
|
||||||
|
fail_test "routes/gsm_spy.py has syntax errors"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if python3 -m py_compile utils/gsm_geocoding.py 2>/dev/null; then
|
||||||
|
pass_test "utils/gsm_geocoding.py has valid syntax"
|
||||||
|
else
|
||||||
|
fail_test "utils/gsm_geocoding.py has syntax errors"
|
||||||
|
fi
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Test 10: Check auto-monitor persistence
|
||||||
|
echo "Test 10: Auto-Monitor Flag Persistence"
|
||||||
|
echo "--------------------------------------"
|
||||||
|
if grep -q "auto_monitor_triggered = False.*# Moved outside loop" routes/gsm_spy.py; then
|
||||||
|
pass_test "auto_monitor_triggered flag moved outside loop"
|
||||||
|
else
|
||||||
|
fail_test "auto_monitor_triggered flag not properly placed"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if grep -q "if current_count >= 3 and not auto_monitor_triggered" routes/gsm_spy.py; then
|
||||||
|
pass_test "Auto-monitor only triggers once per session"
|
||||||
|
else
|
||||||
|
fail_test "Auto-monitor trigger condition incorrect"
|
||||||
|
fi
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Summary
|
||||||
|
echo "=========================================="
|
||||||
|
echo "Test Summary"
|
||||||
|
echo "=========================================="
|
||||||
|
echo -e "Tests passed: ${GREEN}${TESTS_PASSED}${NC}"
|
||||||
|
echo -e "Tests failed: ${RED}${TESTS_FAILED}${NC}"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
if [ $TESTS_FAILED -eq 0 ]; then
|
||||||
|
echo -e "${GREEN}All tests passed! ✓${NC}"
|
||||||
|
echo ""
|
||||||
|
echo "Next steps:"
|
||||||
|
echo "1. Start INTERCEPT: sudo -E venv/bin/python intercept.py"
|
||||||
|
echo "2. Navigate to GSM Spy dashboard in browser"
|
||||||
|
echo "3. Click 'Start Scanner' to test tower detection with geocoding"
|
||||||
|
echo "4. Verify towers appear on map with coordinates"
|
||||||
|
echo "5. Check that auto-monitor starts after 3+ towers found"
|
||||||
|
echo "6. Test Stop button for responsive shutdown (< 2 seconds)"
|
||||||
|
exit 0
|
||||||
|
else
|
||||||
|
echo -e "${RED}Some tests failed. Please review the output above.${NC}"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
200
utils/gsm_geocoding.py
Normal file
200
utils/gsm_geocoding.py
Normal file
@@ -0,0 +1,200 @@
|
|||||||
|
"""GSM Cell Tower Geocoding Service.
|
||||||
|
|
||||||
|
Provides hybrid cache-first geocoding with async API fallback for cell towers.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import queue
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
import requests
|
||||||
|
|
||||||
|
import config
|
||||||
|
from utils.database import get_db
|
||||||
|
|
||||||
|
logger = logging.getLogger('intercept.gsm_geocoding')
|
||||||
|
|
||||||
|
# Queue for pending geocoding requests
|
||||||
|
_geocoding_queue = queue.Queue(maxsize=100)
|
||||||
|
|
||||||
|
|
||||||
|
def lookup_cell_coordinates(mcc: int, mnc: int, lac: int, cid: int) -> dict[str, Any] | None:
|
||||||
|
"""
|
||||||
|
Lookup cell tower coordinates with cache-first strategy.
|
||||||
|
|
||||||
|
Strategy:
|
||||||
|
1. Check gsm_cells table (cache) - fast synchronous lookup
|
||||||
|
2. If not found, return None (caller decides whether to use API)
|
||||||
|
|
||||||
|
Args:
|
||||||
|
mcc: Mobile Country Code
|
||||||
|
mnc: Mobile Network Code
|
||||||
|
lac: Location Area Code
|
||||||
|
cid: Cell ID
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
dict with keys: lat, lon, source='cache', azimuth (optional),
|
||||||
|
range_meters (optional), operator (optional), radio (optional)
|
||||||
|
Returns None if not found in cache.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
with get_db() as conn:
|
||||||
|
result = conn.execute('''
|
||||||
|
SELECT lat, lon, azimuth, range_meters, operator, radio
|
||||||
|
FROM gsm_cells
|
||||||
|
WHERE mcc = ? AND mnc = ? AND lac = ? AND cid = ?
|
||||||
|
''', (mcc, mnc, lac, cid)).fetchone()
|
||||||
|
|
||||||
|
if result:
|
||||||
|
return {
|
||||||
|
'lat': result['lat'],
|
||||||
|
'lon': result['lon'],
|
||||||
|
'source': 'cache',
|
||||||
|
'azimuth': result['azimuth'],
|
||||||
|
'range_meters': result['range_meters'],
|
||||||
|
'operator': result['operator'],
|
||||||
|
'radio': result['radio']
|
||||||
|
}
|
||||||
|
|
||||||
|
return None
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error looking up coordinates from cache: {e}")
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def lookup_cell_from_api(mcc: int, mnc: int, lac: int, cid: int) -> dict[str, Any] | None:
|
||||||
|
"""
|
||||||
|
Lookup cell tower from OpenCellID API and cache result.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
mcc: Mobile Country Code
|
||||||
|
mnc: Mobile Network Code
|
||||||
|
lac: Location Area Code
|
||||||
|
cid: Cell ID
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
dict with keys: lat, lon, source='api', azimuth (optional),
|
||||||
|
range_meters (optional), operator (optional), radio (optional)
|
||||||
|
Returns None if API call fails or cell not found.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
api_url = config.GSM_OPENCELLID_API_URL
|
||||||
|
params = {
|
||||||
|
'key': config.GSM_OPENCELLID_API_KEY,
|
||||||
|
'mcc': mcc,
|
||||||
|
'mnc': mnc,
|
||||||
|
'lac': lac,
|
||||||
|
'cellid': cid,
|
||||||
|
'format': 'json'
|
||||||
|
}
|
||||||
|
|
||||||
|
response = requests.get(api_url, params=params, timeout=10)
|
||||||
|
|
||||||
|
if response.status_code == 200:
|
||||||
|
cell_data = response.json()
|
||||||
|
|
||||||
|
# Cache the result
|
||||||
|
with get_db() as conn:
|
||||||
|
conn.execute('''
|
||||||
|
INSERT OR REPLACE INTO gsm_cells
|
||||||
|
(mcc, mnc, lac, cid, lat, lon, azimuth, range_meters, samples, radio, operator, last_verified)
|
||||||
|
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, CURRENT_TIMESTAMP)
|
||||||
|
''', (
|
||||||
|
mcc, mnc, lac, cid,
|
||||||
|
cell_data.get('lat'),
|
||||||
|
cell_data.get('lon'),
|
||||||
|
cell_data.get('azimuth'),
|
||||||
|
cell_data.get('range'),
|
||||||
|
cell_data.get('samples'),
|
||||||
|
cell_data.get('radio'),
|
||||||
|
cell_data.get('operator')
|
||||||
|
))
|
||||||
|
conn.commit()
|
||||||
|
|
||||||
|
logger.info(f"Cached cell tower from API: MCC={mcc} MNC={mnc} LAC={lac} CID={cid}")
|
||||||
|
|
||||||
|
return {
|
||||||
|
'lat': cell_data.get('lat'),
|
||||||
|
'lon': cell_data.get('lon'),
|
||||||
|
'source': 'api',
|
||||||
|
'azimuth': cell_data.get('azimuth'),
|
||||||
|
'range_meters': cell_data.get('range'),
|
||||||
|
'operator': cell_data.get('operator'),
|
||||||
|
'radio': cell_data.get('radio')
|
||||||
|
}
|
||||||
|
else:
|
||||||
|
logger.warning(f"OpenCellID API returned {response.status_code} for MCC={mcc} MNC={mnc} LAC={lac} CID={cid}")
|
||||||
|
return None
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error calling OpenCellID API: {e}")
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def enrich_tower_data(tower_data: dict[str, Any]) -> dict[str, Any]:
|
||||||
|
"""
|
||||||
|
Enrich tower data with coordinates using cache-first strategy.
|
||||||
|
|
||||||
|
If coordinates found in cache, adds them immediately.
|
||||||
|
If not found, marks as 'pending' and queues for background API lookup.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
tower_data: Dictionary with keys mcc, mnc, lac, cid (and other tower data)
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Enriched tower_data dict with added fields:
|
||||||
|
- lat, lon (if found in cache)
|
||||||
|
- status='pending' (if needs API lookup)
|
||||||
|
- source='cache' (if from cache)
|
||||||
|
"""
|
||||||
|
mcc = tower_data.get('mcc')
|
||||||
|
mnc = tower_data.get('mnc')
|
||||||
|
lac = tower_data.get('lac')
|
||||||
|
cid = tower_data.get('cid')
|
||||||
|
|
||||||
|
# Validate required fields
|
||||||
|
if not all([mcc is not None, mnc is not None, lac is not None, cid is not None]):
|
||||||
|
logger.warning(f"Tower data missing required fields: {tower_data}")
|
||||||
|
return tower_data
|
||||||
|
|
||||||
|
# Try cache lookup
|
||||||
|
coords = lookup_cell_coordinates(mcc, mnc, lac, cid)
|
||||||
|
|
||||||
|
if coords:
|
||||||
|
# Found in cache - add coordinates immediately
|
||||||
|
tower_data['lat'] = coords['lat']
|
||||||
|
tower_data['lon'] = coords['lon']
|
||||||
|
tower_data['source'] = 'cache'
|
||||||
|
|
||||||
|
# Add optional fields if available
|
||||||
|
if coords.get('azimuth') is not None:
|
||||||
|
tower_data['azimuth'] = coords['azimuth']
|
||||||
|
if coords.get('range_meters') is not None:
|
||||||
|
tower_data['range_meters'] = coords['range_meters']
|
||||||
|
if coords.get('operator'):
|
||||||
|
tower_data['operator'] = coords['operator']
|
||||||
|
if coords.get('radio'):
|
||||||
|
tower_data['radio'] = coords['radio']
|
||||||
|
|
||||||
|
logger.debug(f"Cache hit for tower: MCC={mcc} MNC={mnc} LAC={lac} CID={cid}")
|
||||||
|
else:
|
||||||
|
# Not in cache - mark as pending and queue for API lookup
|
||||||
|
tower_data['status'] = 'pending'
|
||||||
|
tower_data['source'] = 'unknown'
|
||||||
|
|
||||||
|
# Queue for background geocoding (non-blocking)
|
||||||
|
try:
|
||||||
|
_geocoding_queue.put_nowait(tower_data.copy())
|
||||||
|
logger.debug(f"Queued tower for geocoding: MCC={mcc} MNC={mnc} LAC={lac} CID={cid}")
|
||||||
|
except queue.Full:
|
||||||
|
logger.warning("Geocoding queue full, dropping tower")
|
||||||
|
|
||||||
|
return tower_data
|
||||||
|
|
||||||
|
|
||||||
|
def get_geocoding_queue() -> queue.Queue:
|
||||||
|
"""Get the geocoding queue for the background worker."""
|
||||||
|
return _geocoding_queue
|
||||||
Reference in New Issue
Block a user