Switch to PyMongo, update config and requirements, fix FTP extension

typo

- Replace Motor (async) with PyMongo (sync) in database manager - Update
environment variable names for FTP and MongoDB config - Remove unused
dependencies from requirements.txt - Fix file extension typo: .slg_v2 →
.sgl_v2 throughout code and docs - Add debug prints for MongoDB env vars
in config - Update FTP monitor to use correct file extension and PyMongo
- Adjust FastAPI descriptions for new extension
This commit is contained in:
rafaeldpsilva
2025-09-11 11:45:19 +01:00
parent b2a5b3d229
commit 2932e0a424
6 changed files with 152 additions and 156 deletions

View File

@@ -9,10 +9,10 @@ The Data Ingestion Service provides comprehensive FTP monitoring and data proces
## Architecture
```
ftp.sa4cps.pt (.slg_v2 files)
ftp.sa4cps.pt (.slg_v2 files)
FTP Monitor (polls every 5 minutes)
Data Processor (supports multiple formats)
Redis Publisher (3 topic channels)
@@ -84,9 +84,9 @@ Set these in the `docker-compose.yml`:
environment:
- FTP_SA4CPS_HOST=ftp.sa4cps.pt # FTP server hostname
- FTP_SA4CPS_PORT=21 # FTP port (default: 21)
- FTP_SA4CPS_USERNAME=anonymous # FTP username
- FTP_SA4CPS_USERNAME= # FTP username
- FTP_SA4CPS_PASSWORD= # FTP password (empty for anonymous)
- FTP_SA4CPS_REMOTE_PATH=/ # Remote directory path
- FTP_SA4CPS_REMOTE_PATH=/ # Remote directory path
```
### Manual Configuration
@@ -101,7 +101,7 @@ configurator = SA4CPSConfigurator()
# Create data source
result = await configurator.create_sa4cps_data_source(
username="your_username",
password="your_password",
password="your_password",
remote_path="/data/energy"
)
@@ -144,7 +144,7 @@ timestamp,sensor_id,energy_kwh,power_w,voltage_v
2024-01-15T10:01:00Z,SENSOR_001,1235.1,865.3,229.8
```
### Space-Delimited Format
### Space-Delimited Format
```
# Energy consumption data
# System: Smart Grid Monitor
@@ -191,7 +191,7 @@ All processed data is converted to a standardized sensor reading format:
### sa4cps_energy_data
Primary energy consumption and power readings:
- Energy consumption (kWh, MWh)
- Power readings (W, kW, MW)
- Power readings (W, kW, MW)
- Efficiency metrics
### sa4cps_sensor_metrics
@@ -201,7 +201,7 @@ Sensor telemetry and environmental data:
- Sensor status/diagnostics
- System health metrics
### sa4cps_raw_data
### sa4cps_raw_data
Raw unprocessed data for debugging:
- Original file content
- Processing metadata
@@ -278,10 +278,10 @@ class CustomSA4CPSProcessor(DataProcessor):
async def _process_slg_v2_line(self, line, header, metadata, line_idx):
# Custom line processing logic
processed = await super()._process_slg_v2_line(line, header, metadata, line_idx)
# Add custom fields
processed['custom_field'] = 'custom_value'
return processed
```
@@ -295,4 +295,4 @@ For issues or questions:
## License
This implementation is part of the SA4CPS project energy monitoring dashboard.
This implementation is part of the SA4CPS project energy monitoring dashboard.