Switch to PyMongo, update config and requirements, fix FTP extension
typo - Replace Motor (async) with PyMongo (sync) in database manager - Update environment variable names for FTP and MongoDB config - Remove unused dependencies from requirements.txt - Fix file extension typo: .slg_v2 → .sgl_v2 throughout code and docs - Add debug prints for MongoDB env vars in config - Update FTP monitor to use correct file extension and PyMongo - Adjust FastAPI descriptions for new extension
This commit is contained in:
@@ -9,10 +9,10 @@ The Data Ingestion Service provides comprehensive FTP monitoring and data proces
|
||||
## Architecture
|
||||
|
||||
```
|
||||
ftp.sa4cps.pt (.slg_v2 files)
|
||||
ftp.sa4cps.pt (.slg_v2 files)
|
||||
↓
|
||||
FTP Monitor (polls every 5 minutes)
|
||||
↓
|
||||
↓
|
||||
Data Processor (supports multiple formats)
|
||||
↓
|
||||
Redis Publisher (3 topic channels)
|
||||
@@ -84,9 +84,9 @@ Set these in the `docker-compose.yml`:
|
||||
environment:
|
||||
- FTP_SA4CPS_HOST=ftp.sa4cps.pt # FTP server hostname
|
||||
- FTP_SA4CPS_PORT=21 # FTP port (default: 21)
|
||||
- FTP_SA4CPS_USERNAME=anonymous # FTP username
|
||||
- FTP_SA4CPS_USERNAME= # FTP username
|
||||
- FTP_SA4CPS_PASSWORD= # FTP password (empty for anonymous)
|
||||
- FTP_SA4CPS_REMOTE_PATH=/ # Remote directory path
|
||||
- FTP_SA4CPS_REMOTE_PATH=/ # Remote directory path
|
||||
```
|
||||
|
||||
### Manual Configuration
|
||||
@@ -101,7 +101,7 @@ configurator = SA4CPSConfigurator()
|
||||
# Create data source
|
||||
result = await configurator.create_sa4cps_data_source(
|
||||
username="your_username",
|
||||
password="your_password",
|
||||
password="your_password",
|
||||
remote_path="/data/energy"
|
||||
)
|
||||
|
||||
@@ -144,7 +144,7 @@ timestamp,sensor_id,energy_kwh,power_w,voltage_v
|
||||
2024-01-15T10:01:00Z,SENSOR_001,1235.1,865.3,229.8
|
||||
```
|
||||
|
||||
### Space-Delimited Format
|
||||
### Space-Delimited Format
|
||||
```
|
||||
# Energy consumption data
|
||||
# System: Smart Grid Monitor
|
||||
@@ -191,7 +191,7 @@ All processed data is converted to a standardized sensor reading format:
|
||||
### sa4cps_energy_data
|
||||
Primary energy consumption and power readings:
|
||||
- Energy consumption (kWh, MWh)
|
||||
- Power readings (W, kW, MW)
|
||||
- Power readings (W, kW, MW)
|
||||
- Efficiency metrics
|
||||
|
||||
### sa4cps_sensor_metrics
|
||||
@@ -201,7 +201,7 @@ Sensor telemetry and environmental data:
|
||||
- Sensor status/diagnostics
|
||||
- System health metrics
|
||||
|
||||
### sa4cps_raw_data
|
||||
### sa4cps_raw_data
|
||||
Raw unprocessed data for debugging:
|
||||
- Original file content
|
||||
- Processing metadata
|
||||
@@ -278,10 +278,10 @@ class CustomSA4CPSProcessor(DataProcessor):
|
||||
async def _process_slg_v2_line(self, line, header, metadata, line_idx):
|
||||
# Custom line processing logic
|
||||
processed = await super()._process_slg_v2_line(line, header, metadata, line_idx)
|
||||
|
||||
|
||||
# Add custom fields
|
||||
processed['custom_field'] = 'custom_value'
|
||||
|
||||
|
||||
return processed
|
||||
```
|
||||
|
||||
@@ -295,4 +295,4 @@ For issues or questions:
|
||||
|
||||
## License
|
||||
|
||||
This implementation is part of the SA4CPS project energy monitoring dashboard.
|
||||
This implementation is part of the SA4CPS project energy monitoring dashboard.
|
||||
|
||||
Reference in New Issue
Block a user