rdpds/sac4cps-backend
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

modular monolythic

Browse Source
This commit is contained in:
rafaeldpsilva
2025-12-20 00:57:59 +00:00
parent 4779eb9ded
commit ccf5f5a5c3
6 changed files with 1052 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

480
monolith/MIGRATION.md Normal file
View File

@@ -0,0 +1,480 @@
# Migration Guide: Microservices to Modular Monolith
This guide explains the transformation from the microservices architecture to the modular monolithic architecture.
## Overview of Changes
### Architecture Transformation
**Before (Microservices)**:
- Multiple independent services (8+ services)
- HTTP-based inter-service communication
- Redis pub/sub for events
- API Gateway for routing
- Service discovery and health checking
- Separate Docker containers per service
**After (Modular Monolith)**:
- Single application with modular structure
- Direct function calls via dependency injection
- In-process event bus
- Integrated routing in main application
- Single Docker container
- Separate databases per module (preserved isolation)
## Key Architectural Differences
### 1. Service Communication
#### Microservices Approach
```python
# HTTP call to another service
async with aiohttp.ClientSession() as session:
url = f"{SENSOR_SERVICE_URL}/sensors/{sensor_id}"
async with session.get(url) as response:
data = await response.json()
```
#### Modular Monolith Approach
```python
# Direct function call with dependency injection
from modules.sensors import SensorService
from core.dependencies import get_sensors_db
sensor_service = SensorService(db=await get_sensors_db(), redis=None)
data = await sensor_service.get_sensor_details(sensor_id)
```
### 2. Event Communication
#### Microservices Approach (Redis Pub/Sub)
```python
# Publishing
await redis.publish("energy_data", json.dumps(data))
# Subscribing
pubsub = redis.pubsub()
await pubsub.subscribe("energy_data")
message = await pubsub.get_message()
```
#### Modular Monolith Approach (Event Bus)
```python
# Publishing
from core.events import event_bus, EventTopics
await event_bus.publish(EventTopics.ENERGY_DATA, data)
# Subscribing
def handle_energy_data(data):
# Process data
pass
event_bus.subscribe(EventTopics.ENERGY_DATA, handle_energy_data)
```
### 3. Database Access
#### Microservices Approach
```python
# Each service has its own database connection
from database import get_database
db = await get_database() # Returns service-specific database
```
#### Modular Monolith Approach
```python
# Centralized database manager with module-specific databases
from core.database import db_manager
sensors_db = db_manager.sensors_db
demand_response_db = db_manager.demand_response_db
```
### 4. Application Structure
#### Microservices Structure
```
microservices/
├── api-gateway/
│ └── main.py (port 8000)
├── sensor-service/
│ └── main.py (port 8007)
├── demand-response-service/
│ └── main.py (port 8003)
├── data-ingestion-service/
│ └── main.py (port 8008)
└── docker-compose.yml (8+ containers)
```
#### Modular Monolith Structure
```
monolith/
├── src/
│ ├── main.py (single entry point)
│ ├── core/ (shared infrastructure)
│ └── modules/
│ ├── sensors/
│ ├── demand_response/
│ └── data_ingestion/
└── docker-compose.yml (1 container)
```
## Migration Steps
### Phase 1: Preparation
1. **Backup existing data**:
```bash
# Backup all MongoDB databases
mongodump --uri="mongodb://admin:password123@localhost:27017" --out=/backup/microservices
```
2. **Document current API endpoints**:
- List all endpoints from each microservice
- Document inter-service communication patterns
- Identify Redis pub/sub channels in use
3. **Review environment variables**:
- Consolidate environment variables
- Update connection strings for external MongoDB and Redis
### Phase 2: Deploy Modular Monolith
1. **Configure environment**:
```bash
cd /path/to/monolith
cp .env.example .env
# Edit .env with MongoDB and Redis connection strings
```
2. **Build and deploy**:
```bash
docker-compose up --build -d
```
3. **Verify health**:
```bash
curl http://localhost:8000/health
curl http://localhost:8000/api/v1/overview
```
### Phase 3: Data Migration (if needed)
The modular monolith uses the **same database structure** as the microservices, so typically no data migration is needed. However, verify:
1. **Database names match**:
- `energy_dashboard_sensors`
- `energy_dashboard_demand_response`
- `digitalmente_ingestion`
2. **Collections are accessible**:
```bash
# Connect to MongoDB
mongosh mongodb://admin:password123@mongodb-host:27017/?authSource=admin
# Check databases
show dbs
# Verify collections in each database
use energy_dashboard_sensors
show collections
```
### Phase 4: API Client Migration
Update API clients to point to the new monolith endpoint:
**Before**:
- Sensor API: `http://api-gateway:8000/api/v1/sensors/*`
- DR API: `http://api-gateway:8000/api/v1/demand-response/*`
**After**:
- All APIs: `http://monolith:8000/api/v1/*`
The API paths remain the same, only the host changes!
### Phase 5: Decommission Microservices
Once the monolith is stable:
1. **Stop microservices**:
```bash
cd /path/to/microservices
docker-compose down
```
2. **Keep backups** for at least 30 days
3. **Archive microservices code** for reference
## Benefits of the Migration
### Operational Simplification
| Aspect | Microservices | Modular Monolith | Improvement |
|--------|---------------|------------------|-------------|
| **Containers** | 8+ containers | 1 container | 87% reduction |
| **Network calls** | HTTP between services | In-process calls | ~100x faster |
| **Deployment complexity** | Coordinate 8+ services | Single deployment | Much simpler |
| **Monitoring** | 8+ health endpoints | 1 health endpoint | Easier |
| **Log aggregation** | 8+ log sources | 1 log source | Simpler |
### Performance Improvements
1. **Reduced latency**:
- Inter-service HTTP calls: ~10-50ms
- Direct function calls: ~0.01-0.1ms
- **Improvement**: 100-1000x faster
2. **Reduced network overhead**:
- No HTTP serialization/deserialization
- No network round-trips
- No service discovery delays
3. **Shared resources**:
- Single database connection pool
- Shared Redis connection (if enabled)
- Shared in-memory caches
### Development Benefits
1. **Easier debugging**:
- Single process to debug
- Direct stack traces across modules
- No distributed tracing needed
2. **Simpler testing**:
- Test entire flow in one process
- No need to mock HTTP calls
- Integration tests run faster
3. **Faster development**:
- Single application to run locally
- Immediate code changes (with reload)
- No service orchestration needed
## Preserved Benefits from Microservices
### Module Isolation
Each module maintains clear boundaries:
- Separate directory structure
- Own models and business logic
- Dedicated database (data isolation)
- Clear public interfaces
### Independent Scaling (Future)
If needed, modules can be extracted back into microservices:
- Clean module boundaries make extraction easy
- Database per module already separated
- Event bus can switch to Redis pub/sub
- Direct calls can switch to HTTP calls
### Team Organization
Teams can still own modules:
- Sensors team owns `modules/sensors/`
- DR team owns `modules/demand_response/`
- Clear ownership and responsibilities
## Rollback Strategy
If you need to rollback to microservices:
1. **Keep microservices code** in the repository
2. **Database unchanged**: Both architectures use the same databases
3. **Redeploy microservices**:
```bash
cd /path/to/microservices
docker-compose up -d
```
4. **Update API clients** to point back to API Gateway
## Monitoring and Observability
### Health Checks
**Single health endpoint**:
```bash
curl http://localhost:8000/health
```
Returns:
```json
{
"service": "Energy Dashboard Monolith",
"status": "healthy",
"components": {
"database": "healthy",
"redis": "healthy",
"event_bus": "healthy"
},
"modules": {
"sensors": "loaded",
"demand_response": "loaded",
"data_ingestion": "loaded"
}
}
```
### Logging
All logs in one place:
```bash
# Docker logs
docker-compose logs -f monolith
# Application logs
docker-compose logs -f monolith | grep "ERROR"
```
### Metrics
System overview endpoint:
```bash
curl http://localhost:8000/api/v1/overview
```
## Common Migration Issues
### Issue: Module Import Errors
**Problem**: `ModuleNotFoundError: No module named 'src.modules'`
**Solution**:
```bash
# Set PYTHONPATH
export PYTHONPATH=/app
# Or in docker-compose.yml
environment:
- PYTHONPATH=/app
```
### Issue: Database Connection Errors
**Problem**: Cannot connect to MongoDB
**Solution**:
1. Verify MongoDB is accessible:
```bash
docker-compose exec monolith ping mongodb-host
```
2. Check connection string in `.env`
3. Ensure network connectivity
### Issue: Redis Connection Errors
**Problem**: Redis connection failed but app should work
**Solution**:
Redis is optional. Set in `.env`:
```
REDIS_ENABLED=false
```
### Issue: Event Subscribers Not Receiving Events
**Problem**: Events published but subscribers not called
**Solution**:
Ensure subscribers are registered before events are published:
```python
# Register subscriber in lifespan startup
@asynccontextmanager
async def lifespan(app: FastAPI):
# Subscribe before publishing
event_bus.subscribe(EventTopics.ENERGY_DATA, handle_energy)
yield
```
## Testing the Migration
### 1. Functional Testing
Test each module's endpoints:
```bash
# Sensors
curl http://localhost:8000/api/v1/sensors/get
curl http://localhost:8000/api/v1/rooms
# Analytics
curl http://localhost:8000/api/v1/analytics/summary
# Health
curl http://localhost:8000/health
```
### 2. Load Testing
Compare performance:
```bash
# Microservices
ab -n 1000 -c 10 http://localhost:8000/api/v1/sensors/get
# Modular Monolith
ab -n 1000 -c 10 http://localhost:8000/api/v1/sensors/get
```
Expected: Modular monolith should be significantly faster.
### 3. WebSocket Testing
Test real-time features:
```javascript
const ws = new WebSocket('ws://localhost:8000/api/v1/ws');
ws.onmessage = (event) => console.log('Received:', event.data);
```
## FAQ
### Q: Do I need to migrate the database?
**A**: No, the modular monolith uses the same database structure as the microservices.
### Q: Can I scale individual modules?
**A**: Not independently. The entire monolith scales together. If you need independent scaling, consider keeping the microservices architecture or using horizontal scaling with load balancers.
### Q: What happens to Redis pub/sub?
**A**: Replaced with an in-process event bus. Redis can still be used for caching if `REDIS_ENABLED=true`.
### Q: Are the API endpoints the same?
**A**: Yes, the API paths remain identical. Only the host changes.
### Q: Can I extract modules back to microservices later?
**A**: Yes, the modular structure makes it easy to extract modules back into separate services if needed.
### Q: How do I add a new module?
**A**: See the "Adding a New Module" section in README.md.
### Q: Is this suitable for production?
**A**: Yes, modular monoliths are production-ready and often more reliable than microservices for small-to-medium scale applications.
## Next Steps
1. **Deploy to staging** and run full test suite
2. **Monitor performance** and compare with microservices
3. **Gradual rollout** to production (canary or blue-green deployment)
4. **Decommission microservices** after 30 days of stable operation
5. **Update documentation** and team training
## Support
For issues or questions about the migration:
1. Check this guide and README.md
2. Review application logs: `docker-compose logs monolith`
3. Test health endpoint: `curl http://localhost:8000/health`
4. Contact the development team