sac4cps-backend/monolith/MIGRATION.md

# Migration Guide: Microservices to Modular Monolith

This guide explains the transformation from the microservices architecture to the modular monolithic architecture.

## Overview of Changes

### Architecture Transformation

**Before (Microservices)**:
- Multiple independent services (8+ services)
- HTTP-based inter-service communication
- Redis pub/sub for events
- API Gateway for routing
- Service discovery and health checking
- Separate Docker containers per service

**After (Modular Monolith)**:
- Single application with modular structure
- Direct function calls via dependency injection
- In-process event bus
- Integrated routing in main application
- Single Docker container
- Separate databases per module (preserved isolation)

## Key Architectural Differences

### 1. Service Communication

#### Microservices Approach
```python
# HTTP call to another service
async with aiohttp.ClientSession() as session:
    url = f"{SENSOR_SERVICE_URL}/sensors/{sensor_id}"
    async with session.get(url) as response:
        data = await response.json()
```

#### Modular Monolith Approach
```python
# Direct function call with dependency injection
from modules.sensors import SensorService
from core.dependencies import get_sensors_db

sensor_service = SensorService(db=await get_sensors_db(), redis=None)
data = await sensor_service.get_sensor_details(sensor_id)
```

### 2. Event Communication

#### Microservices Approach (Redis Pub/Sub)
```python
# Publishing
await redis.publish("energy_data", json.dumps(data))

# Subscribing
pubsub = redis.pubsub()
await pubsub.subscribe("energy_data")
message = await pubsub.get_message()
```

#### Modular Monolith Approach (Event Bus)
```python
# Publishing
from core.events import event_bus, EventTopics
await event_bus.publish(EventTopics.ENERGY_DATA, data)

# Subscribing
def handle_energy_data(data):
    # Process data
    pass

event_bus.subscribe(EventTopics.ENERGY_DATA, handle_energy_data)
```

### 3. Database Access

#### Microservices Approach
```python
# Each service has its own database connection
from database import get_database

db = await get_database()  # Returns service-specific database
```

#### Modular Monolith Approach
```python
# Centralized database manager with module-specific databases
from core.database import db_manager

sensors_db = db_manager.sensors_db
demand_response_db = db_manager.demand_response_db
```

### 4. Application Structure

#### Microservices Structure
```
microservices/
├── api-gateway/
│   └── main.py (port 8000)
├── sensor-service/
│   └── main.py (port 8007)
├── demand-response-service/
│   └── main.py (port 8003)
├── data-ingestion-service/
│   └── main.py (port 8008)
└── docker-compose.yml (8+ containers)
```

#### Modular Monolith Structure
```
monolith/
├── src/
│   ├── main.py (single entry point)
│   ├── core/ (shared infrastructure)
│   └── modules/
│       ├── sensors/
│       ├── demand_response/
│       └── data_ingestion/
└── docker-compose.yml (1 container)
```

## Migration Steps

### Phase 1: Preparation

1. **Backup existing data**:
   ```bash
   # Backup all MongoDB databases
   mongodump --uri="mongodb://admin:password123@localhost:27017" --out=/backup/microservices
   ```

2. **Document current API endpoints**:
   - List all endpoints from each microservice
   - Document inter-service communication patterns
   - Identify Redis pub/sub channels in use

3. **Review environment variables**:
   - Consolidate environment variables
   - Update connection strings for external MongoDB and Redis

### Phase 2: Deploy Modular Monolith

1. **Configure environment**:
   ```bash
   cd /path/to/monolith
   cp .env.example .env
   # Edit .env with MongoDB and Redis connection strings
   ```

2. **Build and deploy**:
   ```bash
   docker-compose up --build -d
   ```

3. **Verify health**:
   ```bash
   curl http://localhost:8000/health
   curl http://localhost:8000/api/v1/overview
   ```

### Phase 3: Data Migration (if needed)

The modular monolith uses the **same database structure** as the microservices, so typically no data migration is needed. However, verify:

1. **Database names match**:
   - `energy_dashboard_sensors`
   - `energy_dashboard_demand_response`
   - `digitalmente_ingestion`

2. **Collections are accessible**:
   ```bash
   # Connect to MongoDB
   mongosh mongodb://admin:password123@mongodb-host:27017/?authSource=admin

   # Check databases
   show dbs

   # Verify collections in each database
   use energy_dashboard_sensors
   show collections
   ```

### Phase 4: API Client Migration

Update API clients to point to the new monolith endpoint:

**Before**:
- Sensor API: `http://api-gateway:8000/api/v1/sensors/*`
- DR API: `http://api-gateway:8000/api/v1/demand-response/*`

**After**:
- All APIs: `http://monolith:8000/api/v1/*`

The API paths remain the same, only the host changes!

### Phase 5: Decommission Microservices

Once the monolith is stable:

1. **Stop microservices**:
   ```bash
   cd /path/to/microservices
   docker-compose down
   ```

2. **Keep backups** for at least 30 days

3. **Archive microservices code** for reference

## Benefits of the Migration

### Operational Simplification

| Aspect | Microservices | Modular Monolith | Improvement |
|--------|---------------|------------------|-------------|
| **Containers** | 8+ containers | 1 container | 87% reduction |
| **Network calls** | HTTP between services | In-process calls | ~100x faster |
| **Deployment complexity** | Coordinate 8+ services | Single deployment | Much simpler |
| **Monitoring** | 8+ health endpoints | 1 health endpoint | Easier |
| **Log aggregation** | 8+ log sources | 1 log source | Simpler |

### Performance Improvements

1. **Reduced latency**:
   - Inter-service HTTP calls: ~10-50ms
   - Direct function calls: ~0.01-0.1ms
   - **Improvement**: 100-1000x faster

2. **Reduced network overhead**:
   - No HTTP serialization/deserialization
   - No network round-trips
   - No service discovery delays

3. **Shared resources**:
   - Single database connection pool
   - Shared Redis connection (if enabled)
   - Shared in-memory caches

### Development Benefits

1. **Easier debugging**:
   - Single process to debug
   - Direct stack traces across modules
   - No distributed tracing needed

2. **Simpler testing**:
   - Test entire flow in one process
   - No need to mock HTTP calls
   - Integration tests run faster

3. **Faster development**:
   - Single application to run locally
   - Immediate code changes (with reload)
   - No service orchestration needed

## Preserved Benefits from Microservices

### Module Isolation

Each module maintains clear boundaries:
- Separate directory structure
- Own models and business logic
- Dedicated database (data isolation)
- Clear public interfaces

### Independent Scaling (Future)

If needed, modules can be extracted back into microservices:
- Clean module boundaries make extraction easy
- Database per module already separated
- Event bus can switch to Redis pub/sub
- Direct calls can switch to HTTP calls

### Team Organization

Teams can still own modules:
- Sensors team owns `modules/sensors/`
- DR team owns `modules/demand_response/`
- Clear ownership and responsibilities

## Rollback Strategy

If you need to rollback to microservices:

1. **Keep microservices code** in the repository

2. **Database unchanged**: Both architectures use the same databases

3. **Redeploy microservices**:
   ```bash
   cd /path/to/microservices
   docker-compose up -d
   ```

4. **Update API clients** to point back to API Gateway

## Monitoring and Observability

### Health Checks

**Single health endpoint**:
```bash
curl http://localhost:8000/health
```

Returns:
```json
{
  "service": "Energy Dashboard Monolith",
  "status": "healthy",
  "components": {
    "database": "healthy",
    "redis": "healthy",
    "event_bus": "healthy"
  },
  "modules": {
    "sensors": "loaded",
    "demand_response": "loaded",
    "data_ingestion": "loaded"
  }
}
```

### Logging

All logs in one place:
```bash
# Docker logs
docker-compose logs -f monolith

# Application logs
docker-compose logs -f monolith | grep "ERROR"
```

### Metrics

System overview endpoint:
```bash
curl http://localhost:8000/api/v1/overview
```

## Common Migration Issues

### Issue: Module Import Errors

**Problem**: `ModuleNotFoundError: No module named 'src.modules'`

**Solution**:
```bash
# Set PYTHONPATH
export PYTHONPATH=/app
# Or in docker-compose.yml
environment:
  - PYTHONPATH=/app
```

### Issue: Database Connection Errors

**Problem**: Cannot connect to MongoDB

**Solution**:
1. Verify MongoDB is accessible:
   ```bash
   docker-compose exec monolith ping mongodb-host
   ```
2. Check connection string in `.env`
3. Ensure network connectivity

### Issue: Redis Connection Errors

**Problem**: Redis connection failed but app should work

**Solution**:
Redis is optional. Set in `.env`:
```
REDIS_ENABLED=false
```

### Issue: Event Subscribers Not Receiving Events

**Problem**: Events published but subscribers not called

**Solution**:
Ensure subscribers are registered before events are published:
```python
# Register subscriber in lifespan startup
@asynccontextmanager
async def lifespan(app: FastAPI):
    # Subscribe before publishing
    event_bus.subscribe(EventTopics.ENERGY_DATA, handle_energy)
    yield
```

## Testing the Migration

### 1. Functional Testing

Test each module's endpoints:

```bash
# Sensors
curl http://localhost:8000/api/v1/sensors/get
curl http://localhost:8000/api/v1/rooms

# Analytics
curl http://localhost:8000/api/v1/analytics/summary

# Health
curl http://localhost:8000/health
```

### 2. Load Testing

Compare performance:

```bash
# Microservices
ab -n 1000 -c 10 http://localhost:8000/api/v1/sensors/get

# Modular Monolith
ab -n 1000 -c 10 http://localhost:8000/api/v1/sensors/get
```

Expected: Modular monolith should be significantly faster.

### 3. WebSocket Testing

Test real-time features:

```javascript
const ws = new WebSocket('ws://localhost:8000/api/v1/ws');
ws.onmessage = (event) => console.log('Received:', event.data);
```

## FAQ

### Q: Do I need to migrate the database?

**A**: No, the modular monolith uses the same database structure as the microservices.

### Q: Can I scale individual modules?

**A**: Not independently. The entire monolith scales together. If you need independent scaling, consider keeping the microservices architecture or using horizontal scaling with load balancers.

### Q: What happens to Redis pub/sub?

**A**: Replaced with an in-process event bus. Redis can still be used for caching if `REDIS_ENABLED=true`.

### Q: Are the API endpoints the same?

**A**: Yes, the API paths remain identical. Only the host changes.

### Q: Can I extract modules back to microservices later?

**A**: Yes, the modular structure makes it easy to extract modules back into separate services if needed.

### Q: How do I add a new module?

**A**: See the "Adding a New Module" section in README.md.

### Q: Is this suitable for production?

**A**: Yes, modular monoliths are production-ready and often more reliable than microservices for small-to-medium scale applications.

## Next Steps

1. **Deploy to staging** and run full test suite
2. **Monitor performance** and compare with microservices
3. **Gradual rollout** to production (canary or blue-green deployment)
4. **Decommission microservices** after 30 days of stable operation
5. **Update documentation** and team training

## Support

For issues or questions about the migration:
1. Check this guide and README.md
2. Review application logs: `docker-compose logs monolith`
3. Test health endpoint: `curl http://localhost:8000/health`
4. Contact the development team