sac4cps-backend/monolith/MIGRATION.md

Migration Guide: Microservices to Modular Monolith

This guide explains the transformation from the microservices architecture to the modular monolithic architecture.

Overview of Changes

Architecture Transformation

Before (Microservices):

• Multiple independent services (8+ services)
• HTTP-based inter-service communication
• Redis pub/sub for events
• API Gateway for routing
• Service discovery and health checking
• Separate Docker containers per service

After (Modular Monolith):

• Single application with modular structure
• Direct function calls via dependency injection
• In-process event bus
• Integrated routing in main application
• Single Docker container
• Separate databases per module (preserved isolation)

Key Architectural Differences

1. Service Communication

Microservices Approach

# HTTP call to another service
async with aiohttp.ClientSession() as session:
    url = f"{SENSOR_SERVICE_URL}/sensors/{sensor_id}"
    async with session.get(url) as response:
        data = await response.json()

Modular Monolith Approach

# Direct function call with dependency injection
from modules.sensors import SensorService
from core.dependencies import get_sensors_db

sensor_service = SensorService(db=await get_sensors_db(), redis=None)
data = await sensor_service.get_sensor_details(sensor_id)

2. Event Communication

Microservices Approach (Redis Pub/Sub)

# Publishing
await redis.publish("energy_data", json.dumps(data))

# Subscribing
pubsub = redis.pubsub()
await pubsub.subscribe("energy_data")
message = await pubsub.get_message()

Modular Monolith Approach (Event Bus)

# Publishing
from core.events import event_bus, EventTopics
await event_bus.publish(EventTopics.ENERGY_DATA, data)

# Subscribing
def handle_energy_data(data):
    # Process data
    pass

event_bus.subscribe(EventTopics.ENERGY_DATA, handle_energy_data)

3. Database Access

Microservices Approach

# Each service has its own database connection
from database import get_database

db = await get_database()  # Returns service-specific database

Modular Monolith Approach

# Centralized database manager with module-specific databases
from core.database import db_manager

sensors_db = db_manager.sensors_db
demand_response_db = db_manager.demand_response_db

4. Application Structure

Microservices Structure

microservices/
├── api-gateway/
│   └── main.py (port 8000)
├── sensor-service/
│   └── main.py (port 8007)
├── demand-response-service/
│   └── main.py (port 8003)
├── data-ingestion-service/
│   └── main.py (port 8008)
└── docker-compose.yml (8+ containers)

Modular Monolith Structure

monolith/
├── src/
│   ├── main.py (single entry point)
│   ├── core/ (shared infrastructure)
│   └── modules/
│       ├── sensors/
│       ├── demand_response/
│       └── data_ingestion/
└── docker-compose.yml (1 container)

Migration Steps

Phase 1: Preparation

1. Backup existing data:

   # Backup all MongoDB databases
   mongodump --uri="mongodb://admin:password123@localhost:27017" --out=/backup/microservices
   

2. Document current API endpoints:

   • List all endpoints from each microservice
   • Document inter-service communication patterns
   • Identify Redis pub/sub channels in use

3. Review environment variables:

   • Consolidate environment variables
   • Update connection strings for external MongoDB and Redis

Phase 2: Deploy Modular Monolith

1. Configure environment:

   cd /path/to/monolith
   cp .env.example .env
   # Edit .env with MongoDB and Redis connection strings
   

2. Build and deploy:

   docker-compose up --build -d
   

3. Verify health:

   curl http://localhost:8000/health
   curl http://localhost:8000/api/v1/overview
   

Phase 3: Data Migration (if needed)

The modular monolith uses the same database structure as the microservices, so typically no data migration is needed. However, verify:

1. Database names match:

   • energy_dashboard_sensors
   • energy_dashboard_demand_response
   • digitalmente_ingestion

2. Collections are accessible:

   # Connect to MongoDB
   mongosh mongodb://admin:password123@mongodb-host:27017/?authSource=admin
   
   # Check databases
   show dbs
   
   # Verify collections in each database
   use energy_dashboard_sensors
   show collections
   

Phase 4: API Client Migration

Update API clients to point to the new monolith endpoint:

Before:

• Sensor API: http://api-gateway:8000/api/v1/sensors/*
• DR API: http://api-gateway:8000/api/v1/demand-response/*

After:

• All APIs: http://monolith:8000/api/v1/*

The API paths remain the same, only the host changes!

Phase 5: Decommission Microservices

Once the monolith is stable:

1. Stop microservices:

   cd /path/to/microservices
   docker-compose down
   

2. Keep backups for at least 30 days

3. Archive microservices code for reference

Benefits of the Migration

Operational Simplification

Aspect	Microservices	Modular Monolith	Improvement
Containers	8+ containers	1 container	87% reduction
Network calls	HTTP between services	In-process calls	~100x faster
Deployment complexity	Coordinate 8+ services	Single deployment	Much simpler
Monitoring	8+ health endpoints	1 health endpoint	Easier
Log aggregation	8+ log sources	1 log source	Simpler

Performance Improvements

1. Reduced latency:

   • Inter-service HTTP calls: ~10-50ms
   • Direct function calls: ~0.01-0.1ms
   • Improvement: 100-1000x faster

2. Reduced network overhead:

   • No HTTP serialization/deserialization
   • No network round-trips
   • No service discovery delays

3. Shared resources:

   • Single database connection pool
   • Shared Redis connection (if enabled)
   • Shared in-memory caches

Development Benefits

1. Easier debugging:

   • Single process to debug
   • Direct stack traces across modules
   • No distributed tracing needed

2. Simpler testing:

   • Test entire flow in one process
   • No need to mock HTTP calls
   • Integration tests run faster

3. Faster development:

   • Single application to run locally
   • Immediate code changes (with reload)
   • No service orchestration needed

Preserved Benefits from Microservices

Module Isolation

Each module maintains clear boundaries:

• Separate directory structure
• Own models and business logic
• Dedicated database (data isolation)
• Clear public interfaces

Independent Scaling (Future)

If needed, modules can be extracted back into microservices:

• Clean module boundaries make extraction easy
• Database per module already separated
• Event bus can switch to Redis pub/sub
• Direct calls can switch to HTTP calls

Team Organization

Teams can still own modules:

• Sensors team owns modules/sensors/
• DR team owns modules/demand_response/
• Clear ownership and responsibilities

Rollback Strategy

If you need to rollback to microservices:

1. Keep microservices code in the repository

2. Database unchanged: Both architectures use the same databases

3. Redeploy microservices:

   cd /path/to/microservices
   docker-compose up -d
   

4. Update API clients to point back to API Gateway

Monitoring and Observability

Health Checks

Single health endpoint:

curl http://localhost:8000/health

Returns:

{
  "service": "Energy Dashboard Monolith",
  "status": "healthy",
  "components": {
    "database": "healthy",
    "redis": "healthy",
    "event_bus": "healthy"
  },
  "modules": {
    "sensors": "loaded",
    "demand_response": "loaded",
    "data_ingestion": "loaded"
  }
}

Logging

All logs in one place:

# Docker logs
docker-compose logs -f monolith

# Application logs
docker-compose logs -f monolith | grep "ERROR"

Metrics

System overview endpoint:

curl http://localhost:8000/api/v1/overview

Common Migration Issues

Issue: Module Import Errors

Problem: ModuleNotFoundError: No module named 'src.modules'

Solution:

# Set PYTHONPATH
export PYTHONPATH=/app
# Or in docker-compose.yml
environment:
  - PYTHONPATH=/app

Issue: Database Connection Errors

Problem: Cannot connect to MongoDB

Solution:

1. Verify MongoDB is accessible:

   docker-compose exec monolith ping mongodb-host
   

2. Check connection string in .env
3. Ensure network connectivity

Issue: Redis Connection Errors

Problem: Redis connection failed but app should work

Solution: Redis is optional. Set in .env:

REDIS_ENABLED=false

Issue: Event Subscribers Not Receiving Events

Problem: Events published but subscribers not called

Solution: Ensure subscribers are registered before events are published:

# Register subscriber in lifespan startup
@asynccontextmanager
async def lifespan(app: FastAPI):
    # Subscribe before publishing
    event_bus.subscribe(EventTopics.ENERGY_DATA, handle_energy)
    yield

Testing the Migration

1. Functional Testing

Test each module's endpoints:

# Sensors
curl http://localhost:8000/api/v1/sensors/get
curl http://localhost:8000/api/v1/rooms

# Analytics
curl http://localhost:8000/api/v1/analytics/summary

# Health
curl http://localhost:8000/health

2. Load Testing

Compare performance:

# Microservices
ab -n 1000 -c 10 http://localhost:8000/api/v1/sensors/get

# Modular Monolith
ab -n 1000 -c 10 http://localhost:8000/api/v1/sensors/get

Expected: Modular monolith should be significantly faster.

3. WebSocket Testing

Test real-time features:

const ws = new WebSocket('ws://localhost:8000/api/v1/ws');
ws.onmessage = (event) => console.log('Received:', event.data);

FAQ

Q: Do I need to migrate the database?

A: No, the modular monolith uses the same database structure as the microservices.

Q: Can I scale individual modules?

A: Not independently. The entire monolith scales together. If you need independent scaling, consider keeping the microservices architecture or using horizontal scaling with load balancers.

Q: What happens to Redis pub/sub?

A: Replaced with an in-process event bus. Redis can still be used for caching if REDIS_ENABLED=true.

Q: Are the API endpoints the same?

A: Yes, the API paths remain identical. Only the host changes.

Q: Can I extract modules back to microservices later?

A: Yes, the modular structure makes it easy to extract modules back into separate services if needed.

Q: How do I add a new module?

A: See the "Adding a New Module" section in README.md.

Q: Is this suitable for production?

A: Yes, modular monoliths are production-ready and often more reliable than microservices for small-to-medium scale applications.

Next Steps

1. Deploy to staging and run full test suite
2. Monitor performance and compare with microservices
3. Gradual rollout to production (canary or blue-green deployment)
4. Decommission microservices after 30 days of stable operation
5. Update documentation and team training

Support

For issues or questions about the migration:

1. Check this guide and README.md
2. Review application logs: docker-compose logs monolith
3. Test health endpoint: curl http://localhost:8000/health
4. Contact the development team