6 Commits

Author	SHA1	Message	Date
rafaeldpsilva	38fb3e6e96	Add dynamic energy data collections per building ... Energy data is now stored in separate collections for each SLGs/Community/Building directory. Collections are created and indexed dynamically. Metadata and API queries have been updated to support this structure.	2025-10-08 14:05:00 +01:00
rafaeldpsilva	41b8753a92	Add scan cache tracking and improve health checks ... - Track scanned FTP directories in MongoDB to avoid redundant scans - Add endpoints to view and clear scan cache - Improve health check logic for better startup and error reporting - Add readiness endpoint for deployment probes - Add test script for health check improvements - Increase logging verbosity for debugging	2025-09-22 15:12:40 +01:00
rafaeldpsilva	aa07347604	Implement iterative FTP scan and skip logic with processed file cache ... - Add iterative directory scanning to prevent infinite recursion - Cache processed files in memory to avoid redundant database lookups - Skip already processed files using cache and database fallback - Add tests for skip logic and iterative scan behavior - Change logging for MongoDB connection and file storage to debug level - Clean up FastAPI app and remove redundant docstrings	2025-09-12 13:43:21 +01:00
rafaeldpsilva	2932e0a424	Switch to PyMongo, update config and requirements, fix FTP extension ... typo - Replace Motor (async) with PyMongo (sync) in database manager - Update environment variable names for FTP and MongoDB config - Remove unused dependencies from requirements.txt - Fix file extension typo: .slg_v2 → .sgl_v2 throughout code and docs - Add debug prints for MongoDB env vars in config - Update FTP monitor to use correct file extension and PyMongo - Adjust FastAPI descriptions for new extension	2025-09-11 11:45:19 +01:00
rafaeldpsilva	b7e734e0d2	Simplify data ingestion service	2025-09-10 15:47:10 +01:00
rafaeldpsilva	13556347b0	Simplify data ingestion service	2025-09-10 15:21:53 +01:00