feat: migrate storage from JSON files to SQLite
Some checks failed
Lint & Test / test (push) Failing after 28s
Some checks failed
Lint & Test / test (push) Failing after 28s
Replace 22 individual JSON store files with a single SQLite database (data/ledgrab.db). All entity stores now use BaseSqliteStore backed by SQLite with WAL mode, write-through caching, and thread-safe access. - Add Database class with SQLite backup/restore API - Add BaseSqliteStore as drop-in replacement for BaseJsonStore - Convert all 16 entity stores to SQLite - Move global settings (MQTT, external URL, auto-backup) to SQLite settings table - Replace JSON backup/restore with SQLite snapshot backups (.db files) - Remove partial export/import feature (backend + frontend) - Update demo seed to write directly to SQLite - Add "Backup Now" button to settings UI - Remove StorageConfig file path fields (single database_file remains)
This commit is contained in:
81
TODO.md
81
TODO.md
@@ -1,47 +1,42 @@
|
||||
# Weather Source Implementation
|
||||
# SQLite Migration
|
||||
|
||||
## Phase 1: Backend — Entity & Provider
|
||||
## Phase 1: Infrastructure
|
||||
- [x] Create `storage/database.py` — SQLite connection wrapper (WAL mode, thread-safe)
|
||||
- [x] Create `storage/base_sqlite_store.py` — same public API as BaseJsonStore, backed by SQLite
|
||||
- [x] Create `storage/migration.py` — auto-migrate JSON files to SQLite on first run
|
||||
- [x] Add `database_file` to `StorageConfig` in config.py
|
||||
- [x] Update demo mode path rewriting for database_file
|
||||
|
||||
- [x] `storage/weather_source.py` — WeatherSource dataclass
|
||||
- [x] `storage/weather_source_store.py` — BaseJsonStore, CRUD, ID prefix `ws_`
|
||||
- [x] `api/schemas/weather_sources.py` — Create/Update/Response Pydantic models
|
||||
- [x] `api/routes/weather_sources.py` — REST CRUD + `POST /{id}/test` endpoint
|
||||
- [x] `core/weather/weather_provider.py` — WeatherData, WeatherProvider ABC, OpenMeteoProvider, WMO_CONDITION_NAMES
|
||||
- [x] `core/weather/weather_manager.py` — Ref-counted runtime pool, polls API, caches WeatherData
|
||||
- [x] `config.py` — Add `weather_sources_file` to StorageConfig
|
||||
- [x] `main.py` — Init store + manager, inject dependencies, shutdown save
|
||||
- [x] `api/__init__.py` — Register router
|
||||
- [x] `api/routes/backup.py` — Add to STORE_MAP
|
||||
## Phase 2: Convert stores (one-by-one)
|
||||
- [x] SyncClockStore
|
||||
- [x] GradientStore
|
||||
- [x] WeatherSourceStore
|
||||
- [x] AutomationStore
|
||||
- [x] ScenePresetStore
|
||||
- [x] TemplateStore
|
||||
- [x] PostprocessingTemplateStore
|
||||
- [x] PatternTemplateStore
|
||||
- [x] AudioTemplateStore
|
||||
- [x] ColorStripProcessingTemplateStore
|
||||
- [x] PictureSourceStore
|
||||
- [x] AudioSourceStore
|
||||
- [x] ValueSourceStore
|
||||
- [x] DeviceStore
|
||||
- [x] OutputTargetStore
|
||||
- [x] ColorStripStore
|
||||
|
||||
## Phase 2: Backend — CSS Stream
|
||||
## Phase 3: Update backup/restore
|
||||
- [x] Refactor backup.py to read from SQLite (export/import/backup/restore)
|
||||
- [x] Keep JSON backup format identical for compatibility
|
||||
- [x] Update AutoBackupEngine to read from SQLite
|
||||
- [x] Add Database to dependency injection
|
||||
|
||||
- [x] `core/processing/weather_stream.py` — WeatherColorStripStream with WMO palette mapping + temperature shift + thunderstorm flash
|
||||
- [x] `core/processing/color_strip_stream_manager.py` — Register `"weather"` stream type + weather_manager dependency
|
||||
- [x] `storage/color_strip_source.py` — WeatherColorStripSource dataclass + registry
|
||||
- [x] `api/schemas/color_strip_sources.py` — Add `"weather"` to Literal + weather_source_id, temperature_influence fields
|
||||
- [x] `core/processing/processor_manager.py` — Pass weather_manager through ProcessorDependencies
|
||||
|
||||
## Phase 3: Frontend — Weather Source Entity
|
||||
|
||||
- [x] `templates/modals/weather-source-editor.html` — Modal with provider select, lat/lon + "Use my location", update interval, test button
|
||||
- [x] `static/js/features/weather-sources.ts` — Modal, CRUD, test (shows weather toast), clone, geolocation, CardSection delegation
|
||||
- [x] `static/js/core/state.ts` — weatherSourcesCache + _cachedWeatherSources
|
||||
- [x] `static/js/types.ts` — WeatherSource interface + ColorStripSource weather fields
|
||||
- [x] `static/js/features/streams.ts` — Weather Sources CardSection + card renderer + tree nav
|
||||
- [x] `templates/index.html` — Include modal template
|
||||
- [x] `static/css/modal.css` — Weather location row styles
|
||||
|
||||
## Phase 4: Frontend — CSS Editor Integration
|
||||
|
||||
- [x] `static/js/features/color-strips.ts` — `"weather"` type, section map, handler, card renderer, populate dropdown
|
||||
- [x] `static/js/core/icons.ts` — Weather icon in CSS type icons
|
||||
- [x] `templates/modals/css-editor.html` — Weather section (EntitySelect for weather source, speed, temperature_influence)
|
||||
|
||||
## Phase 5: i18n + Build
|
||||
|
||||
- [x] `static/locales/en.json` — Weather source + CSS editor keys
|
||||
- [x] `static/locales/ru.json` — Russian translations
|
||||
- [x] `static/locales/zh.json` — Chinese translations
|
||||
- [x] Lint: `ruff check` — passed
|
||||
- [x] Build: `tsc --noEmit` + `npm run build` — passed
|
||||
- [ ] Restart server + test
|
||||
## Phase 4: Cleanup
|
||||
- [ ] Remove individual `*_file` fields from StorageConfig (keep `database_file` only)
|
||||
- [ ] Remove `atomic_write_json` usage from stores (still used by auto_backup settings)
|
||||
- [ ] Remove `freeze_saves` from base_store (only `freeze_writes` needed)
|
||||
- [ ] Remove BaseJsonStore (keep EntityNotFoundError — move to shared location)
|
||||
- [ ] Update _save_all_stores to use _save_all() instead of _save(force=True)
|
||||
- [ ] Update CLAUDE.md and server/CLAUDE.md documentation
|
||||
- [ ] Remove `_json_key`/`_legacy_json_keys` references from old code
|
||||
- [ ] Clean up test files to use Database fixture instead of file paths
|
||||
|
||||
Reference in New Issue
Block a user