feat: migrate storage from JSON files to SQLite
Some checks failed
Lint & Test / test (push) Failing after 28s

Replace 22 individual JSON store files with a single SQLite database
(data/ledgrab.db). All entity stores now use BaseSqliteStore backed by
SQLite with WAL mode, write-through caching, and thread-safe access.

- Add Database class with SQLite backup/restore API
- Add BaseSqliteStore as drop-in replacement for BaseJsonStore
- Convert all 16 entity stores to SQLite
- Move global settings (MQTT, external URL, auto-backup) to SQLite
  settings table
- Replace JSON backup/restore with SQLite snapshot backups (.db files)
- Remove partial export/import feature (backend + frontend)
- Update demo seed to write directly to SQLite
- Add "Backup Now" button to settings UI
- Remove StorageConfig file path fields (single database_file remains)
This commit is contained in:
2026-03-25 00:03:19 +03:00
parent 29fb944494
commit 9dfd2365f4
38 changed files with 941 additions and 880 deletions

81
TODO.md
View File

@@ -1,47 +1,42 @@
# Weather Source Implementation
# SQLite Migration
## Phase 1: Backend — Entity & Provider
## Phase 1: Infrastructure
- [x] Create `storage/database.py` — SQLite connection wrapper (WAL mode, thread-safe)
- [x] Create `storage/base_sqlite_store.py` — same public API as BaseJsonStore, backed by SQLite
- [x] Create `storage/migration.py` — auto-migrate JSON files to SQLite on first run
- [x] Add `database_file` to `StorageConfig` in config.py
- [x] Update demo mode path rewriting for database_file
- [x] `storage/weather_source.py` — WeatherSource dataclass
- [x] `storage/weather_source_store.py` — BaseJsonStore, CRUD, ID prefix `ws_`
- [x] `api/schemas/weather_sources.py` — Create/Update/Response Pydantic models
- [x] `api/routes/weather_sources.py` — REST CRUD + `POST /{id}/test` endpoint
- [x] `core/weather/weather_provider.py` — WeatherData, WeatherProvider ABC, OpenMeteoProvider, WMO_CONDITION_NAMES
- [x] `core/weather/weather_manager.py` — Ref-counted runtime pool, polls API, caches WeatherData
- [x] `config.py` — Add `weather_sources_file` to StorageConfig
- [x] `main.py` — Init store + manager, inject dependencies, shutdown save
- [x] `api/__init__.py` — Register router
- [x] `api/routes/backup.py` — Add to STORE_MAP
## Phase 2: Convert stores (one-by-one)
- [x] SyncClockStore
- [x] GradientStore
- [x] WeatherSourceStore
- [x] AutomationStore
- [x] ScenePresetStore
- [x] TemplateStore
- [x] PostprocessingTemplateStore
- [x] PatternTemplateStore
- [x] AudioTemplateStore
- [x] ColorStripProcessingTemplateStore
- [x] PictureSourceStore
- [x] AudioSourceStore
- [x] ValueSourceStore
- [x] DeviceStore
- [x] OutputTargetStore
- [x] ColorStripStore
## Phase 2: Backend — CSS Stream
## Phase 3: Update backup/restore
- [x] Refactor backup.py to read from SQLite (export/import/backup/restore)
- [x] Keep JSON backup format identical for compatibility
- [x] Update AutoBackupEngine to read from SQLite
- [x] Add Database to dependency injection
- [x] `core/processing/weather_stream.py` — WeatherColorStripStream with WMO palette mapping + temperature shift + thunderstorm flash
- [x] `core/processing/color_strip_stream_manager.py` — Register `"weather"` stream type + weather_manager dependency
- [x] `storage/color_strip_source.py` — WeatherColorStripSource dataclass + registry
- [x] `api/schemas/color_strip_sources.py` — Add `"weather"` to Literal + weather_source_id, temperature_influence fields
- [x] `core/processing/processor_manager.py` — Pass weather_manager through ProcessorDependencies
## Phase 3: Frontend — Weather Source Entity
- [x] `templates/modals/weather-source-editor.html` — Modal with provider select, lat/lon + "Use my location", update interval, test button
- [x] `static/js/features/weather-sources.ts` — Modal, CRUD, test (shows weather toast), clone, geolocation, CardSection delegation
- [x] `static/js/core/state.ts` — weatherSourcesCache + _cachedWeatherSources
- [x] `static/js/types.ts` — WeatherSource interface + ColorStripSource weather fields
- [x] `static/js/features/streams.ts` — Weather Sources CardSection + card renderer + tree nav
- [x] `templates/index.html` — Include modal template
- [x] `static/css/modal.css` — Weather location row styles
## Phase 4: Frontend — CSS Editor Integration
- [x] `static/js/features/color-strips.ts``"weather"` type, section map, handler, card renderer, populate dropdown
- [x] `static/js/core/icons.ts` — Weather icon in CSS type icons
- [x] `templates/modals/css-editor.html` — Weather section (EntitySelect for weather source, speed, temperature_influence)
## Phase 5: i18n + Build
- [x] `static/locales/en.json` — Weather source + CSS editor keys
- [x] `static/locales/ru.json` — Russian translations
- [x] `static/locales/zh.json` — Chinese translations
- [x] Lint: `ruff check` — passed
- [x] Build: `tsc --noEmit` + `npm run build` — passed
- [ ] Restart server + test
## Phase 4: Cleanup
- [ ] Remove individual `*_file` fields from StorageConfig (keep `database_file` only)
- [ ] Remove `atomic_write_json` usage from stores (still used by auto_backup settings)
- [ ] Remove `freeze_saves` from base_store (only `freeze_writes` needed)
- [ ] Remove BaseJsonStore (keep EntityNotFoundError — move to shared location)
- [ ] Update _save_all_stores to use _save_all() instead of _save(force=True)
- [ ] Update CLAUDE.md and server/CLAUDE.md documentation
- [ ] Remove `_json_key`/`_legacy_json_keys` references from old code
- [ ] Clean up test files to use Database fixture instead of file paths