DSMR hardening: restart-free config + decouple from telegram id

Config hot-reload (no restart):
- MqttManager.unsubscribe(); apply_dsmr_subscription(settings) re-applies the
  DSMR subscription (subscribe/unsubscribe/topic-change, fresh snapshot so the
  sample interval also takes effect).
- Called from lifespan AND PUT /api/config, so toggling DSMR ingest in the UI
  takes effect immediately without restarting the app.

Decouple ingest from the DSMR telegram id (overflows / resets to zero):
- Migration 20260624_12: drop UNIQUE(source_id), make recorded_at UNIQUE.
- Idempotency now keys on recorded_at (telegram timestamp); the table's own
  autoincrement PK is the stable identity. source_id kept only as a reference.
- Regression test: colliding telegram ids no longer drop new data.
This commit is contained in:
2026-06-24 11:02:32 +02:00
parent 78f3cc4776
commit 8d4f496ff4
12 changed files with 385 additions and 61 deletions
+18 -12
View File
@@ -22,27 +22,33 @@ from app.db import Base
class DsmrReading(Base):
"""One down-sampled DSMR telegram stored as a full JSON blob.
``recorded_at`` is a real indexed column (not inside the payload) so that
time-range queries are efficient. ``source_id`` holds the telegram's own
integer ``id`` and is used for idempotent upsert (skip if already present).
The entire telegram frame is stored verbatim in ``payload``; no field
allow-list is applied so future commodities (gas, heating, three-phase)
are accommodated without a schema change.
Identity & idempotency are **independent of the DSMR Reader's telegram id**
(that field overflows and must be manually reset to zero — a known DSMR
quirk — so relying on it for uniqueness risks silently dropping new data).
The table's own autoincrement ``id`` PK is the stable internal identity, and
``recorded_at`` (the telegram timestamp) is the UNIQUE de-duplication key: a
single P1 meter emits exactly one telegram per timestamp.
``recorded_at`` is a real column (not inside the payload) so time-range
queries are efficient. The entire telegram frame is stored verbatim in
``payload``; no field allow-list is applied so future commodities (gas,
heating, three-phase) are accommodated without a schema change.
"""
__tablename__ = "dsmr_reading"
id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True)
# UTC timestamp of the sample — real column so it can be indexed efficiently.
# UTC timestamp of the sample — real column, UNIQUE (telegram-id-independent
# idempotency key). The unique index also serves time-range queries.
recorded_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, index=True
DateTime(timezone=True), nullable=False, unique=True
)
# Telegram's own auto-increment id (DSMR Reader assigns it). Used for
# idempotent ingestion: if the same telegram arrives twice we skip it.
# Nullable because some DSMR sources may not emit an id.
source_id: Mapped[int | None] = mapped_column(Integer, unique=True, nullable=True)
# Telegram's own id (DSMR Reader assigns it). Stored only as a reference /
# debugging aid — NOT used for uniqueness or idempotency (it overflows and
# gets reset to zero). Nullable because some DSMR sources may not emit one.
source_id: Mapped[int | None] = mapped_column(Integer, nullable=True)
# Full telegram frame as a JSON object; values are typically JSON strings
# (e.g. "20915.154") — callers must cast to Decimal before arithmetic.