Compare commits

...
39 Commits
Author SHA1 Message Date
tliu93 d5623b9fcb M8-R10: document meter lifecycle recovery runbook
frontend / frontend (push) Successful in 47s
pytest / test (push) Failing after 3m55s
2026-08-24 18:37:46 +02:00
tliu93 b1b6a309cb M8-R09: add meter close unbind and transfer workflows 2026-08-24 18:37:46 +02:00
tliu93 8dc3f71aaf M8-R08: add atomic meter close and binding transfer 2026-08-24 18:37:46 +02:00
tliu93 2be4f78f8a M8-R07B: align roadmap with direct serial Compose configuration 2026-08-24 06:45:31 +02:00
tliu93 1a958cb102 M8-R07A: remove obsolete WarmteLink overlay references 2026-08-24 06:45:31 +02:00
tliu93 c851bad829 M8-R07: integrate WarmteLink device access into Compose 2026-08-24 06:45:31 +02:00
tliu93 09abe05f66 M8-R06: refresh Tibber prices after startup and configuration 2026-08-24 06:45:31 +02:00
tliu93 0924e8df52 M8-R05: recover WarmteLink ingestion and discovery automatically 2026-08-24 06:45:31 +02:00
tliu93 72ac7e7300 M8-R04: interpret WarmteLink timestamps as Amsterdam wall time 2026-08-24 06:45:31 +02:00
tliu93 18083822ea M8-R04A: fix fake discovery receive clock 2026-08-24 06:45:31 +02:00
tliu93 3beeb5a461 M8-R03B: keep meter swap recompute in caller transaction 2026-08-24 06:45:31 +02:00
tliu93 631b14e2ec M8-R03: hand off channel binding during meter swap 2026-08-24 06:45:31 +02:00
tliu93 231c340ea6 M8-R03A: stabilize thermal recompute time-range tests 2026-08-24 06:45:30 +02:00
tliu93 c24b6684cc M8-R02A: reconcile DSMR runtime after source CRUD 2026-08-24 06:45:30 +02:00
tliu93 d9e82038dc M8-R02: report DSMR MQTT source health 2026-08-24 06:45:30 +02:00
tliu93 e59c192097 M8-R01: add configurable MQTT client identity 2026-08-24 00:44:28 +02:00
tliu93 2a47dab272 M8-R01A: align lifespan DSMR reconcile mock contract 2026-08-24 00:35:24 +02:00
tliu93 9db7f63274 M8-R00: align tariff expose tests with source bindings 2026-08-24 00:15:48 +02:00
tliu93 ebf96de4f1 M8-T20: document and validate WarmteLink deployment 2026-08-23 21:28:26 +02:00
tliu93 5b9d60e80a M8-T19: add scope-aware energy contract and cost UI 2026-08-23 21:22:06 +02:00
tliu93 963e43e3e4 M8-T18: add source and multi-commodity meter UI 2026-08-23 21:22:06 +02:00
tliu93 39c11ae606 M8-T17: add Home Assistant thermal entities 2026-08-23 21:22:06 +02:00
tliu93 3eec701448 M8-T16: add thermal meter cost APIs 2026-08-23 21:22:06 +02:00
tliu93 489e5b596a M8-T15: add thermal meter cost engine 2026-08-23 21:22:06 +02:00
tliu93 567ddb9779 M8-T14: add generic meter cost periods 2026-08-23 21:22:06 +02:00
tliu93 0fb51d338c M8-T13: add district heating pricing profile 2026-08-23 21:22:06 +02:00
tliu93 b812d5ac46 M8-T12: scope energy contracts 2026-08-23 21:22:06 +02:00
tliu93 a9458394f2 M8-T11: add WarmteLink discovery and history API 2026-08-23 21:22:06 +02:00
tliu93 4884a19e3d M8-T10: add WarmteLink serial worker manager 2026-08-23 21:22:06 +02:00
tliu93 afe653bafa M8-T09: add WarmteLink ingest state machine 2026-08-23 21:22:06 +02:00
tliu93 25a08c47a4 M8-T08: add WarmteLink reading schema 2026-08-23 21:22:05 +02:00
tliu93 ffc693e995 M8-T07: extract reusable P1 parser 2026-08-23 21:22:05 +02:00
tliu93 5855fff451 M8-T06: add meter source management API 2026-08-23 21:22:05 +02:00
tliu93 1ea2f659e0 M8-T05: bind electricity costs to source bindings 2026-08-23 21:22:05 +02:00
tliu93 2e125dbd53 M8-T04: reconcile DSMR ingest from meter sources 2026-08-23 21:22:05 +02:00
tliu93 28486a83c7 M8-T03: adopt DSMR history into meter sources 2026-08-23 21:22:05 +02:00
tliu93 a78401c2ef M8-T02: add source profiles and binding services 2026-08-23 21:22:05 +02:00
tliu93 009856a50d M8-T01: add meter source identity schema 2026-08-23 21:22:05 +02:00
tliu93 43c2ddce1a M8: clarify Alembic schema migration plan 2026-08-22 22:02:14 +02:00
107 changed files with 22580 additions and 1418 deletions
+7
View File
@@ -7,6 +7,11 @@ APP_DATABASE_URL=sqlite:////app/data/app.db
AUTH_BOOTSTRAP_USERNAME=admin
AUTH_BOOTSTRAP_PASSWORD=change-me
# Required by Docker Compose for the WarmteLink serial device. Set these only in
# your local .env; use a stable /dev/serial/by-id path and its numeric host GID.
# WARMTELINK_DEVICE_PATH=/dev/serial/by-id/<stable-by-id-name>
# WARMTELINK_SERIAL_GID=<host-serial-gid>
# Optional: runtime overrides.
# Leave these commented out to use the application's built-in defaults.
# APP_DEBUG=
@@ -40,6 +45,8 @@ MQTT_BROKER_PORT=1883
MQTT_USERNAME=
MQTT_PASSWORD=
MQTT_TLS_ENABLED=false
# MQTT_CLIENT_ID must be a non-empty ASCII slug; use a distinct value per deployment.
MQTT_CLIENT_ID=home-automation
# Optional: Home Assistant MQTT Discovery.
# Requires MQTT_ENABLED=true and a running MQTT broker.
+20 -2
View File
@@ -21,6 +21,8 @@
- **通用电价合同层**YAML profile 定合同结构(manual 固定/双费率 / tibber 动态电价);`EnergyContract`+`EnergyContractVersion` 存 UI 可填的数值,改价加新版本旧版本保留;price strategy 按 kind 出价
- **实时买卖电费计算**:每 15 分钟按寄存器差值(`_1`=dal/低、`_2`=normal/高)× 买/卖价算计量电费,快照不可变;日/月/年汇总加固定费减 heffingskorting
- **反哺 Home Assistant Energy**:当前买/卖价 + 累计买电支出/卖电收入(`total_increasing`)发成 HA 实体,可直接挂 HA Energy 仪表盘
- **多数据源 Meter 与 WarmteLink**DSMR MQTT 与只读 WarmteLink P1 serial source 统一为 Source → Channel → Binding → MeterWarmteLink 提供 heating `GJ` 与 hot-water `m³` 的 Decimal history、质量与重连
- **热力合同与成本**electricity / thermal scope 可各有一个 active 合同;热力按 15 分钟账本计算 variable、fixed 与 all-in 成本,并可按需暴露给 HA
- pytest 测试与 OpenAPI 导出脚本
- Docker / Compose 部署入口
@@ -44,6 +46,8 @@
- 电价合同(`energy_contract` 表)与版本(`energy_contract_version` 表,values JSON
- Tibber 15 分钟电价缓存(`tibber_price` 表,不可变)
- 每 15 分钟计量电费(`energy_cost_period` 表,快照价,不可变)
- meter source、channel 与 binding`meter_source``meter_source_channel``meter_source_binding`
- WarmteLink scalar 历史(`warmtelink_reading`)与热力 15 分钟成本账本(`meter_cost_period`
配置层只保留一个数据库环境变量:
@@ -55,7 +59,7 @@
python -m scripts.run_migrations
```
该命令会通过 Alembic 将 `app.db` 初始化或升级到最新 head含全部表,包括 M5 新增的 `modbus_device``modbus_reading``exposed_entity_toggle`,以及 M6 新增的 `dsmr_reading``energy_contract``energy_contract_version``tibber_price``energy_cost_period`)。
该命令会通过 Alembic 将 `app.db` 初始化或升级到最新 head包括 Modbus、DSMR、Source/Channel/Binding、WarmteLink、electricity/thermal 合同与成本账本)。
## 当前目录
@@ -63,7 +67,7 @@ python -m scripts.run_migrations
- `app/`: FastAPI 应用代码(包含 JSON API、业务服务、数据模型)
- `frontend/`: React SPA 前端(Vite + React + TypeScript + Mantine
- `alembic_app/`: App DB 的 Alembic migration 环境(管理所有表,含 M5 新增的 `modbus_device``modbus_reading``exposed_entity_toggle`,以及 M6 新增的 `dsmr_reading``energy_contract``energy_contract_version``tibber_price``energy_cost_period`
- `alembic_app/`: App DB 的唯一 Alembic migration 环境(管理所有 app 表,包括 Modbus、DSMR、Meter source、WarmteLink、合同与成本账本
- `tests/`: pytest 测试
- `docs/`: 当前系统说明文档
- `scripts/`: 辅助脚本,例如 OpenAPI 导出
@@ -551,6 +555,20 @@ python scripts/export_openapi.py
- `docker-compose.dev.yml`:本地开发显式叠加层——追加 `build: .`、独立 project /
容器名(`-dev` 后缀)、暴露 8001,并把 DB 指向挂载的 `./data` 副本,可与生产栈在同一台机器上并存
WarmteLink serial access is configured directly by both Compose combinations. Before starting either
one, set these host-specific values in your uncommitted local `.env` (use a stable `/dev/serial/by-id/...`
path, never a transient `/dev/ttyUSB*` name):
```dotenv
WARMTELINK_DEVICE_PATH=/dev/serial/by-id/<stable-by-id-name>
WARMTELINK_SERIAL_GID=<host-serial-gid>
```
Only `app` receives the device as `/dev/warmtelink:rw` and the serial group; `migration` does not.
The app remains non-root, non-privileged, and has no added capabilities. One physical serial port may
have only one owner: stop the app before running the Pre-M8 P1 probe. Never remove `./data`, databases,
or volumes while changing this configuration.
本地开发启动方式(显式叠加 dev 层):
```bash
+1
View File
@@ -20,6 +20,7 @@ from app.models.energy import ( # noqa: F401
TibberPrice,
EnergyCostPeriod,
)
from app.models.meter_source import MeterSource, MeterSourceBinding, MeterSourceChannel # noqa: F401
config = context.config
@@ -0,0 +1,107 @@
"""add protocol-agnostic meter source, channel, and binding tables
Revision ID: 20260822_15_meter_sources
Revises: 20260625_14_meter_uuid
Create Date: 2026-08-22 00:00:00.000000
This revision is additive on upgrade. It deliberately does not backfill
existing DSMR data; that adoption is a later, separately audited migration.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "20260822_15_meter_sources"
down_revision: Union[str, None] = "20260625_14_meter_uuid"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.create_table(
"meter_source",
sa.Column("id", sa.Integer(), autoincrement=True, nullable=False),
sa.Column("uuid", sa.String(length=36), nullable=False),
sa.Column("name", sa.String(length=255), nullable=False),
sa.Column("kind", sa.String(length=64), nullable=False),
sa.Column("enabled", sa.Boolean(), nullable=False),
sa.Column("config", sa.JSON(), nullable=False),
sa.Column("status", sa.String(length=32), nullable=False),
sa.Column("last_seen_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("last_error", sa.String(length=1024), nullable=True),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
sa.PrimaryKeyConstraint("id"),
sa.UniqueConstraint("uuid", name="uq_meter_source_uuid"),
)
op.create_index("ix_meter_source_kind_enabled", "meter_source", ["kind", "enabled"])
op.create_table(
"meter_source_channel",
sa.Column("id", sa.Integer(), autoincrement=True, nullable=False),
sa.Column("uuid", sa.String(length=36), nullable=False),
sa.Column("source_id", sa.Integer(), nullable=False),
sa.Column("channel_key", sa.String(length=128), nullable=False),
sa.Column("label", sa.String(length=255), nullable=False),
sa.Column("suggested_commodity", sa.String(length=32), nullable=True),
sa.Column("unit", sa.String(length=32), nullable=False),
sa.Column("device_type", sa.String(length=64), nullable=True),
sa.Column("fingerprint", sa.String(length=64), nullable=True),
sa.Column("latest_value", sa.Numeric(precision=20, scale=6), nullable=True),
sa.Column("latest_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("latest_quality", sa.String(length=32), nullable=True),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
sa.ForeignKeyConstraint(["source_id"], ["meter_source.id"], ondelete="RESTRICT"),
sa.PrimaryKeyConstraint("id"),
sa.UniqueConstraint("uuid", name="uq_meter_source_channel_uuid"),
sa.UniqueConstraint("source_id", "channel_key", name="uq_meter_source_channel_source_key"),
)
op.create_index("ix_meter_source_channel_source_id", "meter_source_channel", ["source_id"])
op.create_table(
"meter_source_binding",
sa.Column("id", sa.Integer(), autoincrement=True, nullable=False),
sa.Column("uuid", sa.String(length=36), nullable=False),
sa.Column("meter_id", sa.Integer(), nullable=False),
sa.Column("channel_id", sa.Integer(), nullable=False),
sa.Column("started_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("ended_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
sa.ForeignKeyConstraint(["meter_id"], ["meter.id"], ondelete="RESTRICT"),
sa.ForeignKeyConstraint(
["channel_id"], ["meter_source_channel.id"], ondelete="RESTRICT"
),
sa.PrimaryKeyConstraint("id"),
sa.UniqueConstraint("uuid", name="uq_meter_source_binding_uuid"),
)
op.create_index("ix_meter_source_binding_meter_id", "meter_source_binding", ["meter_id"])
op.create_index("ix_meter_source_binding_channel_id", "meter_source_binding", ["channel_id"])
with op.batch_alter_table("energy_cost_period", schema=None) as batch_op:
batch_op.add_column(sa.Column("source_binding_id", sa.Integer(), nullable=True))
batch_op.create_foreign_key(
"fk_energy_cost_period_source_binding_id",
"meter_source_binding",
["source_binding_id"],
["id"],
ondelete="RESTRICT",
)
def downgrade() -> None:
with op.batch_alter_table("energy_cost_period", schema=None) as batch_op:
batch_op.drop_constraint("fk_energy_cost_period_source_binding_id", type_="foreignkey")
batch_op.drop_column("source_binding_id")
op.drop_index("ix_meter_source_binding_channel_id", table_name="meter_source_binding")
op.drop_index("ix_meter_source_binding_meter_id", table_name="meter_source_binding")
op.drop_table("meter_source_binding")
op.drop_index("ix_meter_source_channel_source_id", table_name="meter_source_channel")
op.drop_table("meter_source_channel")
op.drop_index("ix_meter_source_kind_enabled", table_name="meter_source")
op.drop_table("meter_source")
@@ -0,0 +1,269 @@
"""adopt historical DSMR rows into the source and binding model
Revision ID: 20260822_16_dsmr_source_adoption
Revises: 20260822_15_meter_sources
Create Date: 2026-08-22 00:00:00.000000
The upgrade is deliberately data-preserving: it creates one migration-owned
DSMR source/channel, moves the telegram identifier to ``telegram_id``, and
audits every reading and cost row before committing. Old ``app_config`` rows,
payload JSON, and cost snapshots are never deleted or rewritten.
"""
from __future__ import annotations
import uuid
from datetime import datetime, timezone
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "20260822_16_dsmr_source_adoption"
down_revision: Union[str, None] = "20260822_15_meter_sources"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def _as_bool(value: str | None) -> bool:
return value is not None and value.strip().lower() in {"1", "true", "yes", "on"}
def _iso_now() -> str:
return datetime.now(tz=timezone.utc).replace(tzinfo=None).isoformat(sep=" ")
def _config(connection: sa.Connection) -> dict[str, object]:
rows = connection.execute(sa.text("SELECT key, value FROM app_config")).all()
values = {str(key): str(value) for key, value in rows}
# An unconfigured historical DSMR installation needs a disabled identity,
# not guessed connection details. Preserve every legacy value we model
# when any legacy DSMR/MQTT configuration was explicitly present.
legacy_keys = {
"MQTT_BROKER_HOST", "MQTT_BROKER_PORT", "MQTT_USERNAME", "MQTT_PASSWORD",
"MQTT_TLS_ENABLED", "DSMR_MQTT_TOPIC", "DSMR_TARIFF_TOPIC", "DSMR_SAMPLE_INTERVAL_S",
}
if not legacy_keys & values.keys():
return {}
return {
"broker_host": values.get("MQTT_BROKER_HOST", ""),
"broker_port": int(values.get("MQTT_BROKER_PORT", "1883")),
"username": values.get("MQTT_USERNAME", ""),
"password": values.get("MQTT_PASSWORD", ""),
"tls_enabled": _as_bool(values.get("MQTT_TLS_ENABLED")),
"topic": values.get("DSMR_MQTT_TOPIC", "dsmr/json"),
"tariff_topic": values.get("DSMR_TARIFF_TOPIC", "dsmr/meter-stats/electricity_tariff"),
"sample_interval_s": int(values.get("DSMR_SAMPLE_INTERVAL_S", "10")),
}
def _count(connection: sa.Connection, table: str) -> int:
return int(connection.execute(sa.text(f"SELECT COUNT(*) FROM {table}")).scalar_one())
def upgrade() -> None:
connection = op.get_bind()
readings_before = _count(connection, "dsmr_reading")
costs_before = _count(connection, "energy_cost_period")
sources_before = _count(connection, "meter_source")
channels_before = _count(connection, "meter_source_channel")
bindings_before = _count(connection, "meter_source_binding")
now = _iso_now()
# A source exists even without historical configuration/readings. It stays
# disabled unless the old explicit DSMR switch was enabled, so no broker or
# topic is guessed at runtime.
config = _config(connection)
source_result = connection.execute(
sa.text(
"INSERT INTO meter_source "
"(uuid, name, kind, enabled, config, status, last_seen_at, last_error, created_at, updated_at) "
"VALUES (:uuid, :name, 'dsmr_mqtt', :enabled, :config, 'unknown', NULL, NULL, :now, :now)"
),
{
"uuid": str(uuid.uuid4()),
"name": "Migrated DSMR source",
"enabled": _as_bool(
connection.execute(
sa.text("SELECT value FROM app_config WHERE key = 'DSMR_INGEST_ENABLED'")
).scalar_one_or_none()
),
"config": __import__("json").dumps(config),
"now": now,
},
)
source_id = source_result.lastrowid
if source_id is None:
raise RuntimeError("DSMR source adoption failed to create a source")
channel_result = connection.execute(
sa.text(
"INSERT INTO meter_source_channel "
"(uuid, source_id, channel_key, label, suggested_commodity, unit, device_type, fingerprint, "
"latest_value, latest_at, latest_quality, created_at, updated_at) "
"VALUES (:uuid, :source_id, 'electricity-total', 'DSMR electricity total', 'electricity', "
"'kWh', NULL, NULL, NULL, NULL, NULL, :now, :now)"
),
{"uuid": str(uuid.uuid4()), "source_id": source_id, "now": now},
)
channel_id = channel_result.lastrowid
if channel_id is None:
raise RuntimeError("DSMR source adoption failed to create an electricity channel")
if _count(connection, "meter_source") != sources_before + 1:
raise RuntimeError("DSMR source adoption source row-count audit failed")
if _count(connection, "meter_source_channel") != channels_before + 1:
raise RuntimeError("DSMR source adoption channel row-count audit failed")
# Rename/add while nullable, back-fill all rows, then make the FK non-null
# and replace the legacy timestamp-only uniqueness in a SQLite batch rebuild.
with op.batch_alter_table("dsmr_reading", schema=None) as batch_op:
batch_op.alter_column("source_id", new_column_name="telegram_id")
batch_op.add_column(sa.Column("meter_source_id", sa.Integer(), nullable=True))
connection.execute(
sa.text("UPDATE dsmr_reading SET meter_source_id = :source_id WHERE meter_source_id IS NULL"),
{"source_id": source_id},
)
with op.batch_alter_table("dsmr_reading", schema=None) as batch_op:
batch_op.drop_constraint("uq_dsmr_reading_recorded_at", type_="unique")
batch_op.alter_column("meter_source_id", existing_type=sa.Integer(), nullable=False)
batch_op.create_foreign_key(
"fk_dsmr_reading_meter_source_id", "meter_source", ["meter_source_id"], ["id"],
ondelete="RESTRICT",
)
batch_op.create_unique_constraint(
"uq_dsmr_reading_source_recorded_at", ["meter_source_id", "recorded_at"]
)
batch_op.create_index("ix_dsmr_reading_meter_source_id", ["meter_source_id"])
adopted_readings = int(
connection.execute(
sa.text("SELECT COUNT(*) FROM dsmr_reading WHERE meter_source_id = :source_id"),
{"source_id": source_id},
).scalar_one()
)
if adopted_readings != readings_before:
raise RuntimeError("DSMR source adoption reading source audit failed")
# Bind each electricity meter only where it overlaps the actual DSMR data.
data_window = connection.execute(
sa.text("SELECT MIN(recorded_at), MAX(recorded_at) FROM dsmr_reading")
).one()
expected_binding_count = 0
if data_window[0] is not None:
meters = connection.execute(
sa.text(
"SELECT id, started_at, ended_at FROM meter WHERE commodity = 'electricity' "
"ORDER BY started_at, id"
)
).all()
for meter_id, started_at, ended_at in meters:
# Intersect [meter start, meter end) with the inclusive historical
# samples. A closed boundary at the final sample remains valid for
# the preceding interval; an empty intersection gets no fake binding.
if started_at > data_window[1] or (ended_at is not None and ended_at <= data_window[0]):
continue
expected_binding_count += 1
binding_start = max(started_at, data_window[0])
binding_end = ended_at
connection.execute(
sa.text(
"INSERT INTO meter_source_binding "
"(uuid, meter_id, channel_id, started_at, ended_at, created_at, updated_at) "
"VALUES (:uuid, :meter_id, :channel_id, :started_at, :ended_at, :now, :now)"
),
{
"uuid": str(uuid.uuid4()), "meter_id": meter_id, "channel_id": channel_id,
"started_at": binding_start, "ended_at": binding_end, "now": now,
},
)
# A cost period may be linked only if exactly one binding covers both its
# start and end. Historical boundary/unknown rows remain auditable but are
# explicitly degraded instead of being silently attributed to a current meter.
periods = connection.execute(
sa.text("SELECT id, meter_id, period_start, degraded FROM energy_cost_period")
).all()
resolvable_normal_periods: dict[int, int] = {}
unresolved_period_ids: set[int] = set()
for period_id, meter_id, period_start, degraded_before in periods:
candidates = []
if meter_id is not None:
candidates = connection.execute(
sa.text(
"SELECT id FROM meter_source_binding "
"WHERE meter_id = :meter_id AND started_at <= :start "
"AND (ended_at IS NULL OR julianday(ended_at) > julianday(:start, '+15 minutes'))"
),
{"meter_id": meter_id, "start": period_start},
).all()
if len(candidates) == 1:
if not degraded_before:
resolvable_normal_periods[period_id] = candidates[0][0]
connection.execute(
sa.text("UPDATE energy_cost_period SET source_binding_id = :binding_id WHERE id = :id"),
{"binding_id": candidates[0][0], "id": period_id},
)
else:
unresolved_period_ids.add(period_id)
connection.execute(
sa.text(
"UPDATE energy_cost_period SET degraded = 1, source_binding_id = NULL WHERE id = :id"
),
{"id": period_id},
)
readings_after = _count(connection, "dsmr_reading")
costs_after = _count(connection, "energy_cost_period")
if readings_after != readings_before or costs_after != costs_before:
raise RuntimeError("DSMR source adoption row-count audit failed")
if _count(connection, "meter_source_binding") != bindings_before + expected_binding_count:
raise RuntimeError("DSMR source adoption binding row-count audit failed")
if int(
connection.execute(
sa.text("SELECT COUNT(*) FROM meter_source_binding WHERE channel_id = :channel_id"),
{"channel_id": channel_id},
).scalar_one()
) != expected_binding_count:
raise RuntimeError("DSMR source adoption binding channel audit failed")
for period_id, binding_id in resolvable_normal_periods.items():
bound, degraded = connection.execute(
sa.text("SELECT source_binding_id, degraded FROM energy_cost_period WHERE id = :id"),
{"id": period_id},
).one()
if bound != binding_id or degraded:
raise RuntimeError("DSMR source adoption resolvable cost audit failed")
if unresolved_period_ids:
unresolved_count = int(
connection.execute(
sa.text(
"SELECT COUNT(*) FROM energy_cost_period "
"WHERE id IN :period_ids AND (degraded != 1 OR source_binding_id IS NOT NULL)"
).bindparams(sa.bindparam("period_ids", expanding=True)),
{"period_ids": list(unresolved_period_ids)},
).scalar_one()
)
if unresolved_count:
raise RuntimeError("DSMR source adoption unresolved cost audit failed")
orphan_rows = connection.execute(sa.text("PRAGMA foreign_key_check")).all()
if orphan_rows:
raise RuntimeError("DSMR source adoption foreign-key audit failed")
normal_unbound = int(
connection.execute(
sa.text("SELECT COUNT(*) FROM energy_cost_period WHERE degraded = 0 AND source_binding_id IS NULL")
).scalar_one()
)
if normal_unbound:
raise RuntimeError(f"DSMR source adoption left {normal_unbound} normal cost period(s) unbound")
if _count(connection, "meter_source") < 1:
raise RuntimeError("DSMR source adoption source audit failed")
def downgrade() -> None:
# Schema-only downgrade for isolated test databases. It intentionally does
# not delete migration-created source/channel/binding rows.
with op.batch_alter_table("dsmr_reading", schema=None) as batch_op:
batch_op.drop_index("ix_dsmr_reading_meter_source_id")
batch_op.drop_constraint("uq_dsmr_reading_source_recorded_at", type_="unique")
batch_op.drop_constraint("fk_dsmr_reading_meter_source_id", type_="foreignkey")
batch_op.drop_column("meter_source_id")
batch_op.alter_column("telegram_id", new_column_name="source_id")
batch_op.create_unique_constraint("uq_dsmr_reading_recorded_at", ["recorded_at"])
@@ -0,0 +1,52 @@
"""add normalized WarmteLink scalar reading history
Revision ID: 20260822_17_warmtelink_readings
Revises: 20260822_16_dsmr_source_adoption
Create Date: 2026-08-22 00:00:00.000000
The upgrade is additive: existing business rows are neither changed nor
removed. The downgrade is schema-only and is exercised only on isolated test
databases.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "20260822_17_warmtelink_readings"
down_revision: Union[str, None] = "20260822_16_dsmr_source_adoption"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.create_table(
"warmtelink_reading",
sa.Column("id", sa.Integer(), autoincrement=True, nullable=False),
sa.Column("channel_id", sa.Integer(), nullable=False),
sa.Column("recorded_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("received_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("value", sa.Numeric(precision=15, scale=3), nullable=False),
sa.Column("unit", sa.String(length=32), nullable=False),
sa.Column("quality", sa.String(length=32), nullable=False),
sa.Column("equipment_fingerprint", sa.String(length=64), nullable=False),
sa.CheckConstraint(
"quality IN ('valid', 'invalid', 'unverifiable')",
name="ck_warmtelink_reading_quality",
),
sa.ForeignKeyConstraint(
["channel_id"], ["meter_source_channel.id"], ondelete="RESTRICT"
),
sa.PrimaryKeyConstraint("id"),
sa.UniqueConstraint(
"channel_id", "recorded_at", name="uq_warmtelink_reading_channel_recorded_at"
),
)
op.create_index("ix_warmtelink_reading_recorded_at", "warmtelink_reading", ["recorded_at"])
def downgrade() -> None:
op.drop_index("ix_warmtelink_reading_recorded_at", table_name="warmtelink_reading")
op.drop_table("warmtelink_reading")
@@ -0,0 +1,107 @@
"""add a billing scope to energy contracts
Revision ID: 20260822_18_contract_scopes
Revises: 20260822_17_warmtelink_readings
Create Date: 2026-08-22 00:00:00.000000
The upgrade preserves every existing contract, version and cost row. Existing
contracts predate scopes and therefore deterministically belong to electricity.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "20260822_18_contract_scopes"
down_revision: Union[str, None] = "20260822_17_warmtelink_readings"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def _count(connection: sa.Connection, table: str) -> int:
return int(connection.execute(sa.text(f"SELECT COUNT(*) FROM {table}")).scalar_one())
def _orphan_count(connection: sa.Connection) -> int:
version_orphans = connection.execute(
sa.text(
"SELECT COUNT(*) FROM energy_contract_version v "
"LEFT JOIN energy_contract c ON c.id = v.contract_id WHERE c.id IS NULL"
)
).scalar_one()
cost_orphans = connection.execute(
sa.text(
"SELECT COUNT(*) FROM energy_cost_period p "
"LEFT JOIN energy_contract_version v ON v.id = p.contract_version_id "
"WHERE p.contract_version_id IS NOT NULL AND v.id IS NULL"
)
).scalar_one()
return int(version_orphans) + int(cost_orphans)
def _audit_scope_upgrade(connection: sa.Connection, before: dict[str, int], orphan_before: int) -> None:
after = {table: _count(connection, table) for table in before}
if after != before:
raise RuntimeError("contract scope migration row-count audit failed")
if _orphan_count(connection) != orphan_before:
raise RuntimeError("contract scope migration FK audit failed")
invalid_scope_count = connection.execute(
sa.text("SELECT COUNT(*) FROM energy_contract WHERE scope IS NULL OR scope != 'electricity'")
).scalar_one()
if invalid_scope_count:
raise RuntimeError("contract scope migration backfill audit failed")
# Kept on Alembic's Config attributes rather than an environment switch so
# isolated migration tests can deterministically exercise the rollback
# boundary without changing production behavior.
failure_injector = op.get_context().config.attributes.get("m8_t12_post_ddl_audit_failure")
if callable(failure_injector):
failure_injector()
def _apply_scope_schema() -> None:
# SQLite batch mode reconstructs the table. The server default gives every
# historical row its deterministic value during reconstruction.
with op.batch_alter_table("energy_contract", schema=None) as batch_op:
batch_op.add_column(
sa.Column("scope", sa.String(length=32), nullable=False, server_default="electricity")
)
batch_op.create_index("ix_energy_contract_scope", ["scope"])
def upgrade() -> None:
connection = op.get_bind()
before = {
table: _count(connection, table)
for table in ("energy_contract", "energy_contract_version", "energy_cost_period")
}
orphan_before = _orphan_count(connection)
if connection.dialect.name != "sqlite":
_apply_scope_schema()
_audit_scope_upgrade(connection, before, orphan_before)
return
# Alembic marks SQLite batch DDL as non-transactional. SQLite itself can
# nevertheless atomically roll back CREATE/COPY/DROP/RENAME when an
# explicit transaction owns the complete batch operation. Keep the audit
# inside that boundary so a failed audit cannot strand a revision-17 DB
# with a revision-18 table shape.
connection.exec_driver_sql("BEGIN IMMEDIATE")
try:
_apply_scope_schema()
_audit_scope_upgrade(connection, before, orphan_before)
except BaseException:
connection.exec_driver_sql("ROLLBACK")
raise
else:
connection.exec_driver_sql("COMMIT")
def downgrade() -> None:
# Schema reversibility is only exercised against isolated temporary test DBs.
with op.batch_alter_table("energy_contract", schema=None) as batch_op:
batch_op.drop_index("ix_energy_contract_scope")
batch_op.drop_column("scope")
@@ -0,0 +1,95 @@
"""add generic commodity-scoped meter cost periods
Revision ID: 20260822_19_meter_cost_periods
Revises: 20260822_18_contract_scopes
Create Date: 2026-08-22 00:00:00.000000
This additive migration creates a separate audit ledger for non-electricity
meter costs. It deliberately does not alter, migrate, or delete rows from the
existing electricity-only energy_cost_period table.
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "20260822_19_meter_cost_periods"
down_revision: Union[str, None] = "20260822_18_contract_scopes"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
class ExactDecimal(sa.TypeDecorator):
"""Use SQLite text storage while retaining Numeric semantics elsewhere."""
impl = sa.Numeric
cache_ok = True
def __init__(self, precision: int, scale: int) -> None:
self.precision = precision
self.scale = scale
super().__init__(precision=precision, scale=scale)
def load_dialect_impl(self, dialect):
if dialect.name == "sqlite":
return dialect.type_descriptor(sa.String(self.precision + 2))
return dialect.type_descriptor(sa.Numeric(self.precision, self.scale, asdecimal=True))
def upgrade() -> None:
op.create_table(
"meter_cost_period",
sa.Column("id", sa.Integer(), autoincrement=True, nullable=False),
sa.Column("commodity", sa.String(length=32), nullable=False),
sa.Column("period_start", sa.DateTime(timezone=True), nullable=False),
sa.Column("period_end", sa.DateTime(timezone=True), nullable=False),
sa.Column("meter_id", sa.Integer(), nullable=True),
sa.Column("source_binding_id", sa.Integer(), nullable=True),
sa.Column("contract_version_id", sa.Integer(), nullable=True),
# SQLite NUMERIC coercion binds Decimal values as binary floats. Store
# fixed-width decimal text there, while retaining Numeric semantics on
# other supported dialects.
sa.Column("quantity", ExactDecimal(15, 6), nullable=False),
sa.Column("cost", ExactDecimal(15, 9), nullable=False),
sa.Column("currency", sa.String(length=8), nullable=False),
sa.Column("cost_breakdown", sa.JSON(), nullable=False),
sa.Column("pricing_snapshot", sa.JSON(), nullable=False),
sa.Column("quality", sa.String(length=32), nullable=False),
sa.Column("degraded", sa.Boolean(), nullable=False, server_default=sa.false()),
sa.Column("degraded_reason", sa.String(length=255), nullable=True),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
sa.ForeignKeyConstraint(["meter_id"], ["meter.id"], ondelete="RESTRICT"),
sa.ForeignKeyConstraint(
["source_binding_id"], ["meter_source_binding.id"], ondelete="RESTRICT"
),
sa.ForeignKeyConstraint(
["contract_version_id"], ["energy_contract_version.id"], ondelete="RESTRICT"
),
sa.CheckConstraint(
"degraded OR (meter_id IS NOT NULL AND source_binding_id IS NOT NULL "
"AND contract_version_id IS NOT NULL)",
name="ck_meter_cost_period_normal_audit_links",
),
sa.CheckConstraint("period_end > period_start", name="ck_meter_cost_period_positive_interval"),
sa.CheckConstraint(
"NOT degraded OR (degraded_reason IS NOT NULL AND length(trim(degraded_reason)) > 0)",
name="ck_meter_cost_period_degraded_reason",
),
sa.PrimaryKeyConstraint("id"),
sa.UniqueConstraint("commodity", "period_start", name="uq_meter_cost_period_commodity_start"),
)
op.create_index(
"ix_meter_cost_period_commodity_start", "meter_cost_period", ["commodity", "period_start"]
)
op.create_index(
"ix_meter_cost_period_source_binding_id", "meter_cost_period", ["source_binding_id"]
)
def downgrade() -> None:
op.drop_index("ix_meter_cost_period_source_binding_id", table_name="meter_cost_period")
op.drop_index("ix_meter_cost_period_commodity_start", table_name="meter_cost_period")
op.drop_table("meter_cost_period")
+14 -2
View File
@@ -9,7 +9,7 @@ from sqlalchemy.orm import Session
from app.api.routes.api.deps import require_csrf, require_session
from app.config import Settings, get_settings
from app.dependencies import get_app_settings, get_db
from app.integrations.mqtt import MQTT_SETTINGS_KEYS, mqtt_manager
from app.integrations.mqtt import MQTT_SETTINGS_KEYS, mqtt_manager, mqtt_test_client_id
from app.schemas.config import (
ConfigField,
ConfigResponse,
@@ -22,6 +22,7 @@ from app.schemas.config import (
from app.services.auth import AuthenticatedSession
from app.services.config_page import ConfigSaveError, build_config_sections, save_config_updates
from app.services.email import EmailConfigurationError, EmailDeliveryError, send_smtp_test_email
from app.services.tibber_prices import active_tibber_contract_exists, trigger_tibber_refresh
logger = logging.getLogger(__name__)
@@ -65,6 +66,10 @@ def put_config(
# Detect whether any MQTT-related key is being submitted (non-secret change
# or non-blank secret change) so we know to reconnect after saving.
mqtt_keys_submitted = any(k.lower() in MQTT_SETTINGS_KEYS for k in body.updates)
tibber_values_before = (
settings.tibber_api_token,
settings.tibber_home_id,
)
try:
save_config_updates(db, body.updates, settings)
@@ -92,6 +97,13 @@ def put_config(
from app.services.dsmr_ingest import apply_dsmr_subscription
apply_dsmr_subscription(refreshed_settings)
tibber_values_changed = tibber_values_before != (
refreshed_settings.tibber_api_token,
refreshed_settings.tibber_home_id,
)
if tibber_values_changed and active_tibber_contract_exists(db):
trigger_tibber_refresh()
sections_raw = build_config_sections(db, refreshed_settings)
return ConfigUpdateResponse(sections=_sections_from_raw(sections_raw))
@@ -233,7 +245,7 @@ def _run_mqtt_test(settings: Settings) -> None:
client = mqtt.Client(
callback_api_version=mqtt.CallbackAPIVersion.VERSION2,
client_id="home-automation-test",
client_id=mqtt_test_client_id(settings.mqtt_client_id),
)
def _on_connect(
+31 -10
View File
@@ -44,7 +44,7 @@ from __future__ import annotations
import logging
from datetime import UTC, datetime, timedelta
from typing import Any
from typing import Any, Literal
from fastapi import APIRouter, Depends, Query, status
from fastapi.responses import JSONResponse
@@ -133,13 +133,23 @@ def _manual_tariff_from_values(values: dict[str, Any]) -> ManualTariffSchema:
)
def _electricity_prices_response(response: PricesResponse) -> JSONResponse:
"""Preserve the exact pre-scope electricity response body."""
return JSONResponse(
content=response.model_dump(
mode="json", include={"kind", "currency", "points", "tariff"}
)
)
# ---------------------------------------------------------------------------
# GET /api/energy/prices
# ---------------------------------------------------------------------------
@router.get("/prices", response_model=PricesResponse)
@router.get("/prices", response_model=PricesResponse, response_model_exclude_none=True)
def get_prices(
scope: Literal["electricity", "thermal"] = Query("electricity"),
start: datetime | None = Query(
default=None,
description="Inclusive start of the time window (ISO 8601). "
@@ -186,6 +196,17 @@ def get_prices(
"""
now = datetime.now(UTC)
if scope == "thermal":
version = active_contract_version_at(db, now, scope="thermal")
if version is None:
return PricesResponse(kind=None, currency="EUR", points=[], tariff=None)
return PricesResponse(
kind="district_heating", currency=version.contract.currency,
contract_version_id=version.id, effective_from=_as_utc(version.effective_from),
effective_to=_as_utc(version.effective_to) if version.effective_to else None,
values=version.values, points=[], tariff=None,
)
# Default window: today + tomorrow.
if start is None:
start = now.replace(hour=0, minute=0, second=0, microsecond=0)
@@ -199,12 +220,12 @@ def get_prices(
version = active_contract_version_at(db, start_utc)
if version is None:
return PricesResponse(
return _electricity_prices_response(PricesResponse(
kind=None,
currency="EUR",
points=[],
tariff=None,
)
))
contract = version.contract
currency = contract.currency
@@ -247,30 +268,30 @@ def get_prices(
)
)
return PricesResponse(
return _electricity_prices_response(PricesResponse(
kind="tibber",
currency=currency,
points=points,
tariff=None,
)
))
elif contract.kind == "manual":
tariff = _manual_tariff_from_values(version.values or {})
return PricesResponse(
return _electricity_prices_response(PricesResponse(
kind="manual",
currency=currency,
points=[],
tariff=tariff,
)
))
else:
# Unknown kind — return empty response gracefully.
return PricesResponse(
return _electricity_prices_response(PricesResponse(
kind=contract.kind,
currency=currency,
points=[],
tariff=None,
)
))
# ---------------------------------------------------------------------------
+20 -6
View File
@@ -46,7 +46,9 @@ from app.schemas.energy_contract import (
from app.services.auth import AuthenticatedSession
from app.services import timezone as _tz_mod
from app.services.contracts import (
CONTRACT_KIND_SCOPES,
ContractVersionError,
ContractScopeError,
activate_contract,
add_version,
create_contract,
@@ -54,6 +56,7 @@ from app.services.contracts import (
get_contract_or_none,
list_contracts,
)
from app.services.tibber_prices import trigger_tibber_refresh
logger = logging.getLogger(__name__)
@@ -98,6 +101,7 @@ def _contract_detail(db: Session, contract) -> ContractDetailResponse:
id=contract.id,
name=contract.name,
kind=contract.kind,
scope=contract.scope,
active=contract.active,
currency=contract.currency,
created_at=contract.created_at,
@@ -163,16 +167,22 @@ def get_profiles(
@router.get("/contracts", response_model=ContractListResponse)
def list_energy_contracts(
scope: str = "electricity",
db: Session = Depends(get_db),
_auth: AuthenticatedSession = Depends(require_session),
) -> ContractListResponse:
"""List all energy contracts with their active status.
Returns a flat list (no embedded version history); use
Scope defaults to ``electricity`` for old clients. Returns a flat list (no embedded version history); use
GET /api/energy/contracts/{id} to fetch the full version history for a
specific contract.
"""
contracts = list_contracts(db)
if scope not in set(CONTRACT_KIND_SCOPES.values()):
raise HTTPException(
status_code=status.HTTP_422_UNPROCESSABLE_ENTITY,
detail=f"Unknown energy contract scope: {scope!r}",
)
contracts = list_contracts(db, scope=scope)
items = [ContractResponse.model_validate(c) for c in contracts]
return ContractListResponse(items=items, total=len(items))
@@ -208,10 +218,11 @@ def create_energy_contract(
name=body.name,
kind=body.kind,
currency=body.currency,
scope=body.scope,
values=body.values,
effective_from=effective_from,
)
except (ProfileNotFoundError, ProfileValidationError) as exc:
except (ProfileNotFoundError, ProfileValidationError, ContractScopeError) as exc:
_raise_422_for_profile_error(exc)
db.commit()
@@ -256,13 +267,14 @@ def patch_energy_contract(
"""Partially update a contract: rename or change activation status.
- ``name``: updates the human-readable label.
- ``active=true``: activates this contract (all others are deactivated).
- ``active=true``: activates this contract (same-scope contracts are deactivated).
- ``active=false``: deactivates this contract (no effect on others).
At most one contract may be active at any time; the service layer enforces
mutual exclusion.
At most one contract may be active per scope; the service layer enforces
scope-local mutual exclusion.
"""
contract = _get_contract_or_404(db, contract_id)
was_active = contract.active
if body.name is not None:
contract.name = body.name
@@ -275,6 +287,8 @@ def patch_energy_contract(
db.commit()
db.refresh(contract)
if body.active is True and not was_active and contract.kind == "tibber":
trigger_tibber_refresh()
return _contract_detail(db, contract)
+150
View File
@@ -0,0 +1,150 @@
"""Authenticated API for the thermal 15-minute cost ledger."""
from __future__ import annotations
from datetime import UTC, datetime, timedelta
from decimal import Decimal
from typing import Literal
from fastapi import APIRouter, Depends, HTTPException, Query, status
from sqlalchemy import func, select
from sqlalchemy.orm import Session
from app.api.routes.api.deps import require_csrf, require_session
from app.dependencies import get_db
from app.models.energy import MeterCostPeriod
from app.schemas.meter_cost import (
MeterCostPeriodSchema,
MeterCostRecomputeResponse,
MeterCostsResponse,
ThermalCostSummaryResponse,
)
from app.services.auth import AuthenticatedSession
from app.services.meter_cost import recompute_range, summarize
from app.services.timezone import local_midnight_utc, local_now
router = APIRouter(prefix="/api/energy/meter-costs", tags=["api-energy"])
_LIMIT_MAX = 5000
_RECOMPUTE_MAX_DAYS = 31
_QUARTER = timedelta(minutes=15)
def _utc(value: datetime) -> datetime:
return value.replace(tzinfo=UTC) if value.tzinfo is None else value.astimezone(UTC)
def _decimal_strings(value: object) -> object:
if isinstance(value, dict):
return {str(key): _decimal_strings(item) for key, item in value.items()}
if isinstance(value, Decimal):
return format(value, "f")
return str(value) if isinstance(value, (int, float)) else value
def _row_schema(row: MeterCostPeriod) -> MeterCostPeriodSchema:
return MeterCostPeriodSchema(
commodity=row.commodity,
period_start=_utc(row.period_start), period_end=_utc(row.period_end),
meter_id=row.meter_id, source_binding_id=row.source_binding_id,
contract_version_id=row.contract_version_id, quantity=format(row.quantity, "f"),
cost=format(row.cost, "f"), currency=row.currency,
cost_breakdown=_decimal_strings(row.cost_breakdown),
pricing_snapshot=_decimal_strings(row.pricing_snapshot), quality=row.quality,
degraded=row.degraded, degraded_reason=row.degraded_reason,
)
@router.get("", response_model=MeterCostsResponse)
def get_meter_costs(
scope: Literal["thermal"] = Query("thermal"),
commodity: Literal["heating", "hot_water"] | None = None,
start: datetime | None = None,
end: datetime | None = None,
limit: int = Query(500, ge=1, le=_LIMIT_MAX),
offset: int = Query(0, ge=0),
db: Session = Depends(get_db),
_auth: AuthenticatedSession = Depends(require_session),
) -> MeterCostsResponse:
"""List thermal rows in a half-open time window with stable pagination."""
del scope
if start is not None and end is not None and _utc(end) <= _utc(start):
raise HTTPException(status.HTTP_422_UNPROCESSABLE_ENTITY, "'end' must be after 'start'.")
clauses = []
if commodity is not None:
clauses.append(MeterCostPeriod.commodity == commodity)
if start is not None:
clauses.append(MeterCostPeriod.period_start >= _utc(start))
if end is not None:
clauses.append(MeterCostPeriod.period_start < _utc(end))
total = db.scalar(select(func.count()).select_from(MeterCostPeriod).where(*clauses)) or 0
rows = db.execute(
select(MeterCostPeriod).where(*clauses).order_by(MeterCostPeriod.period_start, MeterCostPeriod.id)
.offset(offset).limit(limit)
).scalars().all()
return MeterCostsResponse(items=[_row_schema(row) for row in rows], total=total)
@router.get("/summary", response_model=ThermalCostSummaryResponse)
def get_meter_cost_summary(
scope: Literal["thermal"] = Query("thermal"),
start: datetime | None = None,
end: datetime | None = None,
db: Session = Depends(get_db),
_auth: AuthenticatedSession = Depends(require_session),
) -> ThermalCostSummaryResponse:
"""Summarize thermal variable and once-per-contract daily fixed costs."""
del scope
if start is None or end is None:
today = local_now().date()
start = start or local_midnight_utc(today)
end = end or local_midnight_utc(today + timedelta(days=1))
start, end = _utc(start), _utc(end)
if end <= start:
raise HTTPException(status.HTTP_422_UNPROCESSABLE_ENTITY, "'end' must be after 'start'.")
result = summarize(db, start, end)
breakdown = result["breakdown"]
fixed_breakdown = result["fixed_breakdown"]
fixed = result["fixed_cost"]
return ThermalCostSummaryResponse(
currency=result["currency"], heating=format(breakdown["heating"], "f"),
hot_water_heating=format(breakdown["hot_water_heating"], "f"),
hot_water=format(breakdown["hot_water"], "f"), hot_water_tax=format(breakdown["hot_water_tax"], "f"),
variable_subtotal=format(result["variable_cost"], "f"),
fixed_breakdown={key: format(value, "f") for key, value in fixed_breakdown.items()},
fixed_subtotal=format(fixed, "f"),
all_in=format(result["total_cost"], "f"), period_count=result["period_count"],
degraded_count=result["degraded_count"],
)
@router.post("/recompute", response_model=MeterCostRecomputeResponse)
def post_meter_cost_recompute(
scope: Literal["thermal"] = Query("thermal"),
start: datetime = Query(...), end: datetime = Query(...),
db: Session = Depends(get_db), _auth: AuthenticatedSession = Depends(require_session),
_csrf: None = Depends(require_csrf),
) -> MeterCostRecomputeResponse:
"""Atomically overwrite closed, UTC-quarter thermal rows in a bounded window."""
del scope
start, end = _utc(start), _utc(end)
if end <= start or end - start > timedelta(days=_RECOMPUTE_MAX_DAYS):
raise HTTPException(status.HTTP_422_UNPROCESSABLE_ENTITY, "invalid or overlarge recompute window")
if start.minute % 15 or start.second or start.microsecond or end.minute % 15 or end.second or end.microsecond:
raise HTTPException(status.HTTP_422_UNPROCESSABLE_ENTITY, "start and end must align to UTC quarters")
if end > datetime.now(UTC):
raise HTTPException(status.HTTP_422_UNPROCESSABLE_ENTITY, "recompute window must be closed")
try:
processed = recompute_range(db, start, end, commit=False)
# Ensure pending upserts participate in this transaction before the
# counts are read; a flush/query failure must still roll everything back.
db.flush()
rows = db.execute(select(MeterCostPeriod.degraded).where(
MeterCostPeriod.period_start >= start, MeterCostPeriod.period_start < end
)).scalars().all()
degraded = sum(bool(value) for value in rows)
db.commit()
except Exception:
db.rollback()
raise
return MeterCostRecomputeResponse(processed=processed, normal=len(rows) - degraded, degraded=degraded)
+414
View File
@@ -0,0 +1,414 @@
"""Authenticated HTTP contract for meter sources, channels, and bindings."""
from __future__ import annotations
from datetime import UTC, datetime
from fastapi import APIRouter, Depends, HTTPException, Query, Response, status
from sqlalchemy import select
from sqlalchemy.orm import Session
from app.api.routes.api.deps import require_csrf, require_session
from app.config import get_settings
from app.dependencies import get_db
from app.integrations.meter_sources import SourceProfileError, list_source_profiles, sanitize_source_config
from app.models.energy import DsmrReading, Meter
from app.models.meter_source import MeterSource, MeterSourceBinding, MeterSourceChannel, WarmteLinkReading
from app.schemas.meter_source import (
BindingCreate, BindingListResponse, BindingPatch, BindingResponse, ChannelBindingSummaryResponse,
BindingTransferRequest, BindingTransferResponse,
ChannelReadingResponse,
ChannelReadingsResponse, CommoditiesResponse, CommodityResponse, DiscoverResponse,
DiscoverChannelResponse,
MeterSourceChannelListResponse, MeterSourceChannelResponse, MeterSourceCreate,
MeterSourceListResponse, MeterSourcePatch, MeterSourceResponse, SourceConfigFieldResponse,
SourceProfileResponse, SourceProfilesResponse,
)
from app.services.auth import AuthenticatedSession
from app.services.config_page import build_runtime_settings
from app.services.dsmr_ingest import apply_dsmr_subscription
from app.services.meter_sources import (
BindingNotFoundError, ChannelNotFoundError, MeterNotFoundError,
MeterSourceError, SourceDeleteRestrictedError, SourceNotFoundError, create_binding,
create_source, delete_source, list_bindings, list_sources, transfer_binding, update_binding, update_source,
)
from app.services.energy_cost import recompute_range as electricity_recompute_range
from app.services import timezone as _tz_mod
from app.services.warmtelink_worker import warmtelink_worker_manager
router = APIRouter(prefix="/api/energy", tags=["api-energy-meter-sources"])
def _reconcile_runtimes_after_commit(db: Session) -> None:
"""Best-effort runtime convergence after a durable source CRUD commit."""
try:
warmtelink_worker_manager.reconcile()
except Exception:
# The manager records individual source failures itself. Do not turn a
# successful durable create/update/delete into a misleading HTTP 500.
pass
try:
apply_dsmr_subscription(build_runtime_settings(db, get_settings()))
except Exception:
# DSMR owns independent source clients. Its failure must neither undo
# durable CRUD nor prevent the WarmteLink manager from converging.
pass
finally:
# DSMR health callbacks use short independent sessions. Make a CRUD
# response observe any durable status change they just committed.
db.expire_all()
def _as_utc(value: datetime) -> datetime:
if value.tzinfo is None:
return value.replace(tzinfo=_tz_mod.local_tz()).astimezone(UTC)
return value.astimezone(UTC)
def _source_or_404(db: Session, uuid: str) -> MeterSource:
source = db.execute(select(MeterSource).where(MeterSource.uuid == uuid)).scalar_one_or_none()
if source is None:
raise HTTPException(status_code=404, detail="Meter source not found.")
return source
def _channel_or_404(db: Session, source: MeterSource, uuid: str) -> MeterSourceChannel:
channel = db.execute(
select(MeterSourceChannel).where(
MeterSourceChannel.uuid == uuid, MeterSourceChannel.source_id == source.id
)
).scalar_one_or_none()
if channel is None:
raise HTTPException(status_code=404, detail="Meter source channel not found.")
return channel
def _source_response(source: MeterSource) -> MeterSourceResponse:
return MeterSourceResponse(
uuid=source.uuid, name=source.name, kind=source.kind, enabled=source.enabled,
config=sanitize_source_config(source.kind, source.config), status=source.status,
last_seen_at=source.last_seen_at, last_error=source.last_error,
created_at=source.created_at, updated_at=source.updated_at,
)
def binding_response(binding: MeterSourceBinding) -> BindingResponse:
return BindingResponse(
uuid=binding.uuid, meter_id=binding.meter_id, source_channel_uuid=binding.channel.uuid,
source_uuid=binding.channel.source.uuid, started_at=binding.started_at, ended_at=binding.ended_at,
created_at=binding.created_at, updated_at=binding.updated_at,
)
def _binding_error(exc: MeterSourceError) -> HTTPException:
if isinstance(exc, (SourceNotFoundError, ChannelNotFoundError, MeterNotFoundError, BindingNotFoundError)):
return HTTPException(status_code=404, detail=str(exc))
return HTTPException(status_code=422, detail=str(exc))
def _recompute_binding_commodity(db: Session, commodity: str, start: datetime) -> None:
end = datetime.now(UTC)
if start >= end:
return
if commodity == "electricity":
electricity_recompute_range(db, start, end, commit=False, strict=True)
else:
from app.services.meter_cost import recompute_range
recompute_range(db, start, end, commit=False)
def _republish_after_commit(db: Session) -> None:
try:
from app.services.ha_discovery import publish_discovery
publish_discovery(db)
except Exception:
pass
@router.get("/source-profiles", response_model=SourceProfilesResponse)
def source_profiles(_auth: AuthenticatedSession = Depends(require_session)) -> SourceProfilesResponse:
"""Return profile metadata; default secrets are never populated with stored values."""
profiles = []
for profile in list_source_profiles():
fields = [
SourceConfigFieldResponse(name=f.name, value_type=f.value_type.__name__, default=f.default,
required=f.required, secret=f.secret)
for f in profile.fields
]
profiles.append(SourceProfileResponse(
kind=profile.kind, fields=fields,
defaults={f.name: f.default for f in profile.fields if not f.required},
capabilities=sorted(profile.capabilities), allowed_units=sorted(profile.allowed_units),
))
return SourceProfilesResponse(items=profiles)
@router.get("/commodities", response_model=CommoditiesResponse)
def commodities(_auth: AuthenticatedSession = Depends(require_session)) -> CommoditiesResponse:
return CommoditiesResponse(items=[
CommodityResponse(key="electricity", unit="kWh", capabilities=["meter", "binding", "cost"]),
CommodityResponse(key="heating", unit="GJ", capabilities=["meter", "binding"]),
CommodityResponse(key="hot_water", unit="", capabilities=["meter", "binding"]),
])
@router.get("/sources", response_model=MeterSourceListResponse)
def get_sources(db: Session = Depends(get_db), _auth: AuthenticatedSession = Depends(require_session)) -> MeterSourceListResponse:
items = [_source_response(source) for source in list_sources(db)]
return MeterSourceListResponse(items=items, total=len(items))
@router.post("/sources", response_model=MeterSourceResponse, status_code=status.HTTP_201_CREATED)
def post_source(body: MeterSourceCreate, db: Session = Depends(get_db),
_auth: AuthenticatedSession = Depends(require_session), _csrf: None = Depends(require_csrf)) -> MeterSourceResponse:
try:
source = create_source(db, name=body.name, kind=body.kind, config=body.config, enabled=body.enabled)
db.commit()
_reconcile_runtimes_after_commit(db)
return _source_response(_source_or_404(db, source.uuid))
except (SourceProfileError, MeterSourceError) as exc:
db.rollback()
raise HTTPException(status_code=422, detail=str(exc)) from exc
@router.get("/sources/{source_uuid}", response_model=MeterSourceResponse)
def get_source_detail(source_uuid: str, db: Session = Depends(get_db),
_auth: AuthenticatedSession = Depends(require_session)) -> MeterSourceResponse:
return _source_response(_source_or_404(db, source_uuid))
@router.patch("/sources/{source_uuid}", response_model=MeterSourceResponse)
def patch_source(source_uuid: str, body: MeterSourcePatch, db: Session = Depends(get_db),
_auth: AuthenticatedSession = Depends(require_session), _csrf: None = Depends(require_csrf)) -> MeterSourceResponse:
source = _source_or_404(db, source_uuid)
try:
updated = update_source(db, source.id, name=body.name, enabled=body.enabled, config_patch=body.config)
db.commit()
_reconcile_runtimes_after_commit(db)
return _source_response(_source_or_404(db, updated.uuid))
except (SourceProfileError, MeterSourceError) as exc:
db.rollback()
raise _binding_error(exc) from exc
@router.delete(
"/sources/{source_uuid}", status_code=status.HTTP_204_NO_CONTENT, response_model=None
)
def remove_source(source_uuid: str, db: Session = Depends(get_db),
_auth: AuthenticatedSession = Depends(require_session), _csrf: None = Depends(require_csrf)) -> None:
source = _source_or_404(db, source_uuid)
# DSMR readings are not a relationship on MeterSource to avoid loading a large history.
if db.execute(select(DsmrReading.id).where(DsmrReading.meter_source_id == source.id).limit(1)).scalar() is not None:
raise HTTPException(status_code=409, detail="Meter source has dependent readings.")
try:
delete_source(db, source.id)
db.commit()
_reconcile_runtimes_after_commit(db)
return Response(status_code=status.HTTP_204_NO_CONTENT)
except SourceDeleteRestrictedError as exc:
db.rollback()
raise HTTPException(status_code=409, detail=str(exc)) from exc
@router.post("/sources/{source_uuid}/discover", response_model=DiscoverResponse)
def discover_source(source_uuid: str, db: Session = Depends(get_db),
_auth: AuthenticatedSession = Depends(require_session), _csrf: None = Depends(require_csrf)) -> DiscoverResponse:
source = _source_or_404(db, source_uuid)
if source.kind == "warmtelink_serial":
if not source.enabled:
return DiscoverResponse(
requested=False, supported=True, status="error",
detail="The WarmteLink source is disabled.", channels=_discover_channels(db, source),
)
# This merely schedules lifecycle convergence. It never opens a serial
# descriptor or waits for a frame in the request thread; the one managed
# worker remains the sole owner of serial I/O and can keep reconnecting.
request = warmtelink_worker_manager.request_discovery(source.id)
if request.completed.is_set():
# A worker may have accepted a frame during the bounded wait.
# Refresh only durable accepted metadata, never candidates/raw data.
db.expire_all()
source = _source_or_404(db, source_uuid)
return DiscoverResponse(
requested=request.status != "error", supported=True, status=request.status,
request_id=request.request_id or None, detail=request.detail,
channels=_discover_channels(db, source),
)
return DiscoverResponse(requested=False, supported=True, status="managed_by_runtime",
detail="This source is discovered by its runtime subscription; no connection was opened.")
@router.get("/sources/{source_uuid}/channels", response_model=MeterSourceChannelListResponse)
def source_channels(source_uuid: str, db: Session = Depends(get_db),
_auth: AuthenticatedSession = Depends(require_session)) -> MeterSourceChannelListResponse:
source = _source_or_404(db, source_uuid)
channels = db.execute(select(MeterSourceChannel).where(MeterSourceChannel.source_id == source.id)).scalars().all()
items = []
for channel in channels:
bindings = list_bindings(db, channel_id=channel.id)
meter_ids = [binding.meter_id for binding in bindings]
items.append(MeterSourceChannelResponse(
uuid=channel.uuid, label=channel.label, suggested_commodity=channel.suggested_commodity,
unit=channel.unit, device_type=channel.device_type, latest_value=channel.latest_value,
latest_at=channel.latest_at, latest_quality=channel.latest_quality, binding_count=len(bindings),
bound_meter_ids=meter_ids,
binding_summary=ChannelBindingSummaryResponse(count=len(bindings), meter_ids=meter_ids),
))
return MeterSourceChannelListResponse(items=items, total=len(items), source_status=source.status)
@router.get("/sources/{source_uuid}/channels/{channel_uuid}/readings", response_model=ChannelReadingsResponse)
def channel_readings(source_uuid: str, channel_uuid: str, limit: int = Query(default=100, ge=1, le=1000),
from_: datetime | None = Query(default=None, alias="from"),
to: datetime | None = Query(default=None), db: Session = Depends(get_db),
_auth: AuthenticatedSession = Depends(require_session)) -> ChannelReadingsResponse:
source = _source_or_404(db, source_uuid)
channel = _channel_or_404(db, source, channel_uuid)
if from_ is not None and to is not None and _as_utc(from_) >= _as_utc(to):
raise HTTPException(status_code=422, detail="'from' must be earlier than 'to'.")
if source.kind == "warmtelink_serial":
statement = select(WarmteLinkReading).where(WarmteLinkReading.channel_id == channel.id)
model = WarmteLinkReading
else:
# DSMR remains a source-level protocol history. Its channel is the
# public electricity identity, while payload/telegram diagnostics stay
# private to ingestion and the legacy latest endpoint.
statement = select(DsmrReading).where(DsmrReading.meter_source_id == source.id)
model = DsmrReading
if from_ is not None:
statement = statement.where(model.recorded_at >= _as_utc(from_))
if to is not None:
statement = statement.where(model.recorded_at < _as_utc(to))
rows = list(db.execute(statement.order_by(model.recorded_at.asc()).limit(limit)).scalars())
return ChannelReadingsResponse(
items=[ChannelReadingResponse(
recorded_at=row.recorded_at,
value=getattr(row, "value", None), quality=getattr(row, "quality", None),
) for row in rows],
total=len(rows),
)
def _discover_channels(db: Session, source: MeterSource) -> list[DiscoverChannelResponse]:
"""Return only public, accepted channel metadata for discover responses."""
return [
DiscoverChannelResponse(
uuid=channel.uuid, label=channel.label, unit=channel.unit,
latest_value=channel.latest_value, latest_at=channel.latest_at,
latest_quality=channel.latest_quality,
)
for channel in db.execute(
select(MeterSourceChannel).where(MeterSourceChannel.source_id == source.id)
).scalars()
]
@router.get("/meters/{meter_id}/bindings", response_model=BindingListResponse)
def meter_bindings(meter_id: int, db: Session = Depends(get_db),
_auth: AuthenticatedSession = Depends(require_session)) -> BindingListResponse:
if db.get(Meter, meter_id) is None:
raise HTTPException(status_code=404, detail="Meter not found.")
items = [binding_response(binding) for binding in list_bindings(db, meter_id=meter_id)]
return BindingListResponse(items=items, total=len(items))
@router.post("/meters/{meter_id}/bindings", response_model=BindingResponse, status_code=status.HTTP_201_CREATED)
def post_meter_binding(meter_id: int, body: BindingCreate, db: Session = Depends(get_db),
_auth: AuthenticatedSession = Depends(require_session), _csrf: None = Depends(require_csrf)) -> BindingResponse:
channel = db.execute(select(MeterSourceChannel).where(MeterSourceChannel.uuid == body.source_channel_uuid)).scalar_one_or_none()
if channel is None:
raise HTTPException(status_code=404, detail="Meter source channel not found.")
try:
binding = create_binding(db, meter_id=meter_id, channel_id=channel.id, started_at=_as_utc(body.started_at),
ended_at=_as_utc(body.ended_at) if body.ended_at else None)
meter = db.get(Meter, meter_id)
db.flush()
_recompute_binding_commodity(db, meter.commodity, _as_utc(body.started_at))
db.commit()
db.refresh(binding)
_republish_after_commit(db)
return binding_response(binding)
except MeterSourceError as exc:
db.rollback()
raise _binding_error(exc) from exc
except Exception:
db.rollback()
raise
@router.patch("/bindings/{binding_uuid}", response_model=BindingResponse)
def patch_binding(binding_uuid: str, body: BindingPatch, db: Session = Depends(get_db),
_auth: AuthenticatedSession = Depends(require_session), _csrf: None = Depends(require_csrf)) -> BindingResponse:
binding = db.execute(select(MeterSourceBinding).where(MeterSourceBinding.uuid == binding_uuid)).scalar_one_or_none()
if binding is None:
raise HTTPException(status_code=404, detail="Meter source binding not found.")
try:
# ``ended_at`` has three meaningful states in the service layer: omitted
# keeps the existing boundary, null reopens the interval, and a datetime
# changes the exclusive end. Do not collapse omitted into null here.
changes: dict[str, datetime | None] = {}
if "started_at" in body.model_fields_set:
changes["started_at"] = _as_utc(body.started_at) if body.started_at is not None else None
if "ended_at" in body.model_fields_set:
changes["ended_at"] = _as_utc(body.ended_at) if body.ended_at is not None else None
old_started_at = _as_utc(binding.started_at)
old_ended_at = _as_utc(binding.ended_at) if binding.ended_at is not None else None
updated = update_binding(db, binding.id, **changes)
meter = db.get(Meter, updated.meter_id)
if "started_at" in changes:
earliest = min(old_started_at, _as_utc(updated.started_at))
elif "ended_at" in changes:
new_ended_at = _as_utc(updated.ended_at) if updated.ended_at is not None else None
changed_ends = [value for value in (old_ended_at, new_ended_at) if value is not None]
earliest = min(changed_ends) if changed_ends else old_started_at
else:
earliest = old_started_at
db.flush()
_recompute_binding_commodity(db, meter.commodity, earliest)
db.commit()
db.refresh(updated)
_republish_after_commit(db)
return binding_response(updated)
except MeterSourceError as exc:
db.rollback()
raise _binding_error(exc) from exc
except Exception:
db.rollback()
raise
@router.post("/meters/{meter_id}/bindings/transfer", response_model=BindingTransferResponse)
def post_binding_transfer(
meter_id: int, body: BindingTransferRequest, db: Session = Depends(get_db),
_auth: AuthenticatedSession = Depends(require_session), _csrf: None = Depends(require_csrf),
) -> BindingTransferResponse:
source = db.execute(select(MeterSourceBinding).where(
MeterSourceBinding.uuid == body.from_binding_uuid
)).scalar_one_or_none()
channel = db.execute(select(MeterSourceChannel).where(
MeterSourceChannel.uuid == body.to_source_channel_uuid
)).scalar_one_or_none()
if source is None or channel is None:
raise HTTPException(status_code=404, detail="Meter source binding or channel not found.")
effective_at = _as_utc(body.effective_at)
try:
closed, created = transfer_binding(db, target_meter_id=meter_id, from_binding_id=source.id,
to_channel_id=channel.id, effective_at=effective_at)
meter = db.get(Meter, meter_id)
if closed.meter_id == meter.id:
earliest = effective_at
else:
earliest = min(_as_utc(closed.ended_at), effective_at)
db.flush()
_recompute_binding_commodity(db, meter.commodity, earliest)
db.commit()
db.refresh(closed)
db.refresh(created)
except MeterSourceError as exc:
db.rollback()
raise _binding_error(exc) from exc
except Exception:
db.rollback()
raise
_republish_after_commit(db)
return BindingTransferResponse(closed_binding=binding_response(closed), created_binding=binding_response(created))
+144 -26
View File
@@ -48,16 +48,27 @@ from typing import Optional
from fastapi import APIRouter, Depends, HTTPException, status
from sqlalchemy.orm import Session
from sqlalchemy import select
from app.api.routes.api.deps import require_csrf, require_session
from app.dependencies import get_db
from app.models.energy import Meter
from app.models.meter_source import MeterSourceChannel
from app.schemas.meter import (
MeterCloseRequest,
MeterDeclareRequest,
MeterBindingSummary,
MeterListResponse,
MeterPatchRequest,
MeterResponse,
)
from app.services.meter_sources import (
ChannelNotFoundError,
MeterSourceError,
create_binding,
create_binding_for_meter_swap,
close_open_bindings_for_meter,
)
from app.services import timezone as _tz_mod
from app.services.auth import AuthenticatedSession
from app.services.energy_cost import recompute_range
@@ -65,6 +76,7 @@ from app.services.meters import (
MeterIntervalError,
MeterOverlapError,
declare_meter,
close_meter,
list_meters,
update_meter,
)
@@ -142,7 +154,7 @@ def _trigger_recompute(db: Session, start: datetime, label: str) -> int:
# started_at is in the future — nothing to recompute.
logger.info("%s: started_at (%s) is in the future, skipping recompute.", label, start)
return 0
n = recompute_range(db, start, end)
n = recompute_range(db, start, end, commit=False, strict=True)
logger.info(
"%s: recomputed %d period(s) in window [%s, %s).",
label,
@@ -153,6 +165,33 @@ def _trigger_recompute(db: Session, start: datetime, label: str) -> int:
return n
def _recompute_commodity(db: Session, commodity: str, start: datetime, label: str) -> int:
if commodity == "electricity":
return _trigger_recompute(db, start, label)
from app.services.meter_cost import recompute_range as thermal_recompute_range
end = datetime.now(UTC)
if start >= end:
return 0
return thermal_recompute_range(db, start, end, commit=False)
def _meter_response(meter: Meter) -> MeterResponse:
"""Serialize meter plus binding summaries without exposing source config."""
response = MeterResponse.model_validate(meter)
response.bindings = [
MeterBindingSummary(
uuid=binding.uuid,
source_channel_uuid=binding.channel.uuid,
source_uuid=binding.channel.source.uuid,
started_at=binding.started_at,
ended_at=binding.ended_at,
)
for binding in meter.source_bindings
]
return response
# ---------------------------------------------------------------------------
# GET /api/energy/meters
# ---------------------------------------------------------------------------
@@ -170,7 +209,7 @@ def list_energy_meters(
has the latest ``started_at``.
"""
meters = list_meters(db)
items = [MeterResponse.model_validate(m) for m in meters]
items = [_meter_response(m) for m in meters]
return MeterListResponse(items=items, total=len(items))
@@ -211,6 +250,9 @@ def declare_energy_meter(
started_at_utc = _localize_started_at(body.started_at)
try:
old_meter = db.execute(
select(Meter).where(Meter.commodity == body.commodity, Meter.ended_at.is_(None))
).scalar_one_or_none()
new_meter = declare_meter(
db,
label=body.label,
@@ -219,20 +261,62 @@ def declare_energy_meter(
commodity=body.commodity,
note=body.note,
)
except MeterOverlapError as exc:
db.flush() # assign PK before an optional binding and recompute
# A closed predecessor must never retain an open interval. For a
# meter swap with no selected channel we can safely hand off exactly
# one compatible open channel; ambiguity is fail-closed.
auto_channel = None
if body.source_channel_uuid is None and old_meter is not None and body.reason.value == "meter_swap":
candidates = [b for b in old_meter.source_bindings if b.ended_at is None and b.channel.unit == {"electricity": "kWh", "heating": "GJ", "hot_water": ""}.get(body.commodity)]
if len(candidates) > 1:
raise MeterSourceError("Meter swap has ambiguous open bindings; select a channel explicitly.")
if len(candidates) == 1:
auto_channel = candidates[0].channel
if body.source_channel_uuid is not None:
channel = db.execute(
select(MeterSourceChannel).where(MeterSourceChannel.uuid == body.source_channel_uuid)
).scalar_one_or_none()
if channel is None:
raise ChannelNotFoundError("Meter source channel was not found.")
if body.reason.value == "meter_swap":
create_binding_for_meter_swap(
db,
old_meter_id=old_meter.id if old_meter is not None else None,
new_meter_id=new_meter.id,
channel_id=channel.id,
started_at=started_at_utc,
)
else:
create_binding(
db,
meter_id=new_meter.id,
channel_id=channel.id,
started_at=started_at_utc,
)
elif auto_channel is not None:
create_binding_for_meter_swap(db, old_meter_id=old_meter.id, new_meter_id=new_meter.id,
channel_id=auto_channel.id, started_at=started_at_utc)
if old_meter is not None:
close_open_bindings_for_meter(db, old_meter.id, ended_at=started_at_utc)
# Keep recompute in this transaction: a failure must not leave a new
# meter, its predecessor, or either binding at a half-applied boundary.
now = datetime.now(UTC)
if started_at_utc < now:
db.flush()
_recompute_commodity(db, body.commodity, started_at_utc, "POST /api/energy/meters")
db.commit()
except (MeterIntervalError, MeterOverlapError, MeterSourceError) as exc:
db.rollback()
raise HTTPException(
status_code=status.HTTP_422_UNPROCESSABLE_ENTITY,
status_code=(status.HTTP_404_NOT_FOUND if isinstance(exc, ChannelNotFoundError)
else status.HTTP_422_UNPROCESSABLE_ENTITY),
detail=str(exc),
)
except Exception:
db.rollback()
raise
db.flush() # assign PK before recompute (recompute uses session, needs meter in DB)
# Retroactive recompute: re-judge attribution from the new boundary onward.
now = datetime.now(UTC)
if started_at_utc < now:
_trigger_recompute(db, started_at_utc, "POST /api/energy/meters")
db.commit()
db.refresh(new_meter)
# Trigger HA discovery re-publish so the new active meter's energy-cost
@@ -247,7 +331,31 @@ def declare_energy_meter(
new_meter.label,
started_at_utc.isoformat(),
)
return MeterResponse.model_validate(new_meter)
return _meter_response(new_meter)
@router.post("/meters/{meter_id}/close", response_model=MeterResponse)
def close_energy_meter(
meter_id: int, body: MeterCloseRequest, db: Session = Depends(get_db),
_auth: AuthenticatedSession = Depends(require_session), _csrf: None = Depends(require_csrf),
) -> MeterResponse:
meter = _get_meter_or_404(db, meter_id)
boundary = _localize_started_at(body.ended_at)
try:
close_meter(db, meter, ended_at=boundary)
close_open_bindings_for_meter(db, meter.id, ended_at=boundary)
db.flush()
_recompute_commodity(db, meter.commodity, boundary, f"POST /api/energy/meters/{meter_id}/close")
db.commit()
except (MeterIntervalError, MeterSourceError) as exc:
db.rollback()
raise HTTPException(status_code=422, detail=str(exc)) from exc
except Exception:
db.rollback()
raise
db.refresh(meter)
_trigger_discovery_republish(db)
return _meter_response(meter)
# ---------------------------------------------------------------------------
@@ -298,23 +406,33 @@ def patch_energy_meter(
note=body.note,
started_at=new_started_at_utc,
)
# Retroactive recompute if started_at was changed.
if new_started_at_utc is not None and old_started_at is not None:
# Normalise old_started_at to UTC-aware for comparison.
if old_started_at.tzinfo is None:
old_started_at = old_started_at.replace(tzinfo=UTC)
# Window = [min(old, new), now) — covers all periods whose attribution
# may have changed due to the boundary shift in either direction.
window_start = min(old_started_at, new_started_at_utc)
db.flush()
_recompute_commodity(
db,
meter.commodity,
window_start,
f"PATCH /api/energy/meters/{meter_id}",
)
db.commit()
except MeterIntervalError as exc:
db.rollback()
raise HTTPException(
status_code=status.HTTP_422_UNPROCESSABLE_ENTITY,
detail=str(exc),
)
# Retroactive recompute if started_at was changed.
if new_started_at_utc is not None and old_started_at is not None:
# Normalise old_started_at to UTC-aware for comparison.
if old_started_at.tzinfo is None:
old_started_at = old_started_at.replace(tzinfo=UTC)
# Window = [min(old, new), now) — covers all periods whose attribution
# may have changed due to the boundary shift in either direction.
window_start = min(old_started_at, new_started_at_utc)
_trigger_recompute(db, window_start, f"PATCH /api/energy/meters/{meter_id}")
db.commit()
except Exception:
db.rollback()
raise
db.refresh(meter)
# Trigger HA discovery re-publish so label renames on the active meter
@@ -328,4 +446,4 @@ def patch_energy_meter(
meter.label,
meter.started_at,
)
return MeterResponse.model_validate(meter)
return _meter_response(meter)
+14 -1
View File
@@ -1,7 +1,8 @@
from functools import lru_cache
from pathlib import Path
import re
from pydantic import computed_field
from pydantic import computed_field, field_validator
from pydantic_settings import BaseSettings, SettingsConfigDict
@@ -53,6 +54,7 @@ class Settings(BaseSettings):
mqtt_username: str = ""
mqtt_password: str = ""
mqtt_tls_enabled: bool = False
mqtt_client_id: str = "home-automation"
# Home Assistant MQTT Discovery (T08 wires into CONFIG_FIELDS/UI; T11 does publishing).
ha_discovery_enabled: bool = False
@@ -81,6 +83,17 @@ class Settings(BaseSettings):
extra="ignore",
)
@field_validator("mqtt_client_id", mode="before")
@classmethod
def validate_mqtt_client_id(cls, value: object) -> str:
"""Normalize a broker-safe base client identity used by every MQTT client."""
if not isinstance(value, str):
raise ValueError("MQTT client ID must be a string")
normalized = value.strip()
if not re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9_-]{0,63}", normalized):
raise ValueError("MQTT client ID must be a non-empty ASCII slug")
return normalized
@computed_field
@property
def is_development(self) -> bool:
+332 -1
View File
@@ -27,7 +27,7 @@ from __future__ import annotations
import logging
from dataclasses import dataclass, field
from datetime import timedelta
from datetime import UTC, datetime, timedelta
from typing import Any, Callable, Optional, Protocol
from sqlalchemy.orm import Session
@@ -74,6 +74,17 @@ class DeviceInfo:
their state is being published.
"""
availability_id: Optional[str] = None
"""Stable id used for the shared availability topic, when different from
this HA device's identity. A Meter is identified by its own UUID, while
its liveness comes from the source/channel feeding it.
"""
availability_getter: Optional[Callable[["Session"], bool]] = field(
default=None, repr=False
)
"""Return whether the source behind this device is currently usable."""
@dataclass
class ExposableEntity:
@@ -877,3 +888,323 @@ def _energy_cost_provider(session: Session) -> list[ExposableEntity]:
# Register the energy cost provider at module load time.
register_provider(_energy_cost_provider)
# ---------------------------------------------------------------------------
# M8 source / meter / thermal-cost provider
# ---------------------------------------------------------------------------
_SOURCE_STALE_AFTER = timedelta(minutes=5)
def _utc_now() -> datetime:
"""Small clock seam for live-value bounds and deterministic tests."""
return datetime.now(UTC)
def _as_utc(value: datetime) -> datetime:
return value.replace(tzinfo=UTC) if value.tzinfo is None else value.astimezone(UTC)
def _source_is_online(source: Any, channel: Any | None = None) -> bool:
"""Do not turn an old cumulative value into a plausible live HA state."""
now = _utc_now()
if not source.enabled or source.status != "online" or source.last_seen_at is None:
return False
source_age = now - _as_utc(source.last_seen_at)
if not timedelta(0) <= source_age <= _SOURCE_STALE_AFTER:
return False
if channel is None:
return True
if channel.latest_at is None or channel.latest_quality not in {"valid", "unverifiable"}:
return False
channel_age = now - _as_utc(channel.latest_at)
return timedelta(0) <= channel_age <= _SOURCE_STALE_AFTER
def _dsmr_latest(session: Session, source_id: int, *, start: datetime | None = None,
end: datetime | None = None, not_after: datetime | None = None) -> Any:
"""Latest DSMR row in the source's (optionally bounded) cumulative domain."""
from app.models.energy import DsmrReading
from sqlalchemy import select
query = select(DsmrReading).where(DsmrReading.meter_source_id == source_id)
if start is not None:
query = query.where(DsmrReading.recorded_at >= start)
if end is not None:
query = query.where(DsmrReading.recorded_at < end)
if not_after is not None:
query = query.where(DsmrReading.recorded_at <= not_after)
return session.execute(query.order_by(DsmrReading.recorded_at.desc()).limit(1)).scalar_one_or_none()
def _dsmr_total(reading: Any) -> Any:
"""Return imported electricity total from a real DSMR telegram, or None."""
from decimal import Decimal, InvalidOperation
try:
payload = reading.payload or {}
return Decimal(str(payload["electricity_delivered_1"])) + Decimal(
str(payload["electricity_delivered_2"])
)
except (InvalidOperation, KeyError, TypeError, ValueError):
return None
def _m8_energy_provider(session: Session) -> list[ExposableEntity]:
"""Expose accepted source snapshots and current M8 meters.
The provider intentionally reads the thermal service's public ``summarize``
result for money. Keeping formulas in ``meter_cost`` prevents HA from
becoming a second, subtly different billing implementation.
"""
from app.models.energy import Meter
from app.models.meter_source import MeterSource, MeterSourceBinding, MeterSourceChannel
from sqlalchemy import select
sources = session.execute(select(MeterSource)).scalars().all()
entities: list[ExposableEntity] = []
for source in sources:
source_info = DeviceInfo(
identifiers=("meter-source", source.uuid), name=source.name,
availability_id=source.uuid,
availability_getter=lambda sess, source_id=source.id: _source_online_by_id(sess, source_id),
)
entities.append(ExposableEntity(
key=f"source.{source.uuid}.online", component="binary_sensor", device=source_info,
device_class="connectivity", unit="", name=f"{source.name} Online",
value_getter=lambda sess, source_id=source.id: "ON" if _source_online_by_id(sess, source_id) else "OFF",
))
active_meters = session.execute(
select(Meter).where(
Meter.ended_at.is_(None), Meter.commodity.in_(("electricity", "heating", "hot_water"))
)
).scalars().all()
active_by_commodity = {meter.commodity: meter for meter in active_meters}
for meter in active_meters:
bound = session.execute(
select(MeterSourceBinding, MeterSourceChannel, MeterSource)
.join(MeterSourceChannel, MeterSourceChannel.id == MeterSourceBinding.channel_id)
.join(MeterSource, MeterSource.id == MeterSourceChannel.source_id)
.where(MeterSourceBinding.meter_id == meter.id, MeterSourceBinding.ended_at.is_(None))
).one_or_none()
if bound is None:
continue
binding, channel, source = bound
info = DeviceInfo(
identifiers=("meter", meter.uuid), name=meter.label,
# Keep this opaque and Meter-anchored. In particular, do not use a
# source UUID here: two channels of one source can be independently
# stale/invalid and must not overwrite each other's availability.
availability_id=f"meter-availability-{meter.uuid}",
availability_getter=lambda sess, source_id=source.id, channel_id=channel.id:
_bound_channel_online(sess, source_id, channel_id),
)
if meter.commodity == "electricity":
unit, device_class = "kWh", "energy"
elif meter.commodity == "heating":
unit, device_class = "GJ", "energy"
else:
unit, device_class = "", "volume"
for suffix, getter in (
("total", _meter_total_getter(binding.id, source.id, channel.id)),
("today", _meter_today_getter(binding.id, source.id, channel.id)),
):
entities.append(ExposableEntity(
key=f"meter.{meter.uuid}.{suffix}", component="sensor", device=info,
device_class=device_class, unit=unit,
name=f"{meter.label} {suffix.title()}", value_getter=getter,
state_class="total_increasing",
))
heating, hot_water = active_by_commodity.get("heating"), active_by_commodity.get("hot_water")
if heating is not None and hot_water is not None:
identity = ".".join(sorted((heating.uuid, hot_water.uuid)))
currency = _thermal_currency(session)
cost_info = DeviceInfo(
identifiers=("thermal-cost", identity), name="Thermal Energy Cost",
provides_availability=False,
)
labels = {
"heating": "Heating", "hot_water_heating": "Hot Water Heating", "water": "Water",
"water_tax": "Water Tax", "fixed": "Fixed", "all_in": "All-in",
}
for suffix, window in (("total", None), ("today", "today")):
for metric, label in labels.items():
entities.append(ExposableEntity(
key=f"thermal_cost.{identity}.{metric}_{suffix}", component="sensor", device=cost_info,
device_class="monetary", unit=currency, name=f"Thermal {label} {suffix.title()}",
value_getter=_thermal_cost_getter(metric, window),
state_class="total" if suffix == "total" else "total_increasing",
))
return entities
def _source_online_by_id(session: Session, source_id: int) -> bool:
from app.models.meter_source import MeterSource
source = session.get(MeterSource, source_id)
if source is not None and source.kind == "dsmr_mqtt":
now = _utc_now()
# Inspect the actual latest telegram before calculating freshness: a
# clock-skewed future telegram must not make an older one look live.
latest = _dsmr_latest(session, source_id)
if not source.enabled or latest is None:
return False
age = now - _as_utc(latest.recorded_at)
return timedelta(0) <= age <= _SOURCE_STALE_AFTER
return source is not None and _source_is_online(source)
def _bound_channel_online(session: Session, source_id: int, channel_id: int) -> bool:
from app.models.meter_source import MeterSource, MeterSourceChannel
source, channel = session.get(MeterSource, source_id), session.get(MeterSourceChannel, channel_id)
if source is not None and source.kind == "dsmr_mqtt":
return _source_online_by_id(session, source_id)
return source is not None and channel is not None and _source_is_online(source, channel)
def _meter_total_getter(binding_id: int, source_id: int, channel_id: int) -> Callable[[Session], Any]:
def _getter(session: Session) -> Any:
from app.models.energy import Meter
from app.models.meter_source import MeterSource, MeterSourceBinding, MeterSourceChannel
if not _bound_channel_online(session, source_id, channel_id):
return None
binding = session.get(MeterSourceBinding, binding_id)
if (
binding is None or binding.ended_at is not None or binding.channel_id != channel_id
or (meter := session.get(Meter, binding.meter_id)) is None or meter.ended_at is not None
):
return None
source = session.get(MeterSource, source_id)
channel = session.get(MeterSourceChannel, channel_id)
if source is None or channel is None:
return None
start = max(_as_utc(meter.started_at), _as_utc(binding.started_at))
# Both windows are half-open. The current records have no end, but
# retaining this form makes a future close fail safely.
end = min(
(value for value in (_as_utc(meter.ended_at) if meter.ended_at else None,
_as_utc(binding.ended_at) if binding.ended_at else None) if value is not None),
default=None,
)
now = _utc_now()
if source.kind == "dsmr_mqtt":
latest = _dsmr_latest(session, source_id, start=start, end=end, not_after=now)
if latest is None:
return None
return _dsmr_total(latest)
if channel.latest_at is None or channel.latest_quality not in {"valid", "unverifiable"}:
return None
latest_at = _as_utc(channel.latest_at)
if latest_at < start or latest_at > now or (end is not None and latest_at >= end):
return None
return channel.latest_value
return _getter
def _meter_today_getter(binding_id: int, source_id: int, channel_id: int) -> Callable[[Session], Any]:
def _getter(session: Session) -> Any:
from app.models.energy import Meter
from app.models.meter_source import MeterSource, MeterSourceBinding, WarmteLinkReading
from app.services import timezone as tz
from sqlalchemy import select
binding = session.get(MeterSourceBinding, binding_id)
if binding is None or binding.ended_at is not None or binding.channel_id != channel_id:
return None
meter = session.get(Meter, binding.meter_id)
if meter is None or meter.ended_at is not None:
return None
now = _utc_now()
day = (tz.local_now() - _TODAY_RESET_GRACE).date()
start = max(tz.local_midnight_utc(day), _as_utc(meter.started_at), _as_utc(binding.started_at))
bounds = [tz.local_midnight_utc(day + timedelta(days=1))]
bounds.extend(value for value in (
_as_utc(meter.ended_at) if meter.ended_at else None,
_as_utc(binding.ended_at) if binding.ended_at else None,
) if value is not None)
end = min(bounds)
source = session.get(MeterSource, source_id)
if source is None or not source.enabled:
return None
if source.kind != "dsmr_mqtt" and source.status != "online":
return None
if source.kind == "dsmr_mqtt":
first = _dsmr_latest(session, source_id, start=start, end=end, not_after=now)
if first is None:
return None
from app.models.energy import DsmrReading
from sqlalchemy import select as dsmr_select
rows = session.execute(dsmr_select(DsmrReading).where(
DsmrReading.meter_source_id == source_id, DsmrReading.recorded_at >= start,
DsmrReading.recorded_at < end,
DsmrReading.recorded_at <= now,
).order_by(DsmrReading.recorded_at)).scalars().all()
if len(rows) < 2:
return None
first_total, last_total = _dsmr_total(rows[0]), _dsmr_total(rows[-1])
if first_total is None or last_total is None:
return None
value = last_total - first_total
return value if value >= 0 else None
query = select(WarmteLinkReading).where(
WarmteLinkReading.channel_id == channel_id,
WarmteLinkReading.recorded_at >= start,
WarmteLinkReading.quality.in_(("valid", "unverifiable")),
)
if end is not None:
query = query.where(WarmteLinkReading.recorded_at < end)
query = query.where(WarmteLinkReading.recorded_at <= now)
readings = session.execute(query.order_by(WarmteLinkReading.recorded_at)).scalars().all()
if len(readings) < 2:
return None
value = readings[-1].value - readings[0].value
return value if value >= 0 else None
return _getter
def _thermal_currency(session: Session) -> str:
from app.services.contracts import active_contract_versions
versions = active_contract_versions(session, scope="thermal")
return versions[-1].contract.currency if versions else "EUR"
def _thermal_cost_getter(metric: str, window: str | None) -> Callable[[Session], Any]:
def _getter(session: Session) -> Any:
from app.services import timezone as tz
from app.services.meter_cost import summarize
now = _utc_now()
from app.models.energy import Meter
meters = session.query(Meter).filter(
Meter.commodity.in_(("heating", "hot_water")), Meter.ended_at.is_(None)
).all()
if len(meters) != 2:
return None
epoch_start = max(_as_utc(m.started_at) for m in meters)
if window == "today":
day = (tz.local_now() - _TODAY_RESET_GRACE).date()
start = max(tz.local_midnight_utc(day), epoch_start)
# During the reset grace ``day`` is yesterday, whose local midnight
# remains the cap; otherwise do not summarize readings from later today.
end = min(tz.local_midnight_utc(day + timedelta(days=1)), now)
else:
# A combined thermal identity begins when its newest constituent
# meter epoch begins; including pre-swap rows would mix identities.
start, end = epoch_start, now
result = summarize(session, start, end, now=now)
if result["period_count"] == 0 and result["fixed_cost"] == 0:
return None
if metric == "fixed":
return result["fixed_cost"]
if metric == "all_in":
return result["total_cost"]
if metric == "water":
return result["breakdown"]["hot_water"]
if metric == "water_tax":
return result["breakdown"]["hot_water_tax"]
return result["breakdown"][metric]
return _getter
register_provider(_m8_energy_provider)
+171
View File
@@ -0,0 +1,171 @@
"""Registry and configuration helpers for meter-source integrations.
The registry is deliberately I/O-free. Workers and HTTP handlers use these
helpers to share one config contract without opening a broker or serial port.
"""
from __future__ import annotations
from dataclasses import dataclass
from typing import Any
SECRET_MASK = ""
class SourceProfileError(ValueError):
"""Raised when a source kind or its configuration is invalid."""
@dataclass(frozen=True)
class SourceConfigField:
"""One source configuration field and its public metadata."""
name: str
value_type: type
default: Any = None
required: bool = False
secret: bool = False
@dataclass(frozen=True)
class MeterSourceProfile:
"""A supported source kind's config, capabilities, and channel units."""
kind: str
fields: tuple[SourceConfigField, ...]
capabilities: frozenset[str]
allowed_units: frozenset[str]
DSMR_MQTT_PROFILE = MeterSourceProfile(
kind="dsmr_mqtt",
fields=(
SourceConfigField("broker_host", str, default=""),
SourceConfigField("broker_port", int, default=1883),
SourceConfigField("username", str, default="", secret=True),
SourceConfigField("password", str, default="", secret=True),
SourceConfigField("tls_enabled", bool, default=False),
SourceConfigField("topic", str, default="dsmr/json"),
SourceConfigField("tariff_topic", str, default="dsmr/meter-stats/electricity_tariff"),
SourceConfigField("sample_interval_s", int, default=10),
),
capabilities=frozenset({"discover", "mqtt_subscribe", "tariff"}),
allowed_units=frozenset({"kWh"}),
)
WARMTELINK_SERIAL_PROFILE = MeterSourceProfile(
kind="warmtelink_serial",
fields=(
SourceConfigField("path", str, required=True),
SourceConfigField("baudrate", int, default=115200),
SourceConfigField("data_bits", int, default=7),
SourceConfigField("parity", str, default="N"),
SourceConfigField("stop_bits", int, default=1),
),
capabilities=frozenset({"discover", "read_only_serial"}),
allowed_units=frozenset({"GJ", ""}),
)
SOURCE_PROFILES: dict[str, MeterSourceProfile] = {
DSMR_MQTT_PROFILE.kind: DSMR_MQTT_PROFILE,
WARMTELINK_SERIAL_PROFILE.kind: WARMTELINK_SERIAL_PROFILE,
}
def get_source_profile(kind: str) -> MeterSourceProfile:
"""Return the profile for *kind*, or raise a stable validation error."""
try:
return SOURCE_PROFILES[kind]
except KeyError as exc:
raise SourceProfileError(f"Unsupported meter source kind: {kind!r}") from exc
def list_source_profiles() -> list[MeterSourceProfile]:
"""Return profiles in deterministic kind order for a future API/UI."""
return [SOURCE_PROFILES[kind] for kind in sorted(SOURCE_PROFILES)]
def _check_type(field: SourceConfigField, value: Any) -> None:
# bool is a subclass of int; accept it only for explicitly boolean fields.
if type(value) is not field.value_type:
raise SourceProfileError(
f"Config field {field.name!r} must be a {field.value_type.__name__}."
)
def _validate_field_value(kind: str, field: SourceConfigField, value: Any) -> None:
_check_type(field, value)
if field.name == "path" and not value.startswith("/dev/"):
raise SourceProfileError("warmtelink_serial config path must start with '/dev/'.")
if field.name in {"broker_port", "sample_interval_s", "baudrate"} and value <= 0:
raise SourceProfileError(f"Config field {field.name!r} must be greater than zero.")
if field.name == "data_bits" and value != 7:
raise SourceProfileError("warmtelink_serial data_bits must be 7.")
if field.name == "parity" and value != "N":
raise SourceProfileError("warmtelink_serial parity must be 'N'.")
if field.name == "stop_bits" and value != 1:
raise SourceProfileError("warmtelink_serial stop_bits must be 1.")
def validate_source_config(kind: str, config: dict[str, Any]) -> dict[str, Any]:
"""Validate a complete config and return it with profile defaults filled.
Unknown keys are rejected to make configuration additions explicit. Secret
masking is intentionally not interpreted here: callers must merge a PATCH
with its stored config first.
"""
profile = get_source_profile(kind)
if not isinstance(config, dict):
raise SourceProfileError("Source config must be an object.")
fields = {field.name: field for field in profile.fields}
unknown = set(config) - set(fields)
if unknown:
raise SourceProfileError(f"Unknown {kind} config field(s): {sorted(unknown)!r}")
validated: dict[str, Any] = {}
for field in profile.fields:
if field.name in config:
value = config[field.name]
elif field.required:
raise SourceProfileError(f"Missing required {kind} config field: {field.name!r}")
else:
value = field.default
_validate_field_value(kind, field, value)
validated[field.name] = value
return validated
def sanitize_source_config(kind: str, config: dict[str, Any]) -> dict[str, Any]:
"""Validate and return a response-safe config with secrets masked."""
profile = get_source_profile(kind)
sanitized = validate_source_config(kind, config)
for field in profile.fields:
if field.secret:
sanitized[field.name] = SECRET_MASK
return sanitized
def merge_source_config(kind: str, current: dict[str, Any], patch: dict[str, Any]) -> dict[str, Any]:
"""Merge a partial PATCH into stored config, retaining masked secrets.
An empty secret value is the public response mask and therefore means
"keep the old value". New sources use :func:`validate_source_config`
instead, so an explicitly empty secret can still be initially configured.
"""
profile = get_source_profile(kind)
current_validated = validate_source_config(kind, current)
if not isinstance(patch, dict):
raise SourceProfileError("Source config patch must be an object.")
fields = {field.name: field for field in profile.fields}
unknown = set(patch) - set(fields)
if unknown:
raise SourceProfileError(f"Unknown {kind} config field(s): {sorted(unknown)!r}")
merged = dict(current_validated)
for name, value in patch.items():
field = fields[name]
if field.secret and value == SECRET_MASK:
continue
merged[name] = value
return validate_source_config(kind, merged)
+233 -3
View File
@@ -30,6 +30,7 @@ from __future__ import annotations
import logging
import threading
from collections.abc import Callable
from dataclasses import dataclass
from typing import TYPE_CHECKING
import paho.mqtt.client as mqtt
@@ -47,14 +48,34 @@ MQTT_SETTINGS_KEYS = {
"mqtt_username",
"mqtt_password",
"mqtt_tls_enabled",
"mqtt_client_id",
}
@dataclass
class _SourceClientState:
"""One installed source generation and its in-flight callback count."""
client: mqtt.Client
generation: int
in_flight: int = 0
def _is_configured(settings: Settings) -> bool:
"""Return True if MQTT is enabled *and* the broker host is set."""
return bool(settings.mqtt_enabled and settings.mqtt_broker_host)
def mqtt_source_client_id(base_client_id: str, source_id: int) -> str:
"""Return a stable, deployment-scoped identity for one DSMR source."""
return f"{base_client_id}-dsmr-source-{source_id}"
def mqtt_test_client_id(base_client_id: str) -> str:
"""Return a transient test identity that cannot evict a long-lived client."""
return f"{base_client_id}-test"
class MqttManager:
"""Long-lived MQTT client wrapper.
@@ -72,11 +93,26 @@ class MqttManager:
def __init__(self) -> None:
self._client: mqtt.Client | None = None
self._lock = threading.Lock()
self._lock = threading.RLock()
# Source replacement/removal is serialized independently from callback
# bookkeeping. In particular, paho's loop_stop() joins its network
# thread, whose callback completion also needs ``_lock``.
self._source_lifecycle_lock = threading.Lock()
self._source_idle = threading.Condition(self._lock)
self._connected = False
# topic → handler registry; persists across reconnects so subscriptions
# are automatically re-established when the client reconnects.
self._subscriptions: dict[str, Callable[[bytes], None]] = {}
# DSMR sources are independent connections: their credentials and TLS
# configuration belong to MeterSource.config, not app_config.
self._source_clients: dict[int, mqtt.Client] = {}
self._source_subscriptions: dict[int, dict[str, Callable[[bytes], None]]] = {}
self._source_connected: set[int] = set()
# Each replacement gets a distinct identity. A paho callback can run
# after its client was stopped, so source id alone is not sufficient.
self._source_generations: dict[int, int] = {}
self._source_states: dict[int, _SourceClientState] = {}
self._next_source_generation = 0
# ------------------------------------------------------------------
# Public properties
@@ -115,6 +151,11 @@ class MqttManager:
"""
with self._lock:
self._stop_client()
with self._source_lifecycle_lock:
with self._lock:
source_ids = list(self._source_clients)
for source_id in source_ids:
self._stop_source_client(source_id)
def reconnect(self, settings: Settings) -> None:
"""Disconnect the current client (if any) and reconnect with *settings*.
@@ -196,15 +237,152 @@ class MqttManager:
except Exception:
logger.exception("MQTT unsubscribe error (topic=%s).", topic)
def replace_source(
self,
source_id: int,
*,
host: str,
port: int,
username: str,
password: str,
tls_enabled: bool,
subscriptions: dict[str, Callable[[bytes], None]],
base_client_id: str = "home-automation",
state_handler: Callable[[str], None] | None = None,
) -> bool:
"""Replace one source-owned client and its handlers.
This intentionally does not touch the legacy app-wide client or any
other source client. It is also safe for a source to be temporarily
unconfigured: handlers are retained in the source registry but no
connection is attempted until a host is supplied.
"""
with self._source_lifecycle_lock:
self._stop_source_client(source_id)
if not host:
self._report_source_state(state_handler, "error", source_id)
return False
self._next_source_generation += 1
generation = self._next_source_generation
captured_subscriptions = dict(subscriptions)
client = mqtt.Client(
callback_api_version=mqtt.CallbackAPIVersion.VERSION2,
client_id=mqtt_source_client_id(base_client_id, source_id),
)
def _on_connect(
connected_client: mqtt.Client,
_userdata: object,
_flags: mqtt.ConnectFlags,
reason_code: mqtt.ReasonCode,
_properties: mqtt.Properties | None,
) -> None:
with self._lock:
if not self._is_current_source_client(source_id, generation, connected_client):
return
if reason_code.is_failure:
self._source_connected.discard(source_id)
logger.warning("DSMR MQTT connection refused for source_id=%s", source_id)
state = "error"
else:
self._source_connected.add(source_id)
state = "online"
for topic in captured_subscriptions:
try:
connected_client.subscribe(topic)
except Exception:
logger.exception("DSMR MQTT re-subscribe failed for source_id=%s", source_id)
self._report_source_state(state_handler, state, source_id)
def _on_disconnect(
disconnected_client: mqtt.Client,
_userdata: object,
_flags: mqtt.DisconnectFlags,
_reason_code: mqtt.ReasonCode,
_properties: mqtt.Properties | None,
) -> None:
with self._lock:
if not self._is_current_source_client(source_id, generation, disconnected_client):
return
self._source_connected.discard(source_id)
self._report_source_state(state_handler, "error", source_id)
def _on_message(
message_client: mqtt.Client,
_userdata: object,
message: mqtt.MQTTMessage,
) -> None:
with self._lock:
if not self._is_current_source_client(source_id, generation, message_client):
return
handler = captured_subscriptions.get(message.topic)
state = self._source_states.get(source_id)
if handler is None or state is None:
return
# This permit covers the entire handler call. Teardown first
# invalidates the state and then waits for all permits, so an
# old callback cannot run after teardown returns.
state.in_flight += 1
try:
handler(message.payload)
except Exception:
logger.exception("DSMR source handler raised (source_id=%s)", source_id)
finally:
with self._lock:
state.in_flight -= 1
if state.in_flight == 0:
self._source_idle.notify_all()
client.on_connect = _on_connect
client.on_disconnect = _on_disconnect
client.on_message = _on_message
if tls_enabled:
try:
client.tls_set()
except Exception:
logger.exception("DSMR MQTT TLS setup failed for source_id=%s", source_id)
self._report_source_state(state_handler, "error", source_id)
return False
if username:
client.username_pw_set(username=username, password=password or None)
# Register ownership before network processing begins. A broker
# may deliver CONNACK synchronously from connect(), or on the loop
# thread before connect() returns.
with self._lock:
self._source_clients[source_id] = client
self._source_subscriptions[source_id] = captured_subscriptions
self._source_generations[source_id] = generation
self._source_states[source_id] = _SourceClientState(client, generation)
self._report_source_state(state_handler, "connecting", source_id)
client.loop_start()
try:
client.connect(host=host, port=port, keepalive=60)
except Exception:
logger.exception("DSMR MQTT connect failed (source_id=%s, host=%s)", source_id, host)
self._report_source_state(state_handler, "error", source_id)
self._stop_source_client(source_id)
return False
return True
def remove_source(self, source_id: int) -> None:
"""Drop one source client and its handlers, including queued callbacks."""
with self._source_lifecycle_lock:
self._stop_source_client(source_id)
def source_is_active(self, source_id: int) -> bool:
"""Whether a source-owned client is currently installed for callbacks."""
with self._lock:
return source_id in self._source_clients
# ------------------------------------------------------------------
# Internal helpers — must be called with self._lock held
# Internal helpers
# ------------------------------------------------------------------
def _start_client(self, settings: Settings) -> None:
"""Build a fresh paho Client, configure it, and call loop_start + connect."""
client = mqtt.Client(
callback_api_version=mqtt.CallbackAPIVersion.VERSION2,
client_id="home-automation",
client_id=settings.mqtt_client_id,
)
# Callbacks — VERSION2 on_connect signature:
@@ -333,6 +511,58 @@ class MqttManager:
logger.debug("MQTT loop_stop raised (ignoring).", exc_info=True)
logger.info("MQTT client stopped.")
def _stop_source_client(self, source_id: int) -> None:
"""Detach then stop a source client without blocking callback bookkeeping.
Callers hold ``_source_lifecycle_lock``. The first phase makes the
generation unreachable while holding ``_lock``. Paho operations and
the in-flight wait are deliberately outside that lock: loop_stop()
joins paho's network thread, and an active callback needs ``_lock`` to
release its permit in ``_on_message``'s finally block.
"""
with self._lock:
state = self._source_states.pop(source_id, None)
client = self._source_clients.pop(source_id, None)
self._source_subscriptions.pop(source_id, None)
self._source_connected.discard(source_id)
# Invalidate callbacks even when there was no successfully
# installed client (for example after a failed replacement).
self._source_generations.pop(source_id, None)
if client is not None:
try:
client.disconnect()
except Exception:
logger.debug("DSMR MQTT disconnect raised (source_id=%s)", source_id, exc_info=True)
try:
client.loop_stop()
except Exception:
logger.debug("DSMR MQTT loop_stop raised (source_id=%s)", source_id, exc_info=True)
if state is not None:
with self._lock:
while state.in_flight:
self._source_idle.wait()
def _is_current_source_client(
self, source_id: int, generation: int, client: mqtt.Client
) -> bool:
"""Check callback ownership while ``_lock`` is held."""
return (
self._source_generations.get(source_id) == generation
and self._source_clients.get(source_id) is client
)
@staticmethod
def _report_source_state(
state_handler: Callable[[str], None] | None, state: str, source_id: int
) -> None:
"""Invoke an optional health callback without exposing connection credentials."""
if state_handler is None:
return
try:
state_handler(state)
except Exception:
logger.exception("DSMR MQTT source state update failed for source_id=%s", source_id)
# ---------------------------------------------------------------------------
# Module-level singleton — shared across lifespan and route handlers
+192
View File
@@ -0,0 +1,192 @@
"""Pure, privacy-preserving parser for DSMR and WarmteLink P1 telegrams."""
from __future__ import annotations
from dataclasses import dataclass, field as dataclass_field
from decimal import Decimal, InvalidOperation
from enum import StrEnum
import hashlib
import re
_OBIS_LINE = re.compile(r"^(?P<code>\d+-\d+:\d+\.\d+\.\d+)(?P<values>(?:\([^)]*\))*)$")
_NUMBER_WITH_UNIT = re.compile(r"^(?P<number>[+-]?\d+(?:\.\d+)?)(?:\*(?P<unit>.+))?$")
_CHANNEL_OBIS = re.compile(r"^0-(?P<channel>[1-9]\d*):(24|96)\.")
_EQUIPMENT_ID_CODES = re.compile(r"^0-(?:0|[1-9]\d*):96\.1\.[01]$")
class IntegrityStatus(StrEnum):
"""Whether a frame has a verifiable standard DSMR checksum."""
VALID = "valid"
INVALID = "invalid"
UNVERIFIABLE = "unverifiable"
class P1ParseError(ValueError):
"""A parse error whose message never includes telegram contents."""
@dataclass(frozen=True)
class ObisField:
"""One sanitized OBIS line, including values not understood by the parser."""
code: str
raw_values: tuple[str, ...]
value: Decimal | None = None
unit: str | None = None
comparison_token: str | None = dataclass_field(default=None, repr=False)
@dataclass(frozen=True)
class P1Channel:
"""Fields associated with one M-Bus channel, without its raw identifier."""
number: int
device_type: str | None
equipment_fingerprint: str | None = dataclass_field(repr=False)
readings: tuple[ObisField, ...]
@dataclass(frozen=True)
class P1Telegram:
"""A parsed telegram with only sanitized, persistence-safe data."""
frame_length: int
integrity: IntegrityStatus
integrity_reason: str
timestamp: str | None
equipment_fingerprint: str | None = dataclass_field(repr=False)
fields: tuple[ObisField, ...]
channels: tuple[P1Channel, ...]
class TelegramFramer:
"""Incrementally extract newline-terminated variable-length telegrams."""
def __init__(self) -> None:
self._buffer = bytearray()
def feed(self, chunk: bytes) -> list[bytes]:
"""Append *chunk* and return every complete frame now available."""
self._buffer.extend(chunk)
frames: list[bytes] = []
while (bang := self._buffer.find(b"!")) >= 0:
newline = self._buffer.find(b"\n", bang)
if newline < 0:
break
standard_start = self._buffer.find(b"/")
start = standard_start if 0 <= standard_start < bang else 0
frames.append(bytes(self._buffer[start : newline + 1]))
del self._buffer[: newline + 1]
return frames
def dsmr_crc16(data: bytes) -> int:
"""Return the DSMR CRC-16 over *data* (normally from ``/`` through ``!``)."""
crc = 0
for byte in data:
crc ^= byte
for _ in range(8):
crc = (crc >> 1) ^ 0xA001 if crc & 1 else crc >> 1
return crc & 0xFFFF
def parse_telegram(frame: bytes) -> P1Telegram:
"""Parse one complete frame without retaining raw telegram bytes or IDs."""
bang = frame.find(b"!")
if bang < 0:
raise P1ParseError("telegram has no footer marker")
body = frame[:bang]
footer = frame[bang + 1 :].rstrip(b"\r\n")
integrity, reason = _integrity(frame, bang, footer)
fields, identifiers = _parse_obis_fields(body)
timestamp = _field_value(fields, "0-0:1.0.0")
return P1Telegram(
frame_length=len(frame),
integrity=integrity,
integrity_reason=reason,
timestamp=timestamp,
equipment_fingerprint=identifiers.get("0-0:96.1.1"),
fields=tuple(fields),
channels=_parse_channels(fields, identifiers),
)
def _integrity(frame: bytes, bang: int, footer: bytes) -> tuple[IntegrityStatus, str]:
if not frame.startswith(b"/"):
return IntegrityStatus.UNVERIFIABLE, "missing standard DSMR '/' header"
if len(footer) != 4 or not all(chr(byte) in "0123456789abcdefABCDEF" for byte in footer):
return IntegrityStatus.UNVERIFIABLE, "footer is not a four-digit hexadecimal CRC"
expected = int(footer, 16)
actual = dsmr_crc16(frame[: bang + 1])
if actual == expected:
return IntegrityStatus.VALID, "CRC16 verified from '/' through '!'"
return IntegrityStatus.INVALID, f"CRC16 mismatch: expected {expected:04X}, calculated {actual:04X}"
def _parse_obis_fields(body: bytes) -> tuple[list[ObisField], dict[str, str]]:
fields: list[ObisField] = []
identifiers: dict[str, str] = {}
for line in body.decode("ascii", errors="replace").splitlines()[1:]:
match = _OBIS_LINE.fullmatch(line)
if not match:
continue
code = match.group("code")
raw_values = tuple(re.findall(r"\(([^)]*)\)", match.group("values")))
if _EQUIPMENT_ID_CODES.fullmatch(code):
comparison_token = _fingerprint(raw_values[-1]) if raw_values else None
if comparison_token is not None:
identifiers[code] = comparison_token
fields.append(ObisField(code, ("<redacted>",), comparison_token=comparison_token))
continue
value, unit = _numeric_value(raw_values)
fields.append(ObisField(code, raw_values, value, unit))
return fields, identifiers
def _fingerprint(identifier: str) -> str:
"""Hash an identifier locally; callers never receive its original value."""
return hashlib.sha256(identifier.encode("ascii", errors="replace")).hexdigest()
def _numeric_value(raw_values: tuple[str, ...]) -> tuple[Decimal | None, str | None]:
if not raw_values:
return None, None
match = _NUMBER_WITH_UNIT.fullmatch(raw_values[-1])
if not match:
return None, None
try:
return Decimal(match.group("number")), match.group("unit")
except InvalidOperation:
return None, None
def _field_value(fields: list[ObisField], code: str) -> str | None:
field = next((item for item in fields if item.code == code), None)
return field.raw_values[-1] if field and field.raw_values else None
def _parse_channels(fields: list[ObisField], identifiers: dict[str, str]) -> tuple[P1Channel, ...]:
by_channel: dict[int, list[ObisField]] = {}
for field in fields:
match = _CHANNEL_OBIS.match(field.code)
if match:
by_channel.setdefault(int(match.group("channel")), []).append(field)
return tuple(
P1Channel(
number=number,
device_type=_field_value(channel_fields, f"0-{number}:24.1.0"),
equipment_fingerprint=identifiers.get(f"0-{number}:96.1.0"),
readings=tuple(
field
for field in channel_fields
if field.code == f"0-{number}:24.2.1" and field.value is not None
),
)
for number, channel_fields in sorted(by_channel.items())
)
+159 -2
View File
@@ -25,6 +25,7 @@ Design notes
from __future__ import annotations
import logging
from decimal import Decimal, InvalidOperation
from pathlib import Path
from typing import Any, Optional
@@ -152,13 +153,86 @@ class TibberProfile(BaseModel):
return self
# ---------------------------------------------------------------------------
# DistrictHeatingProfile — user-entered thermal contract structure
# ---------------------------------------------------------------------------
class DistrictHeatingFieldSpec(BaseModel):
"""A Decimal-safe thermal tariff field displayed to the user."""
model_config = {"extra": "forbid"}
unit: str
label: str
help: str
minimum: Decimal = Decimal("0")
default: Decimal | None = None
class DistrictHeatingVariableSpec(BaseModel):
model_config = {"extra": "forbid"}
heating: DistrictHeatingFieldSpec
hot_water_heating: DistrictHeatingFieldSpec
hot_water: DistrictHeatingFieldSpec
hot_water_tax: DistrictHeatingFieldSpec
class DistrictHeatingStandingSpec(BaseModel):
model_config = {"extra": "forbid"}
heating_network: DistrictHeatingFieldSpec
metering: DistrictHeatingFieldSpec
delivery_set: DistrictHeatingFieldSpec
hot_water_network: DistrictHeatingFieldSpec
other: DistrictHeatingFieldSpec
class DistrictHeatingProfile(BaseModel):
"""Complete structure description for a ``district_heating`` contract."""
model_config = {"extra": "forbid"}
kind: str
label: str
variable: DistrictHeatingVariableSpec
standing: DistrictHeatingStandingSpec
@model_validator(mode="after")
def _check_kind(self) -> "DistrictHeatingProfile":
if self.kind != "district_heating":
raise ValueError(
"DistrictHeatingProfile requires kind='district_heating', "
f"got {self.kind!r}"
)
units = {
"heating": "EUR/GJ",
"hot_water_heating": "EUR/m³",
"hot_water": "EUR/m³",
"hot_water_tax": "EUR/m³",
}
for key, unit in units.items():
field = getattr(self.variable, key)
if field.unit != unit or field.minimum != 0 or field.default is not None:
raise ValueError(f"district_heating.variable.{key} must be required {unit} with minimum 0")
for key in DistrictHeatingStandingSpec.model_fields:
field = getattr(self.standing, key)
if field.unit != "EUR/year" or field.minimum != 0 or field.default != 0:
raise ValueError(
f"district_heating.standing.{key} must be EUR/year with default and minimum 0"
)
return self
# A union type for type hints where either profile is acceptable.
AnyProfile = ManualProfile | TibberProfile
AnyProfile = ManualProfile | TibberProfile | DistrictHeatingProfile
# Map kind → Pydantic model class used for validation.
_PROFILE_MODELS: dict[str, type[ManualProfile] | type[TibberProfile]] = {
_PROFILE_MODELS: dict[str, type[ManualProfile] | type[TibberProfile] | type[DistrictHeatingProfile]] = {
"manual": ManualProfile,
"tibber": TibberProfile,
"district_heating": DistrictHeatingProfile,
}
@@ -204,6 +278,11 @@ def load_profile(kind: str) -> AnyProfile:
f"Profile '{kind}': expected a YAML mapping, got {type(raw).__name__}"
)
# YAML's implicit float conversion must never contaminate the thermal profile.
# Existing electricity profiles intentionally retain their established defaults.
if raw.get("kind", kind) == "district_heating":
_reject_yaml_floats(raw, path)
# Choose the right Pydantic model based on the ``kind`` field in the YAML.
yaml_kind = raw.get("kind", kind)
model_cls = _PROFILE_MODELS.get(yaml_kind)
@@ -362,6 +441,82 @@ def _validate_tibber_values(values: dict[str, Any], profile: TibberProfile) -> d
return filled
def _reject_yaml_floats(value: Any, path: Path) -> None:
"""Reject implicit YAML floats for district-heating profile metadata."""
if isinstance(value, float):
raise ProfileValidationError(
f"Profile '{path.stem}' must not contain YAML float values; use integer 0 or strings."
)
if isinstance(value, dict):
for child in value.values():
_reject_yaml_floats(child, path)
elif isinstance(value, list):
for child in value:
_reject_yaml_floats(child, path)
def _decimal_value(section: str, key: str, value: Any) -> str:
"""Validate and normalise one thermal amount without passing through float."""
if isinstance(value, bool) or isinstance(value, float) or not isinstance(value, (str, int, Decimal)):
raise ProfileValidationError(
f"Contract values field '{section}.{key}' must be a Decimal-compatible string or integer, "
f"got {type(value).__name__!r}"
)
try:
amount = Decimal(str(value))
except (InvalidOperation, ValueError) as exc:
raise ProfileValidationError(
f"Contract values field '{section}.{key}' must be a Decimal-compatible value"
) from exc
if not amount.is_finite() or amount < 0:
raise ProfileValidationError(
f"Contract values field '{section}.{key}' must be a non-negative finite Decimal"
)
return format(amount, "f")
def _validate_district_heating_values(
values: dict[str, Any], profile: DistrictHeatingProfile
) -> dict[str, Any]:
"""Validate thermal values and return a complete JSON-safe Decimal snapshot."""
if not isinstance(values, dict):
raise ProfileValidationError("District-heating contract values must be a mapping")
expected_sections = {"variable", "standing"}
unknown_sections = set(values) - expected_sections
if unknown_sections:
raise ProfileValidationError(
f"District-heating contract values contain unknown section(s): {sorted(unknown_sections)}"
)
def normalise_section(
section: str, specs: Any, *, defaults_allowed: bool
) -> dict[str, str]:
supplied = values.get(section, {})
if not isinstance(supplied, dict):
raise ProfileValidationError(f"Contract values section '{section}' must be a mapping")
expected = set(type(specs).model_fields)
unknown = set(supplied) - expected
if unknown:
raise ProfileValidationError(
f"Contract values section '{section}' contains unknown field(s): {sorted(unknown)}"
)
normalised: dict[str, str] = {}
for key in type(specs).model_fields:
if key not in supplied:
field = getattr(specs, key)
if not defaults_allowed or field.default is None:
raise ProfileValidationError(f"Contract values missing required field '{section}.{key}'")
normalised[key] = format(field.default, "f")
else:
normalised[key] = _decimal_value(section, key, supplied[key])
return normalised
return {
"variable": normalise_section("variable", profile.variable, defaults_allowed=False),
"standing": normalise_section("standing", profile.standing, defaults_allowed=True),
}
def validate_values(kind: str, values: dict[str, Any]) -> dict[str, Any]:
"""Validate a contract-values dict against the named profile structure.
@@ -395,5 +550,7 @@ def validate_values(kind: str, values: dict[str, Any]) -> dict[str, Any]:
return _validate_manual_values(values, profile)
if isinstance(profile, TibberProfile):
return _validate_tibber_values(values, profile)
if isinstance(profile, DistrictHeatingProfile):
return _validate_district_heating_values(values, profile)
# Unreachable with current kinds, but guard for future extensions.
raise ProfileValidationError(f"No validator implemented for kind={kind!r}")
@@ -0,0 +1,56 @@
kind: district_heating
label: 区域供热
variable:
heating:
unit: EUR/GJ
label: 供暖热量
help: 按供暖用热量计收;请录入合同中的实际金额。
minimum: 0
hot_water_heating:
unit: EUR/m³
label: 热水加热
help: 按热水体积计收的加热部分;请录入合同中的实际金额。
minimum: 0
hot_water:
unit: EUR/m³
label: 热水用量
help: 按热水体积计收的用量部分;请录入合同中的实际金额。
minimum: 0
hot_water_tax:
unit: EUR/m³
label: 热水税费
help: 按热水体积计收的税费部分;请录入合同中的实际金额。
minimum: 0
standing:
heating_network:
unit: EUR/year
label: 供暖网络费
help: 年度固定费用;默认零,按合同实际金额录入。
minimum: 0
default: 0
metering:
unit: EUR/year
label: 计量费
help: 年度固定费用;默认零,按合同实际金额录入。
minimum: 0
default: 0
delivery_set:
unit: EUR/year
label: 交付装置费
help: 年度固定费用;默认零,按合同实际金额录入。
minimum: 0
default: 0
hot_water_network:
unit: EUR/year
label: 热水网络费
help: 年度固定费用;默认零,按合同实际金额录入。
minimum: 0
default: 0
other:
unit: EUR/year
label: 其他固定费
help: 年度固定费用;默认零,按合同实际金额录入。
minimum: 0
default: 0
+70 -35
View File
@@ -1,7 +1,9 @@
import logging
import os
from contextlib import asynccontextmanager
from datetime import UTC, datetime
from pathlib import Path
from typing import Callable
from fastapi import FastAPI, HTTPException, Request
from fastapi.responses import FileResponse
@@ -16,8 +18,10 @@ from app.api.routes.api.config import router as api_config_router
from app.api.routes.api.data import router as api_data_router
from app.api.routes.api.energy import router as api_energy_router
from app.api.routes.api.energy_contracts import router as api_energy_contracts_router
from app.api.routes.api.meter_costs import router as api_meter_costs_router
from app.api.routes.api.expose import router as api_expose_router
from app.api.routes.api.meters import router as api_meters_router
from app.api.routes.api.meter_sources import router as api_meter_sources_router
from app.api.routes.api.modbus import router as api_modbus_router
from app.api.routes.api.session import router as api_session_router
from app.api.routes import status
@@ -35,8 +39,10 @@ from app.services.dsmr_ingest import apply_dsmr_subscription
from app.services.public_ip import check_public_ipv4_and_notify
from app.services.modbus_poll import poll_all_enabled_devices, BASE_POLL_TICK_SECONDS
from app.services.ha_discovery import publish_discovery, publish_states
from app.services.tibber_prices import refresh_prices
from app.services.tibber_prices import run_tibber_refresh_best_effort
from app.services.energy_cost import compute_closed_periods
from app.services.meter_cost import compute_closed_periods as compute_closed_meter_cost_periods
from app.services.warmtelink_worker import warmtelink_worker_manager
from app.services.timezone import local_tz
from scripts.app_db_adopt import AppDatabaseAdoptionError, validate_app_runtime_db
@@ -86,15 +92,7 @@ def _run_scheduled_tibber_refresh() -> None:
so that a single failed fetch does not crash the scheduler or affect the
other background jobs.
"""
session_local = get_session_local()
session: Session = session_local()
try:
runtime_settings = build_runtime_settings(session, get_settings())
refresh_prices(session, runtime_settings)
except Exception:
logger.exception("_run_scheduled_tibber_refresh: unexpected error")
finally:
session.close()
run_tibber_refresh_best_effort()
def _run_scheduled_energy_cost() -> None:
@@ -110,22 +108,44 @@ def _run_scheduled_energy_cost() -> None:
does not crash the scheduler or affect the other background jobs.
"""
session_local = get_session_local()
session: Session = session_local()
try:
compute_closed_periods(session)
# After billing periods are computed, push fresh energy-cost state values
# to MQTT/HA. publish_states is internally guarded by _should_publish
# (MQTT disabled / not connected → no-op), so this never raises due to
# unconfigured MQTT and does not block the billing job.
def run_scope(label: str, operation: Callable[[Session], None]) -> None:
"""Run one best-effort scope in an isolated transaction/session."""
session: Session | None = None
try:
from app.services.ha_discovery import publish_states
publish_states(session)
session = session_local()
operation(session)
except Exception:
logger.exception("_run_scheduled_energy_cost: publish_states failed (non-fatal)")
except Exception:
logger.exception("_run_scheduled_energy_cost: unexpected error")
finally:
session.close()
logger.exception("_run_scheduled_energy_cost: %s failed", label)
if session is not None:
try:
session.rollback()
except Exception:
# A failed cleanup must not replace the operation/factory
# error or prevent the following independent scope.
logger.exception("_run_scheduled_energy_cost: %s rollback failed", label)
finally:
if session is not None:
try:
session.close()
except Exception:
# Sessions are intentionally isolated; close failures are
# diagnostic only and must remain best-effort too.
logger.exception("_run_scheduled_energy_cost: %s close failed", label)
# Electricity, thermal and HA publishing must not share failed transaction
# state or accidentally commit each other's partially-flushed changes.
run_scope("electricity computation", compute_closed_periods)
run_scope("thermal computation", compute_closed_meter_cost_periods)
def publish(session: Session) -> None:
# publish_states is internally guarded by _should_publish (MQTT
# disabled / disconnected -> no-op), but gets a clean Session anyway.
from app.services.ha_discovery import publish_states
publish_states(session)
run_scope("publish_states (non-fatal)", publish)
def _run_scheduled_ha_state_publish() -> None:
@@ -237,6 +257,10 @@ async def lifespan(_: FastAPI):
replace_existing=True,
max_instances=1,
coalesce=True,
# APScheduler otherwise waits one full interval before its first run.
# This preserves the hourly cadence while requesting a non-blocking
# startup fetch as soon as the scheduler starts.
next_run_time=datetime.now(UTC),
)
# Energy cost billing: compute uncalculated closed 15-minute periods every minute.
# The job is a no-op when no active contract or DSMR data is present, so it is
@@ -272,19 +296,28 @@ async def lifespan(_: FastAPI):
_startup_runtime_settings = build_runtime_settings(_startup_session, get_settings())
finally:
_startup_session.close()
mqtt_manager.connect(_startup_runtime_settings)
serial_started = False
try:
mqtt_manager.connect(_startup_runtime_settings)
# DSMR ingest: subscribe to the configured MQTT topic when enabled. The same
# applier is called from PUT /api/config, so toggling DSMR via the UI takes
# effect without an app restart.
apply_dsmr_subscription(_startup_runtime_settings)
# DSMR sources carry their own runtime configuration and are reconciled
# after the MQTT manager is connected.
apply_dsmr_subscription(_startup_runtime_settings)
# Mark it before reconcile: a partial reconcile can already own a fd or
# a non-daemon thread and must receive the same orderly shutdown.
serial_started = True
warmtelink_worker_manager.start()
yield
# MQTT: clean shutdown before the process exits.
mqtt_manager.disconnect()
scheduler.shutdown(wait=False)
yield
finally:
# Serial descriptors/workers must be handled first on every exit path.
if serial_started:
try:
warmtelink_worker_manager.shutdown()
except Exception:
logger.exception("WarmteLink shutdown failed")
mqtt_manager.disconnect()
scheduler.shutdown(wait=False)
def create_app() -> FastAPI:
@@ -308,7 +341,9 @@ def create_app() -> FastAPI:
app.include_router(api_data_router)
app.include_router(api_energy_router)
app.include_router(api_energy_contracts_router)
app.include_router(api_meter_costs_router)
app.include_router(api_meters_router)
app.include_router(api_meter_sources_router)
app.include_router(api_expose_router)
app.include_router(api_modbus_router)
app.include_router(api_session_router)
+4
View File
@@ -5,12 +5,16 @@ from app.models.config import AppConfigEntry
from app.models.location import Location
from app.models.poo import PooRecord
from app.models.public_ip import PublicIPHistory, PublicIPState
from app.models.meter_source import MeterSource, MeterSourceBinding, MeterSourceChannel
__all__ = [
"AppConfigEntry",
"AuthSession",
"AuthUser",
"Location",
"MeterSource",
"MeterSourceBinding",
"MeterSourceChannel",
"PooRecord",
"PublicIPHistory",
"PublicIPState",
+264 -24
View File
@@ -12,19 +12,134 @@ Six tables:
from __future__ import annotations
import uuid as _uuid
from datetime import datetime
from sqlalchemy import Boolean, DateTime, Float, ForeignKey, Integer, String
from sqlalchemy.orm import Mapped, mapped_column, relationship
from sqlalchemy.types import JSON
from datetime import datetime, timezone
from decimal import Decimal
from typing import Any
from sqlalchemy import (
Boolean,
CheckConstraint,
DateTime,
Float,
ForeignKey,
Index,
Integer,
Numeric,
String,
UniqueConstraint,
event,
text,
)
from sqlalchemy.orm import Mapped, mapped_column, relationship, synonym, validates
from sqlalchemy.types import JSON, TypeDecorator
from app.db import Base
from app.models.meter_source import MeterSourceBinding
def _uuid4_str() -> str:
return str(_uuid.uuid4())
def _decimal_json(value: Any) -> Any:
"""Make auditable JSON portable without admitting binary numeric values."""
if isinstance(value, Decimal):
return format(value, "f")
if isinstance(value, float) or (isinstance(value, int) and not isinstance(value, bool)):
raise ValueError("JSON amounts and quantities must be decimal strings, not numeric JSON values")
if isinstance(value, dict):
return {key: _decimal_json(child) for key, child in value.items()}
if isinstance(value, list):
return [_decimal_json(child) for child in value]
return value
def _validate_fixed_decimal(value: Decimal, precision: int, scale: int, field: str) -> Decimal:
if not isinstance(value, Decimal):
raise ValueError(f"{field} must be a Decimal, not a binary float or other numeric type")
if not value.is_finite():
raise ValueError(f"{field} must be finite")
if -value.as_tuple().exponent > scale:
raise ValueError(f"{field} exceeds scale {scale}")
integer_digits = 0 if value.is_zero() else max(value.copy_abs().adjusted() + 1, 0)
if integer_digits > precision - scale:
raise ValueError(f"{field} exceeds precision {precision},{scale}")
return value
class ExactDecimal(TypeDecorator[Decimal]):
"""Fixed-point Decimal which uses SQLite TEXT, never a binary float."""
impl = Numeric
cache_ok = True
def __init__(self, precision: int, scale: int) -> None:
self.precision = precision
self.scale = scale
super().__init__(precision=precision, scale=scale)
def load_dialect_impl(self, dialect):
if dialect.name == "sqlite":
return dialect.type_descriptor(String(self.precision + 2))
return dialect.type_descriptor(Numeric(self.precision, self.scale, asdecimal=True))
def process_bind_param(self, value: Decimal | None, dialect) -> Decimal | str | None:
if value is None:
return None
value = _validate_fixed_decimal(value, self.precision, self.scale, "decimal value")
if dialect.name == "sqlite":
return format(value, f".{self.scale}f")
return value
def process_result_value(self, value: Decimal | str | None, _dialect) -> Decimal | None:
return None if value is None else Decimal(value)
class DecimalJSON(TypeDecorator[dict]):
"""JSON which serializes Decimal values as strings on every write path."""
impl = JSON
cache_ok = True
def process_bind_param(self, value: Any, _dialect) -> Any:
return None if value is None else _decimal_json(value)
class UTCDateTime(TypeDecorator[datetime]):
"""UTC timestamps that preserve instant identity on SQLite and other dialects."""
impl = DateTime(timezone=True)
cache_ok = True
def __init__(self, field: str) -> None:
self.field = field
super().__init__()
def process_bind_param(self, value: datetime | None, _dialect) -> datetime | None:
return None if value is None else _normalise_utc_period(value, self.field)
def process_result_value(self, value: datetime | None, _dialect) -> datetime | None:
if value is None:
return None
if value.tzinfo is None or value.utcoffset() is None:
return value.replace(tzinfo=timezone.utc)
return value.astimezone(timezone.utc)
def _require_aware_period(start: datetime, end: datetime) -> None:
if start.tzinfo is None or start.utcoffset() is None:
raise ValueError("period_start must be timezone-aware")
if end.tzinfo is None or end.utcoffset() is None:
raise ValueError("period_end must be timezone-aware")
if end <= start:
raise ValueError("period_end must be after period_start")
def _normalise_utc_period(value: datetime, field: str) -> datetime:
if value.tzinfo is None or value.utcoffset() is None:
raise ValueError(f"{field} must be timezone-aware")
return value.astimezone(timezone.utc)
class Meter(Base):
"""One physical electricity meter's installation epoch.
@@ -53,9 +168,7 @@ class Meter(Base):
id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True)
# Stable internal identity — used as HA Discovery unique_id anchor.
uuid: Mapped[str] = mapped_column(
String(36), unique=True, nullable=False, default=_uuid4_str
)
uuid: Mapped[str] = mapped_column(String(36), unique=True, nullable=False, default=_uuid4_str)
# Human-readable label for this physical meter (e.g. address, serial, tariff zone).
label: Mapped[str] = mapped_column(String(255), nullable=False)
@@ -85,6 +198,10 @@ class Meter(Base):
back_populates="meter", cascade="save-update, merge"
)
source_bindings: Mapped[list["MeterSourceBinding"]] = relationship(
back_populates="meter", cascade="save-update, merge"
)
class DsmrReading(Base):
"""One down-sampled DSMR telegram stored as a full JSON blob.
@@ -93,8 +210,9 @@ class DsmrReading(Base):
(that field overflows and must be manually reset to zero — a known DSMR
quirk — so relying on it for uniqueness risks silently dropping new data).
The table's own autoincrement ``id`` PK is the stable internal identity, and
``recorded_at`` (the telegram timestamp) is the UNIQUE de-duplication key: a
single P1 meter emits exactly one telegram per timestamp.
``(meter_source_id, recorded_at)`` is the UNIQUE de-duplication key: each
configured P1 source emits at most one telegram per timestamp, while
different sources may legitimately emit at the same instant.
``recorded_at`` is a real column (not inside the payload) so time-range
queries are efficient. The entire telegram frame is stored verbatim in
@@ -106,29 +224,54 @@ class DsmrReading(Base):
id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True)
# UTC timestamp of the sample — real column, UNIQUE (telegram-id-independent
# idempotency key). The unique index also serves time-range queries.
recorded_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, unique=True
)
# UTC timestamp of the sample. Idempotency is per configured source, so
# distinct P1 sources may legitimately emit at the same instant.
recorded_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
# Telegram's own id (DSMR Reader assigns it). Stored only as a reference /
# debugging aid — NOT used for uniqueness or idempotency (it overflows and
# gets reset to zero). Nullable because some DSMR sources may not emit one.
source_id: Mapped[int | None] = mapped_column(Integer, nullable=True)
telegram_id: Mapped[int | None] = mapped_column(Integer, nullable=True)
# Compatibility for the pre-M8 ingest implementation. This is an ORM
# alias only; the physical database column is ``telegram_id``.
source_id = synonym("telegram_id")
# The configured source is the durable identity of the cumulative reading
# stream. It is non-null after the revision-16 historical adoption.
meter_source_id: Mapped[int] = mapped_column(
ForeignKey("meter_source.id", ondelete="RESTRICT"), nullable=False, index=True
)
# Full telegram frame as a JSON object; values are typically JSON strings
# (e.g. "20915.154") — callers must cast to Decimal before arithmetic.
payload: Mapped[dict] = mapped_column(JSON, nullable=False)
__table_args__ = (
UniqueConstraint(
"meter_source_id", "recorded_at", name="uq_dsmr_reading_source_recorded_at"
),
)
@event.listens_for(DsmrReading, "before_insert")
def _supply_legacy_dsmr_source(_mapper, connection, target: DsmrReading) -> None:
"""Keep the pre-T04 single-source writer working during the schema handoff."""
if target.meter_source_id is None:
target.meter_source_id = connection.execute(
text("SELECT id FROM meter_source WHERE kind = 'dsmr_mqtt' ORDER BY id LIMIT 1")
).scalar_one()
class EnergyContract(Base):
"""Contract head: a named energy contract with a chosen pricing strategy.
``kind`` determines which price strategy is used (``manual`` for fixed
dual-tariff rates entered by the user, ``tibber`` for dynamic API prices).
Only one contract may be ``active`` at a time; the service layer enforces
mutual exclusion. Specific pricing values live in ``EnergyContractVersion``
A contract belongs to an energy ``scope`` (currently electricity; thermal
profiles are reserved for the next milestone). Only one contract may be
``active`` per scope; the service layer enforces mutual exclusion. Specific
pricing values live in ``EnergyContractVersion``
so that price changes can be tracked without modifying historical records.
"""
@@ -144,6 +287,12 @@ class EnergyContract(Base):
# migration simple and the strategy registry extensible.
kind: Mapped[str] = mapped_column(String(32), nullable=False)
# Billing domain. The service registry derives this from ``kind`` so API
# callers cannot move a pricing strategy into an incompatible domain.
scope: Mapped[str] = mapped_column(
String(32), nullable=False, default="electricity", index=True
)
# Whether this is the currently active contract (at most one should be True).
active: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False)
@@ -179,15 +328,11 @@ class EnergyContractVersion(Base):
)
# Start of this version's validity window (inclusive, UTC).
effective_from: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False
)
effective_from: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
# End of this version's validity window (exclusive, UTC). NULL means open-ended
# (i.e. this is the most recent / current version).
effective_to: Mapped[datetime | None] = mapped_column(
DateTime(timezone=True), nullable=True
)
effective_to: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
# Pricing values as a JSON object conforming to the profile structure for
# ``contract.kind`` (validated by the application layer against the YAML profile).
@@ -292,6 +437,12 @@ class EnergyCostPeriod(Base):
ForeignKey("meter.id", ondelete="RESTRICT"), nullable=True
)
# Nullable for historical and degraded rows. Every new normal period
# points at the one binding that supplied both cumulative endpoints.
source_binding_id: Mapped[int | None] = mapped_column(
ForeignKey("meter_source_binding.id", ondelete="RESTRICT"), nullable=True
)
# True when the period was computed with incomplete data (missing readings or
# missing price); serves as a flag for later recomputation.
degraded: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False)
@@ -307,6 +458,95 @@ class EnergyCostPeriod(Base):
# Relationship back to the meter epoch.
meter: Mapped["Meter | None"] = relationship(back_populates="cost_periods")
source_binding: Mapped["MeterSourceBinding | None"] = relationship(
back_populates="cost_periods"
)
class MeterCostPeriod(Base):
"""Auditable commodity-scoped ledger row for one half-open metering period.
A degraded row intentionally permits missing audit links; services must
still require them before writing a normal row.
"""
__tablename__ = "meter_cost_period"
id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True)
commodity: Mapped[str] = mapped_column(String(32), nullable=False)
period_start: Mapped[datetime] = mapped_column(UTCDateTime("period_start"), nullable=False)
period_end: Mapped[datetime] = mapped_column(UTCDateTime("period_end"), nullable=False)
meter_id: Mapped[int | None] = mapped_column(
ForeignKey("meter.id", ondelete="RESTRICT"), nullable=True
)
source_binding_id: Mapped[int | None] = mapped_column(
ForeignKey("meter_source_binding.id", ondelete="RESTRICT"), nullable=True
)
contract_version_id: Mapped[int | None] = mapped_column(
ForeignKey("energy_contract_version.id", ondelete="RESTRICT"), nullable=True
)
# SQLite reliably round-trips at most fifteen significant decimal digits.
# WarmteLink itself reports 0.001 units, so nine cost fractional digits
# retain a six-place tariff times that source precision without float loss.
quantity: Mapped[Decimal] = mapped_column(ExactDecimal(15, 6), nullable=False)
cost: Mapped[Decimal] = mapped_column(ExactDecimal(15, 9), nullable=False)
currency: Mapped[str] = mapped_column(String(8), nullable=False)
cost_breakdown: Mapped[dict] = mapped_column(DecimalJSON(), nullable=False, default=dict)
pricing_snapshot: Mapped[dict] = mapped_column(DecimalJSON(), nullable=False, default=dict)
quality: Mapped[str] = mapped_column(String(32), nullable=False, default="valid")
degraded: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False)
degraded_reason: Mapped[str | None] = mapped_column(String(255), nullable=True)
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
meter: Mapped["Meter | None"] = relationship()
source_binding: Mapped["MeterSourceBinding | None"] = relationship()
contract_version: Mapped["EnergyContractVersion | None"] = relationship()
__table_args__ = (
CheckConstraint(
"degraded OR (meter_id IS NOT NULL AND source_binding_id IS NOT NULL "
"AND contract_version_id IS NOT NULL)",
name="ck_meter_cost_period_normal_audit_links",
),
CheckConstraint("period_end > period_start", name="ck_meter_cost_period_positive_interval"),
CheckConstraint(
"NOT degraded OR (degraded_reason IS NOT NULL AND length(trim(degraded_reason)) > 0)",
name="ck_meter_cost_period_degraded_reason",
),
UniqueConstraint("commodity", "period_start", name="uq_meter_cost_period_commodity_start"),
Index("ix_meter_cost_period_commodity_start", "commodity", "period_start"),
Index("ix_meter_cost_period_source_binding_id", "source_binding_id"),
)
@validates("cost_breakdown", "pricing_snapshot")
def _validate_decimal_json(self, _key: str, value: dict) -> dict:
return _decimal_json(value)
@validates("quantity", "cost")
def _validate_fixed_decimal(self, key: str, value: Decimal) -> Decimal:
precision, scale = (15, 6) if key == "quantity" else (15, 9)
return _validate_fixed_decimal(value, precision, scale, key)
@validates("period_start", "period_end")
def _normalise_period(self, key: str, value: datetime) -> datetime:
return _normalise_utc_period(value, key)
@event.listens_for(MeterCostPeriod, "before_insert")
@event.listens_for(MeterCostPeriod, "before_update")
def _validate_meter_cost_period(_mapper, _connection, target: MeterCostPeriod) -> None:
_require_aware_period(target.period_start, target.period_end)
if not target.degraded and (
target.meter_id is None
or target.source_binding_id is None
or target.contract_version_id is None
):
raise ValueError("normal meter cost periods require meter, binding, and contract version")
if target.degraded and not target.degraded_reason:
raise ValueError("degraded meter cost periods require a degraded_reason")
# Index on recorded_at for efficient time-range queries on DSMR readings.
# (The ORM-level index=True on recorded_at already creates ix_dsmr_reading_recorded_at;
+165
View File
@@ -0,0 +1,165 @@
"""Protocol-agnostic source, channel, and meter-binding identity models."""
from __future__ import annotations
import uuid as _uuid
from datetime import datetime
from decimal import Decimal
from typing import TYPE_CHECKING
from sqlalchemy import (
Boolean,
CheckConstraint,
DateTime,
ForeignKey,
Index,
Integer,
Numeric,
String,
UniqueConstraint,
)
from sqlalchemy.orm import Mapped, mapped_column, relationship
from sqlalchemy.types import JSON
from app.db import Base
if TYPE_CHECKING:
from app.models.energy import EnergyCostPeriod, Meter
def _uuid4_str() -> str:
return str(_uuid.uuid4())
def half_open_intervals_overlap(
started_at: datetime,
ended_at: datetime | None,
other_started_at: datetime,
other_ended_at: datetime | None,
) -> bool:
"""Return whether two ``[started_at, ended_at)`` intervals overlap.
``None`` denotes an open-ended interval. Equal boundaries do not overlap,
which lets a source binding hand off at one exact timestamp.
"""
return (other_ended_at is None or started_at < other_ended_at) and (
ended_at is None or other_started_at < ended_at
)
class MeterSource(Base):
"""A configured protocol connection that discovers one or more channels."""
__tablename__ = "meter_source"
id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True)
uuid: Mapped[str] = mapped_column(String(36), unique=True, nullable=False, default=_uuid4_str)
name: Mapped[str] = mapped_column(String(255), nullable=False)
kind: Mapped[str] = mapped_column(String(64), nullable=False)
enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True)
config: Mapped[dict] = mapped_column(JSON, nullable=False, default=dict)
status: Mapped[str] = mapped_column(String(32), nullable=False, default="unknown")
last_seen_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
last_error: Mapped[str | None] = mapped_column(String(1024), nullable=True)
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
channels: Mapped[list["MeterSourceChannel"]] = relationship(
back_populates="source", cascade="save-update, merge"
)
class MeterSourceChannel(Base):
"""A stable cumulative measurement identity discovered from a source."""
__tablename__ = "meter_source_channel"
id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True)
uuid: Mapped[str] = mapped_column(String(36), unique=True, nullable=False, default=_uuid4_str)
source_id: Mapped[int] = mapped_column(
ForeignKey("meter_source.id", ondelete="RESTRICT"), nullable=False, index=True
)
channel_key: Mapped[str] = mapped_column(String(128), nullable=False)
label: Mapped[str] = mapped_column(String(255), nullable=False)
suggested_commodity: Mapped[str | None] = mapped_column(String(32), nullable=True)
unit: Mapped[str] = mapped_column(String(32), nullable=False)
device_type: Mapped[str | None] = mapped_column(String(64), nullable=True)
fingerprint: Mapped[str | None] = mapped_column(String(64), nullable=True)
latest_value: Mapped[float | None] = mapped_column(Numeric(20, 6), nullable=True)
latest_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
latest_quality: Mapped[str | None] = mapped_column(String(32), nullable=True)
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
source: Mapped["MeterSource"] = relationship(back_populates="channels")
bindings: Mapped[list["MeterSourceBinding"]] = relationship(
back_populates="channel", cascade="save-update, merge"
)
warmtelink_readings: Mapped[list["WarmteLinkReading"]] = relationship(
back_populates="channel", cascade="save-update, merge"
)
__table_args__ = (
UniqueConstraint("source_id", "channel_key", name="uq_meter_source_channel_source_key"),
)
class WarmteLinkReading(Base):
"""One accepted scalar cumulative reading from a WarmteLink channel."""
__tablename__ = "warmtelink_reading"
id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True)
channel_id: Mapped[int] = mapped_column(
ForeignKey("meter_source_channel.id", ondelete="RESTRICT"), nullable=False
)
# These timestamps retain their UTC-aware application semantics. SQLite
# stores them without an offset, so callers must always supply aware UTC.
recorded_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
received_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
# SQLite's ORM Numeric path can exactly round-trip this 12-integer-digit
# range at scale 3. That is ample for a long-lived cumulative meter while
# retaining the protocol's 0.001 resolution without float conversion.
value: Mapped[Decimal] = mapped_column(Numeric(15, 3), nullable=False)
unit: Mapped[str] = mapped_column(String(32), nullable=False)
quality: Mapped[str] = mapped_column(String(32), nullable=False)
equipment_fingerprint: Mapped[str] = mapped_column(String(64), nullable=False)
channel: Mapped["MeterSourceChannel"] = relationship(back_populates="warmtelink_readings")
__table_args__ = (
CheckConstraint(
"quality IN ('valid', 'invalid', 'unverifiable')",
name="ck_warmtelink_reading_quality",
),
UniqueConstraint("channel_id", "recorded_at", name="uq_warmtelink_reading_channel_recorded_at"),
Index("ix_warmtelink_reading_recorded_at", "recorded_at"),
)
class MeterSourceBinding(Base):
"""Connect one source channel to one physical meter for a half-open window."""
__tablename__ = "meter_source_binding"
id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True)
uuid: Mapped[str] = mapped_column(String(36), unique=True, nullable=False, default=_uuid4_str)
meter_id: Mapped[int] = mapped_column(
ForeignKey("meter.id", ondelete="RESTRICT"), nullable=False, index=True
)
channel_id: Mapped[int] = mapped_column(
ForeignKey("meter_source_channel.id", ondelete="RESTRICT"), nullable=False, index=True
)
started_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
ended_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
meter: Mapped["Meter"] = relationship(back_populates="source_bindings")
channel: Mapped["MeterSourceChannel"] = relationship(back_populates="bindings")
cost_periods: Mapped[list["EnergyCostPeriod"]] = relationship(
back_populates="source_binding", cascade="save-update, merge", passive_deletes="all"
)
Index("ix_meter_source_kind_enabled", MeterSource.kind, MeterSource.enabled)
+26 -3
View File
@@ -76,6 +76,24 @@ class PricesResponse(BaseModel):
"Null for tibber contracts and when no active contract exists."
),
)
contract_version_id: int | None = Field(
default=None,
description="Thermal active contract version identifier; omitted for electricity.",
)
effective_from: datetime | None = Field(
default=None,
description="Thermal contract version start; omitted for electricity.",
)
effective_to: datetime | None = Field(
default=None,
description="Thermal contract version end; omitted for electricity.",
)
values: dict[str, dict[str, str]] | None = Field(
default=None,
description="Thermal normalized Decimal-string contract values; omitted for electricity.",
)
model_config = {"ser_json_exclude_none": True}
# ---------------------------------------------------------------------------
@@ -95,13 +113,18 @@ class CostPeriodSchema(BaseModel):
export_revenue: float = Field(description="Revenue from electricity fed to grid (EUR).")
net_cost: float = Field(description="import_cost export_revenue (EUR).")
currency: str = Field(description="ISO 4217 currency code.")
degraded: bool = Field(
description="True when the period was computed with incomplete data."
)
degraded: bool = Field(description="True when the period was computed with incomplete data.")
contract_version_id: int | None = Field(
default=None,
description="FK to the contract version used for this billing period (null when degraded).",
)
source_binding_id: int | None = Field(
default=None,
description=(
"FK to the source binding that supplied both cumulative endpoints "
"(null for legacy or degraded periods)."
),
)
model_config = {"from_attributes": True}
+3
View File
@@ -51,6 +51,7 @@ class ContractResponse(BaseModel):
id: int
name: str
kind: str
scope: str
active: bool
currency: str
created_at: datetime
@@ -69,6 +70,7 @@ class ContractDetailResponse(BaseModel):
id: int
name: str
kind: str
scope: str
active: bool
currency: str
created_at: datetime
@@ -101,6 +103,7 @@ class ContractCreate(BaseModel):
name: str = Field(..., min_length=1, max_length=255)
kind: str = Field(..., min_length=1, max_length=32)
scope: str | None = Field(default=None, min_length=1, max_length=32)
currency: str = Field(default="EUR", min_length=1, max_length=8)
values: dict[str, Any]
effective_from: datetime | None = Field(
+23
View File
@@ -49,10 +49,21 @@ class MeterResponse(BaseModel):
reason: str
note: str | None
created_at: datetime
bindings: list["MeterBindingSummary"] = Field(default_factory=list)
model_config = {"from_attributes": True}
class MeterBindingSummary(BaseModel):
"""Stable, non-sensitive binding identity embedded in meter responses."""
uuid: str
source_channel_uuid: str
source_uuid: str
started_at: datetime
ended_at: datetime | None
class MeterListResponse(BaseModel):
"""Response schema for GET /api/energy/meters.
@@ -107,6 +118,12 @@ class MeterDeclareRequest(BaseModel):
max_length=32,
description="Energy commodity this meter measures. Defaults to 'electricity'.",
)
source_channel_uuid: str | None = Field(
default=None,
min_length=1,
max_length=36,
description="Optional compatible source channel to bind atomically to this meter.",
)
class MeterPatchRequest(BaseModel):
@@ -129,3 +146,9 @@ class MeterPatchRequest(BaseModel):
"Triggers billing recompute over the affected window."
),
)
class MeterCloseRequest(BaseModel):
"""Close the active meter epoch at an exclusive end boundary."""
ended_at: datetime
+62
View File
@@ -0,0 +1,62 @@
"""Schemas for the commodity-scoped thermal cost ledger."""
from __future__ import annotations
from datetime import datetime
from typing import Literal
from pydantic import BaseModel, Field
class MeterCostPeriodSchema(BaseModel):
"""One auditable thermal ledger row; all Decimal values are JSON strings."""
commodity: Literal["heating", "hot_water"]
period_start: datetime
period_end: datetime
meter_id: int | None
source_binding_id: int | None
contract_version_id: int | None
quantity: str
cost: str
currency: str
cost_breakdown: dict[str, str]
pricing_snapshot: dict[str, dict[str, str]]
quality: str
degraded: bool
degraded_reason: str | None
class MeterCostsResponse(BaseModel):
items: list[MeterCostPeriodSchema]
total: int = Field(description="Total matching rows before pagination.")
class ThermalFixedBreakdown(BaseModel):
"""D11 annual-standing charges accrued per settled local day, as Decimal strings."""
heating_network: str
metering: str
delivery_set: str
hot_water_network: str
other: str
class ThermalCostSummaryResponse(BaseModel):
currency: str
heating: str
hot_water_heating: str
hot_water: str
hot_water_tax: str
variable_subtotal: str
fixed_breakdown: ThermalFixedBreakdown
fixed_subtotal: str
all_in: str
period_count: int
degraded_count: int
class MeterCostRecomputeResponse(BaseModel):
processed: int
normal: int
degraded: int
+162
View File
@@ -0,0 +1,162 @@
"""Public schemas for protocol-agnostic meter sources and bindings."""
from __future__ import annotations
from datetime import datetime
from decimal import Decimal
from typing import Any
from pydantic import BaseModel, Field
class SourceConfigFieldResponse(BaseModel):
name: str
value_type: str
default: Any = None
required: bool
secret: bool
class SourceProfileResponse(BaseModel):
kind: str
fields: list[SourceConfigFieldResponse]
defaults: dict[str, Any]
capabilities: list[str]
allowed_units: list[str]
class SourceProfilesResponse(BaseModel):
items: list[SourceProfileResponse]
class MeterSourceCreate(BaseModel):
name: str = Field(..., min_length=1, max_length=255)
kind: str = Field(..., min_length=1, max_length=64)
config: dict[str, Any] = Field(default_factory=dict)
enabled: bool = True
class MeterSourcePatch(BaseModel):
name: str | None = Field(default=None, min_length=1, max_length=255)
config: dict[str, Any] | None = None
enabled: bool | None = None
class MeterSourceResponse(BaseModel):
uuid: str
name: str
kind: str
enabled: bool
config: dict[str, Any]
status: str
last_seen_at: datetime | None
last_error: str | None
created_at: datetime
updated_at: datetime
class MeterSourceListResponse(BaseModel):
items: list[MeterSourceResponse]
total: int
class DiscoverResponse(BaseModel):
requested: bool
supported: bool
status: str
request_id: int | None = None
detail: str | None = None
channels: list["DiscoverChannelResponse"] = Field(default_factory=list)
class DiscoverChannelResponse(BaseModel):
uuid: str
label: str
unit: str
latest_value: Decimal | None
latest_at: datetime | None
latest_quality: str | None
class ChannelBindingSummaryResponse(BaseModel):
count: int
meter_ids: list[int]
class CommodityResponse(BaseModel):
key: str
unit: str
capabilities: list[str]
class CommoditiesResponse(BaseModel):
items: list[CommodityResponse]
class MeterSourceChannelResponse(BaseModel):
uuid: str
label: str
suggested_commodity: str | None
unit: str
device_type: str | None
latest_value: Decimal | None
latest_at: datetime | None
latest_quality: str | None
binding_count: int
bound_meter_ids: list[int]
binding_summary: ChannelBindingSummaryResponse
class MeterSourceChannelListResponse(BaseModel):
items: list[MeterSourceChannelResponse]
total: int
source_status: str
class ChannelReadingResponse(BaseModel):
recorded_at: datetime
value: Decimal | None = None
quality: str | None = None
class ChannelReadingsResponse(BaseModel):
items: list[ChannelReadingResponse]
total: int
class BindingCreate(BaseModel):
source_channel_uuid: str = Field(..., min_length=1, max_length=36)
started_at: datetime
ended_at: datetime | None = None
class BindingPatch(BaseModel):
started_at: datetime | None = None
ended_at: datetime | None = None
class BindingTransferRequest(BaseModel):
from_binding_uuid: str = Field(..., min_length=1, max_length=36)
to_source_channel_uuid: str = Field(..., min_length=1, max_length=36)
effective_at: datetime
class BindingResponse(BaseModel):
uuid: str
meter_id: int
source_channel_uuid: str
source_uuid: str
started_at: datetime
ended_at: datetime | None
created_at: datetime
updated_at: datetime
class BindingListResponse(BaseModel):
items: list[BindingResponse]
total: int
class BindingTransferResponse(BaseModel):
closed_binding: BindingResponse
created_binding: BindingResponse
+13 -29
View File
@@ -109,6 +109,7 @@ CONFIG_FIELDS: tuple[ConfigField, ...] = (
ConfigField("MQTT", "MQTT_USERNAME", "mqtt_username", "MQTT Username"),
ConfigField("MQTT", "MQTT_PASSWORD", "mqtt_password", "MQTT Password", secret=True),
ConfigField("MQTT", "MQTT_TLS_ENABLED", "mqtt_tls_enabled", "MQTT TLS Enabled", input_type="checkbox"),
ConfigField("MQTT", "MQTT_CLIENT_ID", "mqtt_client_id", "MQTT Client ID"),
ConfigField(
"Home Assistant Discovery",
"HA_DISCOVERY_ENABLED",
@@ -129,27 +130,6 @@ CONFIG_FIELDS: tuple[ConfigField, ...] = (
"HA State Topic Prefix",
),
ConfigField("Modbus", "MODBUS_POLLING_ENABLED", "modbus_polling_enabled", "Modbus Polling Enabled", input_type="checkbox"),
ConfigField(
"DSMR",
"DSMR_INGEST_ENABLED",
"dsmr_ingest_enabled",
"DSMR Ingest Enabled",
input_type="checkbox",
),
ConfigField("DSMR", "DSMR_MQTT_TOPIC", "dsmr_mqtt_topic", "DSMR MQTT Topic"),
ConfigField(
"DSMR",
"DSMR_SAMPLE_INTERVAL_S",
"dsmr_sample_interval_s",
"DSMR Sample Interval (s)",
input_type="number",
),
ConfigField(
"DSMR",
"DSMR_TARIFF_TOPIC",
"dsmr_tariff_topic",
"DSMR Tariff Topic",
),
ConfigField(
"Tibber",
"TIBBER_API_TOKEN",
@@ -243,7 +223,12 @@ def save_config_updates(session: Session, form_data: dict[str, str], bootstrap_s
else:
merged_values[field.env_name] = submitted_value
_validate_config_values(merged_values, bootstrap_settings)
validated_settings = _validate_config_values(merged_values, bootstrap_settings)
# Persist the canonical client identity as well as using it at runtime. A
# whitespace-padded value must not survive in app_config and unexpectedly
# reappear in another consumer of the stored settings.
if "MQTT_CLIENT_ID" in merged_values:
merged_values["MQTT_CLIENT_ID"] = validated_settings.mqtt_client_id
_persist_config_values(session, merged_values)
get_settings.cache_clear()
reset_db_caches()
@@ -258,7 +243,9 @@ def save_config_value(
) -> None:
current_values = _read_config_values(session)
current_values[env_name] = value
_validate_config_values(current_values, bootstrap_settings)
validated_settings = _validate_config_values(current_values, bootstrap_settings)
if env_name == "MQTT_CLIENT_ID":
current_values[env_name] = validated_settings.mqtt_client_id
_persist_config_values(session, current_values)
get_settings.cache_clear()
reset_db_caches()
@@ -277,14 +264,14 @@ def _read_config_values(session: Session) -> dict[str, str]:
return {row.key: row.value for row in rows}
def _validate_config_values(config_values: dict[str, str], bootstrap_settings: Settings) -> None:
def _validate_config_values(config_values: dict[str, str], bootstrap_settings: Settings) -> Settings:
payload = _settings_payload(bootstrap_settings)
for field in CONFIG_FIELDS:
if field.env_name in config_values:
payload[field.setting_attr] = config_values[field.env_name]
try:
Settings(_env_file=None, **payload)
return Settings(_env_file=None, **payload)
except Exception as exc:
raise ConfigSaveError("invalid config submission") from exc
@@ -355,13 +342,10 @@ def _settings_payload(settings: Settings) -> dict[str, Any]:
"mqtt_username": settings.mqtt_username,
"mqtt_password": settings.mqtt_password,
"mqtt_tls_enabled": settings.mqtt_tls_enabled,
"mqtt_client_id": settings.mqtt_client_id,
"ha_discovery_enabled": settings.ha_discovery_enabled,
"ha_discovery_prefix": settings.ha_discovery_prefix,
"ha_state_topic_prefix": settings.ha_state_topic_prefix,
"dsmr_ingest_enabled": settings.dsmr_ingest_enabled,
"dsmr_mqtt_topic": settings.dsmr_mqtt_topic,
"dsmr_sample_interval_s": settings.dsmr_sample_interval_s,
"dsmr_tariff_topic": settings.dsmr_tariff_topic,
"tibber_api_token": settings.tibber_api_token,
"tibber_home_id": settings.tibber_home_id,
}
+63 -18
View File
@@ -11,8 +11,8 @@ Design decisions
setting its ``effective_to`` to the new version's ``effective_from``; raises
``ContractVersionError`` if the new date is strictly earlier than the previous
version's ``effective_from``.
- ``activate_contract``: mutual-exclusion; sets all other contracts' ``active``
to False, then sets the given contract's ``active`` to True.
- ``activate_contract``: scope-local mutual exclusion; sets other contracts in
the target scope inactive, then sets the given contract active.
- ``active_contract_version_at``: returns the single version of the currently
active contract that covers *ts* (``effective_from ts < effective_to``,
or open-ended when ``effective_to`` is None).
@@ -32,7 +32,7 @@ import logging
from datetime import UTC, datetime
from typing import Any
from sqlalchemy import select
from sqlalchemy import select, update
from sqlalchemy.orm import Session
from app.integrations.pricing.profiles import validate_values
@@ -41,6 +41,32 @@ from app.models.energy import EnergyContract, EnergyContractVersion
logger = logging.getLogger(__name__)
# This is deliberately separate from the pricing-profile loader. T12 needs to
# reserve the thermal domain before T13 supplies its actual profile.
CONTRACT_KIND_SCOPES: dict[str, str] = {
"manual": "electricity",
"tibber": "electricity",
"district_heating": "thermal",
}
class ContractScopeError(ValueError):
"""Raised when a contract kind is unknown or its supplied scope disagrees."""
def contract_scope_for_kind(kind: str, requested_scope: str | None = None) -> str:
"""Return the registry-owned scope for *kind*, rejecting client mismatches."""
try:
scope = CONTRACT_KIND_SCOPES[kind]
except KeyError as exc:
raise ContractScopeError(f"Unknown energy contract kind: {kind!r}") from exc
if requested_scope is not None and requested_scope != scope:
raise ContractScopeError(
f"Contract kind {kind!r} belongs to scope {scope!r}, not {requested_scope!r}."
)
return scope
# ---------------------------------------------------------------------------
# Internal helpers
# ---------------------------------------------------------------------------
@@ -86,10 +112,14 @@ def get_contract_or_none(session: Session, contract_id: int) -> EnergyContract |
).scalar_one_or_none()
def list_contracts(session: Session) -> list[EnergyContract]:
"""Return all contracts ordered by id (ascending)."""
def list_contracts(session: Session, *, scope: str = "electricity") -> list[EnergyContract]:
"""Return contracts in one scope, ordered by id (ascending)."""
return list(
session.execute(select(EnergyContract).order_by(EnergyContract.id)).scalars().all()
session.execute(
select(EnergyContract)
.where(EnergyContract.scope == scope)
.order_by(EnergyContract.id)
).scalars().all()
)
@@ -117,6 +147,7 @@ def create_contract(
name: str,
kind: str,
currency: str = "EUR",
scope: str | None = None,
values: dict[str, Any],
effective_from: datetime,
) -> EnergyContract:
@@ -129,11 +160,14 @@ def create_contract(
name:
Human-readable label for the contract.
kind:
Pricing strategy identifier (``"manual"`` or ``"tibber"``).
Pricing strategy identifier (``"manual"``, ``"tibber"``, or
``"district_heating"``).
currency:
ISO 4217 currency code (default ``"EUR"``).
values:
Pricing values dict conforming to the named profile's structure.
Pricing values dict conforming to the named profile's structure. The
district-heating profile normalises its Decimal-safe values to strings
before the JSON snapshot is stored.
Validated via ``validate_values(kind, values)`` before any writes.
effective_from:
UTC datetime at which the first pricing version takes effect.
@@ -150,6 +184,7 @@ def create_contract(
ProfileValidationError
If *values* does not conform to the profile structure.
"""
resolved_scope = contract_scope_for_kind(kind, scope)
# Validate (and fill defaults) before any DB write.
filled_values = validate_values(kind, values)
@@ -157,6 +192,7 @@ def create_contract(
contract = EnergyContract(
name=name,
kind=kind,
scope=resolved_scope,
currency=currency,
active=False, # New contracts are inactive; caller must explicitly activate.
created_at=now,
@@ -262,15 +298,18 @@ def add_version(
def activate_contract(session: Session, contract: EnergyContract) -> None:
"""Activate a contract with mutual exclusion.
Sets every other contract's ``active`` flag to False, then sets the given
contract's ``active`` to True. This guarantees at most one active contract
at any time.
Sets every other contract in the same scope inactive, then sets the given
contract active. This guarantees at most one active contract per scope.
Caller must commit after this returns.
"""
# Deactivate all contracts (including the target; we re-activate below).
for other in session.execute(select(EnergyContract)).scalars().all():
other.active = False
# This bulk update is a single write statement inside the caller's
# transaction. SQLite serializes writers, and another scope is never touched.
session.execute(
update(EnergyContract)
.where(EnergyContract.scope == contract.scope, EnergyContract.id != contract.id)
.values(active=False)
)
contract.active = True
contract.updated_at = datetime.now(UTC)
logger.info("Activated contract %r (id=%d)", contract.name, contract.id)
@@ -286,7 +325,9 @@ def deactivate_contract(session: Session, contract: EnergyContract) -> None:
logger.info("Deactivated contract %r (id=%d)", contract.name, contract.id)
def active_contract_versions(session: Session) -> list[EnergyContractVersion]:
def active_contract_versions(
session: Session, *, scope: str = "electricity"
) -> list[EnergyContractVersion]:
"""Return all versions of the currently active contract, ordered by effective_from ascending.
Returns an empty list when there is no active contract. The list spans the
@@ -295,7 +336,9 @@ def active_contract_versions(session: Session) -> list[EnergyContractVersion]:
cost / credit accumulation (Principle C).
"""
active = session.execute(
select(EnergyContract).where(EnergyContract.active.is_(True)).limit(1)
select(EnergyContract)
.where(EnergyContract.active.is_(True), EnergyContract.scope == scope)
.limit(1)
).scalar_one_or_none()
if active is None:
@@ -313,7 +356,7 @@ def active_contract_versions(session: Session) -> list[EnergyContractVersion]:
def active_contract_version_at(
session: Session, ts: datetime
session: Session, ts: datetime, *, scope: str = "electricity"
) -> EnergyContractVersion | None:
"""Return the active contract's version that covers *ts*.
@@ -336,7 +379,9 @@ def active_contract_version_at(
EnergyContractVersion | None
"""
active = session.execute(
select(EnergyContract).where(EnergyContract.active.is_(True)).limit(1)
select(EnergyContract)
.where(EnergyContract.active.is_(True), EnergyContract.scope == scope)
.limit(1)
).scalar_one_or_none()
if active is None:
+380 -246
View File
@@ -1,46 +1,21 @@
"""DSMR telegram ingest service.
Subscribes to the DSMR Reader MQTT topic (``dsmr/json``) and persists
down-sampled DSMR telegram frames to the ``dsmr_reading`` table.
Design decisions
----------------
- **Whole-frame storage**: the entire parsed telegram dict is stored as a JSON
blob in ``DsmrReading.payload``; no field allow-list is applied. This lets
future commodities (gas, heating, three-phase) be accommodated without a
table-schema change.
- **10-second down-sampling** (configurable via ``dsmr_sample_interval_s``):
only telegrams whose ``timestamp`` second falls on an exact multiple of the
interval are persisted. This reduces write volume from ~60 rows/min to ~6
rows/min while guaranteeing that every 15-minute boundary (second=00) is
captured.
- **Idempotency**: the telegram's own ``id`` field is stored as ``source_id``
with a UNIQUE constraint. A second delivery of the same telegram (e.g. after
a broker reconnect) is silently skipped.
- **Network-thread safety**: ``handle_message`` is called from paho's background
loop thread. It opens and closes its own short-lived SQLAlchemy session and
swallows all exceptions so that a buggy payload or transient DB error never
crashes the paho loop or drops the MQTT connection.
- **Numeric values kept as strings**: the DSMR Reader emits all numeric readings
as JSON strings (e.g. ``"20915.154"``). They are stored verbatim; conversion
to ``Decimal`` is deferred to the billing engine (T07) where precision matters.
- **Null phases**: some telegrams omit certain phase readings (``null`` in JSON);
these are stored as-is without special handling.
"""
"""DSMR MQTT ingest, keyed by durable meter-source identity."""
from __future__ import annotations
import json
import logging
import threading
from dataclasses import dataclass
from datetime import datetime, timezone
from typing import TYPE_CHECKING
import sqlalchemy.exc
from sqlalchemy import select
from sqlalchemy.orm import Session
from app.db import get_session_local
from app.models.energy import DsmrReading
from app.models.energy import DsmrReading, Meter
from app.models.meter_source import MeterSource, MeterSourceBinding, MeterSourceChannel
if TYPE_CHECKING:
from app.config import Settings
@@ -48,250 +23,409 @@ if TYPE_CHECKING:
logger = logging.getLogger(__name__)
# Tracks the DSMR topic currently subscribed via the MQTT manager, so a config
# change can unsubscribe the old topic before subscribing the new one.
_current_dsmr_topic: str | None = None
@dataclass(frozen=True, slots=True)
class DsmrSourceSnapshot:
"""The only DSMR runtime configuration a network handler may use."""
# Tracks the DSMR tariff topic currently subscribed via the MQTT manager.
_current_tariff_topic: str | None = None
source_id: int
topic: str
tariff_topic: str
sample_interval_s: int
broker_host: str = ""
broker_port: int = 1883
username: str = ""
password: str = ""
tls_enabled: bool = False
# Current electricity tariff: 1 = dal/off-peak, 2 = normal/peak.
# Written by the paho network thread, read by the publish job — guarded by a lock.
_current_tariff: int | None = None
_subscriptions: dict[int, DsmrSourceSnapshot] = {}
_subscription_client_ids: dict[int, str] = {}
_subscription_lock = threading.RLock()
# A configuration value is not an ownership identity: disable and re-enable
# can produce an equal snapshot. Each installed handler therefore captures a
# fresh token and verifies object identity before it can write or update tariff.
_subscription_tokens: dict[int, object] = {}
_reconcile_lock = threading.RLock()
_tariffs: dict[int, int] = {}
_tariff_lock = threading.Lock()
# Kept only for legacy direct test callers of set_current_tariff(value). Runtime
# handlers never write this value; production callers resolve a binding first.
_current_tariff: int | None = None
_UNSET = object()
def get_current_tariff() -> int | None:
"""Return the most recently received electricity tariff (1 or 2), or None."""
with _tariff_lock:
def get_current_tariff(meter_source_id: int | None = None) -> int | None:
"""Return a source tariff, or resolve the active electricity binding.
The no-argument form is retained for the pre-M8 expose integration. It
opens a short session to select the current electricity binding, so a
source's MQTT callback can never make another source's tariff current.
``_current_tariff`` is solely a test-era fallback when no binding database
is available; runtime MQTT handlers do not update it.
"""
if meter_source_id is None:
session_local = get_session_local()
session = session_local()
try:
source_id = _current_electricity_source_id(session, datetime.now(timezone.utc))
if source_id is not None:
return get_current_tariff(source_id)
except Exception:
logger.debug("DSMR legacy tariff lookup could not resolve a binding", exc_info=True)
finally:
session.close()
return _current_tariff
def set_current_tariff(value: int | None) -> None:
"""Set the current electricity tariff (1 or 2), or clear it with None."""
with _tariff_lock:
global _current_tariff
_current_tariff = value
return _tariffs.get(meter_source_id)
def handle_tariff_message(payload_bytes: bytes) -> None:
"""Parse one DSMR tariff MQTT payload and update the in-memory tariff state.
Called from the paho network thread; *must* swallow all exceptions so that
a bad payload never crashes the loop or drops the broker connection.
Accepts payload as bytes or str (paho can deliver either). Ignores
whitespace. Only accepts integer values 1 or 2; anything else is discarded
and the previous known tariff is preserved.
Parameters
----------
payload_bytes:
Raw bytes from the MQTT message (may also be a str in some paho versions).
"""
try:
# Decode bytes → str if needed; strip surrounding whitespace.
if isinstance(payload_bytes, (bytes, bytearray)):
raw = payload_bytes.decode("utf-8", errors="replace").strip()
def set_current_tariff(meter_source_id: int, value: int | None | object = _UNSET) -> None:
"""Set or clear an individual source's tariff state."""
global _current_tariff
if value is _UNSET:
# Compatibility with older direct callers. Do not route runtime source
# updates through this global fallback.
_current_tariff = meter_source_id if meter_source_id in (1, 2) else None
return
with _tariff_lock:
if value is None:
_tariffs.pop(meter_source_id, None)
else:
raw = str(payload_bytes).strip()
_tariffs[meter_source_id] = value
def _current_electricity_source_id(session: Session, at: datetime) -> int | None:
"""Return the DSMR source bound to electricity at ``at``, if any."""
return session.scalar(
select(MeterSource.id)
.join(MeterSourceChannel, MeterSourceChannel.source_id == MeterSource.id)
.join(MeterSourceBinding, MeterSourceBinding.channel_id == MeterSourceChannel.id)
.join(Meter, Meter.id == MeterSourceBinding.meter_id)
.where(
Meter.commodity == "electricity",
MeterSourceBinding.started_at <= at,
(MeterSourceBinding.ended_at.is_(None)) | (MeterSourceBinding.ended_at > at),
)
.order_by(MeterSourceBinding.started_at.desc())
.limit(1)
)
def get_current_electricity_tariff(session: Session, at: datetime | None = None) -> int | None:
"""Resolve tariff through the current electricity binding, never globally."""
source_id = _current_electricity_source_id(session, at or datetime.now(timezone.utc))
if source_id is None:
return None
return get_current_tariff(source_id)
def handle_tariff_message(payload_bytes: bytes, meter_source_id: int) -> None:
"""Parse one source's tariff payload without raising in paho's thread."""
try:
raw = (
payload_bytes.decode("utf-8", errors="replace").strip()
if isinstance(payload_bytes, (bytes, bytearray))
else str(payload_bytes).strip()
)
value = int(raw)
if value not in (1, 2):
logger.debug(
"dsmr_ingest.handle_tariff_message: unexpected tariff value %d (expected 1 or 2, ignored).",
value,
)
return
set_current_tariff(value)
logger.debug("dsmr_ingest.handle_tariff_message: tariff updated to %d.", value)
if value in (1, 2):
set_current_tariff(meter_source_id, value)
except Exception:
# Malformed payload (e.g. non-numeric); swallow silently to protect network thread.
logger.debug(
"dsmr_ingest.handle_tariff_message: could not parse payload %r (ignored).",
payload_bytes,
)
logger.debug("DSMR tariff payload ignored for source_id=%s", meter_source_id)
def apply_dsmr_subscription(settings: "Settings") -> None:
"""(Re)apply the DSMR MQTT subscriptions to match *settings* — restart-free.
Call this at startup and after every config save. It makes the live MQTT
subscriptions reflect the current ``dsmr_ingest_enabled`` / ``dsmr_mqtt_topic``
/ ``dsmr_sample_interval_s`` / ``dsmr_tariff_topic`` settings without an app
restart:
- **Disabled** unsubscribe any active DSMR and tariff subscriptions.
- **Enabled** (re)subscribe to ``dsmr_mqtt_topic`` with a handler bound to
a *fresh* settings snapshot, so a changed sample interval also takes effect.
Also subscribe to ``dsmr_tariff_topic`` when non-empty.
- **Topic changed** unsubscribe the old topic before subscribing the new one.
Idempotent and safe to call when MQTT is not connected (the subscription is
queued in the manager and established on the next connect).
"""
# Imported here (not at module top) to avoid a circular import at app start.
from app.integrations.mqtt import mqtt_manager
global _current_dsmr_topic, _current_tariff_topic
if not settings.dsmr_ingest_enabled:
if _current_dsmr_topic is not None:
mqtt_manager.unsubscribe(_current_dsmr_topic)
logger.info("DSMR ingest disabled — unsubscribed from topic=%s.", _current_dsmr_topic)
_current_dsmr_topic = None
if _current_tariff_topic is not None:
mqtt_manager.unsubscribe(_current_tariff_topic)
logger.info(
"DSMR ingest disabled — unsubscribed from tariff topic=%s.",
_current_tariff_topic,
)
_current_tariff_topic = None
return
# --- Main DSMR telegram topic ---
topic = settings.dsmr_mqtt_topic
if _current_dsmr_topic is not None and _current_dsmr_topic != topic:
mqtt_manager.unsubscribe(_current_dsmr_topic)
# Re-subscribe (overwrites any existing handler for this topic) with a fresh
# settings snapshot so dsmr_sample_interval_s changes take effect too.
snapshot = settings
mqtt_manager.subscribe(topic, lambda payload: handle_message(payload, snapshot))
_current_dsmr_topic = topic
logger.info("DSMR ingest enabled — subscribed to topic=%s.", topic)
# --- DSMR tariff topic (dual-tariff slot indicator) ---
tariff_topic = settings.dsmr_tariff_topic if settings.dsmr_tariff_topic else ""
if tariff_topic:
if _current_tariff_topic is not None and _current_tariff_topic != tariff_topic:
mqtt_manager.unsubscribe(_current_tariff_topic)
mqtt_manager.subscribe(tariff_topic, lambda payload: handle_tariff_message(payload))
_current_tariff_topic = tariff_topic
logger.info("DSMR tariff topic — subscribed to topic=%s.", tariff_topic)
else:
# tariff_topic is empty → unsubscribe any existing tariff subscription.
if _current_tariff_topic is not None:
mqtt_manager.unsubscribe(_current_tariff_topic)
logger.info(
"DSMR tariff topic cleared — unsubscribed from topic=%s.",
_current_tariff_topic,
)
_current_tariff_topic = None
def _snapshot(source: MeterSource) -> DsmrSourceSnapshot:
config = source.config
return DsmrSourceSnapshot(
source_id=source.id,
topic=str(config.get("topic", "dsmr/json")),
tariff_topic=str(config.get("tariff_topic", "")),
sample_interval_s=int(config.get("sample_interval_s", 10)),
broker_host=str(config.get("broker_host", "")),
broker_port=int(config.get("broker_port", 1883)),
username=str(config.get("username", "")),
password=str(config.get("password", "")),
tls_enabled=bool(config.get("tls_enabled", False)),
)
def handle_message(payload_bytes: bytes, settings: "Settings") -> None:
"""Parse one DSMR MQTT payload and persist it if it passes the sample filter.
Called from the paho network thread; *must* swallow all exceptions so that
a bad payload or transient error does not crash the loop or drop the broker
connection.
Parameters
----------
payload_bytes:
Raw bytes from the MQTT message.
settings:
Runtime settings snapshot (captured at subscription time). Used for
``dsmr_sample_interval_s``.
"""
try:
_handle_message_inner(payload_bytes, settings)
except Exception:
logger.exception("dsmr_ingest.handle_message: unexpected error (swallowed).")
def _handle_message_inner(payload_bytes: bytes, settings: "Settings") -> None:
"""Inner implementation — may raise; caller wraps in try/except."""
# --- 1. Parse JSON ---
try:
data: dict = json.loads(payload_bytes)
except (json.JSONDecodeError, ValueError):
logger.debug("dsmr_ingest: invalid JSON payload (skipped).")
return
if not isinstance(data, dict):
logger.debug("dsmr_ingest: payload is not a JSON object (skipped).")
return
# --- 2. Parse timestamp ---
raw_ts = data.get("timestamp")
if raw_ts is None:
logger.debug("dsmr_ingest: missing 'timestamp' field (skipped).")
return
try:
# Python 3.11+ accepts the trailing 'Z' directly; for 3.10 compat we
# replace 'Z' with '+00:00' before parsing.
ts_str = raw_ts if not isinstance(raw_ts, str) else raw_ts.replace("Z", "+00:00")
ts_utc: datetime = datetime.fromisoformat(ts_str)
# Ensure it is timezone-aware UTC.
if ts_utc.tzinfo is None:
ts_utc = ts_utc.replace(tzinfo=timezone.utc)
except (ValueError, TypeError, AttributeError):
logger.debug(
"dsmr_ingest: cannot parse 'timestamp' value %r (skipped).", raw_ts
)
return
# --- 3. Down-sample: only persist if second falls on interval boundary ---
interval = settings.dsmr_sample_interval_s
if interval > 0 and (ts_utc.second % interval) != 0:
# This telegram is between sample points; discard silently.
return
# --- 4. Extract source_id (telegram's own id) — stored only as a reference,
# NOT used for uniqueness/idempotency (it overflows and gets reset). ---
source_id: int | None = data.get("id")
if source_id is not None and not isinstance(source_id, int):
# Unexpected type — treat as missing rather than raising.
logger.debug(
"dsmr_ingest: 'id' field has unexpected type %s (ignoring).",
type(source_id).__name__,
)
source_id = None
# --- 5. Persist to database ---
# Idempotency is keyed on recorded_at (the telegram timestamp), which is
# telegram-id-independent: a single P1 meter emits one telegram per second,
# and down-sampling keeps at most one per interval-aligned second. The
# UNIQUE(recorded_at) constraint is the backstop for the IntegrityError race.
def _enabled_snapshots() -> list[DsmrSourceSnapshot]:
session_local = get_session_local()
session = session_local()
try:
existing = session.scalar(
select(DsmrReading).where(DsmrReading.recorded_at == ts_utc)
)
if existing is not None:
logger.debug(
"dsmr_ingest: recorded_at=%s already in DB, skipping.",
ts_utc.isoformat(),
sources = session.scalars(
select(MeterSource).where(MeterSource.kind == "dsmr_mqtt", MeterSource.enabled.is_(True))
).all()
return [_snapshot(source) for source in sources]
finally:
session.close()
def apply_dsmr_subscription(settings: "Settings | None" = None) -> None:
"""Reconcile enabled DSMR source subscriptions from the database.
``settings`` provides the DB-merged app-wide MQTT identity. Individual
DSMR broker settings continue to come solely from MeterSource records.
"""
from app.integrations.mqtt import mqtt_manager
from app.config import get_settings
base_client_id = (settings or get_settings()).mqtt_client_id
try:
desired = {snapshot.source_id: snapshot for snapshot in _enabled_snapshots()}
except Exception:
logger.exception("DSMR subscription reconcile failed while reading sources")
return
# Do not retain _subscription_lock while stopping MQTT clients: a callback
# may currently hold it through its complete dispatch, and MqttManager
# waits for that callback before teardown returns.
with _reconcile_lock:
# Every source owns a distinct MQTT client, so equal topics from
# different sources/brokers are dispatchable. A telegram and tariff
# topic on the *same* client would overwrite one handler, however.
rejected_source_ids: set[int] = set()
for source_id, snapshot in list(desired.items()):
if snapshot.tariff_topic and snapshot.topic == snapshot.tariff_topic:
logger.error("DSMR source_id=%s rejected: telegram/tariff topic collision", source_id)
rejected_source_ids.add(source_id)
desired.pop(source_id)
with _subscription_lock:
stale_source_ids = [
source_id
for source_id, current in _subscriptions.items()
if desired.get(source_id) != current
or _subscription_client_ids.get(source_id) != base_client_id
]
for source_id in stale_source_ids:
_subscriptions.pop(source_id, None)
_subscription_client_ids.pop(source_id, None)
_subscription_tokens.pop(source_id, None)
set_current_tariff(source_id, None)
# A disabled source has no installed MQTT owner. Persist that fact
# after invalidating its callback token, so a retained callback cannot
# revive an earlier online state while teardown is in progress.
for source_id in stale_source_ids:
_mark_disabled_source_inactive(source_id)
# A topic collision is a configuration error for an enabled source,
# not a disabled-state transition. It must therefore replace any
# earlier online state even when this process started without a
# matching runtime subscription to tear down.
for source_id in rejected_source_ids:
_mark_rejected_source_error(source_id)
for source_id in stale_source_ids:
mqtt_manager.remove_source(source_id)
for source_id, snapshot in desired.items():
with _subscription_lock:
current = _subscriptions.get(source_id)
current_client_id = _subscription_client_ids.get(source_id)
if (
current == snapshot
and current_client_id == base_client_id
and mqtt_manager.source_is_active(source_id)
):
continue
if current is not None:
# The client went inactive outside reconcile. Invalidate its
# old token before rebuilding the same snapshot.
with _subscription_lock:
if _subscriptions.get(source_id) == current:
_subscriptions.pop(source_id, None)
_subscription_client_ids.pop(source_id, None)
_subscription_tokens.pop(source_id, None)
mqtt_manager.remove_source(source_id)
token = object()
handlers = {
snapshot.topic: lambda payload, captured=snapshot, captured_token=token: (
handle_captured_message(payload, captured, captured_token)
)
}
if snapshot.tariff_topic:
handlers[snapshot.tariff_topic] = (
lambda payload, captured=snapshot, captured_token=token: (
handle_captured_tariff_message(payload, captured, captured_token)
)
)
with _subscription_lock:
_subscriptions[source_id] = snapshot
_subscription_client_ids[source_id] = base_client_id
_subscription_tokens[source_id] = token
applied = mqtt_manager.replace_source(
source_id,
host=snapshot.broker_host,
port=snapshot.broker_port,
username=snapshot.username,
password=snapshot.password,
tls_enabled=snapshot.tls_enabled,
subscriptions=handlers,
base_client_id=base_client_id,
state_handler=lambda state, captured=snapshot, captured_token=token: (
handle_captured_source_state(captured, captured_token, state)
),
)
if not applied:
with _subscription_lock:
if _subscription_tokens.get(source_id) is token:
_subscriptions.pop(source_id, None)
_subscription_client_ids.pop(source_id, None)
_subscription_tokens.pop(source_id, None)
def handle_message(payload_bytes: bytes, snapshot: DsmrSourceSnapshot) -> None:
"""Persist one down-sampled frame under its captured source identity."""
try:
_handle_message_inner(payload_bytes, snapshot)
except Exception:
logger.exception("DSMR ingest handler failed for source_id=%s (swallowed)", snapshot.source_id)
def handle_captured_message(
payload_bytes: bytes, snapshot: DsmrSourceSnapshot, token: object | None = None
) -> None:
"""Run a broker callback only while its exact source generation is active."""
with _subscription_lock:
if token is not None:
if _subscription_tokens.get(snapshot.source_id) is not token:
return
elif _subscriptions.get(snapshot.source_id) != snapshot:
return
handle_message(payload_bytes, snapshot)
def handle_captured_tariff_message(
payload_bytes: bytes, snapshot: DsmrSourceSnapshot, token: object | None = None
) -> None:
"""Ignore tariff callbacks retained from a removed/replaced source."""
with _subscription_lock:
if token is not None:
if _subscription_tokens.get(snapshot.source_id) is not token:
return
elif _subscriptions.get(snapshot.source_id) != snapshot:
return
handle_tariff_message(payload_bytes, snapshot.source_id)
reading = DsmrReading(
recorded_at=ts_utc,
source_id=source_id,
payload=data, # full frame, verbatim
def handle_captured_source_state(snapshot: DsmrSourceSnapshot, token: object, state: str) -> None:
"""Persist one active generation's connection health in a short DB session."""
with _subscription_lock:
if _subscription_tokens.get(snapshot.source_id) is not token:
return
session_local = get_session_local()
session = session_local()
try:
source = session.get(MeterSource, snapshot.source_id)
if source is None or not source.enabled or source.kind != "dsmr_mqtt":
return
source.status = state
source.last_error = "MQTT connection failed." if state == "error" else None
source.updated_at = datetime.now(timezone.utc)
session.commit()
except Exception:
session.rollback()
logger.exception("DSMR source health update failed for source_id=%s", snapshot.source_id)
finally:
session.close()
def _mark_disabled_source_inactive(source_id: int) -> None:
"""Clear an obsolete online health state for a disabled DSMR source.
The caller has already invalidated the source's generation token. This
helper deliberately opens its own short session so reconcile never shares
a callback-thread transaction. Deleted sources simply have no row left
to update.
"""
session_local = get_session_local()
session = session_local()
try:
source = session.get(MeterSource, source_id)
if source is None or source.enabled or source.kind != "dsmr_mqtt":
return
source.status = "unknown"
source.last_error = None
source.updated_at = datetime.now(timezone.utc)
session.commit()
except Exception:
session.rollback()
logger.exception("DSMR disabled source health update failed for source_id=%s", source_id)
finally:
session.close()
def _mark_rejected_source_error(source_id: int) -> None:
"""Persist a non-sensitive error for an enabled source rejected by reconcile."""
session_local = get_session_local()
session = session_local()
try:
source = session.get(MeterSource, source_id)
if source is None or not source.enabled or source.kind != "dsmr_mqtt":
return
source.status = "error"
source.last_error = "DSMR source configuration invalid."
source.updated_at = datetime.now(timezone.utc)
session.commit()
except Exception:
session.rollback()
logger.exception("DSMR rejected source health update failed for source_id=%s", source_id)
finally:
session.close()
def _handle_message_inner(payload_bytes: bytes, snapshot: DsmrSourceSnapshot) -> None:
try:
data = json.loads(payload_bytes)
except (json.JSONDecodeError, ValueError):
return
if not isinstance(data, dict):
return
try:
raw_ts = data["timestamp"]
ts_utc = datetime.fromisoformat(raw_ts.replace("Z", "+00:00"))
if ts_utc.tzinfo is None:
ts_utc = ts_utc.replace(tzinfo=timezone.utc)
except (KeyError, ValueError, TypeError, AttributeError):
return
if snapshot.sample_interval_s > 0 and ts_utc.second % snapshot.sample_interval_s:
return
telegram_id = data.get("id")
if telegram_id is not None and not isinstance(telegram_id, int):
telegram_id = None
session_local = get_session_local()
session = session_local()
try:
exists = session.scalar(
select(DsmrReading.id).where(
DsmrReading.meter_source_id == snapshot.source_id,
DsmrReading.recorded_at == ts_utc,
)
)
session.add(reading)
if exists is None:
session.add(
DsmrReading(
meter_source_id=snapshot.source_id,
recorded_at=ts_utc,
telegram_id=telegram_id,
payload=data,
)
)
source = session.get(MeterSource, snapshot.source_id)
if source is not None and source.enabled and source.kind == "dsmr_mqtt":
source.status = "online"
source.last_seen_at = datetime.now(timezone.utc)
source.last_error = None
source.updated_at = datetime.now(timezone.utc)
session.commit()
logger.debug(
"dsmr_ingest: persisted reading recorded_at=%s source_id=%s.",
ts_utc.isoformat(),
source_id,
)
except sqlalchemy.exc.IntegrityError:
# Race / duplicate: another insert beat us to the same recorded_at.
session.rollback()
logger.debug(
"dsmr_ingest: IntegrityError for recorded_at=%s (duplicate, skipped).",
ts_utc.isoformat(),
)
except Exception:
session.rollback()
logger.exception("dsmr_ingest: DB error (swallowed).")
logger.exception("DSMR database write failed for source_id=%s", snapshot.source_id)
finally:
session.close()
+111 -32
View File
@@ -37,7 +37,7 @@ Design notes
- **Register keys**: DSMR payload uses JSON strings like ``"20915.154"``
for cumulative kWh registers. ``register_at`` converts them to Decimal.
- **Degraded vs skip semantics**:
- *No meter coverage* (``meter_at`` returns None for t0): write a
- *No unique meter coverage* (no sole electricity meter at t0): write a
``degraded=True`` row with ``meter_id=None``.
- *Cross-meter boundary* (m0.id != m1.id for t0/t1): write a ``degraded=True``
row with ``meter_id=m0.id``; losing this one period at the swap boundary is
@@ -67,8 +67,8 @@ Meter-aware compute_period ordering rationale (M7-T03)
The order of checks inside ``compute_period`` is:
1. **Immutability guard** (existing non-degraded row, overwrite=False) return False.
2. **Meter determination** (m0 = meter_at(t0), m1 = meter_at(t1)):
- No meter (m0 is None) write degraded, meter_id=None.
2. **Meter determination** (m0/m1 each resolve to one electricity Meter):
- No unique meter (m0 is None) write degraded, meter_id=None.
- Cross-meter boundary (m0.id != m1.id) write degraded, meter_id=m0.id.
3. **Active contract version check** skip (no write) if absent.
4. **Boundary register readings** within m0's window → write degraded if missing.
@@ -98,8 +98,8 @@ from app.integrations.pricing.strategies import (
get_strategy,
)
from app.models.energy import DsmrReading, EnergyCostPeriod, Meter
from app.models.meter_source import MeterSource, MeterSourceBinding, MeterSourceChannel
from app.services.contracts import active_contract_version_at, active_contract_versions
from app.services.meters import meter_at
from app.services.timezone import local_date, local_now
logger = logging.getLogger(__name__)
@@ -134,10 +134,10 @@ _MAX_DELTA_KWH = Decimal("100")
_SETTLEMENT_OFFSET = timedelta(hours=1, minutes=5)
# DSMR payload register keys (cumulative kWh, JSON string values).
_KEY_D1 = "electricity_delivered_1" # delivered low-tariff (dal / _1)
_KEY_D2 = "electricity_delivered_2" # delivered high-tariff (normal / _2)
_KEY_R1 = "electricity_returned_1" # returned low-tariff
_KEY_R2 = "electricity_returned_2" # returned high-tariff
_KEY_D1 = "electricity_delivered_1" # delivered low-tariff (dal / _1)
_KEY_D2 = "electricity_delivered_2" # delivered high-tariff (normal / _2)
_KEY_R1 = "electricity_returned_1" # returned low-tariff
_KEY_R2 = "electricity_returned_2" # returned high-tariff
# ---------------------------------------------------------------------------
@@ -174,6 +174,23 @@ def _existing_period(session: Session, t0: datetime) -> EnergyCostPeriod | None:
).scalar_one_or_none()
def _unique_electricity_meter_at(session: Session, boundary: datetime) -> Meter | None:
"""Return the sole electricity meter covering *boundary*, if one exists.
Billing must treat overlapping meter epochs as a structural ambiguity rather
than relying on ``meter_at``'s newest-started tie breaker. A cumulative
delta is safe only when exactly one electricity meter covers each endpoint.
"""
candidates = session.execute(
select(Meter).where(
Meter.commodity == "electricity",
Meter.started_at <= boundary,
(Meter.ended_at.is_(None)) | (Meter.ended_at > boundary),
)
).scalars().all()
return candidates[0] if len(candidates) == 1 else None
# ---------------------------------------------------------------------------
# register_at — boundary reading lookup (meter-aware)
# ---------------------------------------------------------------------------
@@ -183,6 +200,8 @@ def register_at(
session: Session,
boundary: datetime,
meter: Meter,
*,
meter_source_id: int | None = None,
) -> dict[str, Decimal] | None:
"""Return the four cumulative kWh register values at *boundary*, within *meter*'s window.
@@ -226,7 +245,7 @@ def register_at(
"""
# Build the meter-window constraints: [started_at, ended_at).
meter_lower = meter.started_at # DsmrReading.recorded_at >= meter.started_at
meter_upper = meter.ended_at # DsmrReading.recorded_at < meter.ended_at (if set)
meter_upper = meter.ended_at # DsmrReading.recorded_at < meter.ended_at (if set)
stmt = (
select(DsmrReading)
@@ -240,6 +259,8 @@ def register_at(
# Apply the upper bound only when the meter is closed (ended_at is not None).
if meter_upper is not None:
stmt = stmt.where(DsmrReading.recorded_at < meter_upper)
if meter_source_id is not None:
stmt = stmt.where(DsmrReading.meter_source_id == meter_source_id)
row: DsmrReading | None = session.execute(stmt).scalar_one_or_none()
@@ -273,6 +294,33 @@ def register_at(
}
def _binding_at(
session: Session, boundary: datetime, meter: Meter
) -> tuple[MeterSourceBinding, int] | None:
"""Resolve the sole DSMR binding for *meter* at one period boundary.
Costing must not infer a cumulative domain from whichever reading happens
to be latest. A binding anchors both the physical meter epoch and its
source stream. Any missing or overlapping binding is therefore
deliberately unresolvable.
"""
candidates = session.execute(
select(MeterSourceBinding, MeterSourceChannel.source_id)
.join(MeterSourceChannel, MeterSourceChannel.id == MeterSourceBinding.channel_id)
.join(MeterSource, MeterSource.id == MeterSourceChannel.source_id)
.where(
MeterSourceBinding.meter_id == meter.id,
MeterSourceBinding.started_at <= boundary,
(MeterSourceBinding.ended_at.is_(None)) | (MeterSourceBinding.ended_at > boundary),
MeterSource.kind == "dsmr_mqtt",
)
).all()
if len(candidates) != 1:
return None
binding, source_id = candidates[0]
return binding, source_id
# ---------------------------------------------------------------------------
# compute_period — single 15-minute period
# ---------------------------------------------------------------------------
@@ -302,9 +350,9 @@ def compute_period(session: Session, t0: datetime, *, overwrite: bool = False) -
Side-effects
------------
- Inserts or updates an ``EnergyCostPeriod`` row keyed on ``period_start=t0``.
- If no meter covers t0 (``meter_at`` returns None for t0): inserts/updates
a degraded row with ``meter_id=None``.
- If the period spans a meter boundary (``meter_at(t0).id != meter_at(t1).id``):
- If no unique meter covers t0: inserts/updates a degraded row with
``meter_id=None``.
- If the period spans a meter boundary (m0.id != m1.id):
inserts/updates a degraded row with ``meter_id=m0.id`` (D5 decision).
- If readings are missing at either boundary within the meter window:
inserts/updates a degraded row with ``meter_id=m0.id``.
@@ -332,7 +380,7 @@ def compute_period(session: Session, t0: datetime, *, overwrite: bool = False) -
# is corrected and a recompute_range is triggered.
#
# Ordering rationale:
# 1. No meter (m0 is None) → degraded(meter_id=None): no epoch for t0.
# 1. No unique meter (m0 is None) → degraded(meter_id=None): no unambiguous epoch for t0.
# 2. Cross-meter boundary (m0.id != m1.id) → degraded(meter_id=m0.id): D5.
# 3. (Single meter, proceed) → contract check → readings → delta guard → price.
#
@@ -341,13 +389,13 @@ def compute_period(session: Session, t0: datetime, *, overwrite: bool = False) -
# first, a missing-contract skip would silently discard the cross-table
# evidence; once a contract is added and recompute runs, the engine would
# incorrectly use cross-table reads.
m0 = meter_at(session, t0)
m1 = meter_at(session, t1)
m0 = _unique_electricity_meter_at(session, t0)
m1 = _unique_electricity_meter_at(session, t1)
if m0 is None:
# No meter epoch covers t0 — degraded with no meter attribution.
# No unambiguous meter epoch covers t0 — degraded with no attribution.
logger.debug(
"compute_period(%s): no active meter at t0 — writing degraded (meter_id=None).",
"compute_period(%s): no unique active meter at t0 — writing degraded (meter_id=None).",
t0.isoformat(),
)
_upsert_degraded(session, t0, now, existing, meter_id=None)
@@ -366,6 +414,16 @@ def compute_period(session: Session, t0: datetime, *, overwrite: bool = False) -
_upsert_degraded(session, t0, now, existing, meter_id=m0.id)
return True
# Both endpoints must resolve to the same binding and source before a
# cumulative subtraction is permitted. This is checked before contract
# lookup so structural inconsistencies remain visible as degraded rows.
bound0 = _binding_at(session, t0, m0)
bound1 = _binding_at(session, t1, m1)
if bound0 is None or bound1 is None or bound0[0].id != bound1[0].id or bound0[1] != bound1[1]:
_upsert_degraded(session, t0, now, existing, meter_id=m0.id)
return True
binding, meter_source_id = bound0
# --- Active contract version at t0 ---
# If there is no active contract covering t0, skip the period entirely.
# We do not write a degraded row — there is no meaningful state to recover
@@ -378,8 +436,8 @@ def compute_period(session: Session, t0: datetime, *, overwrite: bool = False) -
return False
# --- Boundary readings within m0's meter window ---
start_regs = register_at(session, t0, m0)
end_regs = register_at(session, t1, m0)
start_regs = register_at(session, t0, m0, meter_source_id=meter_source_id)
end_regs = register_at(session, t1, m0, meter_source_id=meter_source_id)
if start_regs is None or end_regs is None:
# Missing readings within the meter window → degraded with m0 attribution.
@@ -421,9 +479,7 @@ def compute_period(session: Session, t0: datetime, *, overwrite: bool = False) -
except TibberPriceNotFoundError:
# Missing Tibber price → skip the period; it will be retried once the
# price arrives (e.g. after the next Tibber refresh job runs).
logger.debug(
"compute_period(%s): no Tibber price found — skipping.", t0.isoformat()
)
logger.debug("compute_period(%s): no Tibber price found — skipping.", t0.isoformat())
return False
# --- Upsert the billing record ---
@@ -445,6 +501,7 @@ def compute_period(session: Session, t0: datetime, *, overwrite: bool = False) -
existing.pricing = pricing
existing.contract_version_id = version.id
existing.meter_id = m0.id
existing.source_binding_id = binding.id
existing.degraded = False
existing.computed_at = now
else:
@@ -461,6 +518,7 @@ def compute_period(session: Session, t0: datetime, *, overwrite: bool = False) -
pricing=pricing,
contract_version_id=version.id,
meter_id=m0.id,
source_binding_id=binding.id,
degraded=False,
computed_at=now,
)
@@ -520,6 +578,7 @@ def _upsert_degraded(
existing.pricing = {}
existing.contract_version_id = None
existing.meter_id = meter_id
existing.source_binding_id = None
existing.degraded = True
existing.computed_at = now
else:
@@ -536,6 +595,7 @@ def _upsert_degraded(
pricing={},
contract_version_id=None,
meter_id=meter_id,
source_binding_id=None,
degraded=True,
computed_at=now,
)
@@ -605,7 +665,9 @@ def compute_closed_periods(session: Session) -> int:
# ---------------------------------------------------------------------------
def recompute_range(session: Session, start: datetime, end: datetime) -> int:
def recompute_range(
session: Session, start: datetime, end: datetime, *, commit: bool = True, strict: bool = False
) -> int:
"""Recompute (overwrite) all 15-minute periods in ``[start, end)``.
This is the *explicit opt-in* path for recovering from:
@@ -626,8 +688,16 @@ def recompute_range(session: Session, start: datetime, end: datetime) -> int:
Parameters
----------
session:
Active SQLAlchemy session. The function commits after all periods
have been processed.
Active SQLAlchemy session.
commit:
When true (the default), commit after all periods have been processed.
Callers composing this recompute with other writes may pass false and
own the surrounding transaction themselves.
strict:
When true, propagate a failed period computation to the caller. This
is for lifecycle transactions which must roll back their meter/binding
mutation together with the cost recompute. The default remains
best-effort for existing background and standalone callers.
start:
Inclusive start datetime (floored to the nearest quarter-hour internally).
end:
@@ -658,13 +728,16 @@ def recompute_range(session: Session, start: datetime, end: datetime) -> int:
if did_write:
written += 1
except Exception:
if strict:
raise
logger.exception(
"recompute_range: unexpected error for t0=%s — continuing.",
t0.isoformat(),
)
t0 += timedelta(minutes=_PERIOD_MINUTES)
session.commit()
if commit:
session.commit()
logger.info(
"recompute_range(%s, %s): wrote %d period(s).",
start.isoformat(),
@@ -756,12 +829,16 @@ def summarize(session: Session, start: datetime, end: datetime) -> dict[str, Any
end_utc = _as_utc(end)
# --- Fetch all EnergyCostPeriod rows in [start, end) ---
rows = session.execute(
select(EnergyCostPeriod).where(
EnergyCostPeriod.period_start >= start_utc,
EnergyCostPeriod.period_start < end_utc,
rows = (
session.execute(
select(EnergyCostPeriod).where(
EnergyCostPeriod.period_start >= start_utc,
EnergyCostPeriod.period_start < end_utc,
)
)
).scalars().all()
.scalars()
.all()
)
good_rows = [r for r in rows if not r.degraded]
degraded_rows = [r for r in rows if r.degraded]
@@ -863,7 +940,9 @@ def summarize(session: Session, start: datetime, end: datetime) -> dict[str, Any
version_segments: list[tuple[_date, _date | None, dict]] = []
for v in versions:
v_start_local = local_date(_as_utc(v.effective_from))
v_end_local = local_date(_as_utc(v.effective_to)) if v.effective_to is not None else None
v_end_local = (
local_date(_as_utc(v.effective_to)) if v.effective_to is not None else None
)
version_segments.append((v_start_local, v_end_local, v.values or {}))
for v_start, v_end_excl, v_values in version_segments:
+95 -4
View File
@@ -98,6 +98,15 @@ def _availability_topic(device_uuid: str, prefix: str) -> str:
return f"{prefix}/modbus/{node}/availability"
def _availability_id(entity: ExposableEntity) -> str:
"""Return the identity which owns this entity's liveness topic.
M8 meters deliberately retain their own UUID as HA node/unique identity,
while their availability is supplied by a MeterSource UUID.
"""
return entity.device.availability_id or entity.device.identifiers[1]
def _unique_id(entity: ExposableEntity) -> str:
"""Stable unique_id — device uuid + metric key (never from mutable fields)."""
device_uuid = entity.device.identifiers[1]
@@ -139,8 +148,7 @@ def build_discovery_payload(
if state_prefix is None:
state_prefix = discovery_prefix
device_uuid = entity.device.identifiers[1]
avail_topic = _availability_topic(device_uuid, state_prefix)
avail_topic = _availability_topic(_availability_id(entity), state_prefix)
state_t = _state_topic(entity, state_prefix)
topic = _discovery_topic(entity, discovery_prefix)
@@ -212,6 +220,21 @@ def publish_discovery(session: Session) -> None:
logger.exception("publish_discovery: failed to build catalog; aborting")
return
# Meter UUIDs are intentionally identity-changing epochs. Discovery config
# is retained, so clear only the precisely enumerable old M8 identities;
# never wildcard a provider/topic and risk removing another source's card.
try:
stale_entities = _stale_m8_entities(session)
except Exception:
logger.exception("publish_discovery: unable to enumerate old M8 identities")
stale_entities = []
for old_entity in stale_entities:
try:
old_topic, _ = build_discovery_payload(old_entity, discovery_prefix, state_prefix)
mqtt_manager.publish(old_topic, b"", retain=True)
except Exception:
logger.exception("publish_discovery: unable to clear old identity %r", old_entity.key)
for entry in catalog:
entity = entry.entity
try:
@@ -236,6 +259,62 @@ def publish_discovery(session: Session) -> None:
)
def _stale_m8_entities(session: Session) -> list[ExposableEntity]:
"""Return synthetic discovery entries for superseded thermal identities.
This is deliberately a narrow, best-effort cleanup: ended Meter UUIDs and
historically possible thermal combinations only; current identities are excluded.
"""
from app.integrations.expose import DeviceInfo
from app.models.energy import Meter
from sqlalchemy import select
meters = session.execute(select(Meter).where(
Meter.commodity.in_(("electricity", "heating", "hot_water"))
)).scalars().all()
current = {meter.commodity: meter for meter in meters if meter.ended_at is None}
old = [meter for meter in meters if meter.ended_at is not None]
entities: list[ExposableEntity] = []
for meter in old:
info = DeviceInfo(identifiers=("meter", meter.uuid), name=meter.label)
for suffix in ("total", "today"):
entities.append(ExposableEntity(
key=f"meter.{meter.uuid}.{suffix}", component="sensor", device=info,
device_class=None, unit="", name="obsolete",
))
heatings = [meter for meter in meters if meter.commodity == "heating"]
waters = [meter for meter in meters if meter.commodity == "hot_water"]
current_identity = (
".".join(sorted((current["heating"].uuid, current["hot_water"].uuid)))
if current.get("heating") is not None and current.get("hot_water") is not None else None
)
for heating in heatings:
for water in waters:
if heating.ended_at is None and water.ended_at is None:
continue
# A thermal identity can only have been published when both Meter
# epochs were current at the same instant. Do not form a Cartesian
# product of historical records: that would tombstone identities
# which have never existed in HA.
heating_start, water_start = heating.started_at, water.started_at
heating_end, water_end = heating.ended_at, water.ended_at
if (heating_end is not None and water_start >= heating_end) or (
water_end is not None and heating_start >= water_end
):
continue
identity = ".".join(sorted((heating.uuid, water.uuid)))
if identity == current_identity:
continue
info = DeviceInfo(identifiers=("thermal-cost", identity), name="obsolete")
for metric in ("heating", "hot_water_heating", "water", "water_tax", "fixed", "all_in"):
for suffix in ("total", "today"):
entities.append(ExposableEntity(
key=f"thermal_cost.{identity}.{metric}_{suffix}", component="sensor", device=info,
device_class=None, unit="", name="obsolete",
))
return entities
# ---------------------------------------------------------------------------
# Public: publish states
# ---------------------------------------------------------------------------
@@ -288,7 +367,19 @@ def _publish_entity_state(
Also publishes the availability topic for ``binary_sensor`` "online" entities.
"""
state_t = _state_topic(entity, prefix)
device_uuid = entity.device.identifiers[1]
# Source-backed entities can have a different liveness identity from their
# HA device identity. Publish it before the state; a None value below is
# intentionally not converted to a synthetic zero.
if entity.device.provides_availability and entity.device.availability_getter is not None:
try:
available = bool(entity.device.availability_getter(session))
mqtt_manager.publish(
_availability_topic(_availability_id(entity), prefix),
"online" if available else "offline",
retain=False,
)
except Exception:
logger.exception("availability_getter raised for entity %r", entity.key)
if entity.component == "binary_sensor" and "online" in entity.key:
# The online sensor represents device availability.
@@ -303,7 +394,7 @@ def _publish_entity_state(
# Default to offline when no reading is available.
online = (raw_value == "ON")
avail_payload = "online" if online else "offline"
avail_topic = _availability_topic(device_uuid, prefix)
avail_topic = _availability_topic(_availability_id(entity), prefix)
mqtt_manager.publish(avail_topic, avail_payload, retain=False)
# The state of the binary_sensor itself
state_payload = "ON" if online else "OFF"
+298
View File
@@ -0,0 +1,298 @@
"""Thermal (WarmteLink) 15-minute cost ledger.
This module deliberately does not share the electricity ledger: thermal has
two independently-bound cumulative domains and Decimal database columns.
"""
from __future__ import annotations
import logging
from datetime import UTC, date, datetime, time, timedelta
from decimal import Decimal
from typing import Any
from sqlalchemy import select
from sqlalchemy.orm import Session
from app.models.energy import Meter, MeterCostPeriod
from app.models.meter_source import MeterSource, MeterSourceBinding, MeterSourceChannel, WarmteLinkReading
from app.services.contracts import active_contract_version_at, active_contract_versions
from app.services.energy_cost import floor_to_quarter
from app.services import timezone as timezone_service
logger = logging.getLogger(__name__)
_PERIOD = timedelta(minutes=15)
_FRESHNESS = timedelta(seconds=120)
_LIMITS = {"heating": Decimal("0.1"), "hot_water": Decimal("1")}
_ACCEPTED_QUALITIES = {"valid", "unverifiable"}
_SETTLEMENT_TIME = time(1, 5)
def _utc(value: datetime) -> datetime:
return value.replace(tzinfo=UTC) if value.tzinfo is None else value.astimezone(UTC)
def _decimal(value: Any) -> Decimal:
return value if isinstance(value, Decimal) else Decimal(str(value))
def _existing(session: Session, commodity: str, start: datetime) -> MeterCostPeriod | None:
return session.execute(
select(MeterCostPeriod).where(
MeterCostPeriod.commodity == commodity, MeterCostPeriod.period_start == start
)
).scalar_one_or_none()
def _meter_at(session: Session, commodity: str, instant: datetime) -> Meter | None:
meters = session.execute(
select(Meter).where(
Meter.commodity == commodity,
Meter.started_at <= instant,
(Meter.ended_at.is_(None)) | (Meter.ended_at > instant),
)
).scalars().all()
return meters[0] if len(meters) == 1 else None
def _binding_at(
session: Session, meter: Meter, instant: datetime
) -> tuple[MeterSourceBinding, MeterSourceChannel] | None:
rows = session.execute(
select(MeterSourceBinding, MeterSourceChannel)
.join(MeterSourceChannel, MeterSourceChannel.id == MeterSourceBinding.channel_id)
.join(MeterSource, MeterSource.id == MeterSourceChannel.source_id)
.where(
MeterSourceBinding.meter_id == meter.id,
MeterSourceBinding.started_at <= instant,
(MeterSourceBinding.ended_at.is_(None)) | (MeterSourceBinding.ended_at > instant),
MeterSource.kind == "warmtelink_serial",
)
).all()
return rows[0] if len(rows) == 1 else None
def _reading_at(
session: Session,
channel: MeterSourceChannel,
meter: Meter,
binding: MeterSourceBinding,
target: datetime,
) -> WarmteLinkReading | None:
"""Choose the nearest accepted reading inside this cumulative domain.
Freshness alone is insufficient: a frame immediately before a meter or
source hand-off belongs to a different cumulative register and must never
be used as the other side of a delta.
"""
window_start, window_end = target - _FRESHNESS, target + _FRESHNESS
rows = session.execute(
select(WarmteLinkReading)
.where(
WarmteLinkReading.channel_id == channel.id,
WarmteLinkReading.recorded_at >= window_start,
WarmteLinkReading.recorded_at <= window_end,
)
).scalars().all()
domain_start = max(_utc(meter.started_at), _utc(binding.started_at))
domain_ends = (meter.ended_at, binding.ended_at)
domain_end = min((_utc(value) for value in domain_ends if value is not None), default=None)
accepted = [
row for row in rows
if row.quality in _ACCEPTED_QUALITIES
and _utc(row.recorded_at) >= domain_start
and (domain_end is None or _utc(row.recorded_at) < domain_end)
]
if not accepted:
return None
return min(
accepted,
key=lambda row: (abs((_utc(row.recorded_at) - target).total_seconds()), _utc(row.recorded_at)),
)
def _degrade(
session: Session,
commodity: str,
start: datetime,
end: datetime,
existing: MeterCostPeriod | None,
reason: str,
meter_id: int | None = None,
binding_id: int | None = None,
) -> None:
now = datetime.now(UTC)
fields = dict(
period_end=end, meter_id=meter_id, source_binding_id=binding_id,
contract_version_id=None, quantity=Decimal("0"), cost=Decimal("0"), currency="EUR",
cost_breakdown={}, pricing_snapshot={}, quality="invalid", degraded=True,
degraded_reason=reason, updated_at=now,
)
if existing is None:
session.add(MeterCostPeriod(commodity=commodity, period_start=start, created_at=now, **fields))
else:
for key, value in fields.items():
setattr(existing, key, value)
def compute_period(
session: Session, commodity: str, period_start: datetime, *, overwrite: bool = False
) -> bool:
"""Compute one closed thermal period, recording every unsafe input as degraded."""
if commodity not in _LIMITS:
raise ValueError("commodity must be heating or hot_water")
start = floor_to_quarter(_utc(period_start))
end = start + _PERIOD
existing = _existing(session, commodity, start)
if existing is not None and not existing.degraded and not overwrite:
return False
meter0, meter1 = _meter_at(session, commodity, start), _meter_at(session, commodity, end)
if meter0 is None:
_degrade(session, commodity, start, end, existing, "missing_or_ambiguous_meter")
return True
if meter1 is None or meter1.id != meter0.id:
_degrade(session, commodity, start, end, existing, "cross_meter_epoch", meter0.id)
return True
bound0, bound1 = _binding_at(session, meter0, start), _binding_at(session, meter1, end)
if bound0 is None or bound1 is None:
_degrade(session, commodity, start, end, existing, "missing_or_ambiguous_binding", meter0.id)
return True
binding, channel = bound0
if bound1[0].id != binding.id or bound1[1].id != channel.id:
_degrade(session, commodity, start, end, existing, "cross_source_binding", meter0.id, binding.id)
return True
first = _reading_at(session, channel, meter0, binding, start)
last = _reading_at(session, channel, meter0, binding, end)
if first is None or last is None:
_degrade(session, commodity, start, end, existing, "missing_stale_or_invalid_reading", meter0.id, binding.id)
return True
delta = _decimal(last.value) - _decimal(first.value)
if delta < 0:
_degrade(session, commodity, start, end, existing, "negative_delta", meter0.id, binding.id)
return True
if delta > _LIMITS[commodity]:
_degrade(session, commodity, start, end, existing, "delta_limit_exceeded", meter0.id, binding.id)
return True
version = active_contract_version_at(session, start, scope="thermal")
if version is None:
_degrade(session, commodity, start, end, existing, "missing_contract", meter0.id, binding.id)
return True
values = {key: _decimal(value) for key, value in version.values["variable"].items()}
if commodity == "heating":
breakdown = {"heating": delta * values["heating"]}
else:
breakdown = {
key: delta * values[key]
for key in ("hot_water_heating", "hot_water", "hot_water_tax")
}
cost = sum(breakdown.values(), Decimal("0"))
now = datetime.now(UTC)
fields = dict(
period_end=end, meter_id=meter0.id, source_binding_id=binding.id,
contract_version_id=version.id, quantity=delta, cost=cost, currency=version.contract.currency,
cost_breakdown=breakdown, pricing_snapshot=dict(version.values),
quality="valid" if first.quality == last.quality == "valid" else "unverifiable",
degraded=False, degraded_reason=None, updated_at=now,
)
if existing is None:
session.add(MeterCostPeriod(commodity=commodity, period_start=start, created_at=now, **fields))
else:
for key, value in fields.items():
setattr(existing, key, value)
return True
def compute_closed_periods(session: Session, *, now: datetime | None = None) -> int:
"""Retry incomplete thermal rows and fill recent closed periods without touching good rows."""
now = _utc(now or datetime.now(UTC))
first = floor_to_quarter(now - timedelta(days=7))
written = 0
cursor = first
while cursor + _PERIOD <= now:
for commodity in ("heating", "hot_water"):
if compute_period(session, commodity, cursor):
written += 1
cursor += _PERIOD
session.commit()
return written
def recompute_range(session: Session, start: datetime, end: datetime, *, commit: bool = True) -> int:
"""Recompute a thermal range.
The historical service entry point remains self-committing for the scheduler
and direct callers. HTTP callers pass ``commit=False`` so validation,
recomputation, response statistics, and the single commit share one
transaction owned by the route.
"""
cursor, end = floor_to_quarter(_utc(start)), _utc(end)
now, written = datetime.now(UTC), 0
while cursor < end:
if cursor + _PERIOD <= now:
for commodity in ("heating", "hot_water"):
if compute_period(session, commodity, cursor, overwrite=True):
written += 1
cursor += _PERIOD
if commit:
session.commit()
return written
def _settled_end_date(now: datetime) -> date:
local_now = timezone_service.to_local(now)
return local_now.date() if local_now.timetz().replace(tzinfo=None) >= _SETTLEMENT_TIME else local_now.date() - timedelta(days=1)
def summarize(session: Session, start: datetime, end: datetime, *, now: datetime | None = None) -> dict[str, Any]:
"""Return thermal variable/fixed totals; standing is charged once per contract/day."""
start, end = _utc(start), _utc(end)
rows = session.execute(select(MeterCostPeriod).where(
MeterCostPeriod.period_start >= start, MeterCostPeriod.period_start < end
)).scalars().all()
good = [row for row in rows if not row.degraded]
variable = sum((_decimal(row.cost) for row in good), Decimal("0"))
breakdown: dict[str, Decimal] = {key: Decimal("0") for key in (
"heating", "hot_water_heating", "hot_water", "hot_water_tax")}
for row in good:
for key, value in row.cost_breakdown.items():
breakdown[key] = breakdown.get(key, Decimal("0")) + _decimal(value)
# A summary is half-open. ``end`` at local midnight has no overlap with
# that next local date, and an empty/reversed range owns no standing day.
final_day = timezone_service.local_date(end - timedelta(microseconds=1))
final_day = min(final_day, _settled_end_date(now or datetime.now(UTC)))
day = timezone_service.local_date(start)
fixed_breakdown: dict[str, Decimal] = {key: Decimal("0") for key in (
"heating_network", "metering", "delivery_set", "hot_water_network", "other"
)}
versions = active_contract_versions(session, scope="thermal")
while start < end and day <= final_day:
day_start = datetime.combine(day, time.min, tzinfo=timezone_service.local_tz()).astimezone(UTC)
next_day_start = datetime.combine(
day + timedelta(days=1), time.min, tzinfo=timezone_service.local_tz()
).astimezone(UTC)
local_day_seconds = Decimal(str((next_day_start - day_start).total_seconds()))
# A rate revision part-way through a local date is attributable only
# to its effective interval. This preserves one contract-level daily
# charge while correctly handling first-version and intra-day changes.
for version in versions:
segment_start = max(day_start, _utc(version.effective_from))
version_end = _utc(version.effective_to) if version.effective_to is not None else next_day_start
segment_end = min(next_day_start, version_end)
if segment_start >= segment_end:
continue
values = version.values["standing"]
fraction = Decimal(str((segment_end - segment_start).total_seconds())) / local_day_seconds
for key in fixed_breakdown:
fixed_breakdown[key] += _decimal(values.get(key, "0")) / Decimal("365") * fraction
day += timedelta(days=1)
fixed = sum(fixed_breakdown.values(), Decimal("0"))
return {
"currency": good[0].currency if good else "EUR", "variable_cost": variable,
"fixed_cost": fixed, "fixed_breakdown": fixed_breakdown,
"total_cost": variable + fixed, "breakdown": breakdown,
"period_count": len(good), "degraded_count": len(rows) - len(good),
}
+529
View File
@@ -0,0 +1,529 @@
"""Service layer for source/channel discovery and meter-source bindings.
All mutating functions receive a caller-owned :class:`~sqlalchemy.orm.Session`
and never commit. This lets HTTP handlers compose source and meter changes in
one transaction later without exposing any connection I/O here.
"""
from __future__ import annotations
from datetime import UTC, datetime
from typing import Any
from sqlalchemy import or_, select
from sqlalchemy.orm import Session
from app.integrations.meter_sources import (
SourceProfileError,
get_source_profile,
merge_source_config,
validate_source_config,
)
from app.models.energy import Meter
from app.models.meter_source import (
MeterSource,
MeterSourceBinding,
MeterSourceChannel,
half_open_intervals_overlap,
)
class MeterSourceError(ValueError):
"""Base class for source-domain validation errors."""
class SourceNotFoundError(MeterSourceError):
"""Raised when the requested source does not exist."""
class ChannelNotFoundError(MeterSourceError):
"""Raised when the requested source channel does not exist."""
class MeterNotFoundError(MeterSourceError):
"""Raised when the requested meter does not exist."""
class BindingNotFoundError(MeterSourceError):
"""Raised when the requested binding does not exist."""
class BindingValidationError(MeterSourceError):
"""Raised for an incompatible unit, commodity, or invalid interval."""
class BindingOverlapError(BindingValidationError):
"""Raised when a meter or channel already has an overlapping binding."""
class SourceDeleteRestrictedError(MeterSourceError):
"""Raised when a source has retained channel, binding, or reading history."""
COMMODITY_UNITS = {"electricity": "kWh", "heating": "GJ", "hot_water": ""}
_UNSET = object()
def _utc_now() -> datetime:
return datetime.now(UTC)
def _as_utc(value: datetime) -> datetime:
return value.replace(tzinfo=UTC) if value.tzinfo is None else value.astimezone(UTC)
def get_source(session: Session, source_id: int) -> MeterSource:
source = session.get(MeterSource, source_id)
if source is None:
raise SourceNotFoundError(f"Meter source {source_id} was not found.")
return source
def list_sources(session: Session, *, kind: str | None = None) -> list[MeterSource]:
statement = select(MeterSource).order_by(MeterSource.id)
if kind is not None:
get_source_profile(kind)
statement = statement.where(MeterSource.kind == kind)
return list(session.execute(statement).scalars())
def create_source(
session: Session,
*,
name: str,
kind: str,
config: dict[str, Any],
enabled: bool = True,
) -> MeterSource:
"""Add a source after validating its complete kind-specific config."""
now = _utc_now()
source = MeterSource(
name=name,
kind=kind,
enabled=enabled,
config=validate_source_config(kind, config),
created_at=now,
updated_at=now,
)
session.add(source)
return source
def update_source(
session: Session,
source_id: int,
*,
name: str | None = None,
enabled: bool | None = None,
config_patch: dict[str, Any] | None = None,
) -> MeterSource:
"""Update source metadata and/or merge a partial source config without commit."""
source = get_source(session, source_id)
if name is not None:
source.name = name
if enabled is not None:
source.enabled = enabled
if config_patch is not None:
source.config = merge_source_config(source.kind, source.config, config_patch)
source.updated_at = _utc_now()
return source
def delete_source(session: Session, source_id: int) -> None:
"""Delete an entirely unused source; history is always retained instead."""
source = get_source(session, source_id)
has_channel = session.execute(
select(MeterSourceChannel.id).where(MeterSourceChannel.source_id == source.id).limit(1)
).scalar_one_or_none()
has_binding = session.execute(
select(MeterSourceBinding.id)
.join(MeterSourceChannel)
.where(MeterSourceChannel.source_id == source.id)
.limit(1)
).scalar_one_or_none()
if has_channel is not None or has_binding is not None:
raise SourceDeleteRestrictedError(
f"Meter source {source_id} has dependent channels, bindings, or readings."
)
session.delete(source)
def get_channel(session: Session, channel_id: int) -> MeterSourceChannel:
channel = session.get(MeterSourceChannel, channel_id)
if channel is None:
raise ChannelNotFoundError(f"Meter source channel {channel_id} was not found.")
return channel
def upsert_discovered_channel(
session: Session,
*,
source_id: int,
channel_key: str,
label: str,
unit: str,
suggested_commodity: str | None = None,
device_type: str | None = None,
fingerprint: str | None = None,
latest_value: Any = None,
latest_at: datetime | None = None,
latest_quality: str | None = None,
) -> MeterSourceChannel:
"""Idempotently create or refresh a discovered channel's metadata.
``suggested_commodity`` remains metadata only; this function never creates
a meter or a binding.
"""
source = get_source(session, source_id)
if unit not in get_source_profile(source.kind).allowed_units:
raise SourceProfileError(f"Unit {unit!r} is not allowed for source kind {source.kind!r}.")
channel = session.execute(
select(MeterSourceChannel).where(
MeterSourceChannel.source_id == source.id,
MeterSourceChannel.channel_key == channel_key,
)
).scalar_one_or_none()
now = _utc_now()
if channel is None:
channel = MeterSourceChannel(
source_id=source.id,
channel_key=channel_key,
label=label,
unit=unit,
suggested_commodity=suggested_commodity,
device_type=device_type,
fingerprint=fingerprint,
latest_value=latest_value,
latest_at=latest_at,
latest_quality=latest_quality,
created_at=now,
updated_at=now,
)
session.add(channel)
return channel
if channel.unit != unit:
binding_id = session.execute(
select(MeterSourceBinding.id)
.where(MeterSourceBinding.channel_id == channel.id)
.limit(1)
).scalar_one_or_none()
if binding_id is not None:
raise BindingValidationError(
f"Cannot change unit of bound channel {channel.id} from {channel.unit!r} to {unit!r}."
)
channel.label = label
channel.unit = unit
channel.suggested_commodity = suggested_commodity
channel.device_type = device_type
channel.fingerprint = fingerprint
channel.latest_value = latest_value
channel.latest_at = latest_at
channel.latest_quality = latest_quality
channel.updated_at = now
return channel
def list_bindings(
session: Session, *, meter_id: int | None = None, channel_id: int | None = None
) -> list[MeterSourceBinding]:
statement = select(MeterSourceBinding).order_by(MeterSourceBinding.started_at, MeterSourceBinding.id)
if meter_id is not None:
statement = statement.where(MeterSourceBinding.meter_id == meter_id)
if channel_id is not None:
statement = statement.where(MeterSourceBinding.channel_id == channel_id)
return list(session.execute(statement).scalars())
def _get_meter(session: Session, meter_id: int) -> Meter:
meter = session.get(Meter, meter_id)
if meter is None:
raise MeterNotFoundError(f"Meter {meter_id} was not found.")
return meter
def _validate_binding(
session: Session,
*,
meter_id: int,
channel_id: int,
started_at: datetime,
ended_at: datetime | None,
excluding_ids: set[int] | None = None,
) -> None:
meter = _get_meter(session, meter_id)
channel = get_channel(session, channel_id)
expected_unit = COMMODITY_UNITS.get(meter.commodity)
if expected_unit is None:
raise BindingValidationError(f"Commodity {meter.commodity!r} cannot be bound to a source channel.")
if channel.unit != expected_unit:
raise BindingValidationError(
f"Meter commodity {meter.commodity!r} requires unit {expected_unit!r}, "
f"but channel has {channel.unit!r}."
)
if ended_at is not None and _as_utc(ended_at) <= _as_utc(started_at):
raise BindingValidationError("Binding ended_at must be strictly after started_at.")
if _as_utc(started_at) < _as_utc(meter.started_at):
raise BindingValidationError("Binding must not start before its meter epoch.")
if meter.ended_at is None:
if ended_at is not None:
# A historical binding on an active epoch is valid, but it must be
# wholly within that epoch (whose upper bound is open).
pass
else:
meter_end = _as_utc(meter.ended_at)
if ended_at is None or _as_utc(ended_at) > meter_end:
raise BindingValidationError("Closed meter bindings must end within the meter epoch.")
excluded = excluding_ids or set()
candidates = session.execute(
select(MeterSourceBinding).where(
or_(
MeterSourceBinding.meter_id == meter_id,
MeterSourceBinding.channel_id == channel_id,
)
)
).scalars()
for existing in candidates:
if existing.id in excluded:
continue
if half_open_intervals_overlap(
_as_utc(started_at),
_as_utc(ended_at) if ended_at is not None else None,
_as_utc(existing.started_at),
_as_utc(existing.ended_at) if existing.ended_at is not None else None,
):
side = "meter" if existing.meter_id == meter_id else "channel"
raise BindingOverlapError(f"Binding overlaps existing {side} binding {existing.id}.")
def create_binding(
session: Session,
*,
meter_id: int,
channel_id: int,
started_at: datetime,
ended_at: datetime | None = None,
) -> MeterSourceBinding:
"""Create a compatible non-overlapping half-open source binding."""
now = _utc_now()
if _as_utc(started_at) > now or (ended_at is not None and _as_utc(ended_at) > now):
raise BindingValidationError("Binding boundaries must not be in the future.")
_validate_binding(
session,
meter_id=meter_id,
channel_id=channel_id,
started_at=started_at,
ended_at=ended_at,
)
binding = MeterSourceBinding(
meter_id=meter_id,
channel_id=channel_id,
started_at=started_at,
ended_at=ended_at,
created_at=now,
updated_at=now,
)
session.add(binding)
return binding
def create_binding_for_meter_swap(
session: Session,
*,
old_meter_id: int | None,
new_meter_id: int,
channel_id: int,
started_at: datetime,
) -> MeterSourceBinding:
"""Create a binding during a physical meter swap, handing off one channel if safe.
A channel is transferable only when exactly one of its bindings covered the
instant immediately before ``started_at`` and that binding belongs to the
meter which this declaration just closed. All other occupied or ambiguous
cases retain the normal fail-closed overlap behaviour.
This function deliberately does not commit. The caller must keep the meter
declaration, binding handoff, and any billing recompute in one transaction.
"""
new_meter = _get_meter(session, new_meter_id)
channel = get_channel(session, channel_id)
expected_unit = COMMODITY_UNITS.get(new_meter.commodity)
if expected_unit is None or channel.unit != expected_unit:
raise BindingValidationError(
f"Meter commodity {new_meter.commodity!r} requires unit {expected_unit!r}, "
f"but channel has {channel.unit!r}."
)
boundary = _as_utc(started_at)
if _as_utc(new_meter.started_at) != boundary:
raise BindingValidationError(
"Meter-swap binding must start at the new meter's started_at boundary."
)
covering_bindings = [
binding
for binding in session.execute(
select(MeterSourceBinding).where(MeterSourceBinding.channel_id == channel_id)
).scalars()
if _as_utc(binding.started_at) < boundary
and (binding.ended_at is None or _as_utc(binding.ended_at) >= boundary)
]
if not covering_bindings:
return create_binding(
session,
meter_id=new_meter_id,
channel_id=channel_id,
started_at=started_at,
)
if old_meter_id is None or len(covering_bindings) != 1:
raise BindingOverlapError("Channel is occupied or has an ambiguous binding at meter swap.")
old_meter = _get_meter(session, old_meter_id)
old_binding = covering_bindings[0]
if (
old_meter.commodity != new_meter.commodity
or old_meter.ended_at is None
or _as_utc(old_meter.ended_at) != boundary
or old_binding.meter_id != old_meter.id
):
raise BindingOverlapError("Channel is occupied by a binding that cannot be handed off.")
update_binding(session, old_binding.id, ended_at=started_at)
return create_binding(
session,
meter_id=new_meter_id,
channel_id=channel_id,
started_at=started_at,
)
def update_binding(
session: Session,
binding_id: int,
*,
meter_id: int | None = None,
channel_id: int | None = None,
started_at: datetime | None = None,
ended_at: datetime | None | object = _UNSET,
) -> MeterSourceBinding:
"""Correct a binding while preserving half-open timeline constraints."""
binding = session.get(MeterSourceBinding, binding_id)
if binding is None:
raise BindingNotFoundError(f"Meter source binding {binding_id} was not found.")
new_meter_id = binding.meter_id if meter_id is None else meter_id
new_channel_id = binding.channel_id if channel_id is None else channel_id
new_started_at = binding.started_at if started_at is None else started_at
new_ended_at = binding.ended_at if ended_at is _UNSET else ended_at
now = _utc_now()
if _as_utc(new_started_at) > now or (new_ended_at is not None and _as_utc(new_ended_at) > now):
raise BindingValidationError("Binding boundaries must not be in the future.")
_validate_binding(
session,
meter_id=new_meter_id,
channel_id=new_channel_id,
started_at=new_started_at,
ended_at=new_ended_at,
excluding_ids={binding.id},
)
binding.meter_id = new_meter_id
binding.channel_id = new_channel_id
binding.started_at = new_started_at
binding.ended_at = new_ended_at
binding.updated_at = _utc_now()
return binding
def close_binding(session: Session, binding_id: int, *, ended_at: datetime) -> MeterSourceBinding:
"""Close an existing binding at its exclusive end boundary."""
return update_binding(session, binding_id, ended_at=ended_at)
def close_open_bindings_for_meter(session: Session, meter_id: int, *, ended_at: datetime) -> list[MeterSourceBinding]:
"""Close every open binding on a meter at one shared boundary."""
bindings = list(session.execute(
select(MeterSourceBinding).where(
MeterSourceBinding.meter_id == meter_id, MeterSourceBinding.ended_at.is_(None)
)
).scalars())
for binding in bindings:
update_binding(session, binding.id, ended_at=ended_at)
return bindings
def transfer_binding(
session: Session, *, target_meter_id: int, from_binding_id: int, to_channel_id: int,
effective_at: datetime,
) -> tuple[MeterSourceBinding, MeterSourceBinding]:
"""Atomically close a binding and open its replacement on the target meter."""
source = session.get(MeterSourceBinding, from_binding_id)
if source is None:
raise BindingNotFoundError(f"Meter source binding {from_binding_id} was not found.")
target = _get_meter(session, target_meter_id)
old_meter = _get_meter(session, source.meter_id)
effective_at = _as_utc(effective_at)
now = _utc_now()
if effective_at > now:
raise BindingValidationError("Binding transfer effective_at must not be in the future.")
if old_meter.commodity != target.commodity:
raise BindingValidationError("Binding transfer meters must have the same commodity.")
if source.ended_at is not None:
raise BindingValidationError("Only an open binding can be transferred.")
if old_meter.id == target.id:
close_at = effective_at
else:
# Recovery is deliberately narrow: the source meter must be the one
# and only most-recent closed predecessor in this commodity's timeline.
# A manually closed meter may leave an intentional epoch gap before the
# target is declared, so adjacency is not required.
if old_meter.ended_at is None:
raise BindingValidationError("Source binding must belong to a closed predecessor meter.")
timeline = list(session.execute(
select(Meter).where(Meter.commodity == target.commodity)
).scalars())
predecessors = [
meter for meter in timeline
if meter.id != target.id
and meter.ended_at is not None
and _as_utc(meter.ended_at) <= _as_utc(target.started_at)
]
if not predecessors:
raise BindingValidationError("Source meter is not the unique immediately preceding meter.")
latest_end = max(_as_utc(meter.ended_at) for meter in predecessors)
latest = [meter for meter in predecessors if _as_utc(meter.ended_at) == latest_end]
if len(latest) != 1 or latest[0].id != old_meter.id:
raise BindingValidationError("Source meter is not the unique immediately preceding meter.")
# Reject any overlapping epoch around either endpoint. A separate
# meter inside the gap is already excluded by the predecessor check;
# one extending into either endpoint is an ambiguous timeline too.
for meter in timeline:
if meter.id in {old_meter.id, target.id}:
continue
meter_end = _as_utc(meter.ended_at) if meter.ended_at is not None else None
if (
half_open_intervals_overlap(
_as_utc(old_meter.started_at), _as_utc(old_meter.ended_at),
_as_utc(meter.started_at), meter_end,
)
or half_open_intervals_overlap(
_as_utc(target.started_at),
_as_utc(target.ended_at) if target.ended_at is not None else None,
_as_utc(meter.started_at), meter_end,
)
):
raise BindingValidationError("Source meter has an ambiguous commodity timeline.")
close_at = _as_utc(old_meter.ended_at)
if effective_at < _as_utc(target.started_at):
raise BindingValidationError("Transfer effective_at must be within the target meter epoch.")
if effective_at < _as_utc(source.started_at):
raise BindingValidationError("Transfer effective_at precedes the source binding.")
# Validate the target before mutating the old row, then close/create in one session.
_validate_binding(session, meter_id=target.id, channel_id=to_channel_id,
started_at=effective_at, ended_at=None,
excluding_ids={source.id})
update_binding(session, source.id, ended_at=close_at)
created = create_binding(session, meter_id=target.id, channel_id=to_channel_id,
started_at=effective_at)
return source, created
+76 -2
View File
@@ -53,6 +53,7 @@ from sqlalchemy import select
from sqlalchemy.orm import Session
from app.models.energy import Meter
from app.models.meter_source import MeterSourceBinding
logger = logging.getLogger(__name__)
@@ -104,6 +105,20 @@ class MeterIntervalError(MeterError):
"""
def close_meter(session: Session, meter: Meter, *, ended_at: datetime) -> Meter:
"""Close an active meter at a valid, non-future exclusive boundary."""
boundary = _as_utc(ended_at)
if meter.ended_at is not None:
raise MeterIntervalError("Only an active meter can be closed.")
if boundary <= _as_utc(meter.started_at):
raise MeterIntervalError("Meter ended_at must be strictly after started_at.")
if boundary > datetime.now(UTC):
raise MeterIntervalError("Meter ended_at must not be in the future.")
_validate_bindings_fit_meter_end(session, meter, boundary)
meter.ended_at = boundary
return meter
# ---------------------------------------------------------------------------
# Internal query helpers
# ---------------------------------------------------------------------------
@@ -121,6 +136,23 @@ def _active_meter(session: Session, commodity: str) -> Optional[Meter]:
).scalar_one_or_none()
def _validate_bindings_fit_meter_end(session: Session, meter: Meter, boundary: datetime) -> None:
"""Reject an epoch close that would put any retained binding out of bounds.
Closed binding history is immutable here. Open bindings may subsequently
be closed by the caller at the shared meter boundary, but only when that
produces a non-empty interval.
"""
for binding in session.execute(
select(MeterSourceBinding).where(MeterSourceBinding.meter_id == meter.id)
).scalars():
if binding.ended_at is None:
if _as_utc(binding.started_at) >= boundary:
raise MeterIntervalError("Open binding cannot be closed within the proposed meter epoch.")
elif _as_utc(binding.ended_at) > boundary:
raise MeterIntervalError("Closed binding extends beyond the proposed meter epoch.")
def _meter_before(session: Session, meter: Meter) -> Optional[Meter]:
"""Return the meter whose ``ended_at`` equals *meter*'s ``started_at``.
@@ -296,6 +328,8 @@ def declare_meter(
If *started_at* is strictly earlier than the current active meter's
``started_at`` (chronological backdate below the active epoch's start).
"""
if _as_utc(started_at) > datetime.now(UTC):
raise MeterIntervalError("Meter started_at must not be in the future.")
active = _active_meter(session, commodity)
if active is not None:
@@ -308,6 +342,9 @@ def declare_meter(
"Declare a started_at on or after the active meter's start to avoid "
"a chronologically inconsistent epoch ordering."
)
# Validate before changing the epoch: retained closed binding history
# must never be silently truncated by a later declaration.
_validate_bindings_fit_meter_end(session, active, _as_utc(started_at))
# Close the current active meter at the swap point (contiguous handoff).
active.ended_at = started_at
logger.info(
@@ -400,6 +437,9 @@ def update_meter(
invert).
b. It must be **strictly before** this meter's ``ended_at`` (if set),
so this meter's epoch remains non-empty.
c. Every binding on this meter and its affected predecessor must remain
wholly inside its proposed epoch. The service rejects the correction
rather than rewriting binding history.
Note: triggering a billing recompute (``recompute_range``) after a
retroactive ``started_at`` change is **out of scope** for this service
@@ -429,6 +469,14 @@ def update_meter(
If the new ``started_at`` would produce an invalid (empty or inverted)
epoch for this meter or the immediately preceding one.
"""
# Validate the proposed epoch boundary before touching *any* mutable
# field. PATCH accepts label/note together with started_at, so doing this
# first keeps an invalid future timestamp from leaking a partial in-session
# update before the API's rollback boundary is reached.
proposed_started_at = _as_utc(started_at) if started_at is not None else None
if proposed_started_at is not None and proposed_started_at > datetime.now(UTC):
raise MeterIntervalError("Meter started_at must not be in the future.")
if label is not None:
meter.label = label
logger.info("Updated meter id=%d label=%r", meter.id, label)
@@ -439,10 +487,11 @@ def update_meter(
if started_at is not None:
old_started_at = meter.started_at
assert proposed_started_at is not None
# --- Validate upper bound: new started_at must be < this meter's ended_at (if set).
if meter.ended_at is not None:
if _as_utc(started_at) >= _as_utc(meter.ended_at):
if proposed_started_at >= _as_utc(meter.ended_at):
raise MeterIntervalError(
f"New started_at ({started_at.isoformat()}) must be strictly before "
f"this meter's ended_at ({meter.ended_at.isoformat()}). "
@@ -454,12 +503,37 @@ def update_meter(
# --- Validate lower bound: new started_at must be strictly after prev's started_at.
if prev is not None:
if _as_utc(started_at) <= _as_utc(prev.started_at):
if proposed_started_at <= _as_utc(prev.started_at):
raise MeterIntervalError(
f"New started_at ({started_at.isoformat()}) must be strictly after "
f"the previous meter's started_at ({prev.started_at.isoformat()}). "
"Moving the boundary that far back would collapse the previous meter's epoch."
)
# A boundary correction changes both adjacent meter epochs. Fail closed
# rather than silently rewriting binding history: every existing binding
# must still fit in its proposed epoch before either Meter is mutated.
affected_meters = [
(meter, proposed_started_at, _as_utc(meter.ended_at) if meter.ended_at is not None else None)
]
if prev is not None:
affected_meters.append((prev, _as_utc(prev.started_at), proposed_started_at))
for affected_meter, proposed_start, proposed_end in affected_meters:
bindings = session.execute(
select(MeterSourceBinding).where(MeterSourceBinding.meter_id == affected_meter.id)
).scalars()
for binding in bindings:
if _as_utc(binding.started_at) < proposed_start:
raise MeterIntervalError(
f"Binding {binding.id} starts before meter {affected_meter.id}'s epoch."
)
if proposed_end is not None and (
binding.ended_at is None or _as_utc(binding.ended_at) > proposed_end
):
raise MeterIntervalError(
f"Binding {binding.id} would fall outside meter {affected_meter.id}'s epoch."
)
if prev is not None:
# Maintain continuity: update the previous meter's ended_at to match the new start.
prev.ended_at = started_at
logger.info(
+66 -3
View File
@@ -30,6 +30,7 @@ Design decisions
from __future__ import annotations
import logging
from threading import Lock, Thread
from datetime import UTC, datetime
from sqlalchemy import select
@@ -41,7 +42,10 @@ from app.models.energy import EnergyContract, TibberPrice
logger = logging.getLogger(__name__)
def _active_tibber_contract_exists(session: Session) -> bool:
_background_refresh_lock = Lock()
def active_tibber_contract_exists(session: Session) -> bool:
"""Return True if there is an active contract with kind='tibber'."""
row = session.execute(
select(EnergyContract).where(
@@ -84,14 +88,17 @@ def refresh_prices(session: Session, settings: object) -> int:
logger.debug("refresh_prices: tibber_api_token is empty — no-op")
return 0
if not _active_tibber_contract_exists(session):
if not active_tibber_contract_exists(session):
logger.debug("refresh_prices: no active tibber contract — no-op")
return 0
home_id: str = getattr(settings, "tibber_home_id", "") or ""
home_id_or_none: str | None = home_id.strip() or None
logger.info("refresh_prices: fetching Tibber price range (home_id=%r)", home_id_or_none)
# Neither the API token nor the selected home identifier is safe to emit in
# diagnostics. The fetch client receives them, but logs only describe the
# operation itself.
logger.info("refresh_prices: fetching Tibber price range")
# May raise TibberError or TibberAuthError — let them propagate.
price_points = fetch_price_range(token, home_id_or_none)
@@ -137,3 +144,59 @@ def refresh_prices(session: Session, settings: object) -> int:
logger.info("refresh_prices: upserted %d price points", upserted)
return upserted
def run_tibber_refresh_best_effort() -> bool:
"""Run one refresh with an isolated session, skipping concurrent requests.
This is shared by the hourly scheduler and immediate post-commit triggers.
It intentionally catches all failures: refresh is advisory and must never
make app startup or a successfully committed configuration/contract update
appear to have failed. The boolean reports whether this invocation owned
the work; it is primarily useful for tests and diagnostics.
"""
if not _background_refresh_lock.acquire(blocking=False):
logger.debug("Tibber refresh already running; skipping duplicate request")
return False
session: Session | None = None
try:
# Local imports keep the pure refresh service free of app startup import
# cycles, while every background invocation gets a fresh DB session.
from app.config import get_settings
from app.db import get_session_local
from app.services.config_page import build_runtime_settings
session = get_session_local()()
refresh_prices(session, build_runtime_settings(session, get_settings()))
except Exception as exc:
# Exception text can contain remote request details. Keep diagnostics
# useful without allowing a token or home id to escape through logging.
logger.warning("Tibber price refresh failed (%s)", type(exc).__name__)
if session is not None:
try:
session.rollback()
except Exception:
logger.warning("Tibber price refresh rollback failed")
finally:
if session is not None:
try:
session.close()
except Exception:
logger.warning("Tibber price refresh session close failed")
_background_refresh_lock.release()
return True
def trigger_tibber_refresh() -> None:
"""Request a non-blocking, best-effort Tibber refresh after a DB commit."""
try:
Thread(
target=run_tibber_refresh_best_effort,
name="tibber-price-refresh",
daemon=True,
).start()
except Exception as exc:
# Starting the optional worker must not turn an already committed API
# operation into a failure; avoid logging exception text for secrecy.
logger.warning("Unable to start Tibber price refresh (%s)", type(exc).__name__)
+401
View File
@@ -0,0 +1,401 @@
"""Privacy-preserving WarmteLink frame admission and minute sampling.
The serial worker added later owns I/O. This module deliberately only accepts
already parsed :class:`P1Telegram` instances (or a parser callable at its
small convenience entry point), so rejected telegram bytes never enter the
database or an exception message.
"""
from __future__ import annotations
from collections.abc import Callable
from dataclasses import dataclass
from datetime import UTC, datetime, timedelta
from decimal import Decimal
import re
from zoneinfo import ZoneInfo
from sqlalchemy import select
from sqlalchemy.orm import Session
from app.db import get_session_local
from app.integrations.p1 import IntegrityStatus, P1Channel, P1Telegram, parse_telegram
from app.models.meter_source import MeterSource, WarmteLinkReading
from app.services.meter_sources import upsert_discovered_channel
@dataclass(frozen=True)
class _ChannelSample:
key: str
label: str
value: Decimal
unit: str
device_type: str | None
fingerprint: str | None
identity: tuple[object, ...]
@dataclass(frozen=True)
class _FrameSnapshot:
recorded_at: datetime
fingerprint: str | None
samples: tuple[_ChannelSample, ...]
class WarmteLinkIngestor:
"""Keep unverifiable candidates isolated by source for one worker lifetime."""
def __init__(
self,
*,
clock: Callable[[], datetime] | None = None,
) -> None:
self._clock = clock or (lambda: datetime.now(UTC))
self._previous: dict[int, _FrameSnapshot] = {}
def ingest(
self,
session: Session,
*,
source_id: int,
telegram: P1Telegram,
received_at: datetime | None = None,
) -> bool:
"""Apply one parsed telegram in the caller's transaction.
Returns whether the frame was admitted. Callers that own a session
must commit on success; :func:`handle_frame` is the failure-contained
worker-facing entry point.
"""
source = session.get(MeterSource, source_id)
if source is None:
raise ValueError("WarmteLink source was not found")
if source.kind != "warmtelink_serial":
raise ValueError("Source is not a WarmteLink serial source")
received_at = _utc(self._clock()) if received_at is None else _utc(received_at)
try:
integrity = _integrity_status(telegram.integrity)
except Exception:
self._previous.pop(source_id, None)
self._diagnose(source, "WarmteLink frame could not be normalized", now=received_at)
return False
if integrity is IntegrityStatus.INVALID:
self._previous.pop(source_id, None)
self._diagnose(source, "WarmteLink frame checksum is invalid", now=received_at)
return False
try:
snapshot = _snapshot(telegram, received_at)
fingerprints = _final_fingerprints(snapshot)
except Exception:
# A parser DTO remains an untrusted boundary. Do not let an
# unnormalizable DTO bridge two otherwise matching candidates.
self._previous.pop(source_id, None)
self._diagnose(source, "WarmteLink frame could not be normalized", now=received_at)
return False
if fingerprints is None:
self._previous.pop(source_id, None)
self._diagnose(source, "WarmteLink frame fingerprint is invalid", now=received_at)
return False
if integrity is IntegrityStatus.UNVERIFIABLE:
previous = self._previous.get(source_id)
if previous is None:
self._previous[source_id] = snapshot
self._diagnose(
source, "Awaiting a second matching unverifiable WarmteLink frame", now=received_at
)
return False
reason = _continuity_problem(previous, snapshot)
if reason is not None:
self._previous[source_id] = snapshot
self._diagnose(source, reason, now=received_at)
return False
try:
self._admit(session, source, snapshot, integrity.value, fingerprints, received_at)
except Exception:
# ``ingest`` owns flushes and may be used directly by tests or
# future callers. A failed write must never become a speculative
# predecessor for this source.
self._previous.pop(source_id, None)
raise
if integrity is IntegrityStatus.UNVERIFIABLE:
# Advance the sliding predecessor only once every database write
# for this frame has succeeded. ``handle_frame`` also clears it
# if the caller's later commit fails.
self._previous[source_id] = snapshot
else:
# A verified frame has no need for a speculative predecessor.
self._previous.pop(source_id, None)
return True
def handle_frame(
self,
source_id: int,
frame: bytes,
*,
session_factory: Callable[[], Session] = get_session_local,
parser: Callable[[bytes], P1Telegram] = parse_telegram,
) -> bool:
"""Parse and persist one frame, containing both parse and DB failures.
A failed write is rolled back before a fresh transaction records only
a generic source error. Consequently no partial latest/history update
survives and the following frame may recover normally.
"""
received_at = _utc(self._clock())
try:
telegram = parser(frame)
except Exception:
self._previous.pop(source_id, None)
self._record_error(
session_factory, source_id, "WarmteLink frame could not be parsed", now=received_at
)
return False
try:
with session_factory() as session:
admitted = self.ingest(
session, source_id=source_id, telegram=telegram, received_at=received_at
)
session.commit()
return admitted
except Exception:
self._previous.pop(source_id, None)
self._record_error(session_factory, source_id, "WarmteLink ingest failed", now=received_at)
return False
def _admit(
self,
session: Session,
source: MeterSource,
snapshot: _FrameSnapshot,
quality: str,
fingerprints: tuple[str, ...],
received_at: datetime,
) -> None:
for sample, fingerprint in zip(snapshot.samples, fingerprints, strict=True):
channel = upsert_discovered_channel(
session,
source_id=source.id,
channel_key=sample.key,
label=sample.label,
unit=sample.unit,
suggested_commodity=_suggestion(sample.unit),
device_type=sample.device_type,
fingerprint=fingerprint,
latest_value=sample.value,
latest_at=snapshot.recorded_at,
latest_quality=quality,
)
session.flush()
bucket = snapshot.recorded_at.replace(second=0, microsecond=0)
exists = session.scalar(
select(WarmteLinkReading.id)
.where(
WarmteLinkReading.channel_id == channel.id,
WarmteLinkReading.recorded_at >= bucket,
WarmteLinkReading.recorded_at < bucket + timedelta(minutes=1),
)
.limit(1)
)
if exists is None:
session.add(
WarmteLinkReading(
channel_id=channel.id,
recorded_at=snapshot.recorded_at,
received_at=received_at,
value=sample.value,
unit=sample.unit,
quality=quality,
equipment_fingerprint=fingerprint,
)
)
source.status = "online"
source.last_seen_at = received_at
source.last_error = None
source.updated_at = received_at
def _diagnose(self, source: MeterSource, reason: str, *, now: datetime | None = None) -> None:
now = _utc(self._clock()) if now is None else now
source.status = "error"
source.last_error = reason
source.updated_at = now
def _record_error(
self,
session_factory: Callable[[], Session],
source_id: int,
message: str,
*,
now: datetime,
) -> None:
try:
with session_factory() as session:
source = session.get(MeterSource, source_id)
if source is not None:
self._diagnose(source, message, now=now)
session.commit()
except Exception:
# Error reporting itself must not kill another source worker.
return
def _snapshot(telegram: P1Telegram, received_at: datetime) -> _FrameSnapshot:
recorded_at = _parse_timestamp(telegram.timestamp, received_at)
samples = tuple(_sample(channel) for channel in telegram.channels)
if not samples:
raise ValueError("WarmteLink telegram contains no cumulative channels")
return _FrameSnapshot(recorded_at, telegram.equipment_fingerprint, samples)
def _sample(channel: P1Channel) -> _ChannelSample:
profile = _canonical_channel_profile(channel.number)
if channel.device_type != profile.device_type:
raise ValueError("WarmteLink channel device type is not canonical")
if len(channel.readings) != 1:
raise ValueError("WarmteLink channel has no unambiguous cumulative reading")
reading = channel.readings[0]
if reading.code != profile.reading_code or reading.value is None or reading.unit != profile.raw_unit:
raise ValueError("WarmteLink cumulative reading is incomplete")
# Parser annotations are not a trust boundary: fake or future parser DTOs
# must not put a float (including NaN) or a non-finite Decimal into the
# per-source unverifiable candidate state. Do not coerce here: accepting
# another numeric type would make the persistence and continuity paths
# disagree about the cumulative-value contract.
if not isinstance(reading.value, Decimal) or not reading.value.is_finite():
raise ValueError("WarmteLink cumulative reading is not a finite Decimal")
fingerprint = channel.equipment_fingerprint
return _ChannelSample(
key=profile.key,
label=profile.label,
value=reading.value,
unit=profile.unit,
device_type=profile.device_type,
fingerprint=fingerprint,
identity=(channel.number, profile.device_type, fingerprint, profile.reading_code, profile.unit),
)
@dataclass(frozen=True)
class _CanonicalChannelProfile:
key: str
label: str
device_type: str
reading_code: str
raw_unit: str
unit: str
_CANONICAL_CHANNELS = {
1: _CanonicalChannelProfile(
key="channel-1",
label="WarmteLink channel 1",
device_type="006",
reading_code="0-1:24.2.1",
raw_unit="m3",
unit="",
),
2: _CanonicalChannelProfile(
key="channel-2",
label="WarmteLink channel 2",
device_type="012",
reading_code="0-2:24.2.1",
raw_unit="GJ",
unit="GJ",
),
}
def _canonical_channel_profile(number: object) -> _CanonicalChannelProfile:
# Do not format or coerce parser supplied channel numbers: that could turn
# an arbitrary object into an identity key before it is rejected.
if type(number) is not int:
raise ValueError("WarmteLink channel number is not canonical")
try:
return _CANONICAL_CHANNELS[number]
except KeyError as exc:
raise ValueError("WarmteLink channel is not supported by the profile") from exc
def _integrity_status(value: object) -> IntegrityStatus:
"""Require a real parser integrity enum, never a look-alike value object."""
if not isinstance(value, IntegrityStatus):
raise ValueError("WarmteLink integrity status is not canonical")
return value
def _continuity_problem(previous: _FrameSnapshot, current: _FrameSnapshot) -> str | None:
if current.recorded_at <= previous.recorded_at:
return "Unverifiable WarmteLink timestamp is not strictly increasing"
if current.recorded_at - previous.recorded_at != timedelta(seconds=10):
return "Unverifiable WarmteLink frame cadence is not 10 seconds"
if current.fingerprint != previous.fingerprint:
return "WarmteLink equipment metadata changed"
if tuple(sample.identity for sample in current.samples) != tuple(sample.identity for sample in previous.samples):
return "WarmteLink channel metadata changed or channel set changed"
old_values = {sample.key: sample.value for sample in previous.samples}
if any(sample.value < old_values[sample.key] for sample in current.samples):
return "WarmteLink cumulative value decreased"
return None
_FINGERPRINT_PATTERN = re.compile(r"[0-9a-f]{64}")
def _final_fingerprints(snapshot: _FrameSnapshot) -> tuple[str, ...] | None:
"""Return only canonical SHA-256 hexdigests safe to persist.
A parser DTO is an untrusted boundary: even a field named ``fingerprint``
can contain a raw equipment identifier. Validate the complete DTO before
choosing persisted values: the top-level value and every channel value
must independently be canonical hashes. This deliberately does not use
a top-level fallback for an absent or malformed channel fingerprint.
"""
values = (snapshot.fingerprint, *(sample.fingerprint for sample in snapshot.samples))
if any(not _is_canonical_fingerprint(value) for value in values):
return None
return tuple(sample.fingerprint for sample in snapshot.samples if sample.fingerprint is not None)
def _is_canonical_fingerprint(value: str | None) -> bool:
return value is not None and _FINGERPRINT_PATTERN.fullmatch(value) is not None
_AMSTERDAM = ZoneInfo("Europe/Amsterdam")
_MAX_CLOCK_SKEW = timedelta(minutes=5)
def _parse_timestamp(value: str | None, received_at: datetime) -> datetime:
if value is None or len(value) != 13 or value[-1] not in {"S", "W"} or not value[:-1].isdigit():
raise ValueError("WarmteLink timestamp is unavailable")
naive = datetime.strptime(value[:-1], "%y%m%d%H%M%S")
# The S/W marker is only advisory: deployed devices have emitted W while
# on CEST. Validate both folds by a UTC round trip, which rejects spring
# gaps and leaves one (ordinary) or two (fall-back) real instants.
candidates: list[datetime] = []
for fold in (0, 1):
candidate = naive.replace(tzinfo=_AMSTERDAM, fold=fold).astimezone(UTC)
local = candidate.astimezone(_AMSTERDAM)
if local.replace(tzinfo=None) == naive and local.fold == fold and candidate not in candidates:
candidates.append(candidate)
if not candidates:
raise ValueError("WarmteLink timestamp is unavailable")
received_at = _utc(received_at)
recorded_at = min(candidates, key=lambda candidate: abs(candidate - received_at))
if abs(recorded_at - received_at) > _MAX_CLOCK_SKEW:
raise ValueError("WarmteLink timestamp is unavailable")
return recorded_at
def _suggestion(unit: str) -> str | None:
return {"GJ": "heating", "": "hot_water"}.get(unit)
def _canonical_unit(unit: str) -> str:
"""Map the P1 spelling of cubic metres to the source-profile unit."""
return "" if unit == "m3" else unit
def _utc(value: datetime) -> datetime:
return value.replace(tzinfo=UTC) if value.tzinfo is None else value.astimezone(UTC)
+436
View File
@@ -0,0 +1,436 @@
"""Read-only WarmteLink serial workers and their lifecycle manager.
The worker deliberately owns no long-lived SQLAlchemy session and never
retains telegram bytes after handing a complete frame to the ingestor.
"""
from __future__ import annotations
from collections.abc import Callable
from contextlib import suppress
from dataclasses import dataclass, field
from datetime import UTC, datetime, timedelta
import logging
import threading
from typing import Protocol
import serial
from sqlalchemy import select
from sqlalchemy.orm import Session
from app.db import get_session_local
from app.integrations.p1 import TelegramFramer
from app.models.meter_source import MeterSource
from app.services.warmtelink_ingest import WarmteLinkIngestor
logger = logging.getLogger(__name__)
_BACKOFF_SECONDS = (1, 2, 4, 8, 16, 32, 60)
_JOIN_TIMEOUT_SECONDS = 5
_DISCOVERY_LOCK_TIMEOUT_SECONDS = 0.05
_DISCOVERY_WAIT_SECONDS = 0.1
_DISCOVERY_TIMEOUT_SECONDS = 5
class ReadOnlySerial(Protocol):
def read(self, size: int = 1) -> bytes: ...
def close(self) -> None: ...
SerialFactory = Callable[[dict], ReadOnlySerial]
SessionFactory = Callable[[], Session]
def _default_session_factory() -> Session:
"""Resolve the cached sessionmaker at call time, then open one session."""
return get_session_local()()
class WorkerClock(Protocol):
"""Injectable interruptible clock, keeping retry tests deterministic."""
def wait(self, stop_event: threading.Event, seconds: float) -> bool: ...
class _EventClock:
def wait(self, stop_event: threading.Event, seconds: float) -> bool:
return stop_event.wait(seconds)
def open_warmtelink_serial(config: dict) -> ReadOnlySerial:
"""Open the fixed WarmteLink P1 profile; no write-capable API is exposed."""
return serial.Serial(
port=config["path"], baudrate=115200, bytesize=serial.SEVENBITS,
parity=serial.PARITY_NONE, stopbits=serial.STOPBITS_ONE, timeout=1,
)
@dataclass(frozen=True)
class _WorkerConfig:
source_id: int
config: dict
@dataclass
class DiscoveryRequest:
"""One source-scoped request, completed only by its serial owner."""
request_id: int
source_id: int
deadline: datetime
status: str = "pending"
detail: str | None = None
completed: threading.Event = field(default_factory=threading.Event)
class WarmteLinkWorker:
"""One interruptible, read-only serial loop for one meter source."""
def __init__(
self, source_id: int, config: dict, *, session_factory: SessionFactory = _default_session_factory,
serial_factory: SerialFactory = open_warmtelink_serial,
stop_event: threading.Event | None = None,
ingestor: WarmteLinkIngestor | None = None,
clock: WorkerClock | None = None,
) -> None:
self.source_id = source_id
self.config = dict(config)
self._session_factory = session_factory
self._serial_factory = serial_factory
self._stop_event = stop_event or threading.Event()
self._ingestor = ingestor or WarmteLinkIngestor()
self._clock = clock or _EventClock()
self._serial: ReadOnlySerial | None = None
self._serial_lock = threading.Lock()
self._discovery_lock = threading.Lock()
self._discoveries: list[DiscoveryRequest] = []
# Never inherit a daemon flag from a caller's background thread: a serial
# descriptor and its orderly shutdown must remain visible to the process.
self._thread = threading.Thread(
target=self._run, name=f"warmtelink-{source_id}", daemon=False
)
@property
def thread(self) -> threading.Thread:
return self._thread
def start(self) -> None:
self._thread.start()
def stop(self) -> None:
self._stop_event.set()
self._close_serial()
def request_discovery(self, request: DiscoveryRequest) -> None:
"""Queue a read request; this worker remains the sole serial owner."""
with self._discovery_lock:
self._discoveries.append(request)
def _finish_discoveries(self, status: str, detail: str | None = None) -> None:
now = datetime.now(UTC)
with self._discovery_lock:
pending, self._discoveries = self._discoveries, []
for request in pending:
if request.completed.is_set():
continue
if request.deadline <= now and status == "completed":
request.status, request.detail = "error", "Discovery timed out."
else:
request.status, request.detail = status, detail
request.completed.set()
def _expire_discoveries(self) -> None:
now = datetime.now(UTC)
with self._discovery_lock:
expired = [request for request in self._discoveries if request.deadline <= now]
self._discoveries = [request for request in self._discoveries if request.deadline > now]
for request in expired:
if request.completed.is_set():
continue
request.status, request.detail = "error", "Discovery timed out."
request.completed.set()
def join(self, timeout: float = _JOIN_TIMEOUT_SECONDS) -> bool:
self._thread.join(timeout)
return not self._thread.is_alive()
def _close_serial(self) -> None:
with self._serial_lock:
device, self._serial = self._serial, None
if device is not None:
with suppress(Exception):
device.close()
def _record_error(self, message: str) -> None:
try:
with self._session_factory() as session:
source = session.get(MeterSource, self.source_id)
if source is not None:
source.status = "error"
source.last_error = message
source.updated_at = datetime.now(UTC)
session.commit()
except Exception:
# A source-status failure must not end another source's worker.
return
def _run(self) -> None:
backoff_index = 0
while not self._stop_event.is_set():
try:
device = self._serial_factory(self.config)
with self._serial_lock:
if self._stop_event.is_set():
with suppress(Exception):
device.close()
return
self._serial = device
# A disconnect makes any bytes buffered from the previous
# descriptor untrustworthy. In particular, never let a
# trailing partial telegram be completed by a newly opened
# device.
framer = TelegramFramer()
while not self._stop_event.is_set():
self._expire_discoveries()
chunk = device.read(1024)
if not chunk:
# ``timeout`` reads are normal, but still yield so a bad
# fake/device cannot turn an empty read into a busy spin.
self._clock.wait(self._stop_event, 0.05)
continue
frames = framer.feed(chunk)
for frame in frames:
if self._stop_event.is_set():
break
admitted = self._ingestor.handle_frame(
self.source_id, frame, session_factory=self._session_factory
)
if admitted:
self._finish_discoveries("completed")
# A complete frame proves transport recovery even if its
# contents are rejected by the privacy/admission layer.
backoff_index = 0
except Exception:
self._finish_discoveries("error", "WarmteLink discovery failed.")
self._record_error("WarmteLink serial connection failed")
delay = _BACKOFF_SECONDS[min(backoff_index, len(_BACKOFF_SECONDS) - 1)]
backoff_index += 1
self._clock.wait(self._stop_event, delay)
finally:
self._close_serial()
class WarmteLinkWorkerManager:
"""Reconcile enabled serial sources into exactly one worker each."""
def __init__(
self, *, session_factory: SessionFactory = _default_session_factory,
serial_factory: SerialFactory = open_warmtelink_serial,
worker_factory: Callable[..., WarmteLinkWorker] = WarmteLinkWorker,
) -> None:
self._session_factory = session_factory
self._serial_factory = serial_factory
self._worker_factory = worker_factory
self._workers: dict[int, tuple[_WorkerConfig, WarmteLinkWorker]] = {}
self._lock = threading.Lock()
self._reapers: set[int] = set()
self._next_discovery_id = 0
self._shutting_down = False
@property
def worker_count(self) -> int:
with self._lock:
return len(self._workers)
def reconcile(self) -> None:
# Reading desired state under the same lock which applies it prevents a
# delayed pre-commit snapshot from rolling a newer commit backwards.
with self._lock:
if self._shutting_down:
return
desired = self._read_desired()
self._reconcile_locked(desired)
def start(self) -> None:
"""Enable reconciliation for a newly entered application lifespan."""
with self._lock:
self._shutting_down = False
self.reconcile()
def request_discovery(self, source_id: int) -> DiscoveryRequest:
"""Ask the current source worker for one bounded read/discovery attempt.
This intentionally does not reconcile or open a descriptor. Lifecycle
convergence remains separate; a request can neither replace nor stop a
worker when an HTTP client times out or disconnects.
"""
now = datetime.now(UTC)
request = DiscoveryRequest(0, source_id, now)
if not self._lock.acquire(timeout=_DISCOVERY_LOCK_TIMEOUT_SECONDS):
request.status, request.detail = "error", "Discovery queue is busy."
request.completed.set()
return request
try:
self._next_discovery_id += 1
request.request_id = self._next_discovery_id
request.deadline = now + timedelta(seconds=_DISCOVERY_TIMEOUT_SECONDS)
if self._shutting_down:
request.status, request.detail = "error", "WarmteLink manager is stopped."
request.completed.set()
elif (entry := self._workers.get(source_id)) is None:
request.status, request.detail = "error", "WarmteLink worker is not running."
request.completed.set()
else:
entry[1].request_discovery(request)
timer = threading.Timer(_DISCOVERY_TIMEOUT_SECONDS, self._timeout_discovery, args=(request,))
timer.daemon = True
timer.start()
finally:
self._lock.release()
# A tiny bounded wait makes an immediately available frame observable,
# without turning an HTTP call into serial I/O or an unbounded wait.
request.completed.wait(_DISCOVERY_WAIT_SECONDS)
return request
@staticmethod
def _timeout_discovery(request: DiscoveryRequest) -> None:
"""Resolve a stale HTTP request without touching its healthy worker."""
if not request.completed.is_set():
request.status, request.detail = "error", "Discovery timed out."
request.completed.set()
def _read_desired(self) -> dict[int, _WorkerConfig]:
with self._session_factory() as session:
return {
source.id: _WorkerConfig(source.id, dict(source.config))
for source in session.execute(
select(MeterSource).where(
MeterSource.kind == "warmtelink_serial", MeterSource.enabled.is_(True)
)
).scalars()
}
def _record_manager_error(self, source_id: int) -> None:
"""Best-effort, deliberately non-sensitive lifecycle failure status."""
try:
with self._session_factory() as session:
source = session.get(MeterSource, source_id)
if source is not None:
source.status = "error"
source.last_error = "WarmteLink worker failed"
source.updated_at = datetime.now(UTC)
session.commit()
except Exception:
return
def _reconcile_locked(self, desired: dict[int, _WorkerConfig]) -> None:
stale = [
source_id for source_id, (config, _) in self._workers.items()
if source_id not in desired or desired[source_id] != config
]
blocked: set[int] = set()
for source_id in stale:
_, worker = self._workers[source_id]
try:
worker.stop()
stopped = worker.join()
except Exception:
self._record_manager_error(source_id)
blocked.add(source_id)
continue
if stopped:
self._workers.pop(source_id, None)
else:
logger.error("WarmteLink worker did not stop for source %s", source_id)
blocked.add(source_id)
self._schedule_reaper_locked(source_id, worker)
for source_id, config in desired.items():
if source_id in self._workers or source_id in blocked:
continue
worker: WarmteLinkWorker | None = None
try:
worker = self._worker_factory(
source_id, config.config, session_factory=self._session_factory,
serial_factory=self._serial_factory,
)
self._workers[source_id] = (config, worker)
worker.start()
except Exception:
self._record_manager_error(source_id)
# A failed start normally has no thread. If an unusual worker
# did start before raising, keep it tracked until it is reaped.
if not self._worker_is_alive(worker):
self._workers.pop(source_id, None)
else:
self._schedule_reaper_locked(source_id, worker)
@staticmethod
def _worker_is_alive(worker: object | None) -> bool:
thread = getattr(worker, "thread", None)
return bool(thread is not None and thread.is_alive())
def _schedule_reaper_locked(self, source_id: int, worker: WarmteLinkWorker) -> None:
if source_id in self._reapers:
return
self._reapers.add(source_id)
threading.Thread(
target=self._reap_worker, args=(source_id, worker),
# This bookkeeping watcher must not turn a deliberately bounded
# application shutdown into an unbounded process wait. The actual
# serial worker itself is explicitly non-daemon.
name=f"warmtelink-reaper-{source_id}", daemon=True,
).start()
def _reap_worker(self, source_id: int, worker: WarmteLinkWorker) -> None:
"""Wait for one timed-out worker, then converge without another API call."""
try:
while True:
try:
if worker.join():
break
except Exception:
self._record_manager_error(source_id)
return
# A custom worker can report a bounded join timeout immediately;
# yield before asking again so its reaper cannot busy-spin.
threading.Event().wait(0.05)
with self._lock:
current = self._workers.get(source_id)
if current is not None and current[1] is worker:
self._workers.pop(source_id)
self._reapers.discard(source_id)
should_reconcile = not self._shutting_down
if should_reconcile:
self.reconcile()
finally:
with self._lock:
self._reapers.discard(source_id)
def shutdown(self) -> None:
with self._lock:
self._shutting_down = True
workers = list(self._workers.items())
for source_id, (_, worker) in workers:
try:
worker.stop()
except Exception:
self._record_manager_error(source_id)
for source_id, (_, worker) in workers:
try:
stopped = worker.join()
except Exception:
self._record_manager_error(source_id)
continue
if not stopped:
logger.error("WarmteLink worker did not stop during shutdown for source %s", source_id)
with self._lock:
self._schedule_reaper_locked(source_id, worker)
else:
with self._lock:
current = self._workers.get(source_id)
if current is not None and current[1] is worker:
self._workers.pop(source_id)
warmtelink_worker_manager = WarmteLinkWorkerManager()
+4
View File
@@ -26,3 +26,7 @@ services:
- "127.0.0.1:8002:8000"
environment:
APP_DATABASE_URL: "sqlite:////app/data/app.db"
devices: !override
- "${WARMTELINK_DEVICE_PATH:?Set a stable /dev/serial/by-id path}:/dev/warmtelink:rw"
group_add: !override
- "${WARMTELINK_SERIAL_GID:?Set the host serial device GID}"
+4 -1
View File
@@ -22,8 +22,11 @@ services:
condition: service_completed_successfully
ports:
- "127.0.0.1:8881:8000"
devices:
- "${WARMTELINK_DEVICE_PATH:?Set a stable /dev/serial/by-id path}:/dev/warmtelink:rw"
group_add:
- "${WARMTELINK_SERIAL_GID:?Set the host serial device GID}"
volumes:
- ./data:/app/data
- ./.env:/app/.env:ro
- /etc/localtime:/etc/localtime:ro
+6 -6
View File
@@ -19,7 +19,7 @@
- `main.py`
- FastAPI app factory
- lifespanAPScheduler 启停、MQTT 客户端起停、连接后触发 HA Discovery 发布;M6 新增 `tibber-refresh` 抓价 job + `energy-cost` 1 分钟计费 tick jobM6 启用时注册 DSMR MQTT 订阅
- lifespanAPScheduler 启停、MQTT 客户端起停、连接后触发 HA Discovery 发布;注册 DSMR MQTT source,并启动/关闭每个 enabled WarmteLink 业务只读 serial worker`tibber-refresh`、electricity 与 thermal cost tick 均由 scheduler 驱动
- 基础路由注册
- `config.py`
- 环境变量驱动的 settings(含 M5 新增的 MQTT/HA Discovery/Modbus 配置项;M6 新增 `dsmr_ingest_enabled``dsmr_mqtt_topic``dsmr_sample_interval_s``tibber_api_token`secret)、`tibber_home_id`
@@ -29,22 +29,22 @@
- 通用依赖注入
- `api/`
- HTTP routes
- `api/routes/api/`JSON API`/api/*` 前缀),供 React SPA 调用:会话/鉴权、配置读写、数据查询、记录 CRUD、Modbus 设备 CRUD + readings + metrics + test`/api/modbus/*`)、Expose 勾选 + 重发 discovery`/api/expose`)、MQTT 测试连接(`/api/config/mqtt/test`)、M6 新增合同 CRUD + 版本(`/api/energy/contracts*`)、pricing profile 列表(`/api/energy/profiles`)、价格/费用/汇总/DSMR 最新/重算/Tibber 测试(`/api/energy/prices``/api/energy/costs``/api/energy/costs/summary``/api/energy/dsmr/latest``/api/energy/costs/recompute``/api/energy/tibber/test`
- `api/routes/api/`JSON API`/api/*` 前缀),供 React SPA 调用:会话/鉴权、配置读写、记录 CRUD、Modbus、Expose 与 MQTT 测试;Energy 包含 source profile/source/channel/history/discover/binding/Meter API、scope-aware contracts/prices/costs,以及兼容的 DSMR latest API
- 裸 ingestion 端点:`GET /public-ip/check``POST /homeassistant/publish``POST /poo/record``GET /poo/latest`、TickTick OAuth 等
- `models/`
- SQLAlchemy models
- 所有模型(auth / config / public_ip / location / poo / modbus / expose / energy)共用同一个 `Base`,均落在单一 `app.db`
- 所有模型(auth / config / public_ip / location / poo / modbus / expose / energy / meter_source)共用同一个 `Base`,均落在单一 `app.db`
- M5 新增:`ModbusDevice`(设备部署层)、`ModbusReading`(通用遥测,JSON payload)、`ExposedEntityToggle`HA 实体暴露开关)
- M6 新增:`DsmrReading`(整帧 DSMR telegram10s 降采样)、`EnergyContract`(合同头,含 active 标记)、`EnergyContractVersion`(版本/时段,values JSON,只增不改)、`TibberPrice`15 分钟价缓存,不可变)、`EnergyCostPeriod`(每 15 分钟计量电费,快照价,不可变)
- Energy`MeterSource` / `MeterSourceChannel` / `MeterSourceBinding` 将协议连接、稳定测量 channel 与 Meter epoch 分离;`DsmrReading``WarmteLinkReading` 分别保存 JSON 与 Decimal scalar 历史;electricity `EnergyCostPeriod` 绑定 source bindingthermal `MeterCostPeriod` 保存审计账本;合同按 electricity / thermal scope 共存
- `schemas/`
- Pydantic schemasM5 新增 `modbus.py``expose.py`M6 新增 `energy_contract.py``energy.py`
- Pydantic schemas包括 `modbus.py``expose.py``energy_contract.py``energy.py``meter_source.py`
- `services/`
- 业务服务层
- 当前已迁入 config page 的 DB 持久化逻辑
- 当前已迁入 public IPv4 检查、状态持久化与变化通知逻辑
- 当前已迁入 SMTP 发信与测试发信逻辑
- M5 新增:`modbus_poll.py`(采集 service,逐设备 poll + 落库 + 推 MQTT state)、`ha_discovery.py`(构建 HA Discovery payload、发布 retained config、发布 state
- M6 新增:`tibber_prices.py`httpx GraphQL 抓 15 分钟价,upsert `tibber_price`,幂等;仅 active=tibber 且 token 存在时运行)、`dsmr_ingest.py`MQTT handler,整帧 JSON blob + 10s 降采样落库,`source_id` 幂等)、`energy_cost.py`(计费引擎:每 15 分钟寄存器差 × strategy 出价 → `energy_cost_period` 不可变快照;汇总 Σnet + 固定费 heffingskorting;重算显式 opt-in
- Energy`dsmr_ingest.py` 按 source 入库;`warmtelink_ingest.py` 接纳连续确认的业务只读 P1 scalar,`warmtelink_worker.py` 管理 interruptible serial reconnectpyserial 的 POSIX `O_RDWR` 打开由非 root、非 privileged、无 `m` 的 Docker `rw` device rule 支持;worker 只 read/close,绝不 write);`energy_cost.py``meter_cost.py` 分别计算 electricity/thermal 账本,均拒绝跨 Meter/binding 相减
- `integrations/`
- 外部系统适配层
- Home Assistant outbound adapterREST 通道,原有)
+1 -1
View File
@@ -10,7 +10,7 @@
- [`m6-tibber-dynamic-energy.md`](./m6-tibber-dynamic-energy.md) — 通用电价层 + DSMR 实时电表接入 + 实时买卖电费计算 + HA Energy 反哺
- [`m7-meter-epochs-archival.md`](./m7-meter-epochs-archival.md) — 电表生命周期 / 换表归档(Meter epochs
- [`pre-m8-warmtelink-p1-poc.md`](./pre-m8-warmtelink-p1-poc.md) — WarmteLink P1 真机概念验证(已完成;正式 CLI 长测与供暖变化均经物理表复核)
- [`m8-warmtelink-energy.md`](./m8-warmtelink-energy.md) — WarmteLink P1、多数据源 Meter 与热力计费(Planning 已完成;M8-T01M8-T20 待实现
- [`m8-warmtelink-energy.md`](./m8-warmtelink-energy.md) — WarmteLink P1、多数据源 Meter 与热力计费(M8-T01T20 自动化技术验收已完成;交付后用户人工 walkthrough 待验收
本文件定义**所有任务共用的格式与协作规则**,各个里程碑文档不再重复这些约定。
+195 -61
View File
@@ -1,6 +1,6 @@
# M8 — WarmteLink P1、多数据源 Meter 与热力计费
> **状态:Planning 已完成,M8-T01M8-T20 待实现。**
> **状态:M8-T01M8-T20 自动化技术验收已完成;交付后用户人工 walkthrough 待验收。**
> [Pre-M8 真机概念验证](./pre-m8-warmtelink-p1-poc.md) 已完成并通过翻牌;本文锁定
> M8 的目标架构、迁移顺序、外部契约和可由编排器逐张执行的原子任务。
@@ -18,7 +18,7 @@ M8 完成后,Energy 模块不再把“电表”“采集连接”和“协议
两种热力累计量、15 分钟成本、每日固定费和汇总。
- 在 Energy UI 中分别管理 Sources、Modbus Devices、Meters、合同、价格与成本,并按需把新的
source/meter/cost 实体暴露给 Home Assistant。
- 以非 root、只读串口方式部署;worker 能启动、热更新、断线重连和干净停止。
- 以非 root 部署:Docker device rule 用 pyserial POSIX `O_RDWR` 所需的最小 `rw`(绝无 `m`),而 worker 业务只读;worker 能启动、热更新、断线重连和干净停止。
M8 不承诺 WarmteLink 当前 telegram 未提供的瞬时流量、热功率、供/回水温度,也不推断
“生活热水消耗了多少 GJ”。不把 WarmteLink 塞进 `modbus_device`,不把所有协议读数强行泛化成
@@ -176,6 +176,35 @@ Source 负责连接和健康状态,channel 负责稳定的测量身份,bindi
所有 migration 必须在空库和带历史数据的升级副本上通过,迁移前后逐表对账,不 drop/truncate 任何
业务数据。SQLite batch migration 后显式验证 FK、索引和唯一约束。
### M8 Alembic revision 计划与仓库级不变量
M8 的 schema 变更沿现有单一 `alembic_app` 链按下表串行落地;不得合并成一次不可审计的大迁移,
也不得创建并行 head
| Task / revision | `down_revision` | Schema 变更 |
| --- | --- | --- |
| T01 / `20260822_15_meter_sources` | `20260625_14_meter_uuid` | 新建 `meter_source``meter_source_channel``meter_source_binding`;给 `energy_cost_period` 增加 nullable `source_binding_id` |
| T03 / `20260822_16_dsmr_source_adoption` | `20260822_15_meter_sources` | `dsmr_reading.source_id` 政名为 `telegram_id`,新增 non-null `meter_source_id`,唯一键改为 `(meter_source_id, recorded_at)`,并执行 DSMR/source/binding/cost 历史回填 |
| T08 / `20260822_17_warmtelink_readings` | `20260822_16_dsmr_source_adoption` | 新建 `warmtelink_reading` 标量历史表 |
| T12 / `20260822_18_contract_scopes` | `20260822_17_warmtelink_readings` | 给 `energy_contract` 增加 non-null `scope`,旧行回填 `electricity` |
| T14 / `20260822_19_meter_cost_periods` | `20260822_18_contract_scopes` | 新建 `meter_cost_period` 审计账本 |
每张新增 revision 的任务还必须同时遵守以下仓库级契约:
1. 新模型模块在首次引入时显式 import 到 `alembic_app/env.py`,保证 `Base.metadata` 完整;后续只在
已注册模块中新增模型时无需重复改 env。
2. 同一任务把 `scripts/app_db_adopt.py::APP_BASELINE_REVISION` 更新为该任务的新 head。现有
fail-closed 启动校验和多组部署/模型回归测试都要求该常量与唯一 Alembic head 完全相等;不得把
常量临时留在旧 revision 等后续任务补救。
3. `scripts/run_migrations.py` 继续只调用 app DB adoption/upgrade,不新增第二条迁移链;每个任务验证
旧 head→新 head、空库→head 和重复运行幂等。
4. 实现、review 和 T20 收尾只使用 pytest `tmp_path``/tmp` 下明确命名的临时 SQLite 库以及合成的
历史 fixture。不得挂载、复制、打开或修改运行中 production 的 DB 路径、容器或 volume;所谓
“历史升级副本”在本轮指结构和边界场景等价的隔离合成副本,不含真实生产数据。
5. 每次升级对账至少记录 migration 前后业务表行数、Alembic revision、孤儿 FK、关键唯一键/索引;
任何不一致必须在同一事务中失败并回滚。downgrade 仅用于临时测试库的 schema 可逆性验证,绝不
对生产或用户备份执行。
### DSMR 回填算法
1. 读取旧 runtime config,创建一个 `dsmr_mqtt` source 和稳定 electricity channel;无旧 config 时也
@@ -216,9 +245,13 @@ Source 负责连接和健康状态,channel 负责稳定的测量身份,bindi
- serial 默认 `115200``7` data bits、no parity、`1` stop bit;设备路径必须是 `/dev/...`
- `warmtelink_serial` config 保存路径和串口参数,不保存 telegram/equipment id;敏感字段采用现有
secret mask 约定。
- 新增可选 `docker-compose.warmtelink.yml` overlay,把宿主机稳定 `/dev/serial/by-id/...` 映射为
容器内 `/dev/warmtelink`通过宿主机 serial GID 授权,容器仍非 root。
- 默认 `docker-compose.yml` 在没有硬件时仍可启动,不强制声明不存在的 device。
- 现有 base+dev Compose 通过 `WARMTELINK_DEVICE_PATH``WARMTELINK_SERIAL_GID` 环境变量,把宿主机稳定
`/dev/serial/by-id/...` 直接映射为容器内 `/dev/warmtelink`并以宿主机 serial GID 授权。pyserial 在 POSIX
`O_RDWR` 打开,故 device rule 的最小系统权限为 `rw`(不含 `m`);容器仍非 root/非 privilegedworker
业务只 read/close、绝不 write。
- 当前 base+dev Compose 要求在 `.env`(或等效部署环境)中同时提供
`WARMTELINK_DEVICE_PATH``WARMTELINK_SERIAL_GID`,并固定声明该 device 映射;没有有效硬件路径和
serial GID 的部署不能把 Compose 当作“无硬件可启动”的配置。
- source disable/delete/config change 必须停止旧 workershutdown 不留下线程或打开的 serial fd。
## 8. 前端信息架构
@@ -254,7 +287,7 @@ T01T06 先把现有 DSMR 安全迁到统一 source/bindingT07T11 再接
### M8-T01 — 建立 Source / Channel / Binding 基础模型 [structural]
- **Status**: `todo`
- **Status**: `done`
- **Depends**: none
- **Context**: 先建立协议无关的身份链和时间约束;本卡只改 schema/model,不接运行时或 HTTP。
@@ -262,7 +295,9 @@ T01T06 先把现有 DSMR 安全迁到统一 source/bindingT07T11 再接
- `create app/models/meter_source.py`
- `modify app/models/__init__.py`
- `modify app/models/energy.py`
- `modify alembic_app/env.py`
- `create alembic_app/versions/20260822_15_meter_sources.py`
- `modify scripts/app_db_adopt.py`
- `create tests/test_meter_sources.py`
- `modify tests/test_energy_models.py`
@@ -273,9 +308,12 @@ T01T06 先把现有 DSMR 安全迁到统一 source/bindingT07T11 再接
半开区间相交,数据库 FK 使用 `RESTRICT`relationship 不配置 delete cascade。
3. 给 `EnergyCostPeriod` 增加 nullable `source_binding_id` FK 和 relationship;迁移此时只加列,
不回填 DSMR 历史。
4. migration 在 SQLite batch 模式创建表、索引和约束;downgrade 只能回退 schema,不删除任何
4. `alembic_app/env.py` 注册新模型模块,并把 `APP_BASELINE_REVISION` 同步到 revision 15
不改变 `scripts/run_migrations.py` 的单库行为。
5. migration 在 SQLite batch 模式创建表、索引和约束;downgrade 只能回退 schema,不删除任何
外部数据库文件。
5. 测试 Alembic 空库升级、模型默认值、唯一约束、FK RESTRICT 和无 delete cascade。
6. 测试 Alembic 空库升级、revision 14→15、重复运行、baseline=head、模型默认值、唯一约束、
FK RESTRICT 和无 delete cascade;数据库只使用 `tmp_path` 临时文件。
**Out of scope / 不要碰**
- 不迁移 `dsmr_reading`,不读取旧 config,不实现 API、worker 或 source profile。
@@ -284,16 +322,17 @@ T01T06 先把现有 DSMR 安全迁到统一 source/bindingT07T11 再接
**Acceptance criteria**
- [ ] 三张新表、UUID/唯一键/FK/索引与 §5 一致,`energy_cost_period` 新列可空。
- [ ] 删除被 channel/binding/成本引用的行会被数据库拒绝,不级联丢历史。
- [ ] 空库与从 revision 14 升级都到达 head,测试覆盖约束。
- [ ] 空库与从 revision 14 升级都到达 head`APP_BASELINE_REVISION` 等于唯一 head测试覆盖约束。
- [ ] `pytest``ruff check .` 全绿。
**Reviewer checklist**
- migration 是否完全无 drop/truncate/用户文件操作;SQLite 重建后 FK 是否仍启用。
- `alembic_app/env.py` 是否已注册新模型,baseline 常量是否与 revision 15 完全一致。
- 时间窗是否统一为 `[started_at, ended_at)`,没有把 source 生命周期混进 Meter epoch。
### M8-T02 — Source profile registry 与 binding service
- **Status**: `todo`
- **Status**: `done`
- **Depends**: M8-T01
- **Context**: 在开放 API 前集中 kind config、commodity/unit 兼容与时间线规则,避免各入口各写一套。
@@ -328,15 +367,17 @@ T01T06 先把现有 DSMR 安全迁到统一 source/bindingT07T11 再接
### M8-T03 — 将 DSMR 历史迁入 Source / Binding [structural]
- **Status**: `todo`
- **Status**: `done`
- **Depends**: M8-T02
- **Context**: 先把既有电力链路安全迁到统一模型,之后才能让 runtime 和计费真正按 source 工作。
**Files**
- `modify app/models/energy.py`
- `create alembic_app/versions/20260822_16_dsmr_source_adoption.py`
- `modify scripts/app_db_adopt.py`
- `create tests/test_dsmr_source_migration.py`
- `modify tests/test_energy_models.py`
- `modify tests/test_meter_sources.py`
**Steps**
1. 把 ORM `DsmrReading.source_id` 重命名为 nullable `telegram_id`,新增 non-null
@@ -347,6 +388,10 @@ T01T06 先把现有 DSMR 安全迁到统一 source/bindingT07T11 再接
binding 保持 nullablemigration 注释和测试必须明确这是历史边界,而非当前 binding。
4. 每一阶段在 migration 内核对 source/readings/cost 行数与 orphan FK;不一致立即抛错回滚。
5. 用 revision 14 的历史 fixture 覆盖单 Meter、多次换表、跨界成本、无读数 Meter 和无旧 config。
6. 把 `APP_BASELINE_REVISION` 同步到 revision 16;所有升级 fixture 仅在 `tmp_path` 中构造,不读取
真实 app DB 或 volume。
7. 把 T01 自身的 revision 14→15 schema 对账测试固定升级到明确 revision 15,不以可继续前进的
`head` 作为 T01 终点;T03 的新 fixture 单独负责 revision 14/15→16 历史回填对账。
**Out of scope / 不要碰**
- 不删除旧 DSMR `app_config` 行,不改变 MQTT subscription,不改 API 响应。
@@ -356,15 +401,19 @@ T01T06 先把现有 DSMR 安全迁到统一 source/bindingT07T11 再接
- [ ] 迁移前后 DSMR 与成本行数不减少,orphan FK 为 0,正常可解析周期有 binding。
- [ ] 两个 source 可在同一 timestamp 各存一条 DSMR reading;同 source 重复 timestamp 被拒绝。
- [ ] `telegram_id` 不参与幂等唯一键,旧 telegram id 完整保留。
- [ ] `APP_BASELINE_REVISION` 等于唯一 revision 16 head,升级重复运行幂等。
- [ ] T01 schema-only fixture 固定停在 revision 15T03 fixture 到 revision 16,二者职责不随 head 漂移。
- [ ] 历史升级 fixture、空库升级、`pytest``ruff check .` 全绿。
**Reviewer checklist**
- 对账是否在 migration 中真实执行,而不只是测试断言;失败能否原子回滚。
- baseline 常量是否随 revision 16 同步,fixture 是否完全隔离于真实生产路径。
- 前序 migration 测试是否使用明确 revision 边界,而非把历史阶段误写为永久 `head`
- 是否存在“把所有历史强绑当前 Meter/source”的静默错误或任何 destructive cleanup。
### M8-T04 — DSMR runtime 改为多 Source 配置 [structural]
- **Status**: `todo`
- **Status**: `done`
- **Depends**: M8-T03
- **Context**: schema 回填后,DSMR subscription/ingest 应以数据库 source 为单一运行时配置来源。
@@ -406,7 +455,7 @@ T01T06 先把现有 DSMR 安全迁到统一 source/bindingT07T11 再接
### M8-T05 — 电费计算绑定 Source Binding [structural]
- **Status**: `todo`
- **Status**: `done`
- **Depends**: M8-T04
- **Context**: DSMR 已多 source 后,电力 period 必须只在同一 Meter/binding 累计域内计算。
@@ -416,6 +465,10 @@ T01T06 先把现有 DSMR 安全迁到统一 source/bindingT07T11 再接
- `modify app/schemas/energy.py`
- `modify tests/test_energy_cost.py`
- `modify tests/test_api_energy.py`
- `modify tests/test_energy_expose.py`
- `modify openapi/openapi.json`
- `modify openapi/openapi.yaml`
- `modify frontend/src/api/schema.d.ts`
**Steps**
1. 为 period 两个边界按时间解析唯一 electricity Meter、binding 和 DSMR source;查询 reading 时加入
@@ -426,6 +479,10 @@ T01T06 先把现有 DSMR 安全迁到统一 source/bindingT07T11 再接
不删除旧字段。
4. 用现有 golden tests 锁住单 source 正常 period 的 quantity、cost、rounding、fixed fee 与 summary
增加 source switch、binding boundary、missing/ambiguous binding 测试。
5. 因响应 schema 增加 nullable binding identity,重导 OpenAPI,并在 `frontend/` 运行
`npm run codegen`;两个生成物必须随本卡提交,禁止手改 `schema.d.ts`
6. 固定现有 expose fixed-fee/credit 回归中的 `local_now()` 到明确已越过 01:05 结算点的时刻;测试
不得依赖执行当天恰好处于 UTC 00:00~01:05 之外,也不得为消除红灯改变生产结算语义。
**Out of scope / 不要碰**
- 不实现 thermal cost,不修改合同 scope,不改变正常电价公式。
@@ -435,15 +492,18 @@ T01T06 先把现有 DSMR 安全迁到统一 source/bindingT07T11 再接
- [ ] 正常新周期总能审计到唯一 binding,跨域周期明确 degraded。
- [ ] 既有单 DSMR source 的所有非降级数字逐项不变。
- [ ] recompute 幂等,不能把 source A 起点和 source B 终点相减。
- [ ] `pytest``ruff check .` 全绿
- [ ] fixed-fee/credit golden tests 在 01:05 前后任意实际运行时刻均确定性通过,生产结算点不变
- [ ] `pytest``ruff check .`、OpenAPI/codegen 同步闸门全绿且生成物已提交。
**Reviewer checklist**
- 重点构造 Meter 相同但 source 切换、source 相同但 Meter 换表两种边界。
- 检查 Decimal/rounding 和本地日 fixed-fee 逻辑是否被无意改变。
- 检查 expose golden tests 是否显式固定业务时钟,而不是等待 wall clock 或放宽 01:05 断言。
- 独立重导 OpenAPI 与 codegen,确认 schema 生成物同步且不是手改。
### M8-T06 — Source / Channel / Binding HTTP 契约 [structural]
- **Status**: `todo`
- **Status**: `done`
- **Depends**: M8-T05
- **Context**: 在基础服务与 DSMR 兼容稳定后,提供 §6 的管理接口及原子 Meter+binding 入口。
@@ -485,7 +545,7 @@ T01T06 先把现有 DSMR 安全迁到统一 source/bindingT07T11 再接
### M8-T07 — 提取可复用 P1 parser(零行为变化)
- **Status**: `todo`
- **Status**: `done`
- **Depends**: M8-T02
- **Context**: Pre-M8 parser 已有真机 fixture 证据;先无损提取,避免 worker 与 probe 维护两份协议逻辑。
@@ -521,13 +581,14 @@ T01T06 先把现有 DSMR 安全迁到统一 source/bindingT07T11 再接
### M8-T08 — 建立 WarmteLink 标量读数表
- **Status**: `todo`
- **Status**: `done`
- **Depends**: M8-T07
- **Context**: WarmteLink 与 DSMR payload 结构不同,建立可精确计算的 Decimal 标量历史表。
**Files**
- `modify app/models/meter_source.py`
- `create alembic_app/versions/20260822_17_warmtelink_readings.py`
- `modify scripts/app_db_adopt.py`
- `create tests/test_warmtelink_models.py`
**Steps**
@@ -538,6 +599,7 @@ T01T06 先把现有 DSMR 安全迁到统一 source/bindingT07T11 再接
3. FK 使用 `RESTRICT`,不配置 orphan/delete cascade;选择足以容纳长期累计量与 0.001 精度的
Numeric precision/scale,并测试 Decimal round-trip。
4. migration 覆盖空库/升级、唯一键、FK 和索引;downgrade 不触碰任何外部文件。
5. 把 `APP_BASELINE_REVISION` 同步到 revision 17;所有 migration 测试只使用 `tmp_path` 临时库。
**Out of scope / 不要碰**
- 不实现接纳、sampling、worker、API 或成本。
@@ -547,15 +609,17 @@ T01T06 先把现有 DSMR 安全迁到统一 source/bindingT07T11 再接
- [ ] schema 与 §5 一致,Decimal 无 float 转换,source/channel 删除受限。
- [ ] 同 channel/timestamp 幂等隔离,不同 channel 同时刻允许。
- [ ] 模型和 migration 中不存在 raw telegram/equipment identifier 列。
- [ ] `APP_BASELINE_REVISION` 等于唯一 revision 17 head。
- [ ] `pytest``ruff check .` 全绿。
**Reviewer checklist**
- Numeric precision 是否覆盖合理长期累计值;timezone 与唯一键是否使用设备时间。
- migration 是否数据安全、约束命名稳定且可在 SQLite 正确执行。
- baseline 常量是否随 revision 17 同步,测试是否未打开任何真实数据库。
### M8-T09 — WarmteLink 质量接纳、发现与分钟采样 [structural]
- **Status**: `todo`
- **Status**: `done`
- **Depends**: M8-T08
- **Context**: 把 parser 输出变成可审计的 latest/history;这里锁住最关键的 unverifiable 接纳策略。
@@ -591,7 +655,7 @@ T01T06 先把现有 DSMR 安全迁到统一 source/bindingT07T11 再接
### M8-T10 — Serial worker manager 与应用生命周期 [structural]
- **Status**: `todo`
- **Status**: `done`
- **Depends**: M8-T09, M8-T06
- **Context**: 以可测试的 manager 为每个 enabled WarmteLink source 持有一个只读 worker。
@@ -629,7 +693,7 @@ T01T06 先把现有 DSMR 安全迁到统一 source/bindingT07T11 再接
### M8-T11 — WarmteLink discover、latest 与 history API
- **Status**: `todo`
- **Status**: `done`
- **Depends**: M8-T10
- **Context**: worker 链路稳定后,把一次发现、状态和规范化历史接到已建立的 source API。
@@ -667,7 +731,7 @@ T01T06 先把现有 DSMR 安全迁到统一 source/bindingT07T11 再接
### M8-T12 — 合同 Scope 与按 Scope 激活 [structural]
- **Status**: `todo`
- **Status**: `done`
- **Depends**: M8-T08
- **Context**: electricity 与 thermal 必须能各有一份 active 合同,同时保持旧客户端默认看 electricity。
@@ -677,8 +741,10 @@ T01T06 先把现有 DSMR 安全迁到统一 source/bindingT07T11 再接
- `modify app/schemas/energy_contract.py`
- `modify app/api/routes/api/energy_contracts.py`
- `create alembic_app/versions/20260822_18_contract_scopes.py`
- `modify scripts/app_db_adopt.py`
- `modify tests/test_api_energy_contracts.py`
- `modify tests/test_energy_models.py`
- `modify tests/test_warmtelink_models.py`
- `modify openapi/openapi.json`
- `modify openapi/openapi.yaml`
- `modify frontend/src/api/schema.d.ts`
@@ -692,6 +758,7 @@ T01T06 先把现有 DSMR 安全迁到统一 source/bindingT07T11 再接
list 默认兼容 electricity 行为,新增可选 scope filter。
4. migration 对账合同/version/cost 行数,不改 versions/pricing values,不删除任何合同。
5. 覆盖两 scope 同时 active、同 scope 互斥、kind/scope mismatch、旧 payload 与升级 fixture;重导契约。
6. 把 `APP_BASELINE_REVISION` 同步到 revision 18;升级 fixture 仅使用隔离临时库。
**Out of scope / 不要碰**
- 不新增真实 thermal profile/费率,不改 electricity 成本公式。
@@ -701,15 +768,17 @@ T01T06 先把现有 DSMR 安全迁到统一 source/bindingT07T11 再接
- [ ] 旧合同全部回填 electricity 且行数/version FK 不变。
- [ ] service 对预置的 electricity/thermal rows 可同时保持 active;每 scope 最多一份 active。
- [ ] 旧 manual/tibber API 请求仍成功并得到 electricity scope。
- [ ] `APP_BASELINE_REVISION` 等于唯一 revision 18 head。
- [ ] `pytest``ruff check .`、OpenAPI/codegen 同步闸门全绿。
**Reviewer checklist**
- 并发/事务失败是否可能留下同 scope 双 active 或误停用另一 scope。
- baseline 常量是否随 revision 18 同步,历史对账是否在临时副本内执行。
- migration 是否在 active 旧数据上安全,API 默认值是否真正向后兼容。
### M8-T13 — District-heating 定价 Profile
- **Status**: `todo`
- **Status**: `done`
- **Depends**: M8-T12
- **Context**: 用 profile 固定热力字段、单位和验证,不把用户的实际 Vattenfall 金额写进仓库。
@@ -745,14 +814,16 @@ T01T06 先把现有 DSMR 安全迁到统一 source/bindingT07T11 再接
### M8-T14 — 建立通用 Meter Cost Period 账本
- **Status**: `todo`
- **Status**: `done`
- **Depends**: M8-T13
- **Context**: 热力成本不能硬塞进 electricity 专用账本;先建立按 commodity 审计的独立表。
**Files**
- `modify app/models/energy.py`
- `create alembic_app/versions/20260822_19_meter_cost_periods.py`
- `modify scripts/app_db_adopt.py`
- `create tests/test_meter_cost_models.py`
- `modify tests/test_energy_models.py`
**Steps**
1. 新建 `MeterCostPeriod`,字段按 §5commodity、period start/end、nullable Meter/binding、nullable
@@ -763,6 +834,7 @@ T01T06 先把现有 DSMR 安全迁到统一 source/bindingT07T11 再接
3. quantity/cost 使用定点 NumericORM 不经 floatJSON 中的金额/数量统一序列化为十进制字符串。
4. migration 只创建新表,不迁移/删除 `energy_cost_period`;测试空库/升级、Decimal round-trip、
唯一键、nullable degraded 和 FK RESTRICT。
5. 把 `APP_BASELINE_REVISION` 同步到 revision 19;升级/降级测试只使用 `tmp_path` 临时库。
**Out of scope / 不要碰**
- 不计算或回填热力成本,不泛化/删除现有 electricity 表。
@@ -772,15 +844,17 @@ T01T06 先把现有 DSMR 安全迁到统一 source/bindingT07T11 再接
- [ ] 表结构可表达正常和降级 period,`(commodity, period_start)` 幂等。
- [ ] Decimal 与 JSON snapshot 可精确 round-trip,无 binary float。
- [ ] 现有 electricity 账本及行数完全不受 migration 影响。
- [ ] `APP_BASELINE_REVISION` 等于唯一 revision 19 head。
- [ ] `pytest``ruff check .` 全绿。
**Reviewer checklist**
- 降级 nullable 不能放松正常写入的 service invariantFK 删除策略必须保护审计。
- baseline 常量是否随 revision 19 同步,测试路径是否与生产 DB/volume 完全隔离。
- period timezone/半开区间、唯一键和 Numeric scale 是否足以重算。
### M8-T15 — Thermal 15 分钟成本引擎与调度 [structural]
- **Status**: `todo`
- **Status**: `done`
- **Depends**: M8-T14
- **Context**: 基于两个独立累计 Meter 生成可审计 variable 账本,并在 summary 层只计一次固定费。
@@ -821,7 +895,7 @@ T01T06 先把现有 DSMR 安全迁到统一 source/bindingT07T11 再接
### M8-T16 — Thermal Prices / Costs / Summary / Recompute API
- **Status**: `todo`
- **Status**: `done`
- **Depends**: M8-T15
- **Context**: 对外提供 scope-aware 价格和新的热力账本,不破坏既有 electricity 路由响应。
@@ -830,6 +904,7 @@ T01T06 先把现有 DSMR 安全迁到统一 source/bindingT07T11 再接
- `create app/api/routes/api/meter_costs.py`
- `modify app/schemas/energy.py`
- `modify app/api/routes/api/energy.py`
- `modify app/services/meter_cost.py`
- `modify app/main.py`
- `create tests/test_meter_cost_api.py`
- `modify tests/test_api_energy.py`
@@ -864,7 +939,7 @@ T01T06 先把现有 DSMR 安全迁到统一 source/bindingT07T11 再接
### M8-T17 — Home Assistant Source / Meter / Thermal 实体 [structural]
- **Status**: `todo`
- **Status**: `done`
- **Depends**: M8-T11, M8-T16
- **Context**: 在完整采集和成本链上扩展现有 expose provider,保持默认关闭与稳定 identity。
@@ -902,7 +977,7 @@ T01T06 先把现有 DSMR 安全迁到统一 source/bindingT07T11 再接
### M8-T18 — Sources UI 与多 Commodity Meters [structural]
- **Status**: `todo`
- **Status**: `done`
- **Depends**: M8-T17
- **Context**: 把新数据源/binding 概念变成可配置体验,并把 Modbus 专用页面准确命名。
@@ -952,7 +1027,7 @@ T01T06 先把现有 DSMR 安全迁到统一 source/bindingT07T11 再接
### M8-T19 — Scope-aware Contracts / Prices / Costs UI [structural]
- **Status**: `todo`
- **Status**: `done`
- **Depends**: M8-T18
- **Context**: 完成 electricity/thermal 双 scope 的合同录入、价格快照和成本审计体验。
@@ -996,14 +1071,15 @@ T01T06 先把现有 DSMR 安全迁到统一 source/bindingT07T11 再接
- 人工用小数费率核对请求 body 与展示值未被 float 改写。
- 同时启用两 scope、换 Meter、无一侧 thermal Meter 时,页面文案与 summary 是否诚实。
### M8-T20 — 部署 Overlay、运行文档与端到端收尾 [structural]
### M8-T20 — 部署 Compose 配置、运行文档与端到端收尾 [structural]
- **Status**: `todo`
- **Status**: `done`
- **Depends**: M8-T19
- **Context**: 最后一张卡把串口权限、操作 runbook、真实构建与完整链路变成可重复验收结果
- **Context**: 最后一张卡把串口权限、操作 runbook 与隔离的自动化技术验收变成可重复验收结果;真实硬件/HA 观察作为交付后由用户执行的人工 walkthrough,不是 agent 或 Reviewer 的技术 PASS 前置条件
**Files**
- `create docker-compose.warmtelink.yml`
- `modify docker-compose.yml`
- `modify docker-compose.dev.yml`
- `modify tests/test_deployment.py`
- `create docs/warmtelink-energy.md`
- `modify docs/architecture-overview.md`
@@ -1015,18 +1091,26 @@ T01T06 先把现有 DSMR 安全迁到统一 source/bindingT07T11 再接
- `modify docs/design/m8-warmtelink-energy.md`
**Steps**
1. 新增可选 compose overlay:示例把宿主 `/dev/serial/by-id/<stable-id>` 映射为容器
`/dev/warmtelink`,以显式 serial GID/additional group 授权;服务保持非 root,默认 compose 无硬件
也能启动。不要写入用户真实设备 id。
2. 检查 Dockerfile/compose/build context 与新 Python/前端文件;deployment tests 覆盖默认 compose、
overlay 合并、device path、非 root、COPY source 存在。
3. 写运行文档:识别稳定 by-id、查 GID、启停 overlay、创建 source、discover/bind、质量含义、
reconnect/权限排障、backup/migration、合同录入、HA toggle 与安全回滚。明确不删除旧 config/data
4. 在备份数据库副本演练 Alembic 14→head 并对账;新空库迁移到 head。
5. 跑全量后端/前端/OpenAPI/codegen 闸门和真实 `docker build`;启动迁移后的 app,确认关键路由
不 500
6. 按 §12 人工 walkthrough 走完整链;通过后把 Roadmap/设计索引/M8 状态与 T01T20 Status
更新为完成。打 tag/push 仍需用户另行明确授权
1. 初始交付曾使用独立 Compose 配置文件;该形态已由 M8-R07 后续修订收敛为现有 base+dev Compose。示例通过
`WARMTELINK_DEVICE_PATH=/dev/serial/by-id/<stable-id>``WARMTELINK_SERIAL_GID=<serial-gid>` 直接映射容器
`/dev/warmtelink`,以显式 serial GID/additional group 授权;服务保持非 root。当前 base+dev Compose
明确要求这两个环境变量并固定声明 device 映射,不能作为无硬件启动形态使用。不要写入用户真实设备 id。
2. 检查 Dockerfile/compose/build context 与新 Python/前端文件;deployment tests 覆盖默认及 base+dev compose
合并、device path、pyserial `O_RDWR` 所需的 `rw`/无 `m`、非 root/非 privileged、migration 无 device、COPY
source 存在
3. 写运行文档:识别稳定 by-id、查 GID、在 `.env`(或等效部署环境)配置 stable by-id path 与 serial GID
由 Compose 固定映射串口;创建 source、discover/bind、质量含义、reconnect/权限排障、backup/migration、
合同录入、HA toggle 与安全回滚。明确这两个变量为必填项,不删除旧 config/data
4. `/tmp` 下构造的隔离合成历史数据库演练 Alembic 14→head 并对账;新空库迁移到 head。
不挂载、复制或打开运行中 production 的数据库、容器或 volume
5. 只用 mock/fake serial、合成 source/channel/binding/Meter/contract/cost/HA payload 和 `/tmp`
合成数据库,跑全量后端/前端/OpenAPI/codegen 闸门;演练 rev14→head 与空库→head。构建镜像后,
仅启动一个无 production volume/bind/device 的临时容器,使用容器内临时数据库迁移并确认关键路由不
500;结束时删除该容器、临时 image tag、`/tmp` 数据库和测试凭据。
6. 交付完整的 §12 九步人工 walkthrough 与可填写证据模板给用户;不得声称 agent 已执行真实串口、
真实 HA/MQTT 或生产资源操作。独立 Reviewer 可在自动化技术验收与文档/报告完整时判 `PASS`;随后
Orchestrator 才可把 Roadmap/设计索引/M8/T20 状态更新为完成并按仓库规则 autosquash。打 tag/push
仍需用户另行明确授权。
**Out of scope / 不要碰**
- 不把宿主真实 serial id、GID、equipment id、合同金额或数据库写进仓库。
@@ -1034,15 +1118,16 @@ T01T06 先把现有 DSMR 安全迁到统一 source/bindingT07T11 再接
- 不自动 push、force-push 或打 release tag。
**Acceptance criteria**
- [ ] 默认 compose 无设备可启动;overlay 以非 root 只读访问稳定映射路径,部署测试覆盖
- [ ] 空库/历史副本迁移对账通过,DSMR 正常数字不变,WarmteLink/thermal 全链可运行
- [ ] 文档足够让另一位 operator 从备份、部署、配置走到 HA 与回滚,不含真实 secret/identifier
- [ ] `pytest``ruff check .`、OpenAPI/codegen、全部前端闸门和真实 `docker build` 全绿。
- [ ] 人工 walkthrough 全部通过;Roadmap/M8 任务状态与现实一致
- [ ] **自动化技术验收**:默认 compose 与 synthetic base+dev 配置的结构检查通过;app 仍为非 root/非 privilegeddevice rule 为无 `m``rw`(仅满足 pyserial `O_RDWR` 打开),worker 业务只 read/closemigration 无 device
- [ ] **自动化技术验收**mock/fake serial 覆盖采集、断线与只读边界;合成 source/channel/binding/Meter、thermal/electricity contract/cost 与 HA payload 覆盖完整身份链和成本/HA 行为,绝不连接真实 serial、HA 或 MQTT
- [ ] **自动化技术验收**:仅在 `/tmp` 合成历史副本演练 rev14→head 并对账 DSMR 正常数字、source/binding、Meter、contract 和 cost;空库→head 及重复运行通过,绝不打开、复制、挂载或修改 production DB/config/container/volume
- [ ] **自动化技术验收**`pytest``ruff check .`、OpenAPI/codegen、全部前端闸门和 Docker build 全绿;临时 Docker 运行只使用无 volume/bind/device、无 production 挂载的容器及容器内临时数据库,关键路由不 500,结束后清理容器、image tag、`/tmp` 文件与测试凭据
- [ ] **交付后用户人工 walkthrough**:文档提供完整 §12 九步操作和脱敏证据模板,明确由用户在真实 serial/HA 的备份、可回滚部署上验收;agent/Reviewer 不伪称已执行,且该人工结果不是 Reviewer 技术 PASS 的前置条件
**Reviewer checklist**
- 必须亲自检查 Dockerfile `COPY`、compose merge 后的 user/group/device,不接受只看 unit tests。
- 抽查迁移对账和 rollback 文档,不允许任何自动化删除历史;独立跑完整闸门与 walkthrough
- 独立运行 fake serial、合成 source/channel/binding/Meter/contract/cost/HA payload、`/tmp` rev14→head 和空库→head 的技术验收;抽查迁移对账和 rollback 文档,不允许任何自动化删除历史或接触真实 serial/HA/MQTT/production 资源
- 独立运行完整闸门及无 volume/bind/device 的临时 Docker 容器,核对清理;文档与最终报告必须完整交付 §12 九步给用户且不伪称已执行。上述自动化和交付完整时,Reviewer 可判 `PASS`;真实人工结果由用户交付后验收。
## 11. 每张任务卡的校验矩阵
@@ -1076,13 +1161,25 @@ npm run build
`tests/test_deployment.py::test_dockerfile_copy_sources_exist` 仍绿。Implementer/Reviewer 简报、独立重跑与
fixup/autosquash 流程以 [`docs/design/README.md`](./README.md) 和仓库 `AGENTS.md` 为准。
## 12. M8 最终人工 Walkthrough
M8-T20 的自动化技术验收进一步受以下隔离边界约束:只可使用 mock/fake serial、合成
source/channel/binding/Meter/contract/cost/HA payload、pytest `tmp_path``/tmp` 明确命名的合成
SQLiteDocker 只可运行无 volume/bind/device、无生产挂载的临时容器。不得打开真实 serial、连接真实
HA/MQTT,或访问 production DB/config/container/volume。真实环境观察仅属于 §12 的交付后用户人工
walkthrough。
T20 收尾时必须在备份数据库/可回滚部署上完成以下人工验证:
## 12. 交付后用户人工 Walkthrough
1. 不启用 overlay 启动默认 stack,确认 app、现有 DSMR、Modbus、电价/电费和前端均无回归。
2. 以 stable by-id + serial GID 启用 overlay;容器保持非 root`/dev/warmtelink` 可读,代码没有
write 操作
T20 的 agent/Reviewer 自动化技术验收完成后,最终报告必须将以下九步完整交给用户。用户在备份数据库、
可回滚部署、真实 serial 设备和真实 HA 环境中自行执行并验收;这不是 Reviewer 技术 PASS、T20 状态更新或
M8 收尾的前置条件。agent 不得执行、记录为已执行,或以 mock/临时容器结果替代这些人工观察
1. 在 `.env`(或等效部署环境)设置 `WARMTELINK_DEVICE_PATH=/dev/serial/by-id/<stable-id>`
`WARMTELINK_SERIAL_GID=<serial-gid>`,启动当前 base+dev Compose stack,确认 app、现有 DSMR、Modbus、
电价/电费和前端均无回归;不得把缺少这两个值的配置当作无硬件启动模式。
2. 确认当前 base+dev Compose 的固定映射:容器保持非 root/非 privileged`/dev/warmtelink` 的 device rule
为 pyserial POSIX `O_RDWR` 所需的 `rw`(没有 `m`),并记录可成功打开。该系统权限不等于业务写入:代码只
read/close,绝不调用 write。若要运行 Pre-M8 probe,必须先停止 app,保证串口只有一个 owner,完成后再恢复
当前 `.env` 直配部署。
3. UI 创建 WarmteLink sourcediscover 后看到两个 channel、正确 unit/device type、质量
`unverifiable`;原始 equipment id 在日志、DB、API、UI 均不可见。
4. 分别创建/选择 heating 与 hot_water Meter 并确认 binding;观察 latest 约 10 秒更新、history
@@ -1098,12 +1195,49 @@ T20 收尾时必须在备份数据库/可回滚部署上完成以下人工验证
9. 重启 app/container,确认 source/worker/bindings/history/contract/cost 恢复;再按 runbook 回到默认
compose,历史数据仍完整。
### 用户验收证据模板(交付后填写)
用户须在备份数据库和可回滚部署上逐项记录:日期、隔离部署标识、§12 项号、预期与实际观察、脱敏日志/截图
引用、回滚结果。记录不得包含 stable device id、GID、secret、数据库路径或合同金额。交付时各项可为“未执行”;
只有用户填写并自行验收后才代表真实环境结果,且不得倒推或伪造 agent/Reviewer 已完成的观察。
## 13. Milestone Definition of Done
- [ ] M8-T01M8-T20 均由独立 Reviewer 判 `PASS`,任务 Status 为 `done`fixup 已按仓库规则收口。
- [ ] 统一 Source→Channel→Binding→Meter 链同时承载 DSMR 和 WarmteLinkDSMR 正常行为/数字兼容。
- [ ] WarmteLink 双 channel 只读采集、质量接纳、latest/history、重连、隐私与部署全部符合 D1~D8。
- [ ] electricity 成本可审计到 bindingthermal 合同、15 分钟账本、固定费、API/UI/HA 符合 D9D14。
- [ ] 历史升级对账、空库迁移、后端/前端/OpenAPI/codegen、Docker build 和 §12 walkthrough 全绿
- [ ] 文档、Roadmap 和 milestone 状态反映真实完成度;没有删除用户数据、没有真实 secret/设备身份,
- [x] M8-T01M8-T20 均由独立 Reviewer 依据各自的自动化技术验收`PASS`,任务 Status 为 `done`fixup 已按仓库规则收口。
- [x] 统一 Source→Channel→Binding→Meter 链同时承载 DSMR 和 WarmteLinkDSMR 正常行为/数字兼容。
- [x] WarmteLink 双 channel 只读采集、质量接纳、latest/history、重连、隐私与部署全部符合 D1~D8。
- [x] electricity 成本可审计到 bindingthermal 合同、15 分钟账本、固定费、API/UI/HA 符合 D9D14。
- [x] 历史升级对账、空库迁移、后端/前端/OpenAPI/codegen、Docker build 与无生产挂载的临时 Docker 技术验收全绿;其输入只可为 mock、模拟数据和 `/tmp` 合成数据库
- [x] 最终报告完整交付 §12 九步及用户证据模板;真实 serial/HA walkthrough 由用户在交付后自行验收,不能在报告中伪称 agent 已执行。
- [x] 文档、Roadmap 和 milestone 状态反映真实完成度;没有删除用户数据、没有真实 secret/设备身份,
没有未经授权的 push/tag。
## 14. Post-M8 lifecycle repairM8-R08R10
M8 交付后的 Meter lifecycle 修复链记录在本地 `review-notes/M8-meter-lifecycle-repair-plan.md`
它不新增 ORM / **数据库** schema 或 Alembic revision,不做启动自动修复、一次性数据脚本或历史删除;R08 虽然
更新了 API/Pydantic schema 及 OpenAPI/codegen,但没有变更 ORM 或数据库 schema。已有的 stranded binding 只能
由用户通过 UI 的原子 Transfer 恢复。
| 修复卡 | 当前状态 | 独立 review | 实现 / review 简报 |
| --- | --- | --- | --- |
| M8-R08Close、Unbind、原子 Transfer 与 API 契约 | done | PASS(第 6 轮) | `M8-R08-impl-*``M8-R08-review-6.md` |
| M8-R09Meter 管理 UI | done | PASS(第 12 轮) | `M8-R09-impl-*``M8-R09-review-12.md` |
| M8-R10:文档、最终报告与全量收尾 | done | PASS(第 2 轮) | `M8-R10-impl-1.md``M8-R10-review-2.md` |
R08 的 base commit 为 `c40ee65`(其自动化链包含 5 个 `fixup!`,最终 review 范围止于 `5a50b07`);R09
的 base commit 为 `5c08825`(包含 10 个 `fixup!`,最终 review 范围止于 `c4f4ff7`)。R10 已由第 2 轮
冷启动独立 Reviewer 判定 PASS 后标记 `done`;实现者的本地闸门结果没有被当作人工或 review 验收。
R08/R09 规定的用户操作为:Close active Meter、Unbind open binding、以单请求 same-Meter Transfer 切 source
以及把 closed previous Meter 上的 stranded DSMR binding 恢复到 active Meter。`meter_swap` 在唯一兼容 open
binding 时可自动 handoff;歧义一律 fail closed。恢复可选择晚于新 Meter 起点的时间并留下可见 gap,所有
lifecycle 写入与相应成本重算同事务,失败整笔回滚,HA discovery 只在 commit 后 best-effort republish。
stranded DSMR walkthrough 只能使用开发库中已经存在、由用户报告的 stranded row;不得通过 SQL、API、脚本、
migration 或直接改数据库制造前置状态。若隔离开发库没有这类历史行,应将该项记录为 `N/A/blocked`。Close、
Unbind、Transfer、Recovery 与 handoff 都会改写 Meter/binding 状态,walkthrough 必须将它们视为独立场景,使用
彼此独立且满足前置条件的 Meter/binding(或先恢复各自前置状态),不得按一个会破坏后续前置条件的连续故事操作。
按用户指定顺序,R10 必须先获得独立 Reviewer PASS;之后才由 Orchestrator 对未 push 的 fixup 执行 autosquash
并回填最终交付 SHA 与 commit 数。在此之前,报告只可陈述 pre-autosquash 审计状态,不得伪称历史已干净收口。
+11
View File
@@ -49,3 +49,14 @@
- 更复杂的 backoff 策略
这一轮重点是先把 app -> Home Assistant 的出站契约和可复用结构迁进来。
## Energy、Source 与热力实体
Expose 框架还可以把已勾选的 Energy 实体通过 MQTT Home Assistant Discovery 发布;开关位于应用 Config 页的 HA Expose 面板,默认均为关闭。M8 增加了 source online、按 Meter UUID 锚定的累计量/today,以及 heating、hot-water-heating、water、water-tax、fixed、all-in total/today 等 thermal 实体。
- source 和 Meter identity 不依赖可变 label;换表会产生新 Meter UUID identity。
- thermal 组合成本 identity 由当前 heating/hot_water Meter UUID 的有序组合锚定,任一换表都会产生新 identity,避免不同累计域拼接。
- availability、unit、device/state class 与 today reset 由 provider 声明;operator 应在 HA 中核对,而不应假设同名实体可跨换表连续。
- 关闭 toggle 后 retained discovery 会被清理;关闭暴露不删除 source、Meter、合同、读数或成本历史。
WarmteLink 的 P1 质量会原样保留为 `unverifiable`(若适用),不因发布到 HA 而提升为 `valid`。不要将 raw telegram、equipment id、串口路径、合同金额或 API secret 作为 HA entity/state/attribute 发布。
+54 -1
View File
@@ -104,9 +104,62 @@ MQTT 上报的累计成本实体(`import_cost_total` / `export_revenue_total`
- 若是全新空库,初始表不创建(无历史数据)。
- 回填幂等:重复跑迁移不会创建多条初始表;回填后对账(非降级周期 `meter_id IS NULL` 数必须为 0)。
## M8Source binding 与热力 Meter
M8 将“协议连接”和物理 Meter 分开:`MeterSource` 产生稳定 channel`MeterSourceBinding``[started_at, ended_at)` 内把 channel 接到一个 Meter epoch。DSMR electricity 与 WarmteLink heating `GJ`、hot_water `m³` 都使用这条链。
- 正常成本周期的两端必须解析到**同一** Meter 和 binding;跨 epoch、跨 binding、无 binding、读数陈旧或质量不可接纳时一律 degraded,不跨累计域相减。
- source switch 是关闭/新建 binding,不创建假 Meter swap;实际换表才创建新的 Meter epoch。对于 thermalheating 与 hot_water 独立换表,热力组合 HA identity 因任一 UUID 改变而更新。
- `meter_cost_period` 为 heating 与 hot_water 保存 15 分钟 Decimal quantity/cost、binding、合同版本、price snapshot 和 degraded reason。固定费是合同级日汇总,只计一次。
部署与回滚串口 source 参见 [`warmtelink-energy.md`](./warmtelink-energy.md);保留旧 source/binding/history 可使审计和重算可重复,不能通过删除历史来“修复”边界周期。
## 生命周期操作与 stranded binding 恢复
M8-R08/R09 为 Meter 与 binding 增加了显式的生命周期操作。所有时间都由前端按本地日期时间输入,再按既有
Principle-A 约定交给后端;未来时间会被拒绝。
- **Close Meter**:只能关闭 active Meter。该 Meter 与它的所有 open binding 在同一个 `ended_at`、同一
事务中关闭;关闭后该 commodity 没有 active Meter。
- **Unbind**:可对任意 open binding 执行,只写 binding 的 `ended_at`,绝不删除历史。已关闭 Meter 上残留的
open binding 也可在 UI 中解绑,默认关闭时间为该 Meter 的 `ended_at`
- **Transfer**:以单个原子请求切换 source channel,不采用浏览器端“先关闭再新建”的两步操作。同一 Meter
的 source switch 在同一 `effective_at` 关闭旧 binding、开启新 binding。失败时 binding、Meter 与受影响的
成本重算全部回滚,不会显示或留下部分成功。
新建或更新 binding 必须完整落在所属 Meter epoch 内:
`meter.started_at <= binding.started_at < binding.ended_at <= meter.ended_at`Meter 已关闭时);
open-ended binding 只允许属于 active Meter。Close、Unbind、Transfer 与声明 Meter 都从最早受影响边界重算到
当前时间;electricity 与 heating/hot-water 分别使用对应的成本引擎,重算失败时整笔生命周期变更回滚。数据库
提交成功后才会 best-effort 重新发布 HA discovery;发布失败不会伪装成持久化失败。
### 换表自动交接与人工恢复
声明 `reason=meter_swap` 的新 Meter 时,若未选择 channel 且旧 active Meter 恰好有一条唯一、单位兼容的
open binding,系统会在新 Meter 起点自动把该 channel 原子交接过去。存在多个候选或时间线歧义时,声明会
fail closed 并整体回滚。不是这种唯一自动交接的声明,也会在新边界关闭旧 Meter 的 open binding,避免产生新的
“closed Meter + open binding”。
旧版本或历史异常可能已经留下 stranded binding:前一块 closed Meter 仍有 open DSMR/WarmteLink binding
而当前 active Meter 没有 binding。不要修改数据库、不要跑 migration,也不会在启动时自动修复。请在
**Energy → Meters** 使用该 stranded binding 的 **Recover binding** 操作:选择当前 active Meter 与兼容
channel,提交一次 Transfer。来源必须是同 commodity、唯一且紧邻的前一块 closed Meter;旧 binding 固定在旧
Meter 的 `ended_at` 结束,新 binding 从选择的 `effective_at` 开始。默认是新 Meter 的 `started_at`;选择更晚
时间是允许的,但 UI 会提示这段明确的 unbound gap。非前序 Meter、单位不匹配、channel 在无关区间被占用或歧义
都会被拒绝,不会误关历史。
人工 walkthrough 只能使用开发库里**已经存在且由用户报告的** stranded row。不得为了演示通过 SQL、API、
脚本、migration 或直接改数据库制造这种历史异常;隔离开发库中没有该 row 时,记录此项为 `N/A/blocked` 即可。
Close、Unbind、same-Meter Transfer、Recover binding 与 meter-swap handoff 都会改变 Meter 或 binding 状态,
因此应作为彼此独立的验收场景:每个场景使用各自满足前置条件的 Meter/binding,或在执行前恢复独立前置状态,
不能把它们串成会互相破坏前提的单一故事。
上述恢复不新增 Alembic migration、不执行启动修复,也不删除 Meter、binding、reading 或成本历史;它只把用户
确认的时间线修正为可审计的闭区间。
## 非目标(本里程碑不做)
- Gas / 区域供暖的计费(`commodity != "electricity"` strategy
- Gas 计费 strategy。
- "家庭(home)"分组实体;多合同时间线积分。
- 自动识别换表(DSMR 帧无电表序列号,无法自动识别,靠用户显式声明)。
- `last_reset` 信号(消除 HA 长期统计 blip)。
+16 -12
View File
@@ -2,7 +2,7 @@
本文档记录 `home-automation``v1.0.3` 之后的下一阶段规划。这一阶段不是小修补,而是几次较大的结构性改动:单库化、前端重写、以及远期的移动端试水。
> 每个里程碑的设计与**可执行原子任务**展开在 [`docs/design/`](./design/README.md)M1 [`m1-db-consolidation.md`](./design/m1-db-consolidation.md)、M2 [`m2-frontend-v2.md`](./design/m2-frontend-v2.md)、M3 [`m3-token-mobile.md`](./design/m3-token-mobile.md)、M4 [`m4-login-hardening.md`](./design/m4-login-hardening.md)、M5 [`m5-iot-energy.md`](./design/m5-iot-energy.md)、M6 [`m6-tibber-dynamic-energy.md`](./design/m6-tibber-dynamic-energy.md)、M7 [`m7-meter-epochs-archival.md`](./design/m7-meter-epochs-archival.md)、Pre-M8 [`pre-m8-warmtelink-p1-poc.md`](./design/pre-m8-warmtelink-p1-poc.md)、M8 [`m8-warmtelink-energy.md`](./design/m8-warmtelink-energy.md)。Pre-M8 已完成;M8 Planning 也已完成并拆成 M8-T01~M8-T20,等待后续由编排器按依赖逐张实现
> 每个里程碑的设计与**可执行原子任务**展开在 [`docs/design/`](./design/README.md)M1 [`m1-db-consolidation.md`](./design/m1-db-consolidation.md)、M2 [`m2-frontend-v2.md`](./design/m2-frontend-v2.md)、M3 [`m3-token-mobile.md`](./design/m3-token-mobile.md)、M4 [`m4-login-hardening.md`](./design/m4-login-hardening.md)、M5 [`m5-iot-energy.md`](./design/m5-iot-energy.md)、M6 [`m6-tibber-dynamic-energy.md`](./design/m6-tibber-dynamic-energy.md)、M7 [`m7-meter-epochs-archival.md`](./design/m7-meter-epochs-archival.md)、Pre-M8 [`pre-m8-warmtelink-p1-poc.md`](./design/pre-m8-warmtelink-p1-poc.md)、M8 [`m8-warmtelink-energy.md`](./design/m8-warmtelink-energy.md)。Pre-M8 与 M8-T01~T20 的自动化技术验收均已完成;M8 交付后用户人工 walkthrough 待验收
## 当前基线(v1.0.3
@@ -41,7 +41,7 @@
| **M6** ✅ | 通用电价层 + DSMR 接入 + 实时电费计算 | 通用电价层(manual/tibber profile + 合同版本)+ DSMR 实时电表接入 + 每 15min 寄存器差×价计量电费(不可变快照)+ 日/月/年汇总 + 反哺 HA Energy + 前端合同/价格/费用视图 |
| **M7** ✅ | 电表生命周期 / 换表归档 | 引入 Meter epoch,计费永不跨表算 delta,跨表/无表/异常 delta 一律降级,累计按当前表归零,追溯换表可重算,Meter CRUD API + 前端管理 UI |
| **Pre-M8** ✅ | WarmteLink P1 真机概念验证 | 正式只读 CLI 长测通过;人工开启供暖后累计量 `0.017 → 0.018 GJ` 且与物理表一致,所有 frame 的 CRC 状态仍为 `unverifiable` |
| **M8** 📋 | WarmteLink P1、多数据源 Meter 与热力计费 | Planning 已完成:统一 Source/Channel/Binding、WarmteLink 双 channel、DSMR 迁移、thermal 合同/成本、HA/UI/部署M8-T01M8-T20 待实现 |
| **M8** | WarmteLink P1、多数据源 Meter 与热力计费 | M8-T01~T20 自动化技术验收完成:Source/Channel/Binding、WarmteLink、DSMR 迁移、thermal 合同/成本、HA/UI部署与文档闭环;真实 serial/HA walkthrough 由用户交付后验收 |
| **M3** | 开放与移动端(远期试水) | token 鉴权 + React Native 移动端 |
排序原则:**先清地基,再在干净结构上盖楼。** M2 的新 API 和 React 必须建立在合并后的单库之上;M4 是公网安全加固,在 M5 IoT 集成之前先堵住裸密码这个洞;M5 在安全基座就绪后再做 IoT 接入。
@@ -282,7 +282,7 @@ httpx / paho-mqtt / pyyaml / apscheduler 均为 M5 已有依赖,M6 复用,
---
## M8 — WarmteLink P1、多数据源 Meter 与热力计费(📋 Planning 已完成,待实现
## M8 — WarmteLink P1、多数据源 Meter 与热力计费(✅ 自动化技术验收已完成
### 目标
@@ -298,8 +298,10 @@ DSMR MQTT 与新的 WarmteLink serial source;从一个 WarmteLink source 只
历史和电费安全回填,正常电费数字保持不变,跨 source/binding 周期降级。
- WarmteLink `unverifiable` frame 需要连续双帧确认后接纳,质量标签不漂白;约 10 秒更新 latest、
每分钟保存 history,不保存 raw telegram 或 raw equipment identifier。
- 串口 worker 使用 `115200 7N1`、只读、短 DB session、可热更新和 `1…60s` 退避重连;以可选
compose overlay、稳定 by-id、serial GID 和非 root 容器部署。
- 串口 worker 使用 `115200 7N1`、只读、短 DB session、可热更新和 `1…60s` 退避重连;现有
production base 与 base+dev Compose 通过 `.env` 必填的 `WARMTELINK_DEVICE_PATH`stable by-id
`WARMTELINK_SERIAL_GID` 直配,将设备固定映射为 `/dev/warmtelink:rw`,保持非 root 最小权限;
同一物理串口只能有一个 owner。
- 合同增加 electricity/thermal scope;一份 district-heating 合同覆盖 heating GJ 与 hot-water m³,
生成按 commodity 的 15 分钟成本和 contract-level 每日固定费。仓库不硬编码真实 tariff。
- UI 分成 Sources / Modbus Devices / Meters,并让 Contracts / Prices / Costs 按 scope 切换;新增
@@ -307,14 +309,16 @@ DSMR MQTT 与新的 WarmteLink serial source;从一个 WarmteLink source 只
### 原子实施链
- **M8-T01T06**:统一 source/channel/binding schema、DSMR 历史/runtime/电费迁移和管理 API。
- **M8-T07T11**:共享 P1 parser、WarmteLink 标量存储、质量接纳、serial worker、发现与历史 API。
- **M8-T12T16**:合同 scope、district-heating profile、thermal cost 账本/引擎/API。
- **M8-T17T20**HA、Sources/Meters UIscope-aware 计费 UIcompose/文档/全链收尾。
- **M8-T01T06**已完成统一 source/channel/binding schema、DSMR 历史/runtime/电费迁移和管理 API。
- **M8-T07T11**已完成共享 P1 parser、WarmteLink 标量存储、质量接纳、serial worker、发现与历史 API。
- **M8-T12T16**已完成合同 scope、district-heating profile、thermal cost 账本/引擎/API。
- **M8-T17T19**已完成 HA、Sources/Meters UIscope-aware 计费 UI**M8-T20** 已完成 compose文档与隔离自动化技术收尾。
完成判据不仅是单元闸门全绿,还包括历史副本迁移对账、OpenAPI/codegen、全部前端闸门、真实
`docker build`非 root 串口部署和真机端到端 walkthrough。任何任务都不得删除旧数据库、历史
读数、旧 config 行或 volumepush/tag 仍需用户单独授权。
完成判据不仅是单元闸门全绿,还包括以 mock/fake、合成数据库和隔离 Docker 完成的历史迁移对账、
OpenAPI/codegen、全部前端闸门、真实 `docker build`非 root 串口部署技术验收;并须交付完整的
九步用户 walkthrough 和证据模板。真实 serial/HA 的观察由用户在交付后自行验收,不是 agent/Reviewer
技术 PASS、T20 状态更新或 M8 autosquash/收尾的前置条件,也不得写成已执行。任何任务都不得删除旧
数据库、历史读数、旧 config 行或 volumepush/tag 仍需用户单独授权。
> 完整架构、HTTP 契约、质量/计费规则、依赖图与 M8-T01~M8-T20 任务卡:
> [`docs/design/m8-warmtelink-energy.md`](./design/m8-warmtelink-energy.md)
+110
View File
@@ -0,0 +1,110 @@
# WarmteLink、数据源与热力计费运维手册
本手册说明如何在不改变既有 DSMR、Modbus 和 electricity 功能的前提下,部署只读的 WarmteLink P1 采集、绑定热量/生活热水 Meter、配置热力合同,并按需暴露给 Home Assistant。所有示例均使用占位符;不要把设备标识、GID、数据库路径、token 或真实合同金额提交到仓库。
## 安全边界与开始前备份
- 应用容器继续使用基础 `docker-compose.yml` 中的非 root `user: "1000:1000"`;基础文件和 dev 合并配置都不设置 `privileged` 或额外 capability。
- Docker device cgroup 必须以 `rw` 映射,容器内固定为 `/dev/warmtelink`:这是 pyserial 3.5 在 POSIX 上以 `O_RDWR` 打开串口所需的最小系统权限,并不表示业务可写。规则绝不包含 `m`,且不授予 root、`privileged` 或额外 capability。WarmteLink worker 仍只调用 serial `read` / `close`,绝不调用 `write` 或发送写命令。
- 不删除或覆盖 `app_config``app.db`、旧数据库、Docker volume 或既有 source。禁用/解绑/回滚配置不是删除历史的替代方式。
- 维护前停止写入窗口,使用宿主机的备份流程复制 `./data/app.db` 到受保护的备份位置;确认备份可用后才运行 migration。不要把生产库复制到开发机或用于测试。
## 识别稳定串口并配置 Compose
在宿主机(不是容器)找出稳定 symlink;不要使用会在重启后变化的 `/dev/ttyUSB*` 名称:
```bash
ls -l /dev/serial/by-id/
stable_path=/dev/serial/by-id/<stable-by-id-name>
stat -c '%g %n' "$stable_path"
```
记录输出的数字 GID,而不是猜测 `dialout` 的数值。确认 path 指向预期的字符设备后,在部署机本地 `.env`(不提交)设置:
```bash
WARMTELINK_DEVICE_PATH=/dev/serial/by-id/<stable-by-id-name>
WARMTELINK_SERIAL_GID=<host-serial-gid>
```
production 使用基础 composelocal dev 使用 base 与 dev 合并文件。两种环境都会从本地 `.env` 读取这两个变量:
```bash
docker compose -f docker-compose.yml up -d
docker compose -f docker-compose.yml -f docker-compose.dev.yml up -d --build
```
启动前可用相同文件组合运行 `docker compose ... config`。结果的 `app` 必须仍显示 `user: "1000:1000"`,并只出现从 stable by-id path 到 `/dev/warmtelink``rw` device mapping 与 `group_add` GIDmigration 不得有 device 或 group。不得出现 `privileged`、root user 或 `m` device permission。`rw` 仅满足 pyserial 的 POSIX `O_RDWR` 打开,不改变 worker 的只读业务行为。不得把设备路径或 GID 写入仓库的 `.env.example` 或文档。
同一物理串口在任意时刻只能有一个 owner。运行 Pre-M8 `p1_probe.py` 前,必须先停止 app(包括 dev stack),并在 probe 结束后再启动 app;不要让 probe 与 worker 同时打开该串口。
## Migration 与 source 配置
先在维护窗口运行 migration;它只升级 schema,绝不删除历史表或配置:
```bash
docker compose -f docker-compose.yml run --rm migration
```
登录 Energy 页面,在 **Sources** 创建 `warmtelink_serial` source
| 字段 | 值 |
| --- | --- |
| Path | `/dev/warmtelink` |
| Baud rate | `115200` |
| Data bits / parity / stop bits | `7` / `N` / `1` |
| Enabled | 先关闭,保存并检查配置后再开启 |
这些参数是固定 P1 profile。不要录入 telegram 或 equipment id;应用不会把原始 frame、设备身份或 serial 设置以外的协议标识保存到 source 配置中。开启后选择 **Discover**。成功发现后应有两个只读 channel:heating`GJ`)和 hot water`m³`);受 P1 CRC 限制,正常可接纳的质量可能显示为 `unverifiable`,这不是被错误提升为 `valid`
**Meters** 分别创建或选择 `heating``hot_water` Meter,并各自选择与单位相符的 channel 创建 binding。source switch 只关闭旧 binding、在相同 Meter 上创建新 binding,不是换表;真正的物理换表才使用 Meter swap。跨 binding 或 Meter 边界的 15 分钟成本周期会明确标为 degraded,不能相减伪造成本。
## 运行检查与排障
启用 source 后,latest 通常约每 10 秒更新,持久化 history 最多每分钟一条。检查 source 状态、channel latest/quality 和绑定时间线,而不是从日志中寻找原始 telegram。
| 现象 | 安全排查 |
| --- | --- |
| `offline` 或无法 open serial | 核对 stable symlink 是否仍存在、`stat` 的 GID 是否等于 `WARMTELINK_SERIAL_GID`,再用 `docker compose ... config` 检查 non-root `rw`(非 `r`、非 `m`device rule;不要用 root/privileged 绕过权限。 |
| 无 channels / discover 超时 | 确认 source 启用、`/dev/warmtelink` path 和固定 `115200 7N1`;检查电缆供电后重试 Discover。 |
| 短暂拔线 | worker 标记 offline 并以退避重连;插回后应恢复。检查 history 的 `(channel, recorded_at)` 唯一性,不能手工补重复行。 |
| 成本 degraded | 查两端读数 freshness120 秒)、quality、Meter epoch 与 binding;不要通过修改累计值清除 degraded。 |
若需要停采集,在 UI 禁用该 source,确认 worker 关闭串口后再维护电缆。删除有 channel、binding 或历史的 source 会被 API 拒绝;保留记录以保证审计和成本重算。
## 热力合同、成本与 Home Assistant
**Contracts** 选择 `Thermal` scope,创建 `district_heating` 合同及版本。费率由 operator 按合同人工录入,字段为 heatingEUR/GJ)、hot-water heating / water / taxEUR/m³)与五个年固定费字段;仓库不含任何真实默认金额。thermal 和 electricity 各可有一个 active 合同,彼此不互斥。
成本页的 15 分钟 ledger 分开显示 heating 与 hot-water 三项 variable breakdownfixed 费只在合同级 summary 按本地自然日计提一次,all-in = variable + fixed。用显式 recompute 来验证测试时间窗时,应手算并核对 Decimal 金额,保留原有 electricity 合同和数字不变。
在 Config 的 HA Expose 中只开启需要的 source、Meter 与 thermal entities。核对 unit、state class、availability、today reset 和换表后 identity;关闭 toggle 后应用会清理 retained discovery。不要把 source secret、设备 identity 或合同金额放进 HA entity 名称、日志或截图。
## 安全回滚
1. 在 UI 禁用 WarmteLink source,确认状态离线且 worker 已停止;保留 channels、bindings、history、contracts 与成本账本。
2. 停止 app 后,在 UI 保持 source 禁用;这不会删除 `./data`、数据库、配置或 volumes。需要恢复 WarmteLink 时,确认本地 `.env` 的 stable by-id/GID 后再重新启用 source。
3. 确认 DSMR、Modbus、电价、既有 electricity 成本和前端正常。
4. schema migration 不应以 production downgrade 回滚;只有经过验证的备份恢复流程才处理灾难恢复,且必须由 operator 在隔离维护窗口执行。
## 上线验收清单
- production 与 base+dev Compose 都从本地 `.env` 获取 serial path/GIDapp/migration 均为非 root,只有 app 有 `/dev/warmtelink:rw` 和 serial GID,绝无 `m`、root 或 privileged。该 `rw` 仅为 pyserial 的 `O_RDWR` 打开,worker 业务仍只读,且没有真实设备/GID 被记录在仓库。
- 下列项目是交付后由用户在备份数据库、可回滚部署、真实 serial 设备和真实 HA 环境执行的人工验收;自动化技术验收不能替代这些观察,也不得把它们伪称为已执行。
### 用户人工验收记录(交付后填写)
本模板在交付后由用户填写;它不描述当前宿主环境,也不代表任何项目已通过。每项均填写日期、隔离部署
标识、§12 项号、预期观察、实际观察、脱敏日志/截图引用、回滚结果和结果状态。不得记录 stable device id、
GID、secret、数据库路径或合同金额;交付时所有项目默认均为“未执行”,不得填造真实环境结果。
| §12 项号 | 日期 | 隔离部署标识 | 预期观察 | 实际观察 | 脱敏日志/截图引用 | 回滚结果 | 结果状态 |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 1. 默认 stack 回归 | 待填写 | 待填写 | DSMR、Modbus、电价/电费、前端均无回归 | 待填写 | 待填写 | 待填写 | 未执行 |
| 2. stable by-id 与权限 | 待填写 | 待填写 | 非 root、非 privileged、`rw``m`pyserial 可打开,worker 无 `write` | 待填写 | 待填写 | 待填写 | 未执行 |
| 3. Discover 与隐私 | 待填写 | 待填写 | 两 channel/unit/quality;日志、DB、API、UI 均完成脱敏检查 | 待填写 | 待填写 | 待填写 | 未执行 |
| 4. history/拔插重连 | 待填写 | 待填写 | latest、分钟 history、拔插恢复且无重复记录 | 待填写 | 待填写 | 待填写 | 未执行 |
| 5. source switch / Meter swap | 待填写 | 待填写 | 两条时间线正确、边界 degraded、HA identity 按设计变化 | 待填写 | 待填写 | 待填写 | 未执行 |
| 6. thermal 合同/成本 | 待填写 | 待填写 | 脱敏测试费率手算一致;15 分钟与 01:05 fixed 正确 | 待填写 | 待填写 | 待填写 | 未执行 |
| 7. 双 active scope | 待填写 | 待填写 | electricity 数字对照一致;UI/成本/HA 不串 scope | 待填写 | 待填写 | 待填写 | 未执行 |
| 8. HA toggles | 待填写 | 待填写 | unit、state class、availability、today、identity、retained cleanup 正确 | 待填写 | 待填写 | 待填写 | 未执行 |
| 9. 重启与默认 compose 回滚 | 待填写 | 待填写 | 历史恢复;回默认 compose 后数据完整 | 待填写 | 待填写 | 待填写 | 未执行 |
+1300 -5
View File
File diff suppressed because it is too large Load Diff
+63
View File
@@ -65,6 +65,11 @@ const PROFILES_RESPONSE = {
heffingskorting: { unit: 'EUR/year' },
},
},
{
kind: 'district_heating', label: 'District heating',
variable: { heating: { unit: 'EUR/GJ', default: 0 } },
standing: { delivery_set: { unit: 'EUR/year', default: 0 } },
},
],
}
@@ -79,6 +84,21 @@ const CREATED_CONTRACT = {
versions: [],
}
const D11_PROFILE_RESPONSE = {
profiles: [{
kind: 'district_heating', label: 'District heating',
variable: {
heating: { unit: 'EUR/GJ' }, hot_water_heating: { unit: 'EUR/m³' },
hot_water: { unit: 'EUR/m³' }, hot_water_tax: { unit: 'EUR/m³' },
},
standing: {
heating_network: { unit: 'EUR/year' }, metering: { unit: 'EUR/year' },
delivery_set: { unit: 'EUR/year' }, hot_water_network: { unit: 'EUR/year' },
other: { unit: 'EUR/year' },
},
}],
}
// ---------------------------------------------------------------------------
// Import component
// ---------------------------------------------------------------------------
@@ -174,6 +194,49 @@ describe('ContractForm', () => {
}, { timeout: 3000 })
})
it('limits a thermal create form to its compatible profile and posts its scope', async () => {
const user = userEvent.setup()
mockGet.mockResolvedValue({ data: PROFILES_RESPONSE })
mockPost.mockResolvedValue({ data: CREATED_CONTRACT })
renderWithProviders(<ContractForm scope="thermal" defaultKind="district_heating" onClose={vi.fn()} onSaved={vi.fn()} />)
await waitFor(() => expect(screen.getByTestId('contract-field-variable.heating')).toBeInTheDocument())
await user.type(screen.getByTestId('contract-name'), 'Heat')
await user.click(screen.getByTestId('contract-form-submit'))
await waitFor(() => expect(mockPost).toHaveBeenCalledWith('/api/energy/contracts', expect.objectContaining({ body: expect.objectContaining({ kind: 'district_heating', scope: 'thermal' }) })))
})
it('posts all nine D11 values as unrounded Decimal strings, including zero standing fees', async () => {
const user = userEvent.setup()
mockGet.mockResolvedValue({ data: D11_PROFILE_RESPONSE })
mockPost.mockResolvedValue({ data: CREATED_CONTRACT })
renderWithProviders(<ContractForm scope="thermal" defaultKind="district_heating" onClose={vi.fn()} onSaved={vi.fn()} />)
await waitFor(() => expect(screen.getByTestId('contract-field-variable.heating')).toBeInTheDocument())
await user.type(screen.getByTestId('contract-name'), 'Precise heat')
const values: Record<string, string> = {
'variable.heating': '20.123456789123456789', 'variable.hot_water_heating': '8.200000000000000001',
'variable.hot_water': '1.234567890123456789', 'variable.hot_water_tax': '0.456789012345678901',
'standing.heating_network': '0', 'standing.metering': '0', 'standing.delivery_set': '0',
'standing.hot_water_network': '0', 'standing.other': '0',
}
for (const [key, value] of Object.entries(values)) {
const input = screen.getByTestId(`contract-field-${key}`)
await user.clear(input)
await user.type(input, value)
}
await user.click(screen.getByTestId('contract-form-submit'))
await waitFor(() => expect(mockPost).toHaveBeenCalled())
const body = mockPost.mock.calls[0][1].body
expect(body).toMatchObject({ scope: 'thermal', values: {
variable: {
heating: values['variable.heating'], hot_water_heating: values['variable.hot_water_heating'],
hot_water: values['variable.hot_water'], hot_water_tax: values['variable.hot_water_tax'],
},
standing: {
heating_network: '0', metering: '0', delivery_set: '0', hot_water_network: '0', other: '0',
},
} })
})
it('calls POST /api/energy/contracts/{id}/versions in add-version mode', async () => {
const user = userEvent.setup()
+29 -17
View File
@@ -38,6 +38,8 @@ export interface ContractFormProps {
contractId?: number
/** Existing contract kind (for add-version mode or edit). */
defaultKind?: string
/** The list/create scope currently selected by the parent. */
scope?: 'electricity' | 'thermal'
onClose: () => void
onSaved: () => void
}
@@ -64,7 +66,7 @@ interface LeafField {
/** Dot-separated path within the section, e.g. "buy.normal" */
fieldPath: string
unit: string
defaultValue?: number
defaultValue?: number | string
}
function extractLeafFields(obj: Record<string, unknown>, prefix = ''): LeafField[] {
@@ -77,7 +79,9 @@ function extractLeafFields(obj: Record<string, unknown>, prefix = ''): LeafField
fields.push({
fieldPath: path,
unit: val.unit,
defaultValue: typeof val.default === 'number' ? val.default : undefined,
defaultValue: typeof val.default === 'number' || typeof val.default === 'string'
? val.default
: undefined,
})
} else {
fields.push(...extractLeafFields(val as Record<string, unknown>, path))
@@ -93,6 +97,7 @@ function extractLeafFields(obj: Record<string, unknown>, prefix = ''): LeafField
function buildNestedValues(
sectionFields: Record<string, LeafField[]>,
fieldValues: Record<string, number | string>,
decimalStrings: boolean,
): Record<string, unknown> {
const result: Record<string, unknown> = {}
@@ -100,7 +105,6 @@ function buildNestedValues(
const sectionObj: Record<string, unknown> = {}
for (const field of fields) {
const raw = fieldValues[`${section}.${field.fieldPath}`]
const numVal = typeof raw === 'number' ? raw : parseFloat(String(raw))
// Set nested path
const parts = field.fieldPath.split('.')
let current = sectionObj
@@ -108,7 +112,10 @@ function buildNestedValues(
if (!(parts[i] in current)) current[parts[i]] = {}
current = current[parts[i]] as Record<string, unknown>
}
current[parts[parts.length - 1]] = isNaN(numVal) ? 0 : numVal
// Values are Decimal JSON strings. Do not round-trip user money through JS Number.
current[parts[parts.length - 1]] = decimalStrings
? (raw === undefined || raw === '' ? '0' : String(raw))
: (Number.isFinite(Number(raw)) ? Number(raw) : 0)
}
result[section] = sectionObj
}
@@ -130,7 +137,7 @@ function formatLabel(path: string): string {
// Component
// ---------------------------------------------------------------------------
export function ContractForm({ contractId, defaultKind, onClose, onSaved }: ContractFormProps) {
export function ContractForm({ contractId, defaultKind, scope, onClose, onSaved }: ContractFormProps) {
const isAddVersion = contractId != null
// Profiles query
@@ -161,7 +168,9 @@ export function ContractForm({ contractId, defaultKind, onClose, onSaved }: Cont
const effectiveKind = isAddVersion ? (defaultKind ?? null) : selectedKind
// Build profile options from API response
const profiles = profilesQuery.data?.profiles ?? []
const profiles = (profilesQuery.data?.profiles ?? []).filter((p: Record<string, unknown>) =>
scope === 'thermal' ? p.kind === 'district_heating' : p.kind !== 'district_heating',
)
const profileOptions = profiles.map((p: Record<string, unknown>) => ({
value: p.kind as string,
label: (p.label as string | undefined) ?? (p.kind as string),
@@ -219,8 +228,7 @@ export function ContractForm({ contractId, defaultKind, onClose, onSaved }: Cont
cursor = (cursor as Record<string, unknown>)[part]
}
if (cursor != null && (typeof cursor === 'number' || typeof cursor === 'string')) {
const numVal = typeof cursor === 'number' ? cursor : parseFloat(String(cursor))
seeded[`${section}.${leaf.fieldPath}`] = isNaN(numVal) ? 0 : numVal
seeded[`${section}.${leaf.fieldPath}`] = effectiveKind === 'district_heating' ? String(cursor) : Number(cursor)
}
}
}
@@ -231,6 +239,7 @@ export function ContractForm({ contractId, defaultKind, onClose, onSaved }: Cont
contractDetailQuery.isError,
contractDetailQuery.data,
sectionFields,
effectiveKind,
])
// The effective field values: user edits override prefill; prefill is the base.
@@ -285,7 +294,7 @@ export function ContractForm({ contractId, defaultKind, onClose, onSaved }: Cont
return
}
const values = buildNestedValues(sectionFields, fieldValues)
const values = buildNestedValues(sectionFields, fieldValues, effectiveKind === 'district_heating')
try {
// Convert local date string to a naive local-midnight datetime string (no Z).
@@ -307,6 +316,7 @@ export function ContractForm({ contractId, defaultKind, onClose, onSaved }: Cont
const body = {
name: name.trim(),
kind: effectiveKind,
...(scope ? { scope } : {}),
currency,
values,
...(effectiveFromISO ? { effective_from: effectiveFromISO } : {}),
@@ -409,19 +419,21 @@ export function ContractForm({ contractId, defaultKind, onClose, onSaved }: Cont
</Title>
{fields.map((field) => {
const key = `${section}.${field.fieldPath}`
const raw = fieldValues[key]
const numVal = typeof raw === 'number' ? raw : parseFloat(String(raw))
return (
<NumberInput
return effectiveKind === 'district_heating' ? (
<TextInput
key={key}
label={formatLabel(field.fieldPath)}
description={field.unit}
value={isNaN(numVal) ? 0 : numVal}
onChange={(val) => handleFieldChange(key, val)}
decimalScale={6}
step={0.001}
inputMode="decimal"
value={String(fieldValues[key] ?? '0')}
onChange={(event) => handleFieldChange(key, event.currentTarget.value)}
data-testid={`contract-field-${key}`}
/>
) : (
<NumberInput key={key} label={formatLabel(field.fieldPath)} description={field.unit}
value={typeof fieldValues[key] === 'number' ? fieldValues[key] : Number(fieldValues[key] ?? 0)}
onChange={(value) => handleFieldChange(key, value)} decimalScale={6} step={0.001}
data-testid={`contract-field-${key}`} />
)
})}
</Stack>
@@ -66,6 +66,11 @@ const INACTIVE_CONTRACT = {
updated_at: '2026-06-02T00:00:00Z',
}
const ACTIVE_THERMAL_CONTRACT = {
id: 3, name: 'Active Heat Contract', kind: 'district_heating', active: true, currency: 'EUR',
created_at: '2026-06-03T00:00:00Z', updated_at: '2026-06-03T00:00:00Z',
}
const PROFILES_RESPONSE = {
profiles: [
{
@@ -202,4 +207,39 @@ describe('ContractManager', () => {
expect(screen.getByTestId('contract-form-modal')).toBeInTheDocument()
})
})
it('keeps the selector available while thermal data loads and requests each scope separately', async () => {
const user = userEvent.setup()
mockGet.mockImplementation((path: string, options?: { params?: { query?: { scope?: string } } }) => {
if (path === '/api/energy/contracts' && options?.params?.query?.scope === 'thermal') return new Promise(() => {})
if (path === '/api/energy/contracts') return Promise.resolve({ data: { items: [ACTIVE_CONTRACT], total: 1 } })
return Promise.resolve({ data: PROFILES_RESPONSE })
})
renderWithProviders(<ContractManager />)
await waitFor(() => expect(screen.getByTestId('contracts-scope-selector')).toBeInTheDocument())
await user.click(screen.getByText('Thermal'))
expect(screen.getByTestId('contracts-loading')).toBeInTheDocument()
expect(screen.getByTestId('contracts-scope-selector')).toBeInTheDocument()
await user.click(screen.getByText('Electricity'))
await waitFor(() => expect(screen.getByTestId('contracts-table')).toBeInTheDocument())
expect(mockGet).toHaveBeenCalledWith('/api/energy/contracts', { params: { query: { scope: 'thermal' } } })
})
it('keeps simultaneous active electricity and thermal contracts isolated across repeated switches', async () => {
const user = userEvent.setup()
mockGet.mockImplementation((_path: string, options?: { params?: { query?: { scope?: string } } }) =>
Promise.resolve({ data: { items: options?.params?.query?.scope === 'thermal'
? [ACTIVE_THERMAL_CONTRACT] : [ACTIVE_CONTRACT], total: 1 } }),
)
renderWithProviders(<ContractManager />)
await waitFor(() => expect(screen.getByText('My Active Contract')).toBeInTheDocument())
for (let i = 0; i < 2; i += 1) {
await user.click(screen.getByText('Thermal'))
await waitFor(() => expect(screen.getByText('Active Heat Contract')).toBeInTheDocument())
expect(screen.queryByText('My Active Contract')).not.toBeInTheDocument()
await user.click(screen.getByText('Electricity'))
await waitFor(() => expect(screen.getByText('My Active Contract')).toBeInTheDocument())
expect(screen.queryByText('Active Heat Contract')).not.toBeInTheDocument()
}
})
})
+34 -22
View File
@@ -10,6 +10,7 @@
*/
import { useState } from 'react'
import { useQuery } from '@tanstack/react-query'
import {
Table,
Button,
@@ -24,9 +25,9 @@ import {
Modal,
Accordion,
Code,
SegmentedControl,
} from '@mantine/core'
import {
useContracts,
useUpdateContract,
type ContractResponse,
type ContractDetailResponse,
@@ -244,7 +245,14 @@ function ContractTable({
// ---------------------------------------------------------------------------
export function ContractManager() {
const contractsQuery = useContracts()
const [scope, setScope] = useState<'electricity' | 'thermal'>('electricity')
const contractsQuery = useQuery({
queryKey: ['energy-contracts', scope],
queryFn: async () => {
const res = await apiClient.GET('/api/energy/contracts', { params: { query: { scope } } })
return res.data
},
})
const updateMutation = useUpdateContract()
const [showCreateForm, setShowCreateForm] = useState(false)
@@ -252,6 +260,13 @@ export function ContractManager() {
const [historyContract, setHistoryContract] = useState<ContractResponse | null>(null)
const [activatingId, setActivatingId] = useState<number | null>(null)
function handleScopeChange(nextScope: 'electricity' | 'thermal') {
setScope(nextScope)
setShowCreateForm(false)
setAddVersionContract(null)
setHistoryContract(null)
}
async function handleActivate(id: number) {
setActivatingId(id)
try {
@@ -265,44 +280,40 @@ export function ContractManager() {
// Render states
// ---------------------------------------------------------------------------
if (contractsQuery.isLoading) {
return (
<Center py="xl" data-testid="contracts-loading">
<Loader />
</Center>
)
}
if (contractsQuery.isError || !contractsQuery.data) {
return (
<Alert color="red" data-testid="contracts-load-error">
Failed to load contracts. Please refresh.
</Alert>
)
}
const contracts = contractsQuery.data.items
const contracts = contractsQuery.data?.items ?? []
return (
<Stack gap="lg" data-testid="contract-manager">
<Group justify="space-between" align="center">
<Text fw={500}>Energy Contracts</Text>
<Group gap="sm">
<Text fw={500}>Energy Contracts</Text>
<SegmentedControl
value={scope}
onChange={(value) => handleScopeChange(value as 'electricity' | 'thermal')}
data={[{ label: 'Electricity', value: 'electricity' }, { label: 'Thermal', value: 'thermal' }]}
data-testid="contracts-scope-selector"
/>
</Group>
<Button onClick={() => setShowCreateForm(true)} data-testid="contract-new-button">
New Contract
</Button>
</Group>
<ContractTable
{contractsQuery.isLoading && <Center py="xl" data-testid="contracts-loading"><Loader /></Center>}
{contractsQuery.isError && <Alert color="red" data-testid="contracts-load-error">Failed to load contracts. Please refresh.</Alert>}
{!contractsQuery.isLoading && !contractsQuery.isError && <ContractTable
contracts={contracts}
onActivate={handleActivate}
onAddVersion={(c) => setAddVersionContract(c)}
onViewHistory={(c) => setHistoryContract(c)}
activatingId={activatingId}
/>
/>}
{/* Create new contract */}
{showCreateForm && (
<ContractForm
defaultKind={scope === 'thermal' ? 'district_heating' : undefined}
scope={scope}
onClose={() => setShowCreateForm(false)}
onSaved={() => setShowCreateForm(false)}
/>
@@ -313,6 +324,7 @@ export function ContractManager() {
<ContractForm
contractId={addVersionContract.id}
defaultKind={addVersionContract.kind}
scope={scope}
onClose={() => setAddVersionContract(null)}
onSaved={() => setAddVersionContract(null)}
/>
+186
View File
@@ -73,6 +73,54 @@ const SUMMARY = {
total_payable: 12.5,
}
const THERMAL_SUMMARY = {
currency: 'EUR', heating: '1.10', hot_water_heating: '2.20', hot_water: '3.30',
hot_water_tax: '0.40', variable_subtotal: '7.00', fixed_subtotal: '0.50', all_in: '7.50',
period_count: 4, degraded_count: 2,
fixed_breakdown: { heating_network: '0.1', metering: '0.1', delivery_set: '0', hot_water_network: '0.1', other: '0.2' },
}
const THERMAL_VALUES = {
variable: {
heating: '20.123456789123456789', hot_water_heating: '8.200000000000000001',
hot_water: '1.234567890123456789', hot_water_tax: '0.456789012345678901',
},
standing: {
heating_network: '100.000000000000000001', metering: '0', delivery_set: '20.2',
hot_water_network: '30.3', other: '40.4',
},
}
const THERMAL_ROW = {
commodity: 'heating', period_start: '2026-06-22T10:00:00Z', period_end: '2026-06-22T10:15:00Z',
meter_id: 1, source_binding_id: 2, contract_version_id: 99, quantity: '1.2', cost: '0.123456789', currency: 'EUR',
cost_breakdown: { heating: '0.123456789' }, pricing_snapshot: THERMAL_VALUES,
quality: 'unverifiable', degraded: false, degraded_reason: null,
}
const THERMAL_ROW_OTHER_VERSION = {
...THERMAL_ROW, commodity: 'hot_water', period_start: '2026-06-22T10:15:00Z', period_end: '2026-06-22T10:30:00Z',
contract_version_id: 100, quantity: '2.3', cost: '4.339506172839506170',
cost_breakdown: { hot_water_heating: '1.2', hot_water: '2.8', hot_water_tax: '0.339506172839506170' },
pricing_snapshot: { ...THERMAL_VALUES, variable: { ...THERMAL_VALUES.variable, hot_water: '1.234567890123456789' } },
}
const THERMAL_DEGRADED_MISSING_CONTRACT = {
commodity: 'heating', period_start: '2026-06-22T10:30:00Z', period_end: '2026-06-22T10:45:00Z',
meter_id: 1, source_binding_id: 2, contract_version_id: null, quantity: '0', cost: '0', currency: 'EUR',
cost_breakdown: {}, pricing_snapshot: {}, quality: 'invalid', degraded: true, degraded_reason: 'missing_contract',
}
const THERMAL_DEGRADED_CROSS_EPOCH = {
commodity: 'hot_water', period_start: '2026-06-22T10:45:00Z', period_end: '2026-06-22T11:00:00Z',
meter_id: 2, source_binding_id: null, contract_version_id: null, quantity: '0', cost: '0', currency: 'EUR',
cost_breakdown: {}, pricing_snapshot: {}, quality: 'invalid', degraded: true, degraded_reason: 'cross_meter_epoch',
}
const ACTIVE_HEATING_METER = { id: 10, commodity: 'heating', ended_at: null }
const ACTIVE_HOT_WATER_METER = { id: 11, commodity: 'hot_water', ended_at: null }
const ENDED_HEATING_METER = { id: 9, commodity: 'heating', ended_at: '2026-06-01T00:00:00Z' }
const CLOSED_THERMAL_RANGE = {
startDate: '2020-01-01',
endDate: '2020-01-02',
start: '2020-01-01T00:00:00.000Z',
end: '2020-01-02T00:00:00.000Z',
}
// ---------------------------------------------------------------------------
// Import component
// ---------------------------------------------------------------------------
@@ -220,4 +268,142 @@ describe('CostView', () => {
)
})
})
it('audits thermal rows and recomputes only a closed UTC quarter through the typed client', async () => {
const user = userEvent.setup()
mockGet.mockImplementation((path: string) => {
if (path === '/api/energy/meter-costs') return Promise.resolve({ data: { items: [THERMAL_ROW, THERMAL_ROW_OTHER_VERSION, THERMAL_DEGRADED_MISSING_CONTRACT, THERMAL_DEGRADED_CROSS_EPOCH], total: 4 } })
if (path === '/api/energy/meter-costs/summary') return Promise.resolve({ data: THERMAL_SUMMARY })
if (path === '/api/energy/costs') return Promise.resolve({ data: { items: [], total: 0 } })
if (path === '/api/energy/costs/summary') return Promise.resolve({ data: SUMMARY })
return Promise.resolve({ data: null })
})
mockPost.mockResolvedValue({ data: { processed: 1, normal: 0, degraded: 1 } })
renderWithProviders(<CostView />)
await user.click(screen.getByTestId('costs-scope-selector'))
await user.click(screen.getByText('Thermal'))
await waitFor(() => expect(screen.getByTestId('thermal-cost-summary')).toBeInTheDocument())
expect(screen.getByTestId('thermal-period-count')).toHaveTextContent('4 periods; 2 degraded')
expect(screen.getByTestId('thermal-summary-degraded')).toHaveTextContent('Expand a row to see its recorded reason')
expect(screen.queryByTestId('thermal-degraded-0')).not.toBeInTheDocument()
expect(screen.queryByTestId('thermal-degraded-1')).not.toBeInTheDocument()
expect(screen.getByTestId('thermal-degraded-2')).toHaveTextContent('missing_contract')
expect(screen.getByTestId('thermal-degraded-3')).toHaveTextContent('cross_meter_epoch')
await user.click(screen.getByTestId('thermal-cost-expand-0'))
expect(screen.getByTestId('thermal-cost-audit-0')).toHaveTextContent('Contract version: 99')
expect(screen.getByTestId('thermal-cost-audit-0')).toHaveTextContent('20.123456789123456789')
expect(screen.getByTestId('thermal-cost-audit-0')).toHaveTextContent('heating_network')
await user.click(screen.getByTestId('thermal-cost-expand-1'))
expect(screen.getByTestId('thermal-cost-audit-1')).toHaveTextContent('Contract version: 100')
expect(screen.getByTestId('thermal-cost-audit-1')).toHaveTextContent('1.234567890123456789')
expect(screen.getByTestId('thermal-cost-audit-1')).toHaveTextContent('hot_water_tax')
await user.click(screen.getByTestId('thermal-cost-expand-2'))
expect(screen.getByTestId('thermal-cost-audit-2')).toHaveTextContent('Contract version: none')
expect(screen.getByTestId('thermal-cost-audit-2')).toHaveTextContent('Pricing snapshot: {}')
expect(screen.getByTestId('thermal-fixed-breakdown')).toHaveTextContent('summary only')
expect(screen.getAllByText(/Fixed subtotal|All-in total/)).toHaveLength(2)
expect(screen.getByTestId('thermal-costs-table')).not.toHaveTextContent('Fixed')
await user.click(screen.getByTestId('thermal-cost-range-control'))
await user.click(screen.getByText('Custom'))
expect(screen.getByTestId('thermal-recompute-button')).toBeDisabled()
await user.type(screen.getByTestId('thermal-cost-custom-start'), CLOSED_THERMAL_RANGE.startDate)
await user.type(screen.getByTestId('thermal-cost-custom-end'), CLOSED_THERMAL_RANGE.endDate)
await waitFor(() => expect(screen.getByTestId('thermal-recompute-button')).toBeEnabled())
await user.click(screen.getByTestId('thermal-recompute-button'))
expect(screen.getByTestId('thermal-recompute-confirm-modal')).toHaveTextContent('closed 15-minute')
await user.click(screen.getByTestId('thermal-recompute-confirm'))
await waitFor(() => expect(mockPost).toHaveBeenCalledWith(
'/api/energy/meter-costs/recompute',
{ params: { query: { scope: 'thermal', start: CLOSED_THERMAL_RANGE.start, end: CLOSED_THERMAL_RANGE.end } } },
))
expect(new Date(CLOSED_THERMAL_RANGE.end).getUTCMinutes() % 15).toBe(0)
})
it('keeps the thermal confirmation cancellable and surfaces a 422 recompute failure', async () => {
const user = userEvent.setup()
mockGet.mockImplementation((path: string) => {
if (path === '/api/energy/meter-costs') return Promise.resolve({ data: { items: [THERMAL_ROW], total: 1 } })
if (path === '/api/energy/meter-costs/summary') return Promise.resolve({ data: THERMAL_SUMMARY })
if (path === '/api/energy/costs/summary') return Promise.resolve({ data: SUMMARY })
return Promise.resolve({ data: { items: [], total: 0 } })
})
mockPost.mockRejectedValue(new Error('422'))
renderWithProviders(<CostView />)
await user.click(screen.getByTestId('costs-scope-selector')); await user.click(screen.getByText('Thermal'))
await waitFor(() => expect(screen.getByTestId('thermal-recompute-button')).toBeInTheDocument())
await user.click(screen.getByTestId('thermal-cost-range-control')); await user.click(screen.getByText('Custom'))
expect(screen.getByTestId('thermal-recompute-button')).toBeDisabled()
await user.type(screen.getByTestId('thermal-cost-custom-start'), CLOSED_THERMAL_RANGE.startDate)
await user.type(screen.getByTestId('thermal-cost-custom-end'), CLOSED_THERMAL_RANGE.endDate)
await waitFor(() => expect(screen.getByTestId('thermal-recompute-button')).toBeEnabled())
await user.click(screen.getByTestId('thermal-recompute-button')); await user.click(screen.getByTestId('thermal-recompute-cancel'))
expect(screen.queryByTestId('thermal-recompute-confirm-modal')).not.toBeInTheDocument()
await user.click(screen.getByTestId('thermal-recompute-button')); await user.click(screen.getByTestId('thermal-recompute-confirm'))
await waitFor(() => expect(mockPost).toHaveBeenCalledWith(
'/api/energy/meter-costs/recompute',
{ params: { query: { scope: 'thermal', start: CLOSED_THERMAL_RANGE.start, end: CLOSED_THERMAL_RANGE.end } } },
))
await waitFor(() => expect(screen.getByTestId('thermal-recompute-error')).toBeInTheDocument())
})
it.each([
['only heating', [ACTIVE_HEATING_METER], 'hot-water meter is not configured', 'Not configured'],
['only hot water', [ACTIVE_HOT_WATER_METER], 'heating meter is not configured', 'Not configured'],
['both current meters', [ACTIVE_HEATING_METER, ACTIVE_HOT_WATER_METER], null, '1.10'],
['a replaced heating meter plus its current epoch', [ENDED_HEATING_METER, ACTIVE_HEATING_METER, ACTIVE_HOT_WATER_METER], null, '1.10'],
])('uses active meter epochs for %s without treating zero amounts as missing', async (_name, meterItems, missingText, heatingValue) => {
mockGet.mockImplementation((path: string) => {
if (path === '/api/energy/meters') return Promise.resolve({ data: { items: meterItems, total: meterItems.length } })
if (path === '/api/energy/meter-costs') return Promise.resolve({ data: { items: [THERMAL_ROW], total: 1 } })
if (path === '/api/energy/meter-costs/summary') return Promise.resolve({ data: { ...THERMAL_SUMMARY, heating: heatingValue === 'Not configured' ? '0' : THERMAL_SUMMARY.heating } })
if (path === '/api/energy/costs/summary') return Promise.resolve({ data: SUMMARY })
return Promise.resolve({ data: { items: [], total: 0 } })
})
const user = userEvent.setup()
renderWithProviders(<CostView />)
await user.click(screen.getByTestId('costs-scope-selector'))
await user.click(screen.getByText('Thermal'))
await waitFor(() => expect(screen.getByTestId('thermal-cost-summary')).toBeInTheDocument())
if (missingText) {
expect(screen.getByTestId('thermal-missing-current-meter')).toHaveTextContent(missingText)
expect(screen.getByTestId('thermal-cost-summary')).toHaveTextContent('Not configured')
} else {
expect(screen.queryByTestId('thermal-missing-current-meter')).not.toBeInTheDocument()
expect(screen.getByTestId('thermal-cost-summary')).toHaveTextContent(heatingValue)
}
})
it('paginates the complete thermal ledger and resets offset when its range or scope changes', async () => {
const user = userEvent.setup()
const lastRow = { ...THERMAL_ROW, period_start: '2026-06-22T12:00:00Z', quantity: '501' }
mockGet.mockImplementation((path: string, options?: { params?: { query?: { offset?: number } } }) => {
if (path === '/api/energy/meters') return Promise.resolve({ data: { items: [ACTIVE_HEATING_METER, ACTIVE_HOT_WATER_METER], total: 2 } })
if (path === '/api/energy/meter-costs') {
const offset = options?.params?.query?.offset ?? 0
return Promise.resolve({ data: offset === 0 ? { items: Array.from({ length: 500 }, () => THERMAL_ROW), total: 501 } : { items: [lastRow], total: 501 } })
}
if (path === '/api/energy/meter-costs/summary') return Promise.resolve({ data: THERMAL_SUMMARY })
if (path === '/api/energy/costs/summary') return Promise.resolve({ data: SUMMARY })
return Promise.resolve({ data: { items: [], total: 0 } })
})
renderWithProviders(<CostView />)
await user.click(screen.getByTestId('costs-scope-selector'))
await user.click(screen.getByText('Thermal'))
await waitFor(() => expect(screen.getByTestId('thermal-ledger-count')).toHaveTextContent('Showing 1-500 of 501'))
expect(screen.getByTestId('thermal-ledger-prev')).toBeDisabled()
expect(screen.getByTestId('thermal-ledger-next')).toBeEnabled()
await user.click(screen.getByTestId('thermal-ledger-next'))
await waitFor(() => expect(screen.getByTestId('thermal-ledger-count')).toHaveTextContent('Showing 501-501 of 501'))
expect(screen.getByTestId('thermal-ledger-prev')).toBeEnabled()
expect(screen.getByTestId('thermal-ledger-next')).toBeDisabled()
expect(mockGet).toHaveBeenCalledWith('/api/energy/meter-costs', expect.objectContaining({ params: { query: expect.objectContaining({ scope: 'thermal', offset: 500, limit: 500 }) } }))
await user.click(screen.getByTestId('thermal-cost-range-control'))
await user.click(screen.getByText('This month'))
await waitFor(() => expect(screen.getByTestId('thermal-ledger-count')).toHaveTextContent('Showing 1-500 of 501'))
await user.click(screen.getByTestId('costs-scope-selector'))
await user.click(screen.getByText('Electricity'))
await user.click(screen.getByTestId('costs-scope-selector'))
await user.click(screen.getByText('Thermal'))
await waitFor(() => expect(screen.getByTestId('thermal-ledger-count')).toHaveTextContent('Showing 1-500 of 501'))
})
})
+86
View File
@@ -13,6 +13,7 @@
*/
import { useState } from 'react'
import { useMutation, useQuery, useQueryClient } from '@tanstack/react-query'
import {
Stack,
Text,
@@ -43,6 +44,7 @@ import {
} from 'recharts'
import { useEnergyCosts, useEnergyCostSummary, useRecomputeCosts } from './hooks'
import { formatLocalTime } from '../utils/datetime'
import apiClient from '../api/client'
// ---------------------------------------------------------------------------
// Cost limit — prevent accidental full-table pulls
@@ -109,6 +111,16 @@ function SummaryCard({ label, value, sub, testId }: SummaryCardProps) {
type RangePreset = 'today' | 'month' | 'custom'
export function CostView() {
const [scope, setScope] = useState<'electricity' | 'thermal'>('electricity')
if (scope === 'thermal') return <ThermalCostView onScopeChange={setScope} />
return <ElectricityCostView onScopeChange={setScope} />
}
function ScopeSelector({ scope, onScopeChange }: { scope: 'electricity' | 'thermal'; onScopeChange: (scope: 'electricity' | 'thermal') => void }) {
return <SegmentedControl value={scope} onChange={(value) => onScopeChange(value as 'electricity' | 'thermal')} data={[{ label: 'Electricity', value: 'electricity' }, { label: 'Thermal', value: 'thermal' }]} data-testid="costs-scope-selector" />
}
function ElectricityCostView({ onScopeChange }: { onScopeChange: (scope: 'electricity' | 'thermal') => void }) {
const [rangePreset, setRangePreset] = useState<RangePreset>('today')
// Date strings in YYYY-MM-DD format for custom range
const [customStartStr, setCustomStartStr] = useState('')
@@ -144,6 +156,7 @@ export function CostView() {
<Stack gap="lg" data-testid="cost-view">
{/* Date range selector */}
<Group align="flex-start" gap="md" wrap="wrap">
<ScopeSelector scope="electricity" onScopeChange={onScopeChange} />
<Stack gap="xs">
<Text size="sm" fw={500}>
Date range
@@ -410,3 +423,76 @@ export function CostView() {
</Stack>
)
}
function ThermalCostView({ onScopeChange }: { onScopeChange: (scope: 'electricity' | 'thermal') => void }) {
const [rangePreset, setRangePreset] = useState<RangePreset>('today')
const [customStartStr, setCustomStartStr] = useState('')
const [customEndStr, setCustomEndStr] = useState('')
const [showConfirm, setShowConfirm] = useState(false)
const [recomputeError, setRecomputeError] = useState<string | null>(null)
const [recomputeSuccess, setRecomputeSuccess] = useState<string | null>(null)
const [expandedRows, setExpandedRows] = useState<Set<number>>(() => new Set())
const [ledgerOffset, setLedgerOffset] = useState(0)
const { start, end } = (() => {
if (rangePreset === 'today') return getTodayRange()
if (rangePreset === 'month') return getThisMonthRange()
return {
start: customStartStr ? new Date(customStartStr).toISOString() : undefined,
end: customEndStr ? new Date(customEndStr).toISOString() : undefined,
}
})()
// The server only accepts complete UTC quarters. Never send a future end.
const closedEnd = (() => {
const now = new Date()
now.setUTCMinutes(Math.floor(now.getUTCMinutes() / 15) * 15, 0, 0)
const selectedEnd = end ? new Date(end) : now
return new Date(Math.min(selectedEnd.getTime(), now.getTime())).toISOString()
})()
const recomputeStart = start
const recomputeAvailable = !!recomputeStart && new Date(recomputeStart) < new Date(closedEnd)
const qc = useQueryClient()
const resetLedgerPage = () => {
setLedgerOffset(0)
setExpandedRows(new Set())
}
const rows = useQuery({ queryKey: ['meter-costs', 'thermal', start, end, ledgerOffset], queryFn: async () => {
const result = await apiClient.GET('/api/energy/meter-costs', { params: { query: { scope: 'thermal', start, end, limit: COSTS_MAX_LIMIT, offset: ledgerOffset } } })
return result.data
} })
const meters = useQuery({ queryKey: ['energy-meters', 'thermal'], queryFn: async () => {
const result = await apiClient.GET('/api/energy/meters')
return result.data
} })
const summary = useQuery({ queryKey: ['meter-cost-summary', 'thermal', start, end], queryFn: async () => {
const result = await apiClient.GET('/api/energy/meter-costs/summary', { params: { query: { scope: 'thermal', start, end } } })
return result.data
} })
const recompute = useMutation({ mutationFn: () => apiClient.POST('/api/energy/meter-costs/recompute', { params: { query: { scope: 'thermal', start: recomputeStart!, end: closedEnd } } }), onSuccess: (result) => {
void qc.invalidateQueries({ queryKey: ['meter-costs', 'thermal'] }); void qc.invalidateQueries({ queryKey: ['meter-cost-summary', 'thermal'] })
setRecomputeSuccess(`Recomputed ${result.data?.processed ?? 0} closed periods.`)
} })
const currency = summary.data?.currency ?? rows.data?.items[0]?.currency ?? 'EUR'
const fixed = summary.data?.fixed_breakdown
const hasCurrentHeatingMeter = meters.data?.items.some((meter) => meter.commodity === 'heating' && meter.ended_at === null)
const hasCurrentHotWaterMeter = meters.data?.items.some((meter) => meter.commodity === 'hot_water' && meter.ended_at === null)
const missingCurrentMeters = [
...(hasCurrentHeatingMeter === false ? ['heating'] : []),
...(hasCurrentHotWaterMeter === false ? ['hot-water'] : []),
]
const totalRows = rows.data?.total ?? 0
const shownStart = totalRows === 0 ? 0 : ledgerOffset + 1
const shownEnd = Math.min(ledgerOffset + (rows.data?.items.length ?? 0), totalRows)
return <Stack gap="lg" data-testid="thermal-cost-view">
<Group align="flex-start" gap="md" wrap="wrap"><ScopeSelector scope="thermal" onScopeChange={onScopeChange} /><Stack gap="xs"><Text size="sm" fw={500}>Date range</Text><SegmentedControl value={rangePreset} onChange={(value) => { resetLedgerPage(); setRangePreset(value as RangePreset) }} data={[{ label: 'Today', value: 'today' }, { label: 'This month', value: 'month' }, { label: 'Custom', value: 'custom' }]} data-testid="thermal-cost-range-control" /></Stack>{rangePreset === 'custom' && <Group gap="sm" align="flex-end"><TextInput label="From" type="date" value={customStartStr} onChange={(event) => { resetLedgerPage(); setCustomStartStr(event.currentTarget.value) }} data-testid="thermal-cost-custom-start" /><TextInput label="To" type="date" value={customEndStr} onChange={(event) => { resetLedgerPage(); setCustomEndStr(event.currentTarget.value) }} data-testid="thermal-cost-custom-end" /></Group>}<Button variant="outline" color="orange" onClick={() => { setRecomputeError(null); setRecomputeSuccess(null); setShowConfirm(true) }} disabled={!recomputeAvailable} data-testid="thermal-recompute-button">Recompute</Button></Group>
{(rows.isLoading || summary.isLoading) && <Center><Loader size="sm" /></Center>}
{(rows.isError || summary.isError) && <Alert color="red">Failed to load thermal costs.</Alert>}
{recomputeError && <Alert color="red" data-testid="thermal-recompute-error">{recomputeError}</Alert>}
{recomputeSuccess && <Alert color="green" data-testid="thermal-recompute-success">{recomputeSuccess}</Alert>}
{summary.data && <Stack gap="xs" data-testid="thermal-cost-summary"><Title order={6}>{rangePreset === 'today' ? 'Today' : rangePreset === 'month' ? 'This month' : 'Custom range'} ({currency})</Title><Text size="sm" data-testid="thermal-cost-range">{start ?? 'Select a start date'} {end ?? 'Select an end date'}</Text><SimpleGrid cols={{ base: 2, sm: 3 }}>
<SummaryCard label="Heating" value={hasCurrentHeatingMeter === false ? 'Not configured' : summary.data.heating} /><SummaryCard label="Hot-water heating" value={hasCurrentHotWaterMeter === false ? 'Not configured' : summary.data.hot_water_heating} /><SummaryCard label="Hot water" value={hasCurrentHotWaterMeter === false ? 'Not configured' : summary.data.hot_water} /><SummaryCard label="Hot-water tax" value={hasCurrentHotWaterMeter === false ? 'Not configured' : summary.data.hot_water_tax} /><SummaryCard label="Variable subtotal" value={summary.data.variable_subtotal} /><SummaryCard label="Fixed subtotal" value={summary.data.fixed_subtotal} /><SummaryCard label="All-in total" value={summary.data.all_in} />
</SimpleGrid>{missingCurrentMeters.length > 0 && <Alert color="yellow" data-testid="thermal-missing-current-meter">Current {missingCurrentMeters.join(' and ')} meter{missingCurrentMeters.length > 1 ? 's are' : ' is'} not configured. Historical ledger rows do not establish a current meter.</Alert>}<Text size="sm" data-testid="thermal-period-count">{summary.data.period_count} periods; {summary.data.degraded_count} degraded</Text>{summary.data.degraded_count > 0 && <Alert color="orange" data-testid="thermal-summary-degraded">Some totals include degraded periods. Expand a row to see its recorded reason.</Alert>}{fixed && <Text size="sm" data-testid="thermal-fixed-breakdown">Fixed once per settled local day (summary only): {Object.entries(fixed).map(([key, value]) => `${key} ${value}`).join(' · ')}</Text>}</Stack>}
{rows.data?.items.length === 0 && <Alert color="gray" data-testid="thermal-costs-empty">No thermal cost data for this range. Check that heating or hot-water meters are bound and have settled readings.</Alert>}
{rows.data && <Stack gap="xs"><Text size="sm" c="dimmed" data-testid="thermal-ledger-count">Showing {shownStart}-{shownEnd} of {totalRows}</Text>{rows.data.items.length > 0 && <ScrollArea><Table striped withTableBorder data-testid="thermal-costs-table"><Table.Thead><Table.Tr><Table.Th>Time</Table.Th><Table.Th>Commodity</Table.Th><Table.Th>Quantity</Table.Th><Table.Th>Cost</Table.Th><Table.Th>Breakdown</Table.Th><Table.Th>Status</Table.Th><Table.Th></Table.Th></Table.Tr></Table.Thead><Table.Tbody>{rows.data.items.flatMap((item, index) => [<Table.Tr key={`${item.commodity}-${item.period_start}`} data-testid={`thermal-cost-row-${index}`}><Table.Td>{formatLocalTime(item.period_start)}</Table.Td><Table.Td>{item.commodity}</Table.Td><Table.Td>{item.quantity}</Table.Td><Table.Td>{item.cost} {item.currency}</Table.Td><Table.Td>{Object.entries(item.cost_breakdown).map(([key, value]) => `${key} ${value}`).join(', ')}</Table.Td><Table.Td>{item.degraded ? <Badge color="orange" data-testid={`thermal-degraded-${index}`}>{item.degraded_reason ?? 'degraded'}</Badge> : 'normal'}</Table.Td><Table.Td><Button size="xs" variant="subtle" onClick={() => setExpandedRows((current) => { const next = new Set(current); if (next.has(index)) next.delete(index); else next.add(index); return next })} data-testid={`thermal-cost-expand-${index}`}>{expandedRows.has(index) ? 'Hide audit' : 'Audit'}</Button></Table.Td></Table.Tr>, ...(expandedRows.has(index) ? [<Table.Tr key={`audit-${item.commodity}-${item.period_start}`} data-testid={`thermal-cost-audit-${index}`}><Table.Td colSpan={7}><Text size="xs">Contract version: {item.contract_version_id ?? 'none'}</Text><Text size="xs">Pricing snapshot: {JSON.stringify(item.pricing_snapshot)}</Text></Table.Td></Table.Tr>] : [])])}</Table.Tbody></Table></ScrollArea>}<Group justify="flex-end"><Button size="xs" variant="default" disabled={ledgerOffset === 0} onClick={() => { setExpandedRows(new Set()); setLedgerOffset((current) => Math.max(0, current - COSTS_MAX_LIMIT)) }} data-testid="thermal-ledger-prev">Previous</Button><Button size="xs" variant="default" disabled={ledgerOffset + (rows.data.items.length ?? 0) >= totalRows} onClick={() => { setExpandedRows(new Set()); setLedgerOffset((current) => current + COSTS_MAX_LIMIT) }} data-testid="thermal-ledger-next">Next</Button></Group></Stack>}
{showConfirm && <Modal opened onClose={() => setShowConfirm(false)} title="Recompute thermal costs?" data-testid="thermal-recompute-confirm-modal"><Stack><Text>This explicitly overwrites closed 15-minute thermal ledger rows for {recomputeStart ?? 'the selected start'} {closedEnd}. Continue?</Text>{!recomputeAvailable && <Alert color="yellow">Select a range containing at least one closed UTC quarter.</Alert>}<Group justify="flex-end"><Button variant="default" onClick={() => setShowConfirm(false)} data-testid="thermal-recompute-cancel">Cancel</Button><Button color="orange" loading={recompute.isPending} disabled={!recomputeAvailable} onClick={async () => { try { await recompute.mutateAsync(); setShowConfirm(false) } catch { setRecomputeError('Failed to recompute thermal costs. Please try again.'); setShowConfirm(false) } }} data-testid="thermal-recompute-confirm">Recompute</Button></Group></Stack></Modal>}
</Stack>
}
+10
View File
@@ -50,6 +50,9 @@ describe('DsmrPanel', () => {
renderWithProviders(<DsmrPanel />)
await waitFor(() => expect(screen.getByTestId('dsmr-empty')).toBeInTheDocument())
expect(screen.queryByTestId('dsmr-table')).not.toBeInTheDocument()
expect(screen.getByText(/In this DSMR Source, enable or edit the broker, topic, and profile/)).toBeInTheDocument()
expect(screen.getByText(/confirm the publisher is sending/)).toBeInTheDocument()
expect(screen.queryByText(/Enable DSMR ingest.*Config/)).not.toBeInTheDocument()
})
it('renders the latest telegram as a key/value table; null shown as dash', async () => {
@@ -79,4 +82,11 @@ describe('DsmrPanel', () => {
renderWithProviders(<DsmrPanel />)
await waitFor(() => expect(screen.getByTestId('dsmr-error')).toBeInTheDocument())
})
it('keeps the compatibility endpoint available for DSMR source detail', async () => {
mockGet.mockResolvedValue({ data: { found: true, recorded_at: '2026-06-23T12:16:00Z', payload: { tariff: 'low' } } })
renderWithProviders(<DsmrPanel />)
await waitFor(() => expect(mockGet).toHaveBeenCalledWith('/api/energy/dsmr/latest'))
expect(await screen.findByText('Latest DSMR reading (compatibility view)')).toBeInTheDocument()
})
})
+4 -4
View File
@@ -47,7 +47,7 @@ export function DsmrPanel() {
<Stack gap="md" data-testid="dsmr-panel">
<Group justify="space-between" align="center">
<div>
<Text fw={600}>Latest DSMR reading</Text>
<Text fw={600}>Latest DSMR reading (compatibility view)</Text>
<Text size="xs" c="dimmed">
The most recent parsed telegram persisted to <code>dsmr_reading</code>.
</Text>
@@ -100,9 +100,9 @@ function DsmrContent({ isLoading, isError, data }: DsmrContentProps) {
if (!data.found || !data.payload) {
return (
<Alert color="gray" data-testid="dsmr-empty">
No DSMR data yet. Enable <strong>DSMR ingest</strong> in Config, make sure MQTT
is connected, and confirm the DSMR Reader is publishing to the configured topic
(default <code>dsmr/json</code>). Rows are stored about once every 10 seconds.
No DSMR data yet. In this DSMR Source, enable or edit the broker, topic, and profile
configuration, then confirm the publisher is sending to the configured topic (default
<code>dsmr/json</code>). Rows are stored about once every 10 seconds.
</Alert>
)
}
File diff suppressed because it is too large Load Diff
+395 -24
View File
@@ -34,11 +34,17 @@ import {
useMeters,
useDeclareMeter,
useUpdateMeter,
useSources,
useSourceChannels,
useCreateBinding,
useCloseBinding,
useCloseMeter,
useTransferBinding,
type MeterResponse,
type MeterReason,
} from './hooks'
import { ApiError } from '../api/client'
import { formatLocalDate, parseBackendTimestamp } from '../utils/datetime'
import { formatLocalDate, formatLocalDateTime, parseBackendTimestamp } from '../utils/datetime'
// ---------------------------------------------------------------------------
// Helpers
@@ -73,23 +79,174 @@ function toLocalDateInputString(d: Date): string {
return `${y}-${m}-${day}`
}
function toLocalDateTimeInputString(d = new Date()): string {
const pad = (value: number) => String(value).padStart(2, '0')
return `${toLocalDateInputString(d)}T${pad(d.getHours())}:${pad(d.getMinutes())}`
}
function expectedUnit(commodity: string): string {
return ({ electricity: 'kWh', heating: 'GJ', hot_water: 'm³' } as Record<string, string>)[commodity] ?? ''
}
function apiErrorMessage(err: unknown, fallback: string): string {
if (err instanceof ApiError) {
const body = err.body
if (typeof body === 'string') return body
if (body && typeof body === 'object' && 'detail' in body) {
const detail = (body as { detail?: unknown }).detail
if (typeof detail === 'string') return detail
if (Array.isArray(detail)) {
const messages = detail.map((item) => item && typeof item === 'object' && typeof item.msg === 'string'
? item.msg : String(item)).filter(Boolean)
if (messages.length) return messages.join('; ')
}
if (detail != null) return typeof detail === 'object' ? JSON.stringify(detail) : String(detail)
}
return `${fallback} (error ${err.status}).`
}
return fallback
}
function isValidLocalDateTime(value: string): boolean {
return value.trim() !== '' && !Number.isNaN(new Date(value).getTime())
}
type Eligibility = { eligible: boolean; reason?: string }
function localInstant(value: string): number | null {
const instant = new Date(value).getTime()
return Number.isNaN(instant) ? null : instant
}
function intervalsOverlap(start: number, end: number | null, otherStart: number, otherEnd: number | null): boolean {
return (end === null || otherStart < end) && (otherEnd === null || start < otherEnd)
}
function channelIntervalEligibility(
meters: MeterResponse[], channelUuid: string, startedAt: string, excludeBindingUuids: string[] = [],
): Eligibility {
const start = localInstant(startedAt)
if (start === null) return { eligible: false, reason: 'choose a valid start time first' }
const conflicts = meters.flatMap((meter) => (meter.bindings ?? []).filter((binding) =>
binding.source_channel_uuid === channelUuid && !excludeBindingUuids.includes(binding.uuid) &&
intervalsOverlap(start, null, parseBackendTimestamp(binding.started_at).getTime(), binding.ended_at ? parseBackendTimestamp(binding.ended_at).getTime() : null),
))
if (!conflicts.length) return { eligible: true }
if (conflicts.length > 1) return { eligible: false, reason: 'ambiguous: channel has overlapping binding history; resolve it first' }
return conflicts[0].ended_at === null
? { eligible: false, reason: 'occupied by an open binding; close or transfer it first' }
: { eligible: false, reason: 'overlaps closed binding history; choose a time at or after it ends' }
}
function meterContainsInstant(meter: MeterResponse, instant: number): boolean {
const start = parseBackendTimestamp(meter.started_at).getTime()
const end = meter.ended_at ? parseBackendTimestamp(meter.ended_at).getTime() : null
return instant >= start && (end === null || instant < end)
}
function recoveryTargetFor(meter: MeterResponse, meters: MeterResponse[]): MeterResponse | null {
if (meter.ended_at === null) return null
const active = meters.filter((candidate) => candidate.commodity === meter.commodity && candidate.ended_at === null)
if (active.length !== 1) return null
const target = active[0]
const targetStart = parseBackendTimestamp(target.started_at).getTime()
const predecessors = meters.filter((candidate) => candidate.commodity === meter.commodity && candidate.ended_at !== null &&
parseBackendTimestamp(candidate.ended_at).getTime() <= targetStart)
const latestEnd = Math.max(...predecessors.map((candidate) => parseBackendTimestamp(candidate.ended_at!).getTime()))
const immediate = predecessors.filter((candidate) => parseBackendTimestamp(candidate.ended_at!).getTime() === latestEnd)
if (immediate.length !== 1 || immediate[0].id !== meter.id) return null
const sourceStart = parseBackendTimestamp(meter.started_at).getTime()
const sourceEnd = parseBackendTimestamp(meter.ended_at).getTime()
for (const candidate of meters) {
if (candidate.id === meter.id || candidate.id === target.id || candidate.commodity !== meter.commodity) continue
const candidateStart = parseBackendTimestamp(candidate.started_at).getTime()
const candidateEnd = candidate.ended_at ? parseBackendTimestamp(candidate.ended_at).getTime() : null
if (intervalsOverlap(sourceStart, sourceEnd, candidateStart, candidateEnd) ||
intervalsOverlap(targetStart, null, candidateStart, candidateEnd)) return null
}
return target
}
// ---------------------------------------------------------------------------
// Declare meter form (modal)
// ---------------------------------------------------------------------------
interface DeclareMeterFormProps {
meters: MeterResponse[]
onClose: () => void
onSaved: () => void
}
function DeclareMeterForm({ onClose, onSaved }: DeclareMeterFormProps) {
function DeclareMeterForm({ meters, onClose, onSaved }: DeclareMeterFormProps) {
const [label, setLabel] = useState('')
const [dateStr, setDateStr] = useState('')
const [reason, setReason] = useState<string | null>(null)
const [note, setNote] = useState('')
const [error, setError] = useState<string | null>(null)
const [commodity, setCommodity] = useState<string | null>('electricity')
const [sourceUuid, setSourceUuid] = useState<string | null>(null)
const [channelUuid, setChannelUuid] = useState<string | null>(null)
const sources = useSources()
const channels = useSourceChannels(sourceUuid)
const declareMutation = useDeclareMeter()
const unit = expectedUnit(commodity ?? 'electricity')
const oldMeter = meters.find((meter) => meter.commodity === commodity && meter.ended_at === null)
const channelEligibility = (uuid: string, startedAt: string): Eligibility => {
const channel = channels.data?.items.find((item) => item.uuid === uuid)
if (!channel) return { eligible: false, reason: 'channel is unavailable; reload the source' }
if (channel.unit !== unit) return { eligible: false, reason: `unit mismatch: ${channel.unit}; this meter needs ${unit}` }
if (!startedAt) return { eligible: false, reason: 'choose a start date first' }
const start = localInstant(toLocalMidnightNaive(startedAt))
if (start === null) return { eligible: false, reason: 'choose a valid start date first' }
if (startedAt > toLocalDateInputString(new Date())) return { eligible: false, reason: 'future start dates cannot bind a channel' }
// Channel aggregates include closed binding history. For a meter swap, only
// currently open bindings determine whether this channel can be handed off.
const openBindings = meters.flatMap((meter) =>
(meter.bindings ?? [])
.filter((binding) => binding.source_channel_uuid === channel.uuid && binding.ended_at === null)
.map((binding) => ({ meter, binding })),
)
const oldBinding = openBindings[0]
const isSingleOldBinding = reason === 'meter_swap' && oldMeter !== undefined &&
openBindings.length === 1 && oldBinding !== undefined &&
oldBinding.meter.id === oldMeter.id && oldBinding.meter.commodity === commodity
// A handoff must leave a non-empty interval on the old Meter. The backend
// rejects equality too, so surface it here instead of offering a request
// that is guaranteed to fail.
if (isSingleOldBinding && start <= parseBackendTimestamp(oldBinding.binding.started_at).getTime()) {
return { eligible: false, reason: 'handoff boundary must be strictly after the current binding start' }
}
const canHandoff = isSingleOldBinding
const interval = channelIntervalEligibility(
meters, channel.uuid, toLocalMidnightNaive(startedAt), canHandoff ? [oldBinding.binding.uuid] : [],
)
if (!interval.eligible) return interval
if (openBindings.length && !canHandoff) {
return openBindings.length > 1
? { eligible: false, reason: 'ambiguous: multiple open bindings; resolve them first' }
: { eligible: false, reason: 'occupied by an open binding; choose a different channel or close/transfer it first' }
}
return { eligible: true }
}
const channelOptions = channels.data?.items.map((channel) => {
const result = channelEligibility(channel.uuid, dateStr)
const canHandoff = !isUnboundChannel(channel.uuid) && result.eligible
return {
value: channel.uuid,
label: `${channel.label} (${channel.unit})${canHandoff ? ' — hand off from current meter' : result.reason ? `${result.reason}` : ''}`,
disabled: !result.eligible,
}
}) ?? []
const selectedChannelUuid = channelUuid && channelEligibility(channelUuid, dateStr).eligible
? channelUuid
: null
function isUnboundChannel(uuid: string): boolean {
return !meters.some((meter) => meter.bindings?.some(
(binding) => binding.source_channel_uuid === uuid && binding.ended_at === null,
))
}
async function handleSubmit(e: React.FormEvent) {
e.preventDefault()
@@ -107,6 +264,29 @@ function DeclareMeterForm({ onClose, onSaved }: DeclareMeterFormProps) {
setError('Reason is required.')
return
}
if (sourceUuid && !selectedChannelUuid) {
const unavailable = channelOptions.length === 1 && channelOptions[0].disabled
? channelEligibility(channelOptions[0].value, dateStr).reason
: undefined
setError(unavailable ? `${unavailable[0].toUpperCase()}${unavailable.slice(1)}` : 'Select an eligible source channel or clear the optional source.')
return
}
// Omitting a channel for meter_swap asks the backend to auto-handoff the
// sole open binding. Keep that implicit path subject to the same strict
// boundary rule as an explicitly selected channel.
const start = localInstant(toLocalMidnightNaive(dateStr))
const oldMeter = meters.find((meter) => meter.commodity === commodity && meter.ended_at === null)
// The implicit backend handoff only considers bindings on the current
// commodity's old meter. Bindings are unit-compatible with their meter
// by the binding contract, so an unrelated heating/hot-water binding must
// neither make this ambiguous nor bypass this strict boundary check.
const autoHandoffCandidates = (oldMeter?.bindings ?? []).filter((binding) => binding.ended_at === null)
const oldBinding = autoHandoffCandidates[0]
if (reason === 'meter_swap' && start !== null && oldMeter !== undefined && autoHandoffCandidates.length === 1 &&
oldBinding !== undefined && start <= parseBackendTimestamp(oldBinding.started_at).getTime()) {
setError('Handoff boundary must be strictly after the current binding start.')
return
}
try {
await declareMutation.mutateAsync({
@@ -114,18 +294,12 @@ function DeclareMeterForm({ onClose, onSaved }: DeclareMeterFormProps) {
started_at: toLocalMidnightNaive(dateStr),
reason: reason as MeterReason,
note: note.trim() || undefined,
commodity: 'electricity',
commodity: commodity ?? 'electricity',
...(selectedChannelUuid ? { source_channel_uuid: selectedChannelUuid } : {}),
})
onSaved()
onClose()
} catch (err) {
if (err instanceof ApiError) {
const detail = (err.body as { detail?: string } | null)?.detail
setError(detail ?? `Error ${err.status}: failed to declare meter.`)
} else {
setError('Failed to declare meter. Please try again.')
}
}
} catch (err) { setError(apiErrorMessage(err, 'Failed to declare meter. Please try again.')) }
}
return (
@@ -153,7 +327,11 @@ function DeclareMeterForm({ onClose, onSaved }: DeclareMeterFormProps) {
type="date"
required
value={dateStr}
onChange={(e) => setDateStr(e.currentTarget.value)}
onChange={(e) => {
const nextDateStr = e.currentTarget.value
setDateStr(nextDateStr)
setChannelUuid((uuid) => uuid && !channelEligibility(uuid, nextDateStr).eligible ? null : uuid)
}}
data-testid="meter-started-at"
/>
@@ -162,10 +340,21 @@ function DeclareMeterForm({ onClose, onSaved }: DeclareMeterFormProps) {
required
data={REASON_OPTIONS}
value={reason}
onChange={setReason}
onChange={(value) => { setReason(value); setChannelUuid(null) }}
data-testid="meter-reason"
/>
<Select label="Commodity" value={commodity} onChange={(value) => { setCommodity(value); setChannelUuid(null) }} data={[
{ value: 'electricity', label: 'Electricity' },
{ value: 'heating', label: 'Heating' },
{ value: 'hot_water', label: 'Hot water' },
]} />
<Select label="Bind source (optional)" value={sourceUuid} onChange={(value) => { setSourceUuid(value); setChannelUuid(null) }} data={sources.data?.items.filter((source) => typeof source.uuid === 'string').map((source) => ({ value: source.uuid, label: source.name })) ?? []} />
{reason === 'meter_swap' && <Alert color="blue">If the previous meter has exactly one compatible open binding, declaring this meter automatically hands that channel over atomically. Ambiguous bindings remain unavailable.</Alert>}
{sources.isLoading && <Text size="sm">Loading sources</Text>}{sources.isError && <Alert color="red">Could not load sources. Retry after the connection recovers.</Alert>}
{sourceUuid && channels.isLoading && <Text size="sm">Loading source channels</Text>}{sourceUuid && channels.isError && <Alert color="red">Could not load source channels. Choose another source or retry.</Alert>}
{sourceUuid && <Select label="Compatible source channel (optional)" value={selectedChannelUuid} onChange={setChannelUuid} description="Disabled channels explain unit, current interval, ambiguity, or required time. Closed history remains reusable." data={channelOptions} />}
<Textarea
label="Note (optional)"
value={note}
@@ -252,14 +441,7 @@ function EditMeterForm({ meter, onClose, onSaved }: EditMeterFormProps) {
await updateMutation.mutateAsync({ id: meter.id, body: patchBody })
onSaved(startedAtChanged)
onClose()
} catch (err) {
if (err instanceof ApiError) {
const detail = (err.body as { detail?: string } | null)?.detail
setError(detail ?? `Error ${err.status}: failed to update meter.`)
} else {
setError('Failed to update meter. Please try again.')
}
}
} catch (err) { setError(apiErrorMessage(err, 'Failed to update meter. Please try again.')) }
}
return (
@@ -334,9 +516,10 @@ function EditMeterForm({ meter, onClose, onSaved }: EditMeterFormProps) {
interface MeterTableProps {
meters: MeterResponse[]
onEdit: (meter: MeterResponse) => void
onClose: (meter: MeterResponse) => void
}
function MeterTable({ meters, onEdit }: MeterTableProps) {
function MeterTable({ meters, onEdit, onClose }: MeterTableProps) {
if (meters.length === 0) {
return (
<Text c="dimmed" ta="center" size="sm" data-testid="meters-empty">
@@ -356,6 +539,7 @@ function MeterTable({ meters, onEdit }: MeterTableProps) {
<Table.Th>To</Table.Th>
<Table.Th>Status</Table.Th>
<Table.Th>Reason</Table.Th>
<Table.Th>Binding timeline</Table.Th>
<Table.Th style={{ textAlign: 'right' }}>Actions</Table.Th>
</Table.Tr>
</Table.Thead>
@@ -399,8 +583,24 @@ function MeterTable({ meters, onEdit }: MeterTableProps) {
{meter.reason}
</Badge>
</Table.Td>
<Table.Td>
{meter.bindings?.length ? meter.bindings.map((binding) => (
<Stack key={binding.uuid} gap={0} mb="xs" data-testid={`binding-timeline-${binding.uuid}`}>
<Text size="xs">{binding.source_uuid} {binding.source_channel_uuid}</Text>
<Text size="xs" c="dimmed">
[{formatLocalDateTime(binding.started_at)}, {binding.ended_at ? formatLocalDateTime(binding.ended_at) : 'open-ended'})
{' '}({binding.ended_at ? 'closed' : 'active'})
</Text>
</Stack>
)) : <Text size="xs" c="dimmed">Unbound</Text>}
</Table.Td>
<Table.Td>
<Group justify="flex-end" gap="xs">
{meter.bindings?.filter((binding) => binding.ended_at === null).map((binding) => (
<BindingActions key={binding.uuid} meter={meter} binding={binding} meters={meters} />
))}
{isActive && !meter.bindings?.some((binding) => binding.ended_at === null) && <DirectBindButton meter={meter} meters={meters} />}
{isActive && <Button size="xs" color="red" variant="light" onClick={() => onClose(meter)}>Close meter</Button>}
<Button
size="xs"
variant="outline"
@@ -420,6 +620,173 @@ function MeterTable({ meters, onEdit }: MeterTableProps) {
)
}
function DirectBindButton({ meter, meters }: { meter: MeterResponse; meters: MeterResponse[] }) {
const [opened, setOpened] = useState(false)
return <>{<Button size="xs" variant="subtle" onClick={() => setOpened(true)}>Bind source</Button>}{opened && <DirectBindModal meter={meter} meters={meters} onClose={() => setOpened(false)} />}</>
}
function DirectBindModal({ meter, meters, onClose }: { meter: MeterResponse; meters: MeterResponse[]; onClose: () => void }) {
const [sourceUuid, setSourceUuid] = useState<string | null>(null)
const [channelUuid, setChannelUuid] = useState<string | null>(null)
const [startedAt, setStartedAt] = useState(() => toLocalDateTimeInputString())
const [error, setError] = useState<string | null>(null)
const sources = useSources(); const channels = useSourceChannels(sourceUuid); const create = useCreateBinding()
const eligibilityAt = (channel: { uuid: string; unit: string }, value: string): Eligibility => {
if (channel.unit !== expectedUnit(meter.commodity)) return { eligible: false, reason: `unit mismatch: ${channel.unit}; this meter needs ${expectedUnit(meter.commodity)}` }
const start = localInstant(value)
if (start === null) return { eligible: false, reason: 'choose a valid binding start time first' }
if (value > toLocalDateTimeInputString()) return { eligible: false, reason: 'future binding start times are not allowed' }
if (!meterContainsInstant(meter, start)) return { eligible: false, reason: 'binding start must be within this meter epoch' }
return channelIntervalEligibility(meters, channel.uuid, value)
}
const eligibility = (channel: { uuid: string; unit: string }) => eligibilityAt(channel, startedAt)
const options = channels.data?.items.map((channel) => {
const result = eligibility(channel)
return { value: channel.uuid, label: `${channel.label} (${channel.unit})${result.reason ? `${result.reason}` : ''}`, disabled: !result.eligible }
}) ?? []
const selectedChannel = channels.data?.items.find((channel) => channel.uuid === channelUuid)
const selectedEligible = selectedChannel !== undefined && eligibility(selectedChannel).eligible
async function save() {
if (create.isPending) return
if (!channelUuid) return setError('Select a genuinely unbound, unit-compatible channel.')
if (!isValidLocalDateTime(startedAt)) return setError('Choose a valid binding start time.')
if (!selectedEligible) return setError('The selected channel is no longer eligible. Choose an available channel.')
setError(null)
try { await create.mutateAsync({ id: meter.id, body: { source_channel_uuid: channelUuid, started_at: startedAt } }); onClose() } catch (err) { setError(apiErrorMessage(err, 'Could not bind this source.')) }
}
return <Modal opened onClose={onClose} title="Bind source" data-testid={`direct-bind-modal-${meter.id}`}><form onSubmit={(event) => { event.preventDefault(); void save() }}><Stack>
<Alert color="blue">This active meter has no open binding. Choose a currently unoccupied compatible channel.</Alert>
{sources.isLoading && <Text size="sm">Loading sources</Text>}{sources.isError && <Alert color="red">Could not load sources. Retry after the connection recovers.</Alert>}
<Select label="Source" value={sourceUuid} onChange={(value) => { setSourceUuid(value); setChannelUuid(null) }} data={sources.data?.items.map((source) => ({ value: source.uuid, label: source.name })) ?? []} />
{sourceUuid && channels.isLoading && <Text size="sm">Loading source channels</Text>}{sourceUuid && channels.isError && <Alert color="red">Could not load source channels. Choose another source or retry.</Alert>}
<Select label="Source channel" value={channelUuid} onChange={setChannelUuid} description="Disabled channels explain the unit, interval, ambiguity, or time constraint." data={options} />
<TextInput label="Binding start time" type="datetime-local" value={startedAt} onChange={(event) => { const value = event.currentTarget.value; setStartedAt(value); setChannelUuid((uuid) => { if (!isValidLocalDateTime(value)) return uuid; const channel = channels.data?.items.find((item) => item.uuid === uuid); return channel && !eligibilityAt(channel, value).eligible ? null : uuid }) }} required />
{error && <Alert color="red">{error}</Alert>}
<Group justify="flex-end"><Button type="button" variant="default" onClick={onClose}>Cancel</Button><Button type="submit" loading={create.isPending} disabled={create.isPending || (!!channelUuid && !selectedEligible)}>Bind source</Button></Group>
</Stack></form></Modal>
}
function BindingActions({ meter, binding, meters }: { meter: MeterResponse; binding: NonNullable<MeterResponse['bindings']>[number]; meters: MeterResponse[] }) {
const [unbindOpened, setUnbindOpened] = useState(false)
const [transferOpened, setTransferOpened] = useState(false)
const recoveryTarget = recoveryTargetFor(meter, meters)
// Cross-Meter recovery closes at the old Meter boundary. An anomalous
// retained binding beginning at or after that boundary would create a
// zero-length/negative source interval that the server correctly rejects.
const recoverySourceIsClosable = meter.ended_at === null ||
parseBackendTimestamp(binding.started_at).getTime() < parseBackendTimestamp(meter.ended_at).getTime()
return <>
<Button size="xs" variant="subtle" onClick={() => setUnbindOpened(true)}>Unbind</Button>
{meter.ended_at === null ? <Button size="xs" variant="subtle" onClick={() => setTransferOpened(true)}>Transfer source</Button> : recoveryTarget && recoverySourceIsClosable && <Button size="xs" variant="light" onClick={() => setTransferOpened(true)}>Recover binding</Button>}
{meter.ended_at !== null && recoveryTarget && !recoverySourceIsClosable && <Text size="xs" c="red">Cannot recover: the source binding starts at or after this Meter ended.</Text>}
{unbindOpened && <UnbindModal meter={meter} binding={binding} onClose={() => setUnbindOpened(false)} />}
{transferOpened && <TransferModal target={recoveryTarget ?? meter} sourceBinding={binding} meters={meters} recovery={!!recoveryTarget} onClose={() => setTransferOpened(false)} />}
</>
}
function UnbindModal({ meter, binding, onClose }: { meter: MeterResponse; binding: NonNullable<MeterResponse['bindings']>[number]; onClose: () => void }) {
const [endedAt, setEndedAt] = useState(() => meter.ended_at ? toLocalDateTimeInputString(parseBackendTimestamp(meter.ended_at)) : toLocalDateTimeInputString())
const [error, setError] = useState<string | null>(null)
const close = useCloseBinding()
async function save() {
if (close.isPending) return
if (!isValidLocalDateTime(endedAt)) return setError('Choose a valid unbind time.')
const instant = localInstant(endedAt)
if (instant === null || instant <= parseBackendTimestamp(binding.started_at).getTime()) {
return setError('Unbind time must be strictly after the binding start.')
}
if (endedAt > toLocalDateTimeInputString()) return setError('A future unbind time is not allowed.')
if (meter.ended_at && instant > parseBackendTimestamp(meter.ended_at).getTime()) {
return setError('Unbind time must not be after the meter end.')
}
setError(null)
try { await close.mutateAsync({ uuid: binding.uuid, ended_at: endedAt }); onClose() } catch (err) { setError(apiErrorMessage(err, 'Could not unbind this source. History was not deleted.')) }
}
return <Modal opened onClose={onClose} title="Unbind source" data-testid={`unbind-modal-${binding.uuid}`}><form onSubmit={(event) => { event.preventDefault(); void save() }}><Stack>
<Alert color="blue">Unbinding closes this binding at the selected time. It never deletes binding history.</Alert>
<TextInput label="Unbind time" type="datetime-local" value={endedAt} onChange={(event) => setEndedAt(event.currentTarget.value)} required />
{error && <Alert color="red">{error}</Alert>}
<Group justify="flex-end"><Button type="button" variant="default" onClick={onClose}>Cancel</Button><Button type="submit" loading={close.isPending} disabled={close.isPending}>Unbind</Button></Group>
</Stack></form></Modal>
}
function TransferModal({ target, sourceBinding, meters, recovery, onClose }: { target: MeterResponse; sourceBinding: NonNullable<MeterResponse['bindings']>[number]; meters: MeterResponse[]; recovery: boolean; onClose: () => void }) {
const [sourceUuid, setSourceUuid] = useState<string | null>(null)
const [channelUuid, setChannelUuid] = useState<string | null>(null)
const [effectiveAt, setEffectiveAt] = useState(() => recovery ? toLocalDateTimeInputString(parseBackendTimestamp(target.started_at)) : toLocalDateTimeInputString())
const [error, setError] = useState<string | null>(null)
const sources = useSources(); const channels = useSourceChannels(sourceUuid)
const transfer = useTransferBinding()
const channelEligibilityAt = (channel: { uuid: string; unit: string }, value: string): Eligibility => {
if (channel.unit !== expectedUnit(target.commodity)) return { eligible: false, reason: `unit mismatch: ${channel.unit}; target needs ${expectedUnit(target.commodity)}` }
const start = localInstant(value)
if (start === null) return { eligible: false, reason: 'choose a valid effective time first' }
if (value > toLocalDateTimeInputString()) return { eligible: false, reason: 'future effective times are not allowed' }
if (!meterContainsInstant(target, start)) return { eligible: false, reason: 'effective time must be within the target meter epoch' }
// A same-meter transfer closes its source at `effective_at`. Equality
// would therefore create the forbidden zero-length [start, start)
// interval. Recovery closes at the old meter boundary instead, so it
// deliberately keeps the normal target-epoch rule and may be equal to
// the source binding's (much earlier) start.
if (!recovery && start <= parseBackendTimestamp(sourceBinding.started_at).getTime()) {
return { eligible: false, reason: 'same-meter transfer must be strictly after the source binding start' }
}
return channelIntervalEligibility(meters, channel.uuid, value, [sourceBinding.uuid])
}
const channelEligibility = (channel: { uuid: string; unit: string }) => channelEligibilityAt(channel, effectiveAt)
const options = channels.data?.items.map((channel) => {
const result = channelEligibility(channel)
return { value: channel.uuid, label: `${channel.label} (${channel.unit})${result.reason ? `${result.reason}` : ''}`, disabled: !result.eligible }
}) ?? []
const selectedChannel = channels.data?.items.find((channel) => channel.uuid === channelUuid)
const selectedEligible = selectedChannel !== undefined && channelEligibility(selectedChannel).eligible
const isFuture = effectiveAt > toLocalDateTimeInputString()
async function save() {
if (transfer.isPending) return
if (!channelUuid) return setError('Select a unit-compatible source channel.')
if (!isValidLocalDateTime(effectiveAt) || isFuture) return setError('Choose a valid non-future effective time.')
if (!selectedEligible) return setError('The selected channel is no longer eligible. Choose an available channel.')
setError(null)
try { await transfer.mutateAsync({ id: target.id, body: { from_binding_uuid: sourceBinding.uuid, to_source_channel_uuid: channelUuid, effective_at: effectiveAt } }); onClose() } catch (err) { setError(apiErrorMessage(err, 'Transfer failed. No partial source switch was saved.')) }
}
return <Modal opened onClose={onClose} title={recovery ? 'Recover stranded binding' : 'Transfer source binding'} data-testid={`transfer-modal-${sourceBinding.uuid}`}><form onSubmit={(event) => { event.preventDefault(); void save() }}><Stack>
<Alert color="blue">This is one atomic Transfer request: either the old binding closes and the new one opens together, or neither change is saved.</Alert>
{recovery && <Alert color="yellow">This binding is stranded on a closed meter. Recovery defaults to the new meter start. A later time is allowed, but creates an unbound gap before it.</Alert>}
{sources.isLoading && <Text size="sm">Loading sources</Text>}{sources.isError && <Alert color="red">Could not load sources. Retry after the connection recovers.</Alert>}
<Select label="Source" value={sourceUuid} onChange={(value) => { setSourceUuid(value); setChannelUuid(null) }} data={sources.data?.items.map((source) => ({ value: source.uuid, label: source.name })) ?? []} />
{sourceUuid && channels.isLoading && <Text size="sm">Loading source channels</Text>}{sourceUuid && channels.isError && <Alert color="red">Could not load source channels. Choose another source or retry.</Alert>}
<Select label="Source channel" value={channelUuid} onChange={setChannelUuid} description="Disabled channels name the specific unit or unrelated-open-interval conflict. The server also rejects ambiguity atomically." data={options} />
<TextInput label="Effective time" type="datetime-local" value={effectiveAt} onChange={(event) => { const value = event.currentTarget.value; setEffectiveAt(value); setChannelUuid((uuid) => { if (!isValidLocalDateTime(value)) return uuid; const channel = channels.data?.items.find((item) => item.uuid === uuid); return channel && !channelEligibilityAt(channel, value).eligible ? null : uuid }) }} required data-testid="transfer-effective-at" />
{recovery && effectiveAt && effectiveAt > toLocalDateTimeInputString(parseBackendTimestamp(target.started_at)) && <Alert color="yellow">Warning: this later time leaves an unbound gap from the new meter start until this transfer.</Alert>}
{isFuture && <Alert color="red">A future effective time is not allowed.</Alert>}
{error && <Alert color="red">{error}</Alert>}
<Group justify="flex-end"><Button type="button" variant="default" onClick={onClose}>Cancel</Button><Button type="submit" loading={transfer.isPending} disabled={transfer.isPending || (!!channelUuid && !selectedEligible)}>Transfer binding</Button></Group>
</Stack></form></Modal>
}
function CloseMeterModal({ meter, onClose }: { meter: MeterResponse; onClose: () => void }) {
const [endedAt, setEndedAt] = useState(() => toLocalDateTimeInputString())
const [error, setError] = useState<string | null>(null)
const close = useCloseMeter()
async function save() {
if (close.isPending) return
if (!isValidLocalDateTime(endedAt)) return setError('Choose a valid close time.')
const instant = localInstant(endedAt)
if (instant === null || instant <= parseBackendTimestamp(meter.started_at).getTime()) {
return setError('Close time must be strictly after the meter start.')
}
if (endedAt > toLocalDateTimeInputString()) return setError('A future close time is not allowed.')
setError(null)
try { await close.mutateAsync({ id: meter.id, ended_at: endedAt }); onClose() } catch (err) { setError(apiErrorMessage(err, 'Could not close this meter.')) }
}
return <Modal opened onClose={onClose} title={`Close Meter — ${meter.label}`} data-testid={`close-meter-modal-${meter.id}`}><form onSubmit={(event) => { event.preventDefault(); void save() }}><Stack>
<Alert color="yellow">Closing leaves no active {meter.commodity} meter. Every open binding on this meter closes at the same boundary.</Alert>
<TextInput label="Close time" type="datetime-local" value={endedAt} onChange={(event) => setEndedAt(event.currentTarget.value)} required />
{error && <Alert color="red">{error}</Alert>}
<Group justify="flex-end"><Button type="button" variant="default" onClick={onClose}>Cancel</Button><Button type="submit" color="red" loading={close.isPending} disabled={close.isPending}>Close meter</Button></Group>
</Stack></form></Modal>
}
// ---------------------------------------------------------------------------
// MeterManager — top-level
// ---------------------------------------------------------------------------
@@ -429,6 +796,7 @@ export function MeterManager() {
const [showDeclareForm, setShowDeclareForm] = useState(false)
const [editMeter, setEditMeter] = useState<MeterResponse | null>(null)
const [closeMeter, setCloseMeter] = useState<MeterResponse | null>(null)
const [recomputeNotice, setRecomputeNotice] = useState(false)
// ---------------------------------------------------------------------------
@@ -456,7 +824,7 @@ export function MeterManager() {
return (
<Stack gap="lg" data-testid="meter-manager">
<Group justify="space-between" align="center">
<Text fw={500}>Electricity Meters</Text>
<Text fw={500}>Meters</Text>
<Button
onClick={() => setShowDeclareForm(true)}
data-testid="meter-declare-button"
@@ -477,11 +845,12 @@ export function MeterManager() {
</Notification>
)}
<MeterTable meters={meters} onEdit={(m) => setEditMeter(m)} />
<MeterTable meters={meters} onEdit={(m) => setEditMeter(m)} onClose={(m) => setCloseMeter(m)} />
{/* Declare new meter */}
{showDeclareForm && (
<DeclareMeterForm
meters={meters}
onClose={() => setShowDeclareForm(false)}
onSaved={() => setShowDeclareForm(false)}
/>
@@ -498,6 +867,8 @@ export function MeterManager() {
}}
/>
)}
{closeMeter && <CloseMeterModal meter={closeMeter} onClose={() => setCloseMeter(null)} />}
</Stack>
)
}
+15
View File
@@ -0,0 +1,15 @@
import { describe, expect, it, vi, beforeEach } from 'vitest'
import { screen, waitFor } from '@testing-library/react'
import userEvent from '@testing-library/user-event'
import { renderWithProviders } from '../test-utils'
import { SourceForm } from './SourceForm'
const mockGet = vi.fn(); const mockPatch = vi.fn()
vi.mock('../api/client', () => ({ default: { GET: (...a: unknown[]) => mockGet(...a), POST: vi.fn(), PATCH: (...a: unknown[]) => mockPatch(...a), DELETE: vi.fn() }, ApiError: class ApiError extends Error { constructor(public status: number, public body: unknown) { super(`API error ${status}`) } }, registerLoginRedirect: vi.fn() }))
const profile = { kind: 'dsmr_mqtt', fields: [{ name: 'tls_enabled', value_type: 'bool', default: false }, { name: 'port', value_type: 'int', default: 1883 }, { name: 'topic', value_type: 'string', default: 'telegram' }, { name: 'password', value_type: 'string', secret: true }] }
const source = { uuid: 'source-1', name: 'DSMR', kind: 'dsmr_mqtt', enabled: true, config: { tls_enabled: true, port: 8883, topic: 'old', password: '********' } }
describe('SourceForm typed PATCH and secrets', () => {
beforeEach(() => { vi.clearAllMocks(); mockGet.mockResolvedValue({ data: { items: [profile] } }) })
it('preserves native bool/number/string and omits untouched masked secret', async () => { const user = userEvent.setup(); mockPatch.mockResolvedValue({ data: source }); renderWithProviders(<SourceForm source={source as never} onClose={vi.fn()} />); await user.click(await screen.findByRole('button', { name: 'Save Source' })); await waitFor(() => expect(mockPatch).toHaveBeenCalled()); expect(mockPatch).toHaveBeenCalledWith('/api/energy/sources/{source_uuid}', { params: { path: { source_uuid: 'source-1' } }, body: { name: 'DSMR', enabled: true, config: { tls_enabled: true, port: 8883, topic: 'old' } } }) })
it('sends a replacement secret and shows 422 detail', async () => { const user = userEvent.setup(); mockPatch.mockRejectedValue(new (await import('../api/client')).ApiError(422, { detail: 'invalid broker' })); renderWithProviders(<SourceForm source={source as never} onClose={vi.fn()} />); await user.type(await screen.findByLabelText('password'), 'new-secret'); await user.click(screen.getByRole('button', { name: 'Save Source' })); await waitFor(() => expect(mockPatch).toHaveBeenCalled()); expect(mockPatch.mock.calls[0][1].body.config).toMatchObject({ password: 'new-secret', tls_enabled: true, port: 8883 }); expect(await screen.findByText('invalid broker')).toBeInTheDocument() })
})
+33
View File
@@ -0,0 +1,33 @@
import { useState } from 'react'
import { Alert, Button, Checkbox, Group, Modal, Select, Stack, TextInput } from '@mantine/core'
import { ApiError } from '../api/client'
import { useCreateSource, useSourceProfiles, useUpdateSource, type MeterSourceResponse } from './hooks'
export function SourceForm({ source, onClose }: { source?: MeterSourceResponse; onClose: () => void }) {
const profiles = useSourceProfiles(); const create = useCreateSource(); const update = useUpdateSource()
const [name, setName] = useState(source?.name ?? ''); const [kind, setKind] = useState<string | null>(source?.kind ?? null)
const [enabled, setEnabled] = useState(source?.enabled ?? true); const [config, setConfig] = useState<Record<string, string | boolean | number>>({}); const [error, setError] = useState<string | null>(null)
const profile = profiles.data?.items.find((item) => item.kind === kind)
async function submit(e: React.FormEvent) { e.preventDefault(); setError(null); if (!name.trim() || !kind) return setError('Name and source type are required.')
const values: Record<string, unknown> = {}; profile?.fields.forEach((field) => {
const changed = Object.prototype.hasOwnProperty.call(config, field.name)
const raw = changed ? config[field.name] : (source?.config[field.name] ?? field.default ?? '')
// A masked secret is deliberately absent from edit PATCHes until the user
// explicitly enters a replacement; sending an empty/masked value is unsafe.
if (field.secret && source && (!changed || raw === '')) return
if (field.value_type === 'bool' || field.value_type === 'boolean') values[field.name] = typeof raw === 'boolean' ? raw : raw === 'true'
else if (field.value_type === 'int' || field.value_type === 'integer') values[field.name] = typeof raw === 'number' ? raw : Number(raw)
else values[field.name] = typeof raw === 'string' ? raw : String(raw)
})
try { if (source) await update.mutateAsync({ uuid: source.uuid, body: { name: name.trim(), enabled, config: values } }); else await create.mutateAsync({ name: name.trim(), kind, enabled, config: values }); onClose() } catch (err) { setError(err instanceof ApiError ? String((err.body as { detail?: string })?.detail ?? `Error ${err.status}`) : 'Could not save source.') }
}
return <Modal opened onClose={onClose} title={source ? 'Edit Source' : 'New Source'}><form onSubmit={submit}><Stack>
<TextInput label="Name" required value={name} onChange={(e) => setName(e.currentTarget.value)} />
{profiles.isLoading && <Alert color="blue">Loading source profiles</Alert>}{profiles.isError && <Alert color="red">Failed to load source profiles.</Alert>}
<Select label="Source type" required data={profiles.data?.items.map((p) => ({ value: p.kind, label: p.kind })) ?? []} value={kind} onChange={setKind} disabled={!!source} />
{kind === 'warmtelink_serial' && <Alert color="blue">Serial sources use <code>/dev/serial/by-id/</code>; 115200 7N1.</Alert>}
{profile?.fields.map((field) => field.value_type === 'bool' || field.value_type === 'boolean' ? <Checkbox key={field.name} label={field.name} checked={Boolean(config[field.name] ?? source?.config[field.name] ?? field.default ?? false)} onChange={(e) => setConfig({ ...config, [field.name]: e.currentTarget.checked })} /> : <TextInput key={field.name} label={field.name} required={field.required} type={field.secret ? 'password' : (field.value_type === 'int' || field.value_type === 'integer' ? 'number' : 'text')} placeholder={field.secret && source ? 'Stored secret unchanged when blank' : undefined} value={String(config[field.name] ?? (field.secret ? '' : source?.config[field.name] ?? field.default ?? ''))} onChange={(e) => setConfig({ ...config, [field.name]: e.currentTarget.value })} />)}
<Checkbox label="Enabled" checked={enabled} onChange={(e) => setEnabled(e.currentTarget.checked)} />
{error && <Alert color="red">{error}</Alert>}<Group justify="flex-end"><Button variant="default" onClick={onClose}>Cancel</Button><Button type="submit" loading={create.isPending || update.isPending}>Save Source</Button></Group>
</Stack></form></Modal>
}
@@ -0,0 +1,17 @@
import { describe, expect, it, vi, beforeEach } from 'vitest'
import { screen, waitFor } from '@testing-library/react'
import userEvent from '@testing-library/user-event'
import { renderWithProviders } from '../test-utils'
import { SourceManager } from './SourceManager'
const mockGet = vi.fn(); const mockPost = vi.fn(); const mockDelete = vi.fn()
vi.mock('../api/client', () => ({ default: { GET: (...a: unknown[]) => mockGet(...a), POST: (...a: unknown[]) => mockPost(...a), PATCH: vi.fn(), DELETE: (...a: unknown[]) => mockDelete(...a) }, ApiError: class ApiError extends Error { constructor(public status: number, public body: unknown) { super(`API error ${status}`) } }, registerLoginRedirect: vi.fn() }))
const source = { uuid: 's1', name: 'WarmteLink', kind: 'warmtelink_serial', enabled: true, status: 'online', config: {} }
describe('SourceManager API states', () => {
beforeEach(() => vi.clearAllMocks())
it('renders source list, detail and a completed bounded discovery refresh', async () => { const user = userEvent.setup(); mockGet.mockImplementation((path: string) => Promise.resolve({ data: path === '/api/energy/sources' ? { items: [source] } : path.includes('channels') ? { items: [] } : source })); mockPost.mockResolvedValue({ data: { status: 'completed' } }); renderWithProviders(<SourceManager />); await user.click(await screen.findByText('WarmteLink')); expect(await screen.findByText(/No channels discovered yet/)).toBeInTheDocument(); await user.click(screen.getByRole('button', { name: /Refresh discovered channels/ })); expect(await screen.findByText(/Discovery completed/)).toBeInTheDocument() })
it('explains that an enabled serial source reconnects automatically', async () => { const user = userEvent.setup(); mockGet.mockImplementation((path: string) => Promise.resolve({ data: path === '/api/energy/sources' ? { items: [{ ...source, status: 'error', last_error: 'serial unavailable' }] } : path.includes('channels') ? { items: [] } : { ...source, status: 'error', last_error: 'serial unavailable' } })); renderWithProviders(<SourceManager />); await user.click(await screen.findByText('WarmteLink')); expect(await screen.findAllByText(/reconnecting automatically/i)).not.toHaveLength(0); expect(screen.getByRole('button', { name: /Refresh discovered channels/ })).toBeInTheDocument() })
it('renders source list empty and error states', async () => { mockGet.mockResolvedValueOnce({ data: { items: [] } }); const { unmount } = renderWithProviders(<SourceManager />); expect(await screen.findByText(/No sources configured/)).toBeInTheDocument(); unmount(); mockGet.mockRejectedValueOnce(new Error('offline')); renderWithProviders(<SourceManager />); expect(await screen.findByText(/Failed to load sources/)).toBeInTheDocument() })
it('contains the wide four-column source table in a scroll area', async () => { mockGet.mockResolvedValue({ data: { items: [source] } }); renderWithProviders(<SourceManager />); expect(await screen.findByTestId('sources-table-scrollarea')).toBeInTheDocument(); expect(screen.getByTestId('sources-table')).toHaveStyle({ minWidth: '640px' }) })
it('safely deletes an unreferenced source and clears its selection', async () => { const user = userEvent.setup(); let items = [source]; mockGet.mockImplementation((path: string) => Promise.resolve({ data: path === '/api/energy/sources' ? { items } : path.includes('channels') ? { items: [] } : source })); mockDelete.mockImplementation(async () => { items = []; return { data: undefined } }); renderWithProviders(<SourceManager />); await user.click(await screen.findByText('WarmteLink')); await user.click(screen.getByRole('button', { name: 'Delete source' })); await waitFor(() => expect(mockDelete).toHaveBeenCalledWith('/api/energy/sources/{source_uuid}', { params: { path: { source_uuid: 's1' } } })); expect(await screen.findByText(/No sources configured/)).toBeInTheDocument(); expect(screen.queryByRole('button', { name: 'Delete source' })).not.toBeInTheDocument() })
it('keeps source visible and explains dependencies when deletion returns 409', async () => { const user = userEvent.setup(); mockGet.mockImplementation((path: string) => Promise.resolve({ data: path === '/api/energy/sources' ? { items: [source] } : path.includes('channels') ? { items: [] } : source })); const { ApiError } = await import('../api/client'); mockDelete.mockRejectedValue(new ApiError(409, { detail: 'dependent readings' })); renderWithProviders(<SourceManager />); await user.click(await screen.findByText('WarmteLink')); await user.click(screen.getByRole('button', { name: 'Delete source' })); expect(await screen.findByText(/dependent channels, readings, or meter bindings/)).toBeInTheDocument(); expect(screen.getByRole('button', { name: 'WarmteLink' })).toBeInTheDocument() })
})
+34
View File
@@ -0,0 +1,34 @@
import { useState } from 'react'
import { Alert, Badge, Button, Center, Group, Loader, Paper, ScrollArea, Stack, Table, Text } from '@mantine/core'
import { DsmrPanel } from './DsmrPanel'
import { SourceForm } from './SourceForm'
import { SourceReadings } from './SourceReadings'
import { ApiError } from '../api/client'
import { useDeleteSource, useDiscoverSource, useSource, useSourceChannels, useSources, type MeterSourceResponse } from './hooks'
import { formatLocalDateTime } from '../utils/datetime'
export function SourceManager() {
const sources = useSources(); const [selected, setSelected] = useState<string | null>(null); const [form, setForm] = useState<MeterSourceResponse | undefined | null>(null)
if (sources.isLoading) return <Center data-testid="sources-loading"><Loader /></Center>
if (sources.isError || !sources.data) return <Alert color="red">Failed to load sources.</Alert>
return <Stack data-testid="source-manager"><Group justify="space-between"><Text fw={600}>Sources</Text><Button onClick={() => setForm(undefined)}>New Source</Button></Group>
{sources.data.items.length === 0 ? <Text c="dimmed">No sources configured yet.</Text> : <ScrollArea data-testid="sources-table-scrollarea"><Table data-testid="sources-table" style={{ minWidth: 640 }}><Table.Thead><Table.Tr><Table.Th>Name</Table.Th><Table.Th>Type</Table.Th><Table.Th>Status</Table.Th><Table.Th>Last seen</Table.Th></Table.Tr></Table.Thead><Table.Tbody>{sources.data.items.map((source) => <Table.Tr key={source.uuid}><Table.Td><Button variant="subtle" onClick={() => setSelected(source.uuid)}>{source.name}</Button></Table.Td><Table.Td>{source.kind}</Table.Td><Table.Td><Badge color={source.enabled && source.status === 'online' ? 'green' : 'gray'}>{source.enabled ? source.status : 'disabled'}</Badge>{source.enabled && source.status === 'error' && <Text c="orange" size="xs">Reconnecting automatically.</Text>}{source.last_error && <Text c="red" size="xs">{source.last_error}</Text>}</Table.Td><Table.Td>{source.last_seen_at ? formatLocalDateTime(source.last_seen_at) : '—'}</Table.Td></Table.Tr>)}</Table.Tbody></Table></ScrollArea>}
{selected && <SourceDetail uuid={selected} onEdit={setForm} onDeleted={() => setSelected(null)} />}{form !== null && <SourceForm source={form} onClose={() => setForm(null)} />}
</Stack>
}
function SourceDetail({ uuid, onEdit, onDeleted }: { uuid: string; onEdit: (source: MeterSourceResponse) => void; onDeleted: () => void }) {
const source = useSource(uuid); const channels = useSourceChannels(uuid); const discover = useDiscoverSource(); const remove = useDeleteSource(); const [deleteError, setDeleteError] = useState<string | null>(null)
async function deleteSource() { setDeleteError(null); try { await remove.mutateAsync(uuid); onDeleted() } catch (err) { if (err instanceof ApiError && err.status === 409) setDeleteError('This source cannot be deleted because it still has dependent channels, readings, or meter bindings. Remove those dependencies first; no data was deleted.'); else setDeleteError('Could not delete source. No data was deleted.') } }
if (source.isLoading) return <Loader />; if (source.isError || !source.data) return <Alert color="red">Failed to load source.</Alert>
const detail = source.data!
const result = discover.data?.data
return <Paper withBorder p="md"><Stack><Group justify="space-between"><Text fw={600}>{detail.name}</Text><Group><Button variant="default" onClick={() => onEdit(detail)}>Edit</Button><Button loading={discover.isPending} onClick={() => discover.mutate(uuid)}>Refresh discovered channels</Button><Button color="red" variant="outline" loading={remove.isPending} onClick={deleteSource}>Delete source</Button></Group></Group>
{deleteError && <Alert color="red">{deleteError}</Alert>}
{detail.enabled && detail.status === 'error' && <Alert color="orange">The worker is reconnecting automatically. Refresh only requests a bounded status update; it does not start ingestion.</Alert>}
{discover.isPending && <Alert color="blue">Discovery pending</Alert>}{discover.isError && <Alert color="red">Discovery request failed. Check the source and try again.</Alert>}
{result && <Alert color={result.status === 'error' || result.status === 'timeout' ? 'red' : 'blue'}>Discovery {result.status}: {result.detail ?? (result.status === 'completed' ? 'Channels refreshed.' : 'Waiting for discovery.')}</Alert>}
{detail.kind.includes('dsmr') && <DsmrPanel />}
{channels.isLoading && <Loader />}{channels.isError && <Alert color="red">Failed to load channels.</Alert>}{channels.data?.items.length === 0 && <Text c="dimmed">No channels discovered yet.</Text>}
{channels.data?.items.map((channel) => <Stack key={channel.uuid} gap="xs"><Text>{channel.label} ({channel.unit}) suggestion: {channel.suggested_commodity ?? 'none'} (review before binding)</Text><Text size="sm">Latest: {channel.latest_value ?? '—'}; quality: {channel.latest_quality ?? 'unknown'}; bindings: {channel.binding_count}; meter IDs: {channel.bound_meter_ids.length ? channel.bound_meter_ids.join(', ') : 'none'}</Text><SourceReadings sourceUuid={uuid} channel={channel} /></Stack>)}
</Stack></Paper>
}
@@ -0,0 +1,13 @@
import { describe, expect, it, vi, beforeEach } from 'vitest'
import { screen, waitFor } from '@testing-library/react'
import { renderWithProviders } from '../test-utils'
import { SourceReadings } from './SourceReadings'
const mockGet = vi.fn()
vi.mock('../api/client', () => ({ default: { GET: (...a: unknown[]) => mockGet(...a), POST: vi.fn(), PATCH: vi.fn(), DELETE: vi.fn() }, ApiError: class extends Error {}, registerLoginRedirect: vi.fn() }))
const channel = { uuid: 'channel-1', label: 'Heat', unit: 'GJ', latest_value: '1.2', latest_quality: 'unverifiable' }
describe('SourceReadings', () => {
beforeEach(() => vi.clearAllMocks())
it('renders quality and history from the mocked channel API', async () => { mockGet.mockResolvedValue({ data: { items: [{ recorded_at: '2026-08-01T10:00:00Z', value: '1.1', quality: 'unverifiable' }] } }); renderWithProviders(<SourceReadings sourceUuid="s1" channel={channel as never} />); expect(await screen.findByText(/shown for review, not marked verified/)).toBeInTheDocument(); await waitFor(() => expect(screen.getByText('1.1')).toBeInTheDocument()) })
it('renders an API error and empty history', async () => { mockGet.mockRejectedValueOnce(new Error('down')); const { unmount } = renderWithProviders(<SourceReadings sourceUuid="s1" channel={channel as never} />); expect(await screen.findByText('Failed to load channel history.')).toBeInTheDocument(); unmount(); mockGet.mockResolvedValue({ data: { items: [] } }); renderWithProviders(<SourceReadings sourceUuid="s1" channel={channel as never} />); await waitFor(() => expect(screen.getByText('No channel history yet.')).toBeInTheDocument()) })
it('keeps the latest quality explanation visible while history is loading or fails', async () => { mockGet.mockImplementationOnce(() => new Promise(() => {})); const { unmount } = renderWithProviders(<SourceReadings sourceUuid="s1" channel={channel as never} />); expect(screen.getByText('Quality: unverifiable')).toBeInTheDocument(); expect(screen.getByText(/shown for review, not marked verified/)).toBeInTheDocument(); unmount(); mockGet.mockRejectedValueOnce(new Error('down')); renderWithProviders(<SourceReadings sourceUuid="s1" channel={channel as never} />); expect(await screen.findByText('Failed to load channel history.')).toBeInTheDocument(); expect(screen.getByText('Quality: unverifiable')).toBeInTheDocument(); expect(screen.getByText(/shown for review, not marked verified/)).toBeInTheDocument() })
})
+12
View File
@@ -0,0 +1,12 @@
import { Alert, Badge, Center, Loader, Stack, Table, Text } from '@mantine/core'
import { useChannelReadings, type MeterSourceChannelResponse } from './hooks'
import { formatLocalDateTime } from '../utils/datetime'
export function SourceReadings({ sourceUuid, channel }: { sourceUuid: string; channel: MeterSourceChannelResponse }) {
const query = useChannelReadings(sourceUuid, channel.uuid)
return <Stack gap="xs"><Text fw={500}>{channel.label} latest {channel.latest_value ?? '—'} {channel.unit}</Text>
<Text size="sm">Quality: {channel.latest_quality ?? 'unknown'}</Text>
{channel.latest_quality === 'unverifiable' && <Alert color="yellow">This reading is unverifiable: it is shown for review, not marked verified.</Alert>}
{query.isLoading ? <Center><Loader /></Center> : query.isError || !query.data ? <Alert color="red">Failed to load channel history.</Alert> : query.data.items.length === 0 ? <Text c="dimmed">No channel history yet.</Text> : <Table><Table.Thead><Table.Tr><Table.Th>Recorded</Table.Th><Table.Th>Value</Table.Th><Table.Th>Quality</Table.Th></Table.Tr></Table.Thead><Table.Tbody>{query.data.items.map((row) => <Table.Tr key={row.recorded_at}><Table.Td>{formatLocalDateTime(row.recorded_at)}</Table.Td><Table.Td>{row.value ?? '—'}</Table.Td><Table.Td><Badge color={row.quality === 'unverifiable' ? 'yellow' : 'gray'}>{row.quality ?? 'unknown'}</Badge></Table.Td></Table.Tr>)}</Table.Tbody></Table>}
</Stack>
}
+48
View File
@@ -14,6 +14,7 @@
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest'
import { screen, waitFor, fireEvent } from '@testing-library/react'
import userEvent from '@testing-library/user-event'
import { renderWithProviders } from '../test-utils'
// ---------------------------------------------------------------------------
@@ -244,6 +245,53 @@ describe('TibberPrices', () => {
expect(screen.getByTestId('tariff-sell-dal')).toHaveTextContent('0.0900')
})
it('keeps thermal prices scoped and renders the complete D11 Decimal snapshot with units', async () => {
const user = userEvent.setup()
const thermal = {
kind: 'district_heating', currency: 'EUR', points: [], tariff: null,
contract_version_id: 42, effective_from: '2026-01-01T00:00:00Z', effective_to: '2026-12-31T00:00:00Z',
values: {
variable: {
heating: '20.123456789123456789', hot_water_heating: '8.200000000000000001',
hot_water: '1.234567890123456789', hot_water_tax: '0.456789012345678901',
},
standing: {
heating_network: '100.000000000000000001', metering: '0', delivery_set: '20.2',
hot_water_network: '30.3', other: '40.4',
},
},
}
mockGet.mockImplementation((_path: string, options?: { params?: { query?: { scope?: string } } }) =>
Promise.resolve({ data: options?.params?.query?.scope === 'thermal'
? thermal
: { kind: 'manual', currency: 'EUR', points: [], tariff: { buy_dal: 0.1, buy_normal: 0.2, sell_dal: 0.03, sell_normal: 0.04 } } }),
)
renderWithProviders(<TibberPrices />)
await waitFor(() => expect(screen.getByTestId('manual-tariff-table')).toBeInTheDocument())
await user.click(screen.getByText('Thermal'))
await waitFor(() => expect(screen.getByTestId('thermal-price-snapshot')).toBeInTheDocument())
const snapshot = screen.getByTestId('thermal-price-snapshot')
expect(snapshot).toHaveTextContent('Version 42')
expect(snapshot).toHaveTextContent('effective 2026-01-01T00:00:00Z to 2026-12-31T00:00:00Z')
expect(snapshot).toHaveTextContent('not a 15-minute market spot price')
expect(snapshot).toHaveTextContent('20.123456789123456789 EUR/GJ')
expect(snapshot).toHaveTextContent('8.200000000000000001 EUR/m³')
expect(snapshot).toHaveTextContent('1.234567890123456789 EUR/m³')
expect(snapshot).toHaveTextContent('0.456789012345678901 EUR/m³')
for (const value of Object.values(thermal.values.standing)) {
expect(snapshot).toHaveTextContent(`${value} EUR/year`)
}
expect(mockGet).toHaveBeenCalledWith('/api/energy/prices', expect.objectContaining({
params: { query: expect.objectContaining({ scope: 'thermal' }) },
}))
await user.click(screen.getByText('Electricity'))
await waitFor(() => expect(screen.getByTestId('manual-tariff-table')).toBeInTheDocument())
expect(screen.queryByTestId('thermal-price-snapshot')).not.toBeInTheDocument()
})
it('marks the currently active price slot with a dot and a caption', async () => {
installChartSize()
+56 -11
View File
@@ -22,6 +22,7 @@ import {
Badge,
Group,
Paper,
SegmentedControl,
} from '@mantine/core'
import {
LineChart,
@@ -34,7 +35,8 @@ import {
ReferenceDot,
ResponsiveContainer,
} from 'recharts'
import { useEnergyPrices } from './hooks'
import { useQuery } from '@tanstack/react-query'
import apiClient from '../api/client'
import { formatLocalDate, formatLocalTime, parseBackendTimestamp } from '../utils/datetime'
const BUY_COLOR = '#2196f3'
@@ -46,6 +48,18 @@ const FALLBACK_SLOT_MS = 60 * 60 * 1000
/** How often the "current price" marker re-evaluates which slot is active. */
const NOW_TICK_MS = 30 * 1000
/** D11 thermal profile units. The API snapshot is Decimal strings, while the
* currency comes from its contract metadata. Keep this display-only: no rates
* are derived or prefilled in the browser. */
function thermalUnit(section: string, key: string, currency: string): string {
if (section === 'standing') return `${currency}/year`
if (key === 'heating') return `${currency}/GJ`
if (key === 'hot_water_heating' || key === 'hot_water' || key === 'hot_water_tax') {
return `${currency}/m³`
}
return currency
}
// ---------------------------------------------------------------------------
// Time range helpers
// ---------------------------------------------------------------------------
@@ -288,46 +302,59 @@ function ManualTariffTable({ tariff, currency }: ManualTariffTableProps) {
// ---------------------------------------------------------------------------
export function TibberPrices() {
const [scope, setScope] = useState<'electricity' | 'thermal'>('electricity')
const start = getTodayStart()
const end = getTomorrowEnd()
const { data, isLoading, isError } = useQuery({
queryKey: ['energy-prices', scope, start, end],
queryFn: async () => {
const res = await apiClient.GET('/api/energy/prices', { params: { query: { scope, start, end } } })
return res.data
},
})
const { data, isLoading, isError } = useEnergyPrices(start, end)
const selector = (
<SegmentedControl
value={scope}
onChange={(value) => setScope(value as 'electricity' | 'thermal')}
data={[{ label: 'Electricity', value: 'electricity' }, { label: 'Thermal', value: 'thermal' }]}
data-testid="prices-scope-selector"
/>
)
if (isLoading) {
return (
<Center py="xl" data-testid="prices-loading">
<Loader />
</Center>
<Stack><Group><Text fw={500}>Energy Prices</Text>{selector}</Group><Center py="xl" data-testid="prices-loading"><Loader /></Center></Stack>
)
}
if (isError) {
return (
<Alert color="red" data-testid="prices-error">
<Stack><Group><Text fw={500}>Energy Prices</Text>{selector}</Group><Alert color="red" data-testid="prices-error">
Failed to load energy prices. Please refresh.
</Alert>
</Alert></Stack>
)
}
if (!data) {
return (
<Alert color="gray" data-testid="prices-no-data">
<Stack><Group><Text fw={500}>Energy Prices</Text>{selector}</Group><Alert color="gray" data-testid="prices-no-data">
No pricing data available.
</Alert>
</Alert></Stack>
)
}
// No active contract
if (!data.kind) {
return (
<Paper withBorder p="md" data-testid="prices-no-contract">
<Stack><Group><Text fw={500}>Energy Prices</Text>{selector}</Group><Paper withBorder p="md" data-testid="prices-no-contract">
<Stack gap="xs">
<Text fw={500}>No active contract</Text>
<Text size="sm" c="dimmed">
Activate an energy contract on the Contracts tab to see pricing data.
</Text>
</Stack>
</Paper>
</Paper></Stack>
)
}
@@ -337,6 +364,7 @@ export function TibberPrices() {
<Stack gap="lg" data-testid="tibber-prices">
<Group gap="sm" align="center">
<Text fw={500}>Energy Prices</Text>
{selector}
<Badge variant="outline" size="sm">
{data.kind}
</Badge>
@@ -364,6 +392,23 @@ export function TibberPrices() {
Manual tariff data not available.
</Alert>
)}
{scope === 'thermal' && data.values && (
<Paper withBorder p="md" data-testid="thermal-price-snapshot">
<Stack gap="xs">
<Text fw={500}>Thermal contract snapshot</Text>
<Text size="sm">Version {data.contract_version_id ?? '—'} · effective {data.effective_from ?? '—'} to {data.effective_to ?? 'open'}</Text>
<Text size="sm" c="dimmed">This is a contract snapshot, not a 15-minute market spot price.</Text>
{Object.entries(data.values).map(([section, values]) => (
<Text size="sm" key={section} data-testid={`thermal-price-section-${section}`}>
{section}: {Object.entries(values).map(([key, value]) =>
`${key} ${value} ${thermalUnit(section, key, currency)}`,
).join(', ')}
</Text>
))}
</Stack>
</Paper>
)}
</Stack>
)
}
+11
View File
@@ -150,6 +150,17 @@ describe('useCreateContract', () => {
expect(mockPost).toHaveBeenCalledWith('/api/energy/contracts', { body })
})
it('preserves the caller-selected thermal scope in the typed create payload', async () => {
mockPost.mockResolvedValue({ data: { id: 9 } })
const { Wrapper } = makeWrapper()
const { useCreateContract } = await import('./hooks')
const { result } = renderHook(() => useCreateContract(), { wrapper: Wrapper })
await act(async () => {
await result.current.mutateAsync({ name: 'Heat', kind: 'district_heating', scope: 'thermal', currency: 'EUR', values: {} })
})
expect(mockPost).toHaveBeenCalledWith('/api/energy/contracts', expect.objectContaining({ body: expect.objectContaining({ scope: 'thermal' }) }))
})
})
describe('useEnergyPrices', () => {
+52
View File
@@ -81,6 +81,7 @@ function makeWrapper() {
// ---------------------------------------------------------------------------
describe('useDevices', () => {
// Source hooks use the same typed client and QueryClient invalidation boundary.
beforeEach(() => vi.clearAllMocks())
it('calls GET /api/modbus/devices and returns device list', async () => {
@@ -98,6 +99,57 @@ describe('useDevices', () => {
})
})
describe('useDeclareMeter source binding invalidation', () => {
beforeEach(() => vi.clearAllMocks())
it('invalidates every cache affected by atomic meter and binding creation', async () => {
mockPost.mockResolvedValue({ data: { id: 9 } })
const { qc, Wrapper } = makeWrapper()
const affectedKeys = [
['energy-meters'], ['energy-source-channels'], ['energy-meter-bindings', 9],
['energy-sources'], ['energy-source', 'source-1'], ['energy-channel-readings', 'source-1', 'channel-1'],
['expose-catalog'], ['energy-costs', 'electricity'], ['energy-costs-summary', 'electricity'],
['meter-costs', 'thermal', 'month'], ['meter-cost-summary', 'thermal'],
]
for (const queryKey of affectedKeys) qc.setQueryData(queryKey, { cached: true })
const { useDeclareMeter } = await import('./hooks')
const { result } = renderHook(() => useDeclareMeter(), { wrapper: Wrapper })
await act(async () => { await result.current.mutateAsync({ label: 'Heat', commodity: 'heating', started_at: '2026-08-01T00:00:00Z', reason: 'initial', source_channel_uuid: 'channel-1' } as never) })
expect(mockPost).toHaveBeenCalledWith('/api/energy/meters', expect.objectContaining({ body: expect.objectContaining({ source_channel_uuid: 'channel-1' }) }))
for (const queryKey of affectedKeys) expect(qc.getQueryState(queryKey)?.isInvalidated).toBe(true)
})
})
describe('meter lifecycle mutations', () => {
beforeEach(() => vi.clearAllMocks())
it.each([
['Declare auto-handoff', 'post', async (hooks: typeof import('./hooks')) => hooks.useDeclareMeter, { label: 'swap', commodity: 'electricity', started_at: '2026-08-24T12:34', reason: 'meter_swap' }],
['Close meter', 'post', async (hooks: typeof import('./hooks')) => hooks.useCloseMeter, { id: 9, ended_at: '2026-08-24T12:34' }],
['Unbind', 'patch', async (hooks: typeof import('./hooks')) => hooks.useCloseBinding, { uuid: 'old', ended_at: '2026-08-24T12:34' }],
['Transfer', 'post', async (hooks: typeof import('./hooks')) => hooks.useTransferBinding, { id: 10, body: { from_binding_uuid: 'old', to_source_channel_uuid: 'new', effective_at: '2026-08-24T12:34' } }],
['Direct bind', 'post', async (hooks: typeof import('./hooks')) => hooks.useCreateBinding, { id: 10, body: { source_channel_uuid: 'new', started_at: '2026-08-24T12:34' } }],
['Update electricity start', 'patch', async (hooks: typeof import('./hooks')) => hooks.useUpdateMeter, { id: 10, body: { started_at: '2026-08-24T12:34' } }],
['Update heating label', 'patch', async (hooks: typeof import('./hooks')) => hooks.useUpdateMeter, { id: 11, body: { label: 'Heating meter' } }],
['Update hot water start', 'patch', async (hooks: typeof import('./hooks')) => hooks.useUpdateMeter, { id: 12, body: { started_at: '2026-08-24T12:34' } }],
])('%s invalidates every lifecycle view using a fresh QueryClient', async (_name, method, getHook, payload) => {
mockPost.mockResolvedValue({ data: {} })
mockPatch.mockResolvedValue({ data: {} })
const { qc, Wrapper } = makeWrapper()
const affectedKeys = [
['energy-meters'], ['energy-sources'], ['energy-source', 'source-1'], ['energy-source-channels'], ['energy-meter-bindings'],
['energy-channel-readings', 'source-1', 'channel-1'], ['energy-costs', 'electricity'], ['energy-costs-summary', 'electricity'],
['meter-costs', 'thermal', 'month'], ['meter-cost-summary', 'thermal'], ['expose-catalog'],
]
for (const key of affectedKeys) qc.setQueryData(key, { cached: true })
const hooks = await import('./hooks')
const useHook = await getHook(hooks)
const result = renderHook(() => useHook(), { wrapper: Wrapper })
await act(async () => { await result.result.current.mutateAsync(payload as never) })
expect(method === 'post' ? mockPost : mockPatch).toHaveBeenCalled()
for (const key of affectedKeys) expect(qc.getQueryState(key)?.isInvalidated).toBe(true)
})
})
describe('useProfiles', () => {
beforeEach(() => vi.clearAllMocks())
+51 -12
View File
@@ -237,6 +237,15 @@ export type SummaryResponse = components['schemas']['SummaryResponse']
export type DsmrLatestResponse = components['schemas']['DsmrLatestResponse']
export type TibberTestResponse = components['schemas']['TibberTestResponse']
export type TibberTestPriceSchema = components['schemas']['TibberTestPriceSchema']
export type SourceProfileResponse = components['schemas']['SourceProfileResponse']
export type MeterSourceResponse = components['schemas']['MeterSourceResponse']
export type MeterSourceCreate = components['schemas']['MeterSourceCreate']
export type MeterSourcePatch = components['schemas']['MeterSourcePatch']
export type MeterSourceChannelResponse = components['schemas']['MeterSourceChannelResponse']
export type BindingResponse = components['schemas']['BindingResponse']
export type BindingCreate = components['schemas']['BindingCreate']
export type BindingTransferRequest = components['schemas']['BindingTransferRequest']
export type MeterCloseRequest = components['schemas']['MeterCloseRequest']
// ---------------------------------------------------------------------------
// Query: list all energy contracts
@@ -470,12 +479,9 @@ export function useDeclareMeter() {
return useMutation({
mutationFn: (body: MeterDeclareRequest) =>
apiClient.POST('/api/energy/meters', { body }),
onSuccess: () => {
void qc.invalidateQueries({ queryKey: ['energy-meters'] })
// Invalidate cost-related queries: a new meter may trigger recompute server-side.
void qc.invalidateQueries({ queryKey: ['energy-costs'] })
void qc.invalidateQueries({ queryKey: ['energy-costs-summary'] })
},
// A meter_swap can hand a binding over even when the optional channel was
// omitted from this request, so every lifecycle write shares this boundary.
onSuccess: () => invalidateLifecycleQueries(qc),
})
}
@@ -491,15 +497,48 @@ export function useUpdateMeter() {
params: { path: { meter_id: id } },
body,
}),
onSuccess: () => {
void qc.invalidateQueries({ queryKey: ['energy-meters'] })
// Retroactive started_at correction triggers recompute server-side.
void qc.invalidateQueries({ queryKey: ['energy-costs'] })
void qc.invalidateQueries({ queryKey: ['energy-costs-summary'] })
},
onSuccess: () => invalidateLifecycleQueries(qc),
})
}
// Source → channel → binding hooks. These deliberately use the generated
// OpenAPI types; UI suggestions remain just suggestions until a user binds one.
export function useSourceProfiles() {
return useQuery({ queryKey: ['energy-source-profiles'], queryFn: async () => {
const res = await apiClient.GET('/api/energy/source-profiles'); return res.data
}, staleTime: 5 * 60 * 1000 })
}
export function useSources() {
return useQuery({ queryKey: ['energy-sources'], queryFn: async () => {
const res = await apiClient.GET('/api/energy/sources'); return res.data
}, refetchInterval: 5_000 })
}
export function useSource(uuid: string | null) {
return useQuery({ queryKey: ['energy-source', uuid], enabled: !!uuid, queryFn: async () => {
const res = await apiClient.GET('/api/energy/sources/{source_uuid}', { params: { path: { source_uuid: uuid! } } }); return res.data
}, refetchInterval: 3_000 })
}
function invalidateLifecycleQueries(qc: ReturnType<typeof useQueryClient>) {
void qc.invalidateQueries({ queryKey: ['energy-sources'] }); void qc.invalidateQueries({ queryKey: ['energy-source'] });
void qc.invalidateQueries({ queryKey: ['energy-source-channels'] }); void qc.invalidateQueries({ queryKey: ['energy-meters'] });
void qc.invalidateQueries({ queryKey: ['energy-channel-readings'] }); void qc.invalidateQueries({ queryKey: ['energy-meter-bindings'] })
void qc.invalidateQueries({ queryKey: ['expose-catalog'] })
void qc.invalidateQueries({ queryKey: ['energy-costs'] }); void qc.invalidateQueries({ queryKey: ['energy-costs-summary'] })
void qc.invalidateQueries({ queryKey: ['meter-costs', 'thermal'] })
void qc.invalidateQueries({ queryKey: ['meter-cost-summary', 'thermal'] })
}
export function useCreateSource() { const qc = useQueryClient(); return useMutation({ mutationFn: (body: MeterSourceCreate) => apiClient.POST('/api/energy/sources', { body }), onSuccess: () => invalidateLifecycleQueries(qc) }) }
export function useUpdateSource() { const qc = useQueryClient(); return useMutation({ mutationFn: ({ uuid, body }: { uuid: string; body: MeterSourcePatch }) => apiClient.PATCH('/api/energy/sources/{source_uuid}', { params: { path: { source_uuid: uuid } }, body }), onSuccess: () => invalidateLifecycleQueries(qc) }) }
export function useDeleteSource() { const qc = useQueryClient(); return useMutation({ mutationFn: (uuid: string) => apiClient.DELETE('/api/energy/sources/{source_uuid}', { params: { path: { source_uuid: uuid } } }), onSuccess: () => invalidateLifecycleQueries(qc) }) }
export function useDiscoverSource() { const qc = useQueryClient(); return useMutation({ mutationFn: (uuid: string) => apiClient.POST('/api/energy/sources/{source_uuid}/discover', { params: { path: { source_uuid: uuid } } }), onSuccess: () => invalidateLifecycleQueries(qc) }) }
export function useSourceChannels(uuid: string | null) { return useQuery({ queryKey: ['energy-source-channels', uuid], enabled: !!uuid, queryFn: async () => { const res = await apiClient.GET('/api/energy/sources/{source_uuid}/channels', { params: { path: { source_uuid: uuid! } } }); return res.data }, refetchInterval: 3_000 }) }
export function useChannelReadings(sourceUuid: string | null, channelUuid: string | null) { return useQuery({ queryKey: ['energy-channel-readings', sourceUuid, channelUuid], enabled: !!sourceUuid && !!channelUuid, queryFn: async () => { const res = await apiClient.GET('/api/energy/sources/{source_uuid}/channels/{channel_uuid}/readings', { params: { path: { source_uuid: sourceUuid!, channel_uuid: channelUuid! }, query: { limit: 60 } } }); return res.data }, refetchInterval: 5_000 }) }
export function useMeterBindings(id: number | null) { return useQuery({ queryKey: ['energy-meter-bindings', id], enabled: id != null, queryFn: async () => { const res = await apiClient.GET('/api/energy/meters/{meter_id}/bindings', { params: { path: { meter_id: id! } } }); return res.data } }) }
export function useCreateBinding() { const qc = useQueryClient(); return useMutation({ mutationFn: ({ id, body }: { id: number; body: BindingCreate }) => apiClient.POST('/api/energy/meters/{meter_id}/bindings', { params: { path: { meter_id: id } }, body }), onSuccess: () => invalidateLifecycleQueries(qc) }) }
export function useCloseBinding() { const qc = useQueryClient(); return useMutation({ mutationFn: ({ uuid, ended_at }: { uuid: string; ended_at: string }) => apiClient.PATCH('/api/energy/bindings/{binding_uuid}', { params: { path: { binding_uuid: uuid } }, body: { ended_at } }), onSuccess: () => invalidateLifecycleQueries(qc) }) }
export function useCloseMeter() { const qc = useQueryClient(); return useMutation({ mutationFn: ({ id, ended_at }: { id: number; ended_at: string }) => apiClient.POST('/api/energy/meters/{meter_id}/close', { params: { path: { meter_id: id } }, body: { ended_at } satisfies MeterCloseRequest }), onSuccess: () => invalidateLifecycleQueries(qc) }) }
export function useTransferBinding() { const qc = useQueryClient(); return useMutation({ mutationFn: ({ id, body }: { id: number; body: BindingTransferRequest }) => apiClient.POST('/api/energy/meters/{meter_id}/bindings/transfer', { params: { path: { meter_id: id } }, body }), onSuccess: () => invalidateLifecycleQueries(qc) }) }
// ---------------------------------------------------------------------------
// Query: time-range readings for a device (window + limit — never full-table)
// ---------------------------------------------------------------------------
+14
View File
@@ -15,6 +15,7 @@
import { describe, it, expect, vi, beforeEach } from 'vitest'
import { screen, waitFor, fireEvent } from '@testing-library/react'
import userEvent from '@testing-library/user-event'
import { renderWithProviders } from '../test-utils'
import { EnergyPage } from './EnergyPage'
@@ -135,6 +136,19 @@ describe('EnergyPage — device list', () => {
})
})
describe('EnergyPage — M8 navigation labels', () => {
beforeEach(() => { vi.clearAllMocks(); setupDefaultMocks() })
it('has keyboard-operable Sources, Modbus Devices and Meters tabs, without a DSMR tab', async () => {
const user = userEvent.setup(); renderEnergy()
const sources = screen.getByRole('tab', { name: 'Sources' })
expect(screen.getByRole('tab', { name: 'Modbus Devices' })).toBeInTheDocument()
expect(screen.getByRole('tab', { name: 'Meters' })).toBeInTheDocument()
expect(screen.queryByRole('tab', { name: /^DSMR$/ })).not.toBeInTheDocument()
sources.focus(); await user.keyboard('{Enter}')
expect(await screen.findByTestId('panel-sources')).toBeInTheDocument()
})
})
describe('EnergyPage — create device', () => {
beforeEach(() => {
vi.clearAllMocks()
+10 -9
View File
@@ -45,7 +45,7 @@ import { ContractManager } from '../energy/ContractManager'
import { MeterManager } from '../energy/MeterManager'
import { TibberPrices } from '../energy/TibberPrices'
import { CostView } from '../energy/CostView'
import { DsmrPanel } from '../energy/DsmrPanel'
import { SourceManager } from '../energy/SourceManager'
import type { ModbusDevice, ModbusTestReadResponse, MetricInfo } from '../energy/hooks'
import { ApiError } from '../api/client'
import { formatMetricValue } from '../energy/format'
@@ -603,7 +603,7 @@ function DevicesTab() {
return (
<Stack gap="lg">
<Group justify="space-between" align="center">
<Title order={2}>Energy Devices</Title>
<Title order={2}>Energy Modbus Devices</Title>
<Button onClick={openCreate} data-testid="device-new-button">
New Device
</Button>
@@ -659,8 +659,11 @@ export function EnergyPage() {
<Container size="xl" pt="xl" pb="xl" data-testid="energy-page">
<Tabs defaultValue="devices">
<Tabs.List mb="lg">
<Tabs.Tab value="sources" data-testid="tab-sources">
Sources
</Tabs.Tab>
<Tabs.Tab value="devices" data-testid="tab-devices">
Devices
Modbus Devices
</Tabs.Tab>
<Tabs.Tab value="meters" data-testid="tab-meters">
Meters
@@ -674,11 +677,12 @@ export function EnergyPage() {
<Tabs.Tab value="costs" data-testid="tab-costs">
Costs
</Tabs.Tab>
<Tabs.Tab value="dsmr" data-testid="tab-dsmr">
DSMR
</Tabs.Tab>
</Tabs.List>
<Tabs.Panel value="sources" data-testid="panel-sources">
<SourceManager />
</Tabs.Panel>
<Tabs.Panel value="devices" data-testid="panel-devices">
<DevicesTab />
</Tabs.Panel>
@@ -699,9 +703,6 @@ export function EnergyPage() {
<CostView />
</Tabs.Panel>
<Tabs.Panel value="dsmr" data-testid="panel-dsmr">
<DsmrPanel />
</Tabs.Panel>
</Tabs>
</Container>
)
+2350 -2
View File
File diff suppressed because it is too large Load Diff
+1513 -4
View File
File diff suppressed because it is too large Load Diff
+1 -1
View File
@@ -15,7 +15,7 @@ if str(PROJECT_ROOT) not in sys.path:
from app.config import get_settings
APP_BASELINE_REVISION = "20260625_14_meter_uuid"
APP_BASELINE_REVISION = "20260822_19_meter_cost_periods"
class AppDatabaseAdoptionError(RuntimeError):
+11 -185
View File
@@ -3,195 +3,15 @@
from __future__ import annotations
import argparse
from dataclasses import dataclass
from decimal import Decimal, InvalidOperation
from enum import StrEnum
import errno
import re
import sys
import time
from typing import BinaryIO, Callable, TextIO
import serial
from app.integrations.p1 import IntegrityStatus, ObisField, P1Telegram, TelegramFramer, parse_telegram
_OBIS_LINE = re.compile(r"^(?P<code>\d+-\d+:\d+\.\d+\.\d+)(?P<values>(?:\([^)]*\))*)$")
_NUMBER_WITH_UNIT = re.compile(r"^(?P<number>[+-]?\d+(?:\.\d+)?)(?:\*(?P<unit>.+))?$")
_CHANNEL_OBIS = re.compile(r"^0-(?P<channel>[1-9]\d*):(24|96)\.")
class IntegrityStatus(StrEnum):
"""Whether a frame has a verifiable standard DSMR checksum."""
VALID = "valid"
INVALID = "invalid"
UNVERIFIABLE = "unverifiable"
@dataclass(frozen=True)
class ObisField:
"""One OBIS line, including values not understood by this proof of concept."""
code: str
raw_values: tuple[str, ...]
value: Decimal | None = None
unit: str | None = None
@dataclass(frozen=True)
class P1Channel:
"""Fields associated with one M-Bus channel, discovered from its OBIS code."""
number: int
device_type: str | None
equipment_id: str | None
readings: tuple[ObisField, ...]
@dataclass(frozen=True)
class P1Telegram:
"""A parsed telegram while retaining its raw framing and all OBIS fields."""
raw: bytes
header: bytes
footer: bytes
integrity: IntegrityStatus
integrity_reason: str
timestamp: str | None
fields: tuple[ObisField, ...]
channels: tuple[P1Channel, ...]
def dsmr_crc16(data: bytes) -> int:
"""Return the DSMR CRC-16 over *data* (normally from ``/`` through ``!``)."""
crc = 0
for byte in data:
crc ^= byte
for _ in range(8):
crc = (crc >> 1) ^ 0xA001 if crc & 1 else crc >> 1
return crc & 0xFFFF
class TelegramFramer:
"""Incrementally extract newline-terminated telegrams from byte chunks.
A standard telegram starts with ``/``. WarmteLink's observed telegrams do
not, so a non-standard frame is retained from the current buffer start until
its ``!`` footer line instead of fabricating a standard header.
"""
def __init__(self) -> None:
self._buffer = bytearray()
def feed(self, chunk: bytes) -> list[bytes]:
"""Append *chunk* and return every complete frame now available."""
self._buffer.extend(chunk)
frames: list[bytes] = []
while (bang := self._buffer.find(b"!")) >= 0:
newline = self._buffer.find(b"\n", bang)
if newline < 0:
break
standard_start = self._buffer.find(b"/")
start = standard_start if 0 <= standard_start < bang else 0
frames.append(bytes(self._buffer[start : newline + 1]))
del self._buffer[: newline + 1]
return frames
def parse_telegram(frame: bytes) -> P1Telegram:
"""Parse a complete frame without guessing missing DSMR framing bytes."""
bang = frame.find(b"!")
if bang < 0:
raise ValueError("telegram has no footer marker '!'")
body = frame[:bang]
footer = frame[bang + 1 :].rstrip(b"\r\n")
header = body.splitlines()[0] if body else b""
integrity, reason = _integrity(frame, bang, footer)
fields = _parse_obis_fields(body)
timestamp = _field_value(fields, "0-0:1.0.0")
channels = _parse_channels(fields)
return P1Telegram(
raw=frame,
header=header,
footer=footer,
integrity=integrity,
integrity_reason=reason,
timestamp=timestamp,
fields=tuple(fields),
channels=channels,
)
def _integrity(frame: bytes, bang: int, footer: bytes) -> tuple[IntegrityStatus, str]:
if not frame.startswith(b"/"):
return IntegrityStatus.UNVERIFIABLE, "missing standard DSMR '/' header"
if len(footer) != 4 or not all(chr(byte) in "0123456789abcdefABCDEF" for byte in footer):
return IntegrityStatus.UNVERIFIABLE, "footer is not a four-digit hexadecimal CRC"
expected = int(footer, 16)
actual = dsmr_crc16(frame[: bang + 1])
if actual == expected:
return IntegrityStatus.VALID, "CRC16 verified from '/' through '!'"
return IntegrityStatus.INVALID, f"CRC16 mismatch: expected {expected:04X}, calculated {actual:04X}"
def _parse_obis_fields(body: bytes) -> list[ObisField]:
fields: list[ObisField] = []
for line in body.decode("ascii", errors="replace").splitlines()[1:]:
match = _OBIS_LINE.fullmatch(line)
if not match:
continue
raw_values = tuple(re.findall(r"\(([^)]*)\)", match.group("values")))
value, unit = _numeric_value(raw_values)
fields.append(ObisField(match.group("code"), raw_values, value, unit))
return fields
def _numeric_value(raw_values: tuple[str, ...]) -> tuple[Decimal | None, str | None]:
if not raw_values:
return None, None
match = _NUMBER_WITH_UNIT.fullmatch(raw_values[-1])
if not match:
return None, None
try:
return Decimal(match.group("number")), match.group("unit")
except InvalidOperation:
return None, None
def _field_value(fields: list[ObisField], code: str) -> str | None:
field = next((item for item in fields if item.code == code), None)
return field.raw_values[-1] if field and field.raw_values else None
def _parse_channels(fields: list[ObisField]) -> tuple[P1Channel, ...]:
by_channel: dict[int, list[ObisField]] = {}
for field in fields:
match = _CHANNEL_OBIS.match(field.code)
if match:
by_channel.setdefault(int(match.group("channel")), []).append(field)
return tuple(
P1Channel(
number=number,
device_type=_field_value(channel_fields, f"0-{number}:24.1.0"),
equipment_id=_field_value(channel_fields, f"0-{number}:96.1.0"),
readings=tuple(
field
for field in channel_fields
if field.code == f"0-{number}:24.2.1" and field.value is not None
),
)
for number, channel_fields in sorted(by_channel.items())
)
__all__ = ["IntegrityStatus", "TelegramFramer", "build_parser", "parse_telegram", "run_probe"]
def build_parser() -> argparse.ArgumentParser:
@@ -279,6 +99,12 @@ def _format_field(field: ObisField) -> str:
return f" {field.code}: {raw_values}{value}".rstrip()
def _comparison_values(field: ObisField) -> tuple[str, ...]:
"""Return a local change-detection token without exposing identifiers."""
return (field.comparison_token,) if field.comparison_token is not None else field.raw_values
def _print_telegram(
telegram: P1Telegram,
frame_number: int,
@@ -289,14 +115,14 @@ def _print_telegram(
) -> dict[str, tuple[str, ...]]:
cadence_text = "first frame" if cadence is None else f"cadence={cadence:.1f}s"
print(
f"frame {frame_number}: {telegram.integrity.value}; bytes={len(telegram.raw)}; {cadence_text}",
f"frame {frame_number}: {telegram.integrity.value}; bytes={telegram.frame_length}; {cadence_text}",
file=output,
)
print(f" integrity: {telegram.integrity_reason}", file=output)
current_fields = {field.code: field.raw_values for field in telegram.fields}
current_fields = {field.code: _comparison_values(field) for field in telegram.fields}
fields = telegram.fields
if show_changes and previous_fields:
fields = tuple(field for field in fields if previous_fields.get(field.code) != field.raw_values)
fields = tuple(field for field in fields if previous_fields.get(field.code) != _comparison_values(field))
print(f" changed fields: {len(fields)}", file=output)
for field in fields:
print(_format_field(field), file=output)
+125 -109
View File
@@ -187,15 +187,13 @@ def test_put_config_with_csrf_header_updates_app_name(
assert app_name_field["value"] == "Updated via API"
def test_put_config_reapplies_dsmr_subscription(
def test_put_config_reapplies_dsmr_source_subscription(
client: TestClient, test_database_urls
) -> None:
"""Saving config must re-apply the DSMR subscription so enabling DSMR ingest
takes effect without an app restart (the route calls apply_dsmr_subscription
with the refreshed settings)."""
"""A config save triggers source subscription reconciliation."""
_login(client)
payload = _full_config_payload({"DSMR_INGEST_ENABLED": "true"})
payload = _full_config_payload()
with patch("app.services.dsmr_ingest.apply_dsmr_subscription") as spy:
response = client.put(
"/api/config",
@@ -205,8 +203,7 @@ def test_put_config_reapplies_dsmr_subscription(
assert response.status_code == 200
spy.assert_called_once()
applied_settings = spy.call_args.args[0]
assert applied_settings.dsmr_ingest_enabled is True
assert spy.call_args.args
def test_put_config_blank_secret_keeps_existing_value(
@@ -457,6 +454,7 @@ def test_get_config_includes_mqtt_section(client: TestClient) -> None:
assert "MQTT_USERNAME" in env_names
assert "MQTT_PASSWORD" in env_names
assert "MQTT_TLS_ENABLED" in env_names
assert "MQTT_CLIENT_ID" in env_names
def test_get_config_includes_ha_discovery_section(client: TestClient) -> None:
@@ -581,6 +579,18 @@ def test_put_config_invalid_mqtt_port_returns_422_and_does_not_write(
assert rows.get("MQTT_BROKER_PORT") != "not-a-number"
def test_put_config_invalid_mqtt_client_id_returns_422(client: TestClient) -> None:
_login(client)
response = client.put(
"/api/config",
json={"updates": _full_config_payload({"MQTT_CLIENT_ID": "invalid id"})},
headers={"X-CSRF-Token": "token"},
)
assert response.status_code == 422
# ---------------------------------------------------------------------------
# M5-polish2 Area B: bool fields have input_type="checkbox"
# ---------------------------------------------------------------------------
@@ -594,7 +604,6 @@ EXPECTED_CHECKBOX_FIELDS = {
"MQTT_TLS_ENABLED",
"HA_DISCOVERY_ENABLED",
"MODBUS_POLLING_ENABLED",
"DSMR_INGEST_ENABLED",
}
@@ -726,29 +735,46 @@ def test_put_config_mqtt_reconnect_uses_db_merged_settings(
)
def test_put_config_mqtt_client_id_reconnects_and_trims_value(
client: TestClient, test_database_urls
) -> None:
_login(client)
mock_mgr = MagicMock()
with patch("app.api.routes.api.config.mqtt_manager", mock_mgr):
response = client.put(
"/api/config",
json={"updates": _full_config_payload({"MQTT_CLIENT_ID": " home-automation-dev "})},
headers={"X-CSRF-Token": "token"},
)
assert response.status_code == 200
reconnect_settings = mock_mgr.reconnect.call_args.args[0]
assert reconnect_settings.mqtt_client_id == "home-automation-dev"
conn = sqlite3.connect(test_database_urls["app_path"])
try:
stored_value = conn.execute(
"SELECT value FROM app_config WHERE key = 'MQTT_CLIENT_ID'"
).fetchone()
finally:
conn.close()
assert stored_value == ("home-automation-dev",)
# ---------------------------------------------------------------------------
# M6-T02: DSMR + Tibber CONFIG_FIELDS
# ---------------------------------------------------------------------------
def test_get_config_includes_dsmr_section(client: TestClient) -> None:
"""GET /api/config must include a DSMR section with expected fields including DSMR_TARIFF_TOPIC."""
def test_get_config_excludes_legacy_dsmr_section(client: TestClient) -> None:
"""DSMR is configured through MeterSource, never the legacy config form."""
_login(client)
response = client.get("/api/config")
body = response.json()
section_names = {s["name"] for s in body["sections"]}
assert "DSMR" in section_names, f"DSMR section missing; got {section_names}"
dsmr_section = next(s for s in body["sections"] if s["name"] == "DSMR")
env_names = {f["env_name"] for f in dsmr_section["fields"]}
assert "DSMR_INGEST_ENABLED" in env_names
assert "DSMR_MQTT_TOPIC" in env_names
assert "DSMR_SAMPLE_INTERVAL_S" in env_names
assert "DSMR_TARIFF_TOPIC" in env_names, (
f"DSMR_TARIFF_TOPIC must be present in DSMR section; got {env_names}"
)
assert "DSMR" not in section_names
def test_get_config_includes_tibber_section(client: TestClient) -> None:
@@ -783,36 +809,46 @@ def test_get_config_tibber_api_token_is_secret(client: TestClient) -> None:
)
def test_get_config_dsmr_ingest_enabled_is_checkbox(client: TestClient) -> None:
"""DSMR_INGEST_ENABLED must have input_type='checkbox' for correct frontend rendering."""
def test_get_config_excludes_all_legacy_dsmr_fields(client: TestClient) -> None:
"""Old DSMR KV values stay in DB but are not returned by the config API."""
_login(client)
response = client.get("/api/config")
body = response.json()
dsmr_section = next(s for s in body["sections"] if s["name"] == "DSMR")
enabled_field = next(f for f in dsmr_section["fields"] if f["env_name"] == "DSMR_INGEST_ENABLED")
assert enabled_field["input_type"] == "checkbox", (
f"DSMR_INGEST_ENABLED input_type should be 'checkbox', got {enabled_field['input_type']!r}"
)
fields = {field["env_name"] for section in body["sections"] for field in section["fields"]}
assert not fields.intersection({"DSMR_INGEST_ENABLED", "DSMR_MQTT_TOPIC", "DSMR_SAMPLE_INTERVAL_S", "DSMR_TARIFF_TOPIC"})
def test_get_config_dsmr_sample_interval_input_type_is_number(client: TestClient) -> None:
"""DSMR_SAMPLE_INTERVAL_S must have input_type='number'."""
def test_config_save_preserves_legacy_dsmr_kv_rows(client: TestClient, test_database_urls) -> None:
"""A config-only save neither reads nor deletes retired DSMR configuration."""
_login(client)
conn = sqlite3.connect(test_database_urls["app_path"])
try:
conn.execute(
"INSERT INTO app_config (key, value, updated_at) VALUES (?, ?, CURRENT_TIMESTAMP)",
("DSMR_MQTT_TOPIC", "legacy/topic"),
)
conn.execute(
"INSERT INTO app_config (key, value, updated_at) VALUES (?, ?, CURRENT_TIMESTAMP)",
("DSMR_SAMPLE_INTERVAL_S", "37"),
)
conn.commit()
finally:
conn.close()
response = client.get("/api/config")
body = response.json()
dsmr_section = next(s for s in body["sections"] if s["name"] == "DSMR")
interval_field = next(
f for f in dsmr_section["fields"] if f["env_name"] == "DSMR_SAMPLE_INTERVAL_S"
)
assert interval_field["input_type"] == "number", (
f"DSMR_SAMPLE_INTERVAL_S input_type should be 'number', got {interval_field['input_type']!r}"
response = client.put(
"/api/config",
json={"updates": _full_config_payload({"APP_NAME": "config-only save"})},
headers={"X-CSRF-Token": "any-non-empty-value"},
)
assert response.status_code == 200
conn = sqlite3.connect(test_database_urls["app_path"])
try:
rows = dict(conn.execute("SELECT key, value FROM app_config WHERE key LIKE 'DSMR_%'"))
finally:
conn.close()
assert rows == {"DSMR_MQTT_TOPIC": "legacy/topic", "DSMR_SAMPLE_INTERVAL_S": "37"}
def test_put_config_blank_tibber_api_token_keeps_existing(
@@ -882,75 +918,6 @@ def test_put_config_new_tibber_api_token_overwrites_existing(
assert rows.get("TIBBER_API_TOKEN") == "new-tibber-token"
def test_put_config_invalid_dsmr_sample_interval_returns_422_and_does_not_write(
client: TestClient, test_database_urls
) -> None:
"""Non-integer DSMR_SAMPLE_INTERVAL_S must return 422 and not persist the bad value."""
_login(client)
payload = _full_config_payload({"DSMR_SAMPLE_INTERVAL_S": "not-a-number"})
response = client.put(
"/api/config",
json={"updates": payload},
headers={"X-CSRF-Token": "token"},
)
assert response.status_code == 422
conn = sqlite3.connect(test_database_urls["app_path"])
try:
rows = dict(conn.execute("SELECT key, value FROM app_config").fetchall())
finally:
conn.close()
assert rows.get("DSMR_SAMPLE_INTERVAL_S") != "not-a-number"
def test_put_config_dsmr_tariff_topic_persists_and_reflects_in_get(
client: TestClient, test_database_urls
) -> None:
"""DSMR_TARIFF_TOPIC must persist via PUT and be readable via GET /api/config."""
_login(client)
new_topic = "meter/tariff/slot"
payload = _full_config_payload({"DSMR_TARIFF_TOPIC": new_topic})
with patch("app.services.dsmr_ingest.apply_dsmr_subscription"):
response = client.put(
"/api/config",
json={"updates": payload},
headers={"X-CSRF-Token": "token"},
)
assert response.status_code == 200
# The updated value must appear in the GET response.
get_resp = client.get("/api/config")
body = get_resp.json()
dsmr_section = next(s for s in body["sections"] if s["name"] == "DSMR")
tariff_field = next(f for f in dsmr_section["fields"] if f["env_name"] == "DSMR_TARIFF_TOPIC")
assert tariff_field["value"] == new_topic, (
f"Expected DSMR_TARIFF_TOPIC to be {new_topic!r}, got {tariff_field['value']!r}"
)
def test_put_config_dsmr_tariff_topic_in_settings_payload(client: TestClient) -> None:
"""DSMR_TARIFF_TOPIC must appear in _settings_payload (GET /api/config returns it)."""
_login(client)
# The default value from Settings must appear in the DSMR section.
response = client.get("/api/config")
body = response.json()
dsmr_section = next(s for s in body["sections"] if s["name"] == "DSMR")
tariff_field = next(
(f for f in dsmr_section["fields"] if f["env_name"] == "DSMR_TARIFF_TOPIC"), None
)
assert tariff_field is not None, "DSMR_TARIFF_TOPIC must appear in DSMR config section"
# Default value should be the DSMR reader meter-stats topic.
assert tariff_field["value"] == "dsmr/meter-stats/electricity_tariff", (
f"Unexpected default for DSMR_TARIFF_TOPIC: {tariff_field['value']!r}"
)
def test_get_config_tibber_api_token_value_masked_after_save(
client: TestClient, test_database_urls
) -> None:
@@ -979,6 +946,55 @@ def test_get_config_tibber_api_token_value_masked_after_save(
assert "some-tibber-token" not in resp_after.text
def test_put_tibber_config_triggers_refresh_only_for_active_tibber_contract(
client: TestClient, test_database_urls
) -> None:
"""Saved Tibber credentials request a refresh only when their contract is active."""
_login(client)
conn = sqlite3.connect(test_database_urls["app_path"])
try:
conn.execute(
"INSERT INTO energy_contract (name, kind, scope, active, currency, created_at, updated_at) "
"VALUES (?, ?, ?, ?, ?, CURRENT_TIMESTAMP, CURRENT_TIMESTAMP)",
("Tibber", "tibber", "electricity", True, "EUR"),
)
conn.commit()
finally:
conn.close()
with patch("app.api.routes.api.config.trigger_tibber_refresh") as trigger:
response = client.put(
"/api/config",
json={"updates": _full_config_payload({"TIBBER_API_TOKEN": "new-secret-token"})},
headers={"X-CSRF-Token": "token"},
)
assert response.status_code == 200
trigger.assert_called_once()
with patch("app.api.routes.api.config.trigger_tibber_refresh") as trigger:
response = client.put(
"/api/config",
json={"updates": _full_config_payload({"TIBBER_API_TOKEN": ""})},
headers={"X-CSRF-Token": "token"},
)
assert response.status_code == 200
trigger.assert_not_called()
def test_put_tibber_config_does_not_refresh_without_active_tibber_contract(client: TestClient) -> None:
_login(client)
with patch("app.api.routes.api.config.trigger_tibber_refresh") as trigger:
response = client.put(
"/api/config",
json={"updates": _full_config_payload({"TIBBER_HOME_ID": "new-home-id"})},
headers={"X-CSRF-Token": "token"},
)
assert response.status_code == 200
trigger.assert_not_called()
def test_post_mqtt_test_uses_db_broker_host(
client: TestClient, test_database_urls
) -> None:
+43 -11
View File
@@ -304,6 +304,35 @@ def test_prices_no_contract_returns_empty(energy_client):
assert body["points"] == []
assert body["tariff"] is None
assert "currency" in body
assert client.get("/api/energy/prices?scope=electricity").json() == body
def test_prices_thermal_returns_active_contract_snapshot(energy_client):
client, engine, _app = energy_client
now = datetime.now(UTC)
values = {
"variable": {"heating": "20.123456", "hot_water_heating": "4", "hot_water": "2", "hot_water_tax": "1"},
"standing": {"heating_network": "0", "metering": "0", "delivery_set": "0", "hot_water_network": "0", "other": "0"},
}
with Session(engine) as session:
contract = EnergyContract(name="thermal", kind="district_heating", scope="thermal", active=True,
currency="EUR", created_at=now, updated_at=now)
session.add(contract)
session.flush()
session.add(EnergyContractVersion(contract_id=contract.id, effective_from=now - timedelta(days=1),
values=values, created_at=now))
session.commit()
_login(client)
response = client.get("/api/energy/prices?scope=thermal")
assert response.status_code == 200
body = response.json()
assert body == {
"kind": "district_heating", "currency": "EUR", "points": [],
"contract_version_id": body["contract_version_id"],
"effective_from": body["effective_from"], "values": values,
}
assert client.get("/api/energy/prices").json()["points"] == []
assert client.get("/api/energy/prices?scope=invalid").status_code == 422
# ---------------------------------------------------------------------------
@@ -334,6 +363,7 @@ def test_prices_manual_contract_returns_tariff(energy_client):
assert abs(tariff["sell_dal"] - 0.10) < 1e-6
# sell_normal = 0.10
assert abs(tariff["sell_normal"] - 0.10) < 1e-6
assert client.get("/api/energy/prices?scope=electricity").json() == body
# ---------------------------------------------------------------------------
@@ -369,6 +399,7 @@ def test_prices_tibber_contract_returns_points(energy_client):
assert abs(p["buy"] - 0.245) < 1e-6
assert abs(p["sell"] - (0.245 - 0.1108)) < 1e-4
assert p["level"] == "NORMAL"
assert client.get("/api/energy/prices", params={"scope": "electricity", "start": start, "end": end}).json() == body
def test_prices_tibber_sell_reflects_sell_fee(energy_client):
@@ -430,9 +461,7 @@ def test_prices_tibber_limit_caps_results(energy_client):
start = (datetime.now(UTC) - timedelta(hours=3)).isoformat()
end = (datetime.now(UTC) + timedelta(hours=2)).isoformat()
resp = client.get(
"/api/energy/prices", params={"start": start, "end": end, "limit": 2}
)
resp = client.get("/api/energy/prices", params={"start": start, "end": end, "limit": 2})
assert resp.status_code == 200
body = resp.json()
assert len(body["points"]) <= 2
@@ -508,9 +537,16 @@ def test_costs_schema_fields_present(energy_client):
item = resp.json()["items"][0]
for field in (
"period_start",
"d1_kwh", "d2_kwh", "r1_kwh", "r2_kwh",
"import_cost", "export_revenue", "net_cost",
"currency", "degraded",
"d1_kwh",
"d2_kwh",
"r1_kwh",
"r2_kwh",
"import_cost",
"export_revenue",
"net_cost",
"currency",
"degraded",
"source_binding_id",
):
assert field in item, f"Missing field: {field}"
@@ -556,9 +592,7 @@ def test_summary_returns_correct_structure(energy_client):
start = (datetime.now(UTC) - timedelta(hours=3)).isoformat()
end = datetime.now(UTC).isoformat()
resp = client.get(
"/api/energy/costs/summary", params={"start": start, "end": end}
)
resp = client.get("/api/energy/costs/summary", params={"start": start, "end": end})
assert resp.status_code == 200
body = resp.json()
for field in (
@@ -903,5 +937,3 @@ def test_tibber_test_token_not_in_response(energy_client):
assert resp.status_code == 200
assert secret_token not in resp.text
+225 -3
View File
@@ -48,6 +48,7 @@ from __future__ import annotations
from datetime import UTC, datetime, timedelta
from typing import Any
from unittest.mock import patch
import pytest
from fastapi.testclient import TestClient
@@ -55,6 +56,7 @@ from sqlalchemy import create_engine, select
from sqlalchemy.orm import Session
from app.models.energy import EnergyContract, EnergyContractVersion
from app.services.contracts import activate_contract
# ---------------------------------------------------------------------------
# Shared helpers
@@ -103,6 +105,15 @@ _TIBBER_VALUES: dict[str, Any] = {
},
}
_DISTRICT_HEATING_VALUES: dict[str, Any] = {
"variable": {
"heating": "12.34",
"hot_water_heating": "1.20",
"hot_water": "2.30",
"hot_water_tax": "0.10",
},
}
def _manual_payload(**overrides) -> dict[str, Any]:
base: dict[str, Any] = {
@@ -126,6 +137,17 @@ def _tibber_payload(**overrides) -> dict[str, Any]:
return base
def _district_heating_payload(**overrides) -> dict[str, Any]:
base: dict[str, Any] = {
"name": "District Heating",
"kind": "district_heating",
"currency": "EUR",
"values": _DISTRICT_HEATING_VALUES,
}
base.update(overrides)
return base
# ---------------------------------------------------------------------------
# Fixtures
# ---------------------------------------------------------------------------
@@ -155,7 +177,7 @@ def test_profiles_unauthenticated_returns_401(contracts_client):
assert resp.status_code == 401
def test_profiles_returns_both_kinds(contracts_client):
def test_profiles_returns_all_kinds(contracts_client):
client, _ = contracts_client
_login(client)
resp = client.get("/api/energy/profiles")
@@ -165,6 +187,7 @@ def test_profiles_returns_both_kinds(contracts_client):
kinds = {p["kind"] for p in body["profiles"]}
assert "manual" in kinds
assert "tibber" in kinds
assert "district_heating" in kinds
def test_profiles_contain_structure(contracts_client):
@@ -176,9 +199,12 @@ def test_profiles_contain_structure(contracts_client):
for profile in body["profiles"]:
assert "kind" in profile
assert "label" in profile
assert "energy" in profile
assert "standing" in profile
assert "credits" in profile
if profile["kind"] == "district_heating":
assert "variable" in profile
else:
assert "energy" in profile
assert "credits" in profile
# ---------------------------------------------------------------------------
@@ -345,6 +371,57 @@ def test_create_tibber_contract_success(contracts_client):
assert len(body["versions"]) == 1
def test_create_district_heating_contract_normalises_snapshot_and_scope(contracts_client):
client, _ = contracts_client
_login(client)
response = client.post(
"/api/energy/contracts",
json=_district_heating_payload(),
headers={"X-CSRF-Token": _CSRF},
)
assert response.status_code == 201
body = response.json()
assert body["kind"] == "district_heating"
assert body["scope"] == "thermal"
assert body["versions"][0]["values"] == {
"variable": _DISTRICT_HEATING_VALUES["variable"],
"standing": {
"heating_network": "0", "metering": "0", "delivery_set": "0",
"hot_water_network": "0", "other": "0",
},
}
assert client.get("/api/energy/contracts").json()["items"] == []
assert client.get("/api/energy/contracts?scope=thermal").json()["total"] == 1
mismatch = client.post(
"/api/energy/contracts",
json=_district_heating_payload(scope="electricity"),
headers={"X-CSRF-Token": _CSRF},
)
assert mismatch.status_code == 422
@pytest.mark.parametrize(
"values",
[
{"variable": {"heating": "1"}},
{"variable": {**_DISTRICT_HEATING_VALUES["variable"], "heating": -1}},
{"variable": {**_DISTRICT_HEATING_VALUES["variable"], "heating": 1.5}},
{"variable": {**_DISTRICT_HEATING_VALUES["variable"], "extra": "1"}},
],
)
def test_create_district_heating_rejects_invalid_decimal_values(contracts_client, values):
client, engine = contracts_client
_login(client)
response = client.post(
"/api/energy/contracts",
json=_district_heating_payload(values=values),
headers={"X-CSRF-Token": _CSRF},
)
assert response.status_code == 422
with Session(engine) as session:
assert session.execute(select(EnergyContract)).scalars().all() == []
def test_create_contract_defaults_effective_from(contracts_client):
"""When effective_from is omitted, the version is created with a recent timestamp.
@@ -538,6 +615,52 @@ def test_activate_contract_mutual_exclusion(contracts_client):
assert active_contracts[0].id == id_b
def test_activating_tibber_contract_triggers_refresh_only_after_commit(contracts_client):
"""Inactive→active Tibber is the only contract transition that requests refresh."""
client, engine = contracts_client
_login(client)
created = client.post(
"/api/energy/contracts",
json=_tibber_payload(),
headers={"X-CSRF-Token": _CSRF},
)
contract_id = created.json()["id"]
with patch("app.api.routes.api.energy_contracts.trigger_tibber_refresh") as trigger:
response = client.patch(
f"/api/energy/contracts/{contract_id}",
json={"active": True},
headers={"X-CSRF-Token": _CSRF},
)
assert response.status_code == 200
assert trigger.call_count == 1
with Session(engine) as session:
assert session.get(EnergyContract, contract_id).active is True
with patch("app.api.routes.api.energy_contracts.trigger_tibber_refresh") as trigger:
response = client.patch(
f"/api/energy/contracts/{contract_id}",
json={"active": True},
headers={"X-CSRF-Token": _CSRF},
)
assert response.status_code == 200
trigger.assert_not_called()
manual = client.post(
"/api/energy/contracts",
json=_manual_payload(),
headers={"X-CSRF-Token": _CSRF},
).json()
with patch("app.api.routes.api.energy_contracts.trigger_tibber_refresh") as trigger:
response = client.patch(
f"/api/energy/contracts/{manual['id']}",
json={"active": True},
headers={"X-CSRF-Token": _CSRF},
)
assert response.status_code == 200
trigger.assert_not_called()
def test_deactivate_contract(contracts_client):
"""PATCH active=false deactivates the contract without touching others."""
client, _ = contracts_client
@@ -567,6 +690,72 @@ def test_deactivate_contract(contracts_client):
assert resp.json()["active"] is False
def test_scope_defaults_filtering_and_kind_mismatch(contracts_client):
"""Old clients default to electricity; a supplied incompatible scope is rejected."""
client, engine = contracts_client
_login(client)
created = client.post(
"/api/energy/contracts",
json=_manual_payload(),
headers={"X-CSRF-Token": _CSRF},
)
assert created.status_code == 201
assert created.json()["scope"] == "electricity"
assert client.get("/api/energy/contracts").json()["total"] == 1
assert client.get("/api/energy/contracts?scope=thermal").json()["items"] == []
mismatch = client.post(
"/api/energy/contracts",
json=_manual_payload(scope="thermal"),
headers={"X-CSRF-Token": _CSRF},
)
assert mismatch.status_code == 422
with Session(engine) as session:
assert len(session.execute(select(EnergyContract)).scalars().all()) == 1
def test_activation_is_scope_local_and_transaction_rollback_is_safe(contracts_client):
"""A thermal activation neither deactivates electricity nor survives rollback."""
client, engine = contracts_client
_login(client)
electricity = client.post(
"/api/energy/contracts",
json=_manual_payload(name="Electricity"),
headers={"X-CSRF-Token": _CSRF},
).json()
client.patch(
f"/api/energy/contracts/{electricity['id']}",
json={"active": True},
headers={"X-CSRF-Token": _CSRF},
)
now = datetime.now(UTC)
with Session(engine) as session:
thermal = EnergyContract(
name="Future thermal", kind="district_heating", scope="thermal", active=False,
currency="EUR", created_at=now, updated_at=now,
)
session.add(thermal)
session.commit()
thermal_id = thermal.id
with Session(engine) as session:
thermal = session.get(EnergyContract, thermal_id)
assert thermal is not None
activate_contract(session, thermal)
session.rollback() # Simulate a later write failure in this transaction.
with Session(engine) as session:
rows = {row.scope: row for row in session.execute(select(EnergyContract)).scalars()}
assert rows["electricity"].active is True
assert rows["thermal"].active is False
activate_contract(session, rows["thermal"])
session.commit()
with Session(engine) as session:
active = session.execute(select(EnergyContract).where(EnergyContract.active.is_(True))).scalars().all()
assert {row.scope for row in active} == {"electricity", "thermal"}
# ---------------------------------------------------------------------------
# POST /api/energy/contracts/{id}/versions
# ---------------------------------------------------------------------------
@@ -748,3 +937,36 @@ def test_add_version_closes_previous_and_appends(contracts_client):
.all()
)
assert len(all_versions) == 2
def test_district_heating_version_timeline_keeps_normalised_snapshots(contracts_client):
client, _ = contracts_client
_login(client)
t0 = datetime(2026, 1, 1, tzinfo=UTC)
created = client.post(
"/api/energy/contracts",
json=_district_heating_payload(effective_from=t0.isoformat()),
headers={"X-CSRF-Token": _CSRF},
)
assert created.status_code == 201
contract_id = created.json()["id"]
t1 = datetime(2026, 6, 1, tzinfo=UTC)
updated_values = {
"variable": {**_DISTRICT_HEATING_VALUES["variable"], "heating": "13.500"},
"standing": {"metering": "10.00"},
}
response = client.post(
f"/api/energy/contracts/{contract_id}/versions",
json={"effective_from": t1.isoformat(), "values": updated_values},
headers={"X-CSRF-Token": _CSRF},
)
assert response.status_code == 201
old, new = response.json()["versions"]
assert old["effective_to"] is not None
assert old["values"]["variable"]["heating"] == "12.34"
assert new["effective_to"] is None
assert new["values"]["variable"]["heating"] == "13.500"
assert new["values"]["standing"] == {
"heating_network": "0", "metering": "10.00", "delivery_set": "0",
"hot_water_network": "0", "other": "0",
}
+673 -3
View File
@@ -31,15 +31,17 @@ Retroactive recompute integration
from __future__ import annotations
from datetime import UTC, datetime
from datetime import UTC, datetime, timedelta, timezone
from decimal import Decimal
from unittest.mock import patch
import pytest
from fastapi.testclient import TestClient
from sqlalchemy import create_engine, select
from sqlalchemy import create_engine, event, select
from sqlalchemy.orm import Session
from app.models.energy import Meter
from app.models.energy import EnergyCostPeriod, Meter, MeterCostPeriod
from app.models.meter_source import MeterSource, MeterSourceBinding, MeterSourceChannel
# ---------------------------------------------------------------------------
# Shared helpers
@@ -67,6 +69,51 @@ def _declare_payload(**overrides) -> dict:
return base
def _add_bound_channel(
engine,
*,
meter_id: int,
started_at: datetime,
ended_at: datetime | None = None,
unit: str = "kWh",
) -> str:
"""Persist one test-only channel binding and return its public UUID."""
with Session(engine) as session:
source = MeterSource(
name="Test DSMR",
kind="dsmr_mqtt",
enabled=True,
config={},
status="online",
created_at=started_at,
updated_at=started_at,
)
session.add(source)
session.flush()
channel = MeterSourceChannel(
source_id=source.id,
channel_key=f"test-total-{meter_id}",
label="Test total",
unit=unit,
created_at=started_at,
updated_at=started_at,
)
session.add(channel)
session.flush()
session.add(
MeterSourceBinding(
meter_id=meter_id,
channel_id=channel.id,
started_at=started_at,
ended_at=ended_at,
created_at=started_at,
updated_at=started_at,
)
)
session.commit()
return channel.uuid
# ---------------------------------------------------------------------------
# Fixtures
# ---------------------------------------------------------------------------
@@ -252,6 +299,207 @@ def test_declare_meter_swap_closes_previous(meters_client):
assert ended_naive == t1
def test_declare_meter_swap_hands_off_previous_meter_channel_atomically(meters_client):
client, engine = meters_client
_login(client)
t0 = datetime(2024, 6, 1, tzinfo=UTC)
boundary = datetime(2025, 3, 15, 12, tzinfo=UTC)
with patch("app.api.routes.api.meters.recompute_range", return_value=0):
old_response = client.post(
"/api/energy/meters",
json=_declare_payload(label="Old meter", started_at=t0.isoformat()),
headers={"X-CSRF-Token": _CSRF},
)
assert old_response.status_code == 201
old_id = old_response.json()["id"]
channel_uuid = _add_bound_channel(engine, meter_id=old_id, started_at=t0)
response = client.post(
"/api/energy/meters",
json=_declare_payload(
label="New meter",
started_at=boundary.isoformat(),
reason="meter_swap",
source_channel_uuid=channel_uuid,
),
headers={"X-CSRF-Token": _CSRF},
)
assert response.status_code == 201
new_id = response.json()["id"]
with Session(engine) as session:
bindings = session.execute(
select(MeterSourceBinding).order_by(MeterSourceBinding.id)
).scalars().all()
old_binding_ended_at = bindings[0].ended_at
if old_binding_ended_at is not None and old_binding_ended_at.tzinfo is None:
old_binding_ended_at = old_binding_ended_at.replace(tzinfo=UTC)
new_binding_started_at = bindings[1].started_at
if new_binding_started_at.tzinfo is None:
new_binding_started_at = new_binding_started_at.replace(tzinfo=UTC)
assert [(bindings[0].meter_id, old_binding_ended_at), (bindings[1].meter_id, bindings[1].ended_at)] == [
(old_id, boundary),
(new_id, None),
]
assert new_binding_started_at == boundary
def test_declare_meter_swap_rejects_other_meter_channel_and_rolls_back(meters_client):
client, engine = meters_client
_login(client)
t0 = datetime(2024, 6, 1, tzinfo=UTC)
boundary = datetime(2025, 3, 15, 12, tzinfo=UTC)
with patch("app.api.routes.api.meters.recompute_range", return_value=0):
old_response = client.post(
"/api/energy/meters",
json=_declare_payload(label="Old meter", started_at=t0.isoformat()),
headers={"X-CSRF-Token": _CSRF},
)
old_id = old_response.json()["id"]
other = Meter(
label="Other meter",
commodity="electricity",
started_at=t0,
ended_at=boundary + timedelta(days=1),
reason="initial",
created_at=t0,
)
with Session(engine) as session:
session.add(other)
session.commit()
other_id = other.id
channel_uuid = _add_bound_channel(engine, meter_id=other_id, started_at=t0)
response = client.post(
"/api/energy/meters",
json=_declare_payload(
label="Rejected meter",
started_at=boundary.isoformat(),
reason="meter_swap",
source_channel_uuid=channel_uuid,
),
headers={"X-CSRF-Token": _CSRF},
)
assert response.status_code == 422
with Session(engine) as session:
assert session.execute(select(Meter).where(Meter.label == "Rejected meter")).scalar_one_or_none() is None
assert session.get(Meter, old_id).ended_at is None
binding = session.execute(select(MeterSourceBinding)).scalar_one()
assert binding.ended_at is None
def test_declare_meter_non_swap_cannot_take_previous_meter_channel(meters_client):
client, engine = meters_client
_login(client)
t0 = datetime(2024, 6, 1, tzinfo=UTC)
boundary = datetime(2025, 3, 15, 12, tzinfo=UTC)
with patch("app.api.routes.api.meters.recompute_range", return_value=0):
old_response = client.post(
"/api/energy/meters",
json=_declare_payload(label="Old meter", started_at=t0.isoformat()),
headers={"X-CSRF-Token": _CSRF},
)
old_id = old_response.json()["id"]
channel_uuid = _add_bound_channel(engine, meter_id=old_id, started_at=t0)
response = client.post(
"/api/energy/meters",
json=_declare_payload(
label="Moved meter",
started_at=boundary.isoformat(),
reason="home_move",
source_channel_uuid=channel_uuid,
),
headers={"X-CSRF-Token": _CSRF},
)
assert response.status_code == 422
with Session(engine) as session:
assert session.get(Meter, old_id).ended_at is None
assert session.execute(select(MeterSourceBinding)).scalar_one().ended_at is None
def test_declare_meter_recompute_failure_rolls_back_handoff(meters_client):
client, engine = meters_client
_login(client)
t0 = datetime(2024, 6, 1, tzinfo=UTC)
boundary = datetime(2025, 3, 15, 12, tzinfo=UTC)
with patch("app.api.routes.api.meters.recompute_range", return_value=0):
old_response = client.post(
"/api/energy/meters",
json=_declare_payload(label="Old meter", started_at=t0.isoformat()),
headers={"X-CSRF-Token": _CSRF},
)
old_id = old_response.json()["id"]
channel_uuid = _add_bound_channel(engine, meter_id=old_id, started_at=t0)
with patch("app.api.routes.api.meters.recompute_range", side_effect=RuntimeError("recompute failed")):
with pytest.raises(RuntimeError, match="recompute failed"):
client.post(
"/api/energy/meters",
json=_declare_payload(
label="Failed meter",
started_at=boundary.isoformat(),
reason="meter_swap",
source_channel_uuid=channel_uuid,
),
headers={"X-CSRF-Token": _CSRF},
)
with Session(engine) as session:
assert session.execute(select(Meter).where(Meter.label == "Failed meter")).scalar_one_or_none() is None
assert session.get(Meter, old_id).ended_at is None
assert session.execute(select(MeterSourceBinding)).scalar_one().ended_at is None
def test_declare_meter_final_commit_failure_rolls_back_handoff_and_recompute(meters_client):
"""A real recompute remains uncommitted until the route's final commit succeeds."""
client, engine = meters_client
_login(client)
boundary = datetime.now(UTC) - timedelta(minutes=45)
boundary = boundary.replace(minute=boundary.minute - boundary.minute % 15, second=0, microsecond=0)
old_start = boundary - timedelta(days=1)
with Session(engine) as session:
old_meter = Meter(
label="Old meter",
commodity="electricity",
started_at=old_start,
reason="initial",
created_at=old_start,
)
session.add(old_meter)
session.commit()
old_id = old_meter.id
channel_uuid = _add_bound_channel(engine, meter_id=old_id, started_at=old_start)
def fail_final_commit(_session: Session) -> None:
raise RuntimeError("final commit failed")
event.listen(Session, "before_commit", fail_final_commit)
try:
with pytest.raises(RuntimeError, match="final commit failed"):
client.post(
"/api/energy/meters",
json=_declare_payload(
label="New meter",
started_at=boundary.isoformat(),
reason="meter_swap",
source_channel_uuid=channel_uuid,
),
headers={"X-CSRF-Token": _CSRF},
)
finally:
event.remove(Session, "before_commit", fail_final_commit)
with Session(engine) as session:
assert session.execute(select(Meter).where(Meter.label == "New meter")).scalar_one_or_none() is None
assert session.get(Meter, old_id).ended_at is None
binding = session.execute(select(MeterSourceBinding)).scalar_one()
assert binding.ended_at is None
assert session.execute(select(EnergyCostPeriod)).scalars().all() == []
def test_declare_meter_overlap_returns_422(meters_client):
"""Declaring a meter with started_at before active meter's started_at → 422."""
client, _ = meters_client
@@ -355,6 +603,8 @@ def test_declare_meter_retroactive_triggers_recompute(meters_client):
assert resp.status_code == 201
# recompute_range should have been called with start == t_past
assert mock_recompute.called
assert mock_recompute.call_args.kwargs["commit"] is False
assert mock_recompute.call_args.kwargs["strict"] is True
call_args = mock_recompute.call_args
recompute_start = call_args[0][1] # positional arg index 1 (session is 0)
# Normalise for comparison
@@ -494,6 +744,8 @@ def test_patch_meter_started_at_retroactive_triggers_recompute(meters_client):
assert resp.status_code == 200
# recompute should be triggered
assert mock_recompute.called
assert mock_recompute.call_args.kwargs["commit"] is False
assert mock_recompute.call_args.kwargs["strict"] is True
call_args = mock_recompute.call_args
recompute_start = call_args[0][1]
if recompute_start.tzinfo is None:
@@ -535,6 +787,59 @@ def test_patch_meter_started_at_interval_violation_returns_422(meters_client):
assert resp.status_code == 422
@pytest.mark.parametrize("shift", ["later", "earlier"])
def test_patch_meter_started_at_rejects_boundary_shift_that_strands_binding(
meters_client, mock_publish_discovery, shift
):
"""Rejected boundary shifts leave adjacent meters/bindings untouched and emit no side effects."""
client, engine = meters_client
_login(client)
t0 = datetime(2024, 1, 1, tzinfo=UTC)
boundary = datetime(2025, 1, 1, tzinfo=UTC)
with patch("app.api.routes.api.meters.recompute_range", return_value=0):
meter_a = client.post(
"/api/energy/meters", json=_declare_payload(label="A", started_at=t0.isoformat()),
headers={"X-CSRF-Token": _CSRF},
).json()
meter_b = client.post(
"/api/energy/meters", json=_declare_payload(
label="B", started_at=boundary.isoformat(), reason="meter_swap"
), headers={"X-CSRF-Token": _CSRF},
).json()
if shift == "later":
_add_bound_channel(engine, meter_id=meter_b["id"], started_at=boundary)
proposed = boundary + timedelta(days=1)
else:
_add_bound_channel(engine, meter_id=meter_a["id"], started_at=t0)
with Session(engine) as session:
binding = session.scalar(select(MeterSourceBinding))
assert binding is not None
binding.ended_at = boundary
session.commit()
proposed = boundary - timedelta(days=1)
mock_publish_discovery.reset_mock()
with patch("app.api.routes.api.meters.recompute_range", return_value=0) as recompute:
response = client.patch(
f"/api/energy/meters/{meter_b['id']}", json={"started_at": proposed.isoformat()},
headers={"X-CSRF-Token": _CSRF},
)
assert response.status_code == 422
recompute.assert_not_called()
mock_publish_discovery.assert_not_called()
with Session(engine) as observer:
assert observer.get(Meter, meter_a["id"]).ended_at.replace(tzinfo=UTC) == boundary
assert observer.get(Meter, meter_b["id"]).started_at.replace(tzinfo=UTC) == boundary
binding = observer.scalar(select(MeterSourceBinding))
assert binding is not None
if shift == "later":
assert binding.meter_id == meter_b["id"] and binding.ended_at is None
else:
assert binding.meter_id == meter_a["id"]
assert binding.ended_at.replace(tzinfo=UTC) == boundary
def test_patch_meter_no_recompute_when_started_at_not_changed(meters_client):
"""PATCH that only changes label does NOT trigger recompute."""
client, _ = meters_client
@@ -560,6 +865,201 @@ def test_patch_meter_no_recompute_when_started_at_not_changed(meters_client):
assert not mock_recompute.called
@pytest.mark.parametrize("representation", ["aware_utc", "naive_local"])
def test_patch_meter_rejects_future_started_at_before_any_side_effect(
meters_client, mock_publish_discovery, monkeypatch, representation
):
"""Future aware and local-naive starts leave all persisted state untouched."""
client, engine = meters_client
_login(client)
started = datetime.now(UTC) - timedelta(hours=2)
with patch("app.api.routes.api.meters.recompute_range", return_value=0):
declared = client.post(
"/api/energy/meters",
json=_declare_payload(
label="Original", note="Original note", started_at=started.isoformat()
),
headers={"X-CSRF-Token": _CSRF},
)
meter_id = declared.json()["id"]
mock_publish_discovery.reset_mock()
future = datetime.now(UTC) + timedelta(hours=2)
if representation == "aware_utc":
proposed = future.isoformat()
else:
monkeypatch.setattr(
"app.services.timezone.local_tz", lambda: timezone(timedelta(hours=2))
)
proposed = (future + timedelta(hours=2)).replace(tzinfo=None).isoformat()
with patch("app.api.routes.api.meters.recompute_range", return_value=0) as recompute:
response = client.patch(
f"/api/energy/meters/{meter_id}",
json={"label": "Changed", "note": "Changed note", "started_at": proposed},
headers={"X-CSRF-Token": _CSRF},
)
assert response.status_code == 422
recompute.assert_not_called()
mock_publish_discovery.assert_not_called()
with Session(engine) as observer:
meter = observer.get(Meter, meter_id)
assert meter is not None
assert meter.label == "Original"
assert meter.note == "Original note"
assert meter.started_at.replace(tzinfo=UTC) == started
@pytest.mark.parametrize("commodity", ["heating", "hot_water"])
def test_patch_thermal_started_at_uses_thermal_recompute(meters_client, monkeypatch, commodity):
"""A successful thermal correction never routes through electricity recompute."""
from app.services import meter_cost
client, _ = meters_client
_login(client)
started = datetime.now(UTC) - timedelta(hours=3)
calls = []
monkeypatch.setattr(
meter_cost,
"recompute_range",
lambda db, start, end, *, commit: calls.append((start, end, commit)) or 0,
)
with patch("app.api.routes.api.meters.recompute_range", return_value=0) as electricity:
declared = client.post(
"/api/energy/meters",
json=_declare_payload(
commodity=commodity,
label=f"{commodity} meter",
started_at=started.isoformat(),
),
headers={"X-CSRF-Token": _CSRF},
)
assert declared.status_code == 201
calls.clear()
electricity.reset_mock()
response = client.patch(
f"/api/energy/meters/{declared.json()['id']}",
json={"started_at": (started + timedelta(minutes=30)).isoformat()},
headers={"X-CSRF-Token": _CSRF},
)
assert response.status_code == 200
assert len(calls) == 1
assert calls[0][2] is False
electricity.assert_not_called()
def _create_adjacent_thermal_patch_state(engine, commodity: str):
"""Create an editable thermal boundary with two untouched bindings."""
old_start = datetime.now(UTC) - timedelta(hours=5)
boundary = old_start + timedelta(hours=1)
shifted = boundary + timedelta(minutes=30)
current_binding_start = shifted + timedelta(minutes=30)
with Session(engine) as session:
previous = Meter(
label="Previous thermal meter",
commodity=commodity,
started_at=old_start,
ended_at=boundary,
reason="initial",
created_at=old_start,
)
current = Meter(
label="Current thermal meter",
commodity=commodity,
started_at=boundary,
reason="meter_swap",
created_at=boundary,
)
session.add_all([previous, current])
session.commit()
previous_id, current_id = previous.id, current.id
unit = {"heating": "GJ", "hot_water": ""}[commodity]
_add_bound_channel(
engine, meter_id=previous_id, started_at=old_start, ended_at=boundary, unit=unit
)
_add_bound_channel(engine, meter_id=current_id, started_at=current_binding_start, unit=unit)
return previous_id, current_id, boundary, shifted, current_binding_start
@pytest.mark.parametrize("commodity", ["heating", "hot_water"])
@pytest.mark.parametrize("failure", ["recompute", "flush", "commit"])
def test_patch_thermal_failure_rolls_back_lifecycle_and_cost_state(
meters_client, mock_publish_discovery, monkeypatch, commodity, failure
):
"""Thermal PATCH failures roll back meters, bindings, cost writes, and HA."""
from app.services import meter_cost
client, engine = meters_client
_login(client)
previous_id, current_id, boundary, shifted, binding_start = _create_adjacent_thermal_patch_state(
engine, commodity
)
def recompute_with_uncommitted_cost(db, start, end, *, commit):
assert commit is False
db.add(
MeterCostPeriod(
commodity=commodity,
period_start=shifted,
period_end=shifted + timedelta(minutes=15),
quantity=Decimal("0"),
cost=Decimal("0"),
currency="EUR",
cost_breakdown={},
pricing_snapshot={},
quality="invalid",
degraded=True,
degraded_reason="test rollback",
created_at=shifted,
updated_at=shifted,
)
)
if failure == "recompute":
raise RuntimeError("thermal recompute failed")
return 0
monkeypatch.setattr(meter_cost, "recompute_range", recompute_with_uncommitted_cost)
if failure == "flush":
monkeypatch.setattr(
Session,
"flush",
lambda *_args, **_kwargs: (_ for _ in ()).throw(RuntimeError("thermal flush failed")),
)
if failure == "commit":
def fail_commit(_session: Session) -> None:
raise RuntimeError("thermal commit failed")
event.listen(Session, "before_commit", fail_commit)
try:
expected = f"thermal {failure} failed"
with pytest.raises(RuntimeError, match=expected):
client.patch(
f"/api/energy/meters/{current_id}",
json={"started_at": shifted.isoformat()},
headers={"X-CSRF-Token": _CSRF},
)
finally:
if failure == "commit":
event.remove(Session, "before_commit", fail_commit)
if failure == "flush":
monkeypatch.undo()
mock_publish_discovery.assert_not_called()
with Session(engine) as observer:
previous = observer.get(Meter, previous_id)
current = observer.get(Meter, current_id)
assert previous is not None and previous.ended_at.replace(tzinfo=UTC) == boundary
assert current is not None and current.started_at.replace(tzinfo=UTC) == boundary
bindings = observer.execute(
select(MeterSourceBinding).order_by(MeterSourceBinding.meter_id)
).scalars().all()
assert bindings[0].ended_at.replace(tzinfo=UTC) == boundary
assert bindings[1].started_at.replace(tzinfo=UTC) == binding_start
assert observer.execute(select(MeterCostPeriod)).scalars().all() == []
# ---------------------------------------------------------------------------
# Timeline continuity (recompute mocked to avoid slow computation over empty quarters)
# ---------------------------------------------------------------------------
@@ -753,3 +1253,173 @@ def test_declare_meter_succeeds_when_publish_discovery_raises(meters_client):
# The meter must be created successfully despite the discovery failure.
assert resp.status_code == 201
assert resp.json()["label"] == "Best Effort Meter"
@pytest.mark.parametrize("commodity", ["heating", "hot_water"])
def test_thermal_declare_and_close_use_meter_cost_recompute(meters_client, monkeypatch, commodity):
"""Thermal lifecycle routes use the meter-cost helper's actual signature."""
from app.services import meter_cost
client, _ = meters_client
_login(client)
calls = []
monkeypatch.setattr(meter_cost, "recompute_range", lambda db, start, end, *, commit: calls.append((start, end, commit)) or 0)
started = datetime.now(UTC) - timedelta(hours=2)
declared = client.post("/api/energy/meters", headers={"X-CSRF-Token": _CSRF}, json={
"label": f"{commodity} meter", "started_at": started.isoformat(), "reason": "initial", "commodity": commodity,
})
assert declared.status_code == 201
closed = client.post(f"/api/energy/meters/{declared.json()['id']}/close", headers={"X-CSRF-Token": _CSRF}, json={
"ended_at": (started + timedelta(hours=1)).isoformat(),
})
assert closed.status_code == 200
assert len(calls) == 2 and all(call[2] is False for call in calls)
def test_close_meter_closes_open_bindings_and_enforces_auth_csrf(meters_client):
client, engine = meters_client
started = datetime.now(UTC) - timedelta(hours=2)
_login(client)
with patch("app.api.routes.api.meters.recompute_range", return_value=0):
declared = client.post("/api/energy/meters", headers={"X-CSRF-Token": _CSRF}, json=_declare_payload(
label="closable", started_at=started.isoformat(), reason="initial",
))
assert declared.status_code == 201
_add_bound_channel(engine, meter_id=declared.json()["id"], started_at=started)
boundary = started + timedelta(hours=1)
assert client.post(f"/api/energy/meters/{declared.json()['id']}/close", json={"ended_at": boundary.isoformat()}).status_code == 403
with patch("app.api.routes.api.meters.recompute_range", return_value=0):
closed = client.post(f"/api/energy/meters/{declared.json()['id']}/close", headers={"X-CSRF-Token": _CSRF}, json={
"ended_at": boundary.isoformat(),
})
assert closed.status_code == 200
with Session(engine) as session:
binding = session.scalar(select(MeterSourceBinding))
assert binding is not None and binding.ended_at.replace(tzinfo=UTC) == boundary
@pytest.mark.parametrize("commodity,unit", [("electricity", "kWh"), ("heating", "GJ"), ("hot_water", "")])
@pytest.mark.parametrize("operation", ["close", "declare"])
def test_lifecycle_rejects_retained_closed_binding_beyond_proposed_end(
meters_client, mock_publish_discovery, commodity, unit, operation,
):
"""Close and declare fail closed before recompute or HA for every commodity."""
client, engine = meters_client
_login(client)
start = datetime.now(UTC) - timedelta(hours=3)
boundary = start + timedelta(hours=1)
retained_end = start + timedelta(hours=2)
declared = client.post(
"/api/energy/meters", headers={"X-CSRF-Token": _CSRF},
json=_declare_payload(label="retained-history", started_at=start.isoformat(), reason="initial", commodity=commodity),
)
assert declared.status_code == 201
meter_id = declared.json()["id"]
_add_bound_channel(engine, meter_id=meter_id, started_at=start, ended_at=retained_end, unit=unit)
mock_publish_discovery.reset_mock()
with patch("app.api.routes.api.meters._recompute_commodity", side_effect=AssertionError("must not recompute")):
if operation == "close":
response = client.post(
f"/api/energy/meters/{meter_id}/close", headers={"X-CSRF-Token": _CSRF},
json={"ended_at": boundary.isoformat()},
)
else:
response = client.post(
"/api/energy/meters", headers={"X-CSRF-Token": _CSRF},
json=_declare_payload(
label="replacement", started_at=boundary.isoformat(), reason="meter_swap", commodity=commodity,
),
)
assert response.status_code == 422
assert mock_publish_discovery.call_count == 0
with Session(engine) as observer:
meter = observer.get(Meter, meter_id)
binding = observer.scalar(select(MeterSourceBinding).where(MeterSourceBinding.meter_id == meter_id))
assert meter is not None and meter.ended_at is None
assert binding is not None and binding.ended_at.replace(tzinfo=UTC) == retained_end
assert observer.scalars(select(Meter).where(Meter.commodity == commodity)).all() == [meter]
def test_close_flushes_lifecycle_boundary_before_strict_recompute(meters_client):
"""The strict recompute query observes the just-closed meter and binding."""
client, engine = meters_client
_login(client)
started = datetime.now(UTC) - timedelta(hours=2)
boundary = started + timedelta(hours=1)
with patch("app.api.routes.api.meters.recompute_range", return_value=0):
declared = client.post(
"/api/energy/meters", headers={"X-CSRF-Token": _CSRF},
json=_declare_payload(label="flush-visible", started_at=started.isoformat(), reason="initial"),
)
meter_id = declared.json()["id"]
_add_bound_channel(engine, meter_id=meter_id, started_at=started)
def observe(session, *_args, **kwargs):
assert kwargs == {"commit": False, "strict": True}
observed_meter = session.get(Meter, meter_id)
observed_binding = session.scalar(select(MeterSourceBinding))
assert observed_meter is not None and observed_meter.ended_at is not None
assert observed_binding is not None and observed_binding.ended_at is not None
return 0
with patch("app.api.routes.api.meters.recompute_range", side_effect=observe):
response = client.post(
f"/api/energy/meters/{meter_id}/close", headers={"X-CSRF-Token": _CSRF},
json={"ended_at": boundary.isoformat()},
)
assert response.status_code == 200
def test_close_strict_compute_failure_rolls_back_persisted_lifecycle_state(meters_client, monkeypatch):
"""A real per-period strict failure rolls back the close in a fresh Session."""
from app.services import energy_cost
client, engine = meters_client
_login(client)
started = datetime.now(UTC) - timedelta(hours=2)
boundary = started + timedelta(hours=1)
with patch("app.api.routes.api.meters.recompute_range", return_value=0):
declared = client.post(
"/api/energy/meters", headers={"X-CSRF-Token": _CSRF},
json=_declare_payload(label="strict-rollback", started_at=started.isoformat(), reason="initial"),
)
meter_id = declared.json()["id"]
_add_bound_channel(engine, meter_id=meter_id, started_at=started)
monkeypatch.setattr(
energy_cost, "compute_period", lambda *_args, **_kwargs: (_ for _ in ()).throw(RuntimeError("strict boom"))
)
with pytest.raises(RuntimeError, match="strict boom"):
client.post(
f"/api/energy/meters/{meter_id}/close", headers={"X-CSRF-Token": _CSRF},
json={"ended_at": boundary.isoformat()},
)
with Session(engine) as observer:
assert observer.get(Meter, meter_id).ended_at is None
assert observer.scalar(select(MeterSourceBinding)).ended_at is None
def test_close_flush_failure_rolls_back_persisted_lifecycle_state(meters_client, monkeypatch):
"""The mandatory pre-recompute flush shares the route rollback boundary."""
client, engine = meters_client
_login(client)
started = datetime.now(UTC) - timedelta(hours=2)
with patch("app.api.routes.api.meters.recompute_range", return_value=0):
declared = client.post(
"/api/energy/meters", headers={"X-CSRF-Token": _CSRF},
json=_declare_payload(label="flush-rollback", started_at=started.isoformat(), reason="initial"),
)
meter_id = declared.json()["id"]
_add_bound_channel(engine, meter_id=meter_id, started_at=started)
monkeypatch.setattr(Session, "flush", lambda *_args, **_kwargs: (_ for _ in ()).throw(RuntimeError("flush boom")))
with pytest.raises(RuntimeError, match="flush boom"):
client.post(
f"/api/energy/meters/{meter_id}/close", headers={"X-CSRF-Token": _CSRF},
json={"ended_at": (started + timedelta(hours=1)).isoformat()},
)
monkeypatch.undo()
with Session(engine) as observer:
assert observer.get(Meter, meter_id).ended_at is None
assert observer.scalar(select(MeterSourceBinding)).ended_at is None
+38
View File
@@ -112,6 +112,7 @@ def test_app_start_seeds_missing_config_from_env_without_overwriting_existing_va
assert rows["APP_NAME"] == "Database Owned Name"
assert rows["HOME_ASSISTANT_BASE_URL"] == "http://bootstrap-ha.local:8123"
assert rows["AUTH_SESSION_COOKIE_NAME"] == "home_automation_session"
assert rows["MQTT_CLIENT_ID"] == "home-automation"
get_settings.cache_clear()
reset_db_caches()
@@ -151,3 +152,40 @@ def test_app_start_syncs_app_hostname_from_env_even_when_db_has_old_value(
get_settings.cache_clear()
reset_db_caches()
def test_lifespan_schedules_immediate_and_hourly_tibber_refresh(tmp_path, monkeypatch: pytest.MonkeyPatch) -> None:
"""The single Tibber interval job starts immediately while retaining its hourly trigger."""
import app.main as main
app_database_url = _prepare_app_db(tmp_path)
added_jobs = []
class _Scheduler:
def __init__(self, **_kwargs): pass
def add_job(self, func, **kwargs): added_jobs.append((func, kwargs))
def start(self): pass
def shutdown(self, **_kwargs): pass
monkeypatch.setenv("APP_DATABASE_URL", app_database_url)
monkeypatch.setenv("AUTH_BOOTSTRAP_USERNAME", "admin")
monkeypatch.setenv("AUTH_BOOTSTRAP_PASSWORD", "test-password")
monkeypatch.setattr(main, "BackgroundScheduler", _Scheduler)
monkeypatch.setattr(main.mqtt_manager, "connect", lambda _settings: None)
monkeypatch.setattr(main.mqtt_manager, "disconnect", lambda: None)
monkeypatch.setattr(main, "apply_dsmr_subscription", lambda _settings: None)
monkeypatch.setattr(main.warmtelink_worker_manager, "start", lambda: None)
monkeypatch.setattr(main.warmtelink_worker_manager, "shutdown", lambda: None)
get_settings.cache_clear()
reset_db_caches()
anyio.run(_run_lifespan, create_app())
tibber_jobs = [kwargs for func, kwargs in added_jobs if func is main._run_scheduled_tibber_refresh]
assert len(tibber_jobs) == 1
assert tibber_jobs[0]["id"] == "tibber-refresh"
assert tibber_jobs[0]["max_instances"] == 1
assert tibber_jobs[0]["next_run_time"] is not None
get_settings.cache_clear()
reset_db_caches()
+33
View File
@@ -72,6 +72,39 @@ def test_compose_uses_migration_job_before_app() -> None:
assert dev["services"]["app"]["build"] == "."
def test_compose_keeps_app_non_root_and_maps_minimal_warmtelink_serial_access() -> None:
"""Base Compose maps the configured device with only pyserial's required access.
Environment interpolation is deliberately left unresolved: operators supply
a host-specific stable by-id path and its numeric serial GID in local .env.
The dev compose file inherits this app configuration unchanged.
"""
base = _read_yaml("docker-compose.yml")
dev = _read_yaml("docker-compose.dev.yml")
base_app = base["services"]["app"]
base_migration = base["services"]["migration"]
assert base_app["user"] == "1000:1000"
assert base_migration["user"] == "1000:1000"
assert "privileged" not in base_app
assert "privileged" not in base_migration
assert "cap_add" not in base_app
assert "cap_add" not in base_migration
assert base_app["devices"] == [
"${WARMTELINK_DEVICE_PATH:?Set a stable /dev/serial/by-id path}:/dev/warmtelink:rw"
]
device_rule = base_app["devices"][0]
permissions = device_rule.rsplit(":", maxsplit=1)[1]
assert permissions == "rw"
assert "m" not in permissions
assert base_app["group_add"] == ["${WARMTELINK_SERIAL_GID:?Set the host serial device GID}"]
assert dev["services"]["app"]["devices"] == base_app["devices"]
assert dev["services"]["app"]["group_add"] == base_app["group_add"]
assert "devices" not in base_migration
assert "group_add" not in base_migration
assert "privileged" not in base_migration
def test_image_defaults_to_uvicorn_only() -> None:
dockerfile = (PROJECT_ROOT / "Dockerfile").read_text()
entrypoint = (PROJECT_ROOT / "docker/entrypoint.sh").read_text()
+427 -37
View File
@@ -20,8 +20,9 @@ Covers:
from __future__ import annotations
import json
from datetime import datetime, timezone
from pathlib import Path
from unittest.mock import MagicMock, patch
from unittest.mock import patch
import pytest
from alembic import command
@@ -30,6 +31,7 @@ from sqlalchemy import create_engine, select
from sqlalchemy.orm import Session
from app.models.energy import DsmrReading
from app.models.meter_source import MeterSource
# ---------------------------------------------------------------------------
@@ -63,11 +65,9 @@ def _make_settings(
dsmr_ingest_enabled: bool = True,
dsmr_mqtt_topic: str = "dsmr/json",
):
s = MagicMock()
s.dsmr_sample_interval_s = dsmr_sample_interval_s
s.dsmr_ingest_enabled = dsmr_ingest_enabled
s.dsmr_mqtt_topic = dsmr_mqtt_topic
return s
del dsmr_ingest_enabled
from app.services.dsmr_ingest import DsmrSourceSnapshot
return DsmrSourceSnapshot(1, dsmr_mqtt_topic, "", dsmr_sample_interval_s)
# The reference telegram sample from §6.3 of the design doc.
@@ -175,6 +175,264 @@ def test_second_00_persists_full_frame(dsmr_db):
assert payload["phase_voltage_l2"] is None
def test_accepted_message_marks_source_online_and_updates_last_seen(dsmr_db):
"""A valid, accepted telegram clears stale diagnostics without changing its payload."""
engine, SessionLocal = dsmr_db
settings = _make_settings(dsmr_sample_interval_s=10)
with Session(engine) as session:
source = session.get(MeterSource, 1)
assert source is not None
source.enabled = True
source.status = "error"
source.last_error = "old error"
session.commit()
_call_handle_message(_SAMPLE_TELEGRAM, settings, SessionLocal)
with Session(engine) as session:
source = session.get(MeterSource, 1)
assert source is not None
assert source.status == "online"
assert source.last_error is None
assert source.last_seen_at is not None
def test_source_state_callback_persists_only_current_generation(dsmr_db, monkeypatch):
"""Connection callbacks use a short session and stale generations leave health untouched."""
engine, SessionLocal = dsmr_db
from app.services import dsmr_ingest
snapshot = _make_settings()
current = object()
monkeypatch.setattr(dsmr_ingest, "_subscription_tokens", {snapshot.source_id: current})
with Session(engine) as session:
source = session.get(MeterSource, snapshot.source_id)
assert source is not None
source.enabled = True
session.commit()
with patch.object(dsmr_ingest, "get_session_local", return_value=SessionLocal):
dsmr_ingest.handle_captured_source_state(snapshot, current, "connecting")
dsmr_ingest.handle_captured_source_state(snapshot, current, "error")
dsmr_ingest.handle_captured_source_state(snapshot, object(), "online")
with Session(engine) as session:
source = session.get(MeterSource, snapshot.source_id)
assert source is not None
assert source.status == "error"
assert source.last_error == "MQTT connection failed."
def test_source_health_is_isolated_between_sources(dsmr_db, monkeypatch):
"""Connection and message health changes only touch their own source row."""
engine, SessionLocal = dsmr_db
from app.services import dsmr_ingest
from app.services.dsmr_ingest import DsmrSourceSnapshot
first = _make_settings()
second = DsmrSourceSnapshot(2, "second/topic", "", 10)
first_token = object()
second_token = object()
monkeypatch.setattr(
dsmr_ingest, "_subscription_tokens", {first.source_id: first_token, second.source_id: second_token}
)
with Session(engine) as session:
first_source = session.get(MeterSource, first.source_id)
assert first_source is not None
first_source.enabled = True
session.add(MeterSource(
id=2, name="Second DSMR", kind="dsmr_mqtt", enabled=True, config={},
status="online", created_at=datetime.now(timezone.utc), updated_at=datetime.now(timezone.utc),
))
session.commit()
with patch.object(dsmr_ingest, "get_session_local", return_value=SessionLocal):
dsmr_ingest.handle_captured_source_state(first, first_token, "connecting")
dsmr_ingest.handle_captured_source_state(second, second_token, "error")
dsmr_ingest.handle_message(_payload(_SAMPLE_TELEGRAM), first)
with Session(engine) as session:
first_source = session.get(MeterSource, first.source_id)
second_source = session.get(MeterSource, second.source_id)
assert first_source is not None and second_source is not None
assert first_source.status == "online"
assert first_source.last_error is None
assert first_source.last_seen_at is not None
assert second_source.status == "error"
assert second_source.last_error == "MQTT connection failed."
assert second_source.last_seen_at is None
def test_disable_reconcile_persists_unknown_and_rejects_retained_state_callback(dsmr_db, monkeypatch):
"""Disable invalidates the generation before clearing a stale online health state."""
engine, SessionLocal = dsmr_db
from app.services import dsmr_ingest
class FakeMqtt:
def __init__(self) -> None:
self.removed: list[int] = []
self.state_handler = None
self.active: set[int] = set()
def replace_source(self, source_id: int, **kwargs: object) -> bool:
self.active.add(source_id)
self.state_handler = kwargs["state_handler"]
return True
def remove_source(self, source_id: int) -> None:
self.removed.append(source_id)
self.active.discard(source_id)
def source_is_active(self, source_id: int) -> bool:
return source_id in self.active
fake = FakeMqtt()
snapshot = _make_settings()
monkeypatch.setattr(dsmr_ingest, "_subscriptions", {})
monkeypatch.setattr(dsmr_ingest, "_subscription_client_ids", {})
monkeypatch.setattr(dsmr_ingest, "_subscription_tokens", {})
monkeypatch.setattr(dsmr_ingest, "_tariffs", {})
monkeypatch.setattr(dsmr_ingest, "get_session_local", lambda: SessionLocal)
monkeypatch.setattr("app.integrations.mqtt.mqtt_manager", fake)
monkeypatch.setattr(dsmr_ingest, "_enabled_snapshots", lambda: [snapshot])
with Session(engine) as session:
source = session.get(MeterSource, snapshot.source_id)
assert source is not None
source.enabled = True
source.status = "online"
session.commit()
dsmr_ingest.apply_dsmr_subscription()
assert fake.state_handler is not None
with Session(engine) as session:
source = session.get(MeterSource, snapshot.source_id)
assert source is not None
source.enabled = False
session.commit()
monkeypatch.setattr(dsmr_ingest, "_enabled_snapshots", lambda: [])
dsmr_ingest.apply_dsmr_subscription()
fake.state_handler("online")
with Session(engine) as session:
source = session.get(MeterSource, snapshot.source_id)
assert source is not None
assert source.status == "unknown"
assert source.last_error is None
assert fake.removed == [snapshot.source_id]
def test_topic_collision_persists_error_when_reconcile_removes_active_client(dsmr_db, monkeypatch):
"""A rejected enabled source cannot retain health from its removed client."""
engine, SessionLocal = dsmr_db
from app.services import dsmr_ingest
class FakeMqtt:
def __init__(self) -> None:
self.removed: list[int] = []
self.active: set[int] = set()
def replace_source(self, source_id: int, **kwargs: object) -> bool:
self.active.add(source_id)
return True
def remove_source(self, source_id: int) -> None:
self.removed.append(source_id)
self.active.discard(source_id)
def source_is_active(self, source_id: int) -> bool:
return source_id in self.active
fake = FakeMqtt()
valid = _make_settings(dsmr_mqtt_topic="dsmr/telegram")
collision = _make_settings(dsmr_mqtt_topic="dsmr/telegram")
collision = dsmr_ingest.DsmrSourceSnapshot(
collision.source_id,
collision.topic,
collision.topic,
collision.sample_interval_s,
)
monkeypatch.setattr(dsmr_ingest, "_subscriptions", {})
monkeypatch.setattr(dsmr_ingest, "_subscription_client_ids", {})
monkeypatch.setattr(dsmr_ingest, "_subscription_tokens", {})
monkeypatch.setattr(dsmr_ingest, "_tariffs", {})
monkeypatch.setattr(dsmr_ingest, "get_session_local", lambda: SessionLocal)
monkeypatch.setattr("app.integrations.mqtt.mqtt_manager", fake)
with Session(engine) as session:
source = session.get(MeterSource, valid.source_id)
assert source is not None
source.enabled = True
source.status = "online"
session.commit()
monkeypatch.setattr(dsmr_ingest, "_enabled_snapshots", lambda: [valid])
dsmr_ingest.apply_dsmr_subscription()
monkeypatch.setattr(dsmr_ingest, "_enabled_snapshots", lambda: [collision])
dsmr_ingest.apply_dsmr_subscription()
with Session(engine) as session:
source = session.get(MeterSource, valid.source_id)
assert source is not None
assert source.enabled is True
assert source.status == "error"
assert source.last_error == "DSMR source configuration invalid."
assert fake.removed == [valid.source_id]
assert fake.active == set()
def test_topic_collision_persists_error_on_startup_without_runtime_client(dsmr_db, monkeypatch):
"""A collision corrects stale persisted online health without an installed client."""
engine, SessionLocal = dsmr_db
from app.services import dsmr_ingest
collision = dsmr_ingest.DsmrSourceSnapshot(1, "dsmr/telegram", "dsmr/telegram", 10)
monkeypatch.setattr(dsmr_ingest, "_subscriptions", {})
monkeypatch.setattr(dsmr_ingest, "_subscription_client_ids", {})
monkeypatch.setattr(dsmr_ingest, "_subscription_tokens", {})
monkeypatch.setattr(dsmr_ingest, "get_session_local", lambda: SessionLocal)
monkeypatch.setattr(dsmr_ingest, "_enabled_snapshots", lambda: [collision])
with Session(engine) as session:
source = session.get(MeterSource, collision.source_id)
assert source is not None
source.enabled = True
source.status = "online"
session.commit()
dsmr_ingest.apply_dsmr_subscription()
with Session(engine) as session:
source = session.get(MeterSource, collision.source_id)
assert source is not None
assert source.status == "error"
assert source.last_error == "DSMR source configuration invalid."
def test_topic_collision_does_not_change_a_disabled_source(dsmr_db, monkeypatch):
"""A concurrent disable is not overwritten by collision error handling."""
engine, SessionLocal = dsmr_db
from app.services import dsmr_ingest
collision = dsmr_ingest.DsmrSourceSnapshot(1, "dsmr/telegram", "dsmr/telegram", 10)
monkeypatch.setattr(dsmr_ingest, "_subscriptions", {})
monkeypatch.setattr(dsmr_ingest, "_subscription_client_ids", {})
monkeypatch.setattr(dsmr_ingest, "_subscription_tokens", {})
monkeypatch.setattr(dsmr_ingest, "get_session_local", lambda: SessionLocal)
monkeypatch.setattr(dsmr_ingest, "_enabled_snapshots", lambda: [collision])
with Session(engine) as session:
source = session.get(MeterSource, collision.source_id)
assert source is not None
source.enabled = False
source.status = "unknown"
source.last_error = None
session.commit()
dsmr_ingest.apply_dsmr_subscription()
with Session(engine) as session:
source = session.get(MeterSource, collision.source_id)
assert source is not None
assert source.status == "unknown"
assert source.last_error is None
def test_second_10_persists(dsmr_db):
"""A telegram with second=10 (another 10s boundary) must also be persisted."""
_, SessionLocal = dsmr_db
@@ -240,6 +498,20 @@ def test_different_timestamps_are_independent(dsmr_db):
assert _count_readings(SessionLocal) == 2
def test_two_sources_can_store_the_same_timestamp_independently(dsmr_db):
"""The source identity, not timestamp alone, defines DSMR idempotency."""
_, SessionLocal = dsmr_db
first = _make_settings(dsmr_sample_interval_s=10)
from app.services.dsmr_ingest import DsmrSourceSnapshot
second = DsmrSourceSnapshot(2, "second/topic", "", 10)
_call_handle_message(_SAMPLE_TELEGRAM, first, SessionLocal)
_call_handle_message(_SAMPLE_TELEGRAM, second, SessionLocal)
rows = _get_readings(SessionLocal)
assert {row.meter_source_id for row in rows} == {1, 2}
def test_telegram_id_collision_does_not_drop_new_data(dsmr_db):
"""Regression: the telegram id overflows / gets reset to zero in DSMR firmware.
Two DISTINCT telegrams (different timestamps) that happen to share the SAME
@@ -259,12 +531,12 @@ def test_telegram_id_collision_does_not_drop_new_data(dsmr_db):
# ---------------------------------------------------------------------------
# 4. Missing source_id — still persisted with source_id=None
# 4. Missing telegram id — still persisted with telegram_id=None
# ---------------------------------------------------------------------------
def test_missing_id_persisted_with_source_id_none(dsmr_db):
"""Telegram without an 'id' field must be stored with source_id=None."""
def test_missing_id_persisted_with_telegram_id_none(dsmr_db):
"""Telegram without an 'id' field must be stored with telegram_id=None."""
engine, SessionLocal = dsmr_db
settings = _make_settings(dsmr_sample_interval_s=10)
@@ -276,7 +548,7 @@ def test_missing_id_persisted_with_source_id_none(dsmr_db):
readings = session.scalars(select(DsmrReading)).all()
assert len(readings) == 1
assert readings[0].source_id is None
assert readings[0].telegram_id is None
# ---------------------------------------------------------------------------
@@ -419,80 +691,87 @@ def test_timestamp_with_utc_offset_suffix(dsmr_db):
@pytest.fixture()
def reset_tariff(monkeypatch):
"""Reset _current_tariff to None before and after each tariff test."""
from app.services import dsmr_ingest as _di
monkeypatch.setattr(_di, "_tariffs", {})
monkeypatch.setattr(_di, "_current_tariff", None)
yield
# monkeypatch auto-restores on teardown
def test_tariff_message_value_2_sets_tariff(reset_tariff):
"""Payload b'2' must set the current tariff to 2 (normal/peak)."""
from app.services.dsmr_ingest import handle_tariff_message, get_current_tariff
handle_tariff_message(b"2")
assert get_current_tariff() == 2
handle_tariff_message(b"2", 1)
assert get_current_tariff(1) == 2
def test_tariff_message_value_1_sets_tariff(reset_tariff):
"""Payload b'1' must set the current tariff to 1 (dal/off-peak)."""
from app.services.dsmr_ingest import handle_tariff_message, get_current_tariff
handle_tariff_message(b"1")
assert get_current_tariff() == 1
handle_tariff_message(b"1", 1)
assert get_current_tariff(1) == 1
def test_tariff_message_updates_from_2_to_1(reset_tariff):
"""Subsequent payloads must overwrite the previous tariff value."""
from app.services.dsmr_ingest import handle_tariff_message, get_current_tariff
handle_tariff_message(b"2")
assert get_current_tariff() == 2
handle_tariff_message(b"1")
assert get_current_tariff() == 1
handle_tariff_message(b"2", 1)
assert get_current_tariff(1) == 2
handle_tariff_message(b"1", 1)
assert get_current_tariff(1) == 1
def test_tariffs_are_isolated_by_source(reset_tariff):
from app.services.dsmr_ingest import handle_tariff_message, get_current_tariff
handle_tariff_message(b"1", 1)
handle_tariff_message(b"2", 2)
assert get_current_tariff(1) == 1
assert get_current_tariff(2) == 2
def test_tariff_message_strips_whitespace(reset_tariff):
"""Payloads with surrounding whitespace (e.g. b'2\\n') must be accepted."""
from app.services.dsmr_ingest import handle_tariff_message, get_current_tariff
handle_tariff_message(b"2\n")
assert get_current_tariff() == 2
handle_tariff_message(b"2\n", 1)
assert get_current_tariff(1) == 2
handle_tariff_message(b" 1 ")
assert get_current_tariff() == 1
handle_tariff_message(b" 1 ", 1)
assert get_current_tariff(1) == 1
def test_tariff_message_invalid_non_numeric_does_not_update(reset_tariff):
"""Non-numeric payload must not update the tariff; previous value is preserved."""
from app.services.dsmr_ingest import handle_tariff_message, get_current_tariff, set_current_tariff
set_current_tariff(2)
handle_tariff_message(b"x")
set_current_tariff(1, 2)
handle_tariff_message(b"x", 1)
# Must NOT raise and must NOT change the tariff.
assert get_current_tariff() == 2
assert get_current_tariff(1) == 2
def test_tariff_message_invalid_empty_does_not_update(reset_tariff):
"""Empty payload must not update the tariff; previous value is preserved."""
from app.services.dsmr_ingest import handle_tariff_message, get_current_tariff, set_current_tariff
set_current_tariff(1)
handle_tariff_message(b"")
assert get_current_tariff() == 1
set_current_tariff(1, 1)
handle_tariff_message(b"", 1)
assert get_current_tariff(1) == 1
def test_tariff_message_out_of_range_value_does_not_update(reset_tariff):
"""Payload with out-of-range integer (not 1 or 2) must not update the tariff."""
from app.services.dsmr_ingest import handle_tariff_message, get_current_tariff, set_current_tariff
set_current_tariff(2)
handle_tariff_message(b"3") # 3 is not a valid tariff
assert get_current_tariff() == 2
set_current_tariff(1, 2)
handle_tariff_message(b"3", 1) # 3 is not a valid tariff
assert get_current_tariff(1) == 2
handle_tariff_message(b"0") # 0 is not a valid tariff
assert get_current_tariff() == 2
handle_tariff_message(b"0", 1) # 0 is not a valid tariff
assert get_current_tariff(1) == 2
def test_tariff_message_does_not_raise_on_any_input(reset_tariff):
@@ -501,4 +780,115 @@ def test_tariff_message_does_not_raise_on_any_input(reset_tariff):
# All of these must complete without raising.
for payload in (b"", b"x", b"99", b"\xff\xfe", b"None", b"2.0"):
handle_tariff_message(payload) # must not raise
handle_tariff_message(payload, 1) # must not raise
def test_electricity_tariff_resolves_current_binding_and_handoff(dsmr_db, reset_tariff):
"""Runtime tariffs remain isolated and are selected through the active binding."""
from datetime import timedelta
from app.models.energy import Meter
from app.models.meter_source import MeterSource, MeterSourceBinding, MeterSourceChannel
from app.services.dsmr_ingest import get_current_electricity_tariff, handle_tariff_message
_, SessionLocal = dsmr_db
now = datetime.now(timezone.utc).replace(microsecond=0)
with SessionLocal() as session:
meter = Meter(
label="electricity",
commodity="electricity",
started_at=now - timedelta(days=2),
ended_at=None,
reason="initial",
note=None,
created_at=now,
)
first = MeterSource(
name="first", kind="dsmr_mqtt", enabled=True, config={}, created_at=now, updated_at=now
)
second = MeterSource(
name="second", kind="dsmr_mqtt", enabled=True, config={}, created_at=now, updated_at=now
)
session.add_all([meter, first, second])
session.flush()
first_channel = MeterSourceChannel(
source_id=first.id, channel_key="electricity", label="first", unit="kWh",
created_at=now, updated_at=now
)
second_channel = MeterSourceChannel(
source_id=second.id, channel_key="electricity", label="second", unit="kWh",
created_at=now, updated_at=now
)
session.add_all([first_channel, second_channel])
session.flush()
handoff = now - timedelta(hours=1)
session.add_all([
MeterSourceBinding(meter_id=meter.id, channel_id=first_channel.id, started_at=now - timedelta(days=2), ended_at=handoff, created_at=now, updated_at=now),
MeterSourceBinding(meter_id=meter.id, channel_id=second_channel.id, started_at=handoff, ended_at=None, created_at=now, updated_at=now),
])
session.commit()
handle_tariff_message(b"1", first.id)
handle_tariff_message(b"2", second.id)
assert get_current_electricity_tariff(session, now - timedelta(days=3)) is None
assert get_current_electricity_tariff(session, handoff - timedelta(seconds=1)) == 1
assert get_current_electricity_tariff(session, handoff) == 2
assert get_current_electricity_tariff(session, now + timedelta(days=3)) == 2
def test_legacy_getter_resolves_runtime_tariff_through_current_binding(dsmr_db, reset_tariff):
"""The unchanged no-argument caller selects the bound source, not a global tariff."""
from datetime import timedelta
from app.models.energy import Meter
from app.models.meter_source import MeterSource, MeterSourceBinding, MeterSourceChannel
from app.services import dsmr_ingest
_, SessionLocal = dsmr_db
now = datetime.now(timezone.utc).replace(microsecond=0)
with SessionLocal() as session:
meter = Meter(
label="electricity", commodity="electricity", started_at=now - timedelta(days=1),
ended_at=None, reason="initial", note=None, created_at=now,
)
bound = MeterSource(
name="bound", kind="dsmr_mqtt", enabled=True, config={}, created_at=now, updated_at=now
)
other = MeterSource(
name="other", kind="dsmr_mqtt", enabled=True, config={}, created_at=now, updated_at=now
)
session.add_all([meter, bound, other])
session.flush()
channel = MeterSourceChannel(
source_id=bound.id, channel_key="electricity", label="bound", unit="kWh",
created_at=now, updated_at=now,
)
session.add(channel)
session.flush()
session.add(MeterSourceBinding(
meter_id=meter.id, channel_id=channel.id, started_at=now - timedelta(days=1), ended_at=None,
created_at=now, updated_at=now,
))
session.commit()
dsmr_ingest.handle_tariff_message(b"1", bound.id)
dsmr_ingest.handle_tariff_message(b"2", other.id)
with patch.object(dsmr_ingest, "get_session_local", return_value=SessionLocal):
assert dsmr_ingest.get_current_tariff() == 1
@pytest.mark.parametrize("replacement", ["disable", "delete", "config-change"])
def test_retained_source_handler_cannot_write_after_reconcile(dsmr_db, monkeypatch, replacement):
"""A callback fetched before disable/delete/reconfigure is rejected before DB access."""
from app.services import dsmr_ingest
from app.services.dsmr_ingest import DsmrSourceSnapshot
_, SessionLocal = dsmr_db
old = DsmrSourceSnapshot(1, "old", "", 10, broker_host="one.test")
if replacement in {"disable", "delete"}:
active = {}
else:
active = {1: DsmrSourceSnapshot(1, "new", "", 10, broker_host="changed.test")}
monkeypatch.setattr(dsmr_ingest, "_subscriptions", active)
with patch.object(dsmr_ingest, "get_session_local", return_value=SessionLocal):
dsmr_ingest.handle_captured_message(_payload(_SAMPLE_TELEGRAM), old)
assert _count_readings(SessionLocal) == 0
+285
View File
@@ -0,0 +1,285 @@
"""Isolated revision-14/15 fixtures for the DSMR source-adoption migration."""
from __future__ import annotations
import json
from datetime import datetime, timedelta, timezone
from pathlib import Path
import pytest
import sqlalchemy.exc
from alembic import command
from alembic.config import Config
from sqlalchemy import create_engine, event, inspect, text
def _config(database_url: str) -> Config:
config = Config("alembic_app.ini")
config.set_main_option("sqlalchemy.url", database_url)
return config
def _engine(database_url: str):
engine = create_engine(database_url, connect_args={"check_same_thread": False})
@event.listens_for(engine, "connect")
def _foreign_keys(connection, _record) -> None:
connection.execute("PRAGMA foreign_keys=ON")
return engine
def _insert_meter(connection, label: str, started: datetime, ended: datetime | None) -> int:
connection.execute(
text(
"INSERT INTO meter (uuid, label, commodity, started_at, ended_at, reason, note, created_at) "
"VALUES (:uuid, :label, 'electricity', :started, :ended, 'initial', NULL, :started)"
),
{"uuid": f"{label:0<8}-0000-4000-8000-000000000000", "label": label,
"started": started, "ended": ended},
)
return int(connection.execute(text("SELECT last_insert_rowid()")).scalar_one())
def _insert_contract_version(connection, start: datetime) -> int:
connection.execute(
text(
"INSERT INTO energy_contract (name, kind, active, currency, created_at, updated_at) "
"VALUES ('Historic contract', 'manual', 1, 'EUR', :at, :at)"
),
{"at": start},
)
contract_id = int(connection.execute(text("SELECT last_insert_rowid()")).scalar_one())
connection.execute(
text(
"INSERT INTO energy_contract_version "
"(contract_id, effective_from, effective_to, \"values\", created_at) "
"VALUES (:contract_id, :at, NULL, :values, :at)"
),
{"contract_id": contract_id, "at": start, "values": json.dumps({"historic": True})},
)
return int(connection.execute(text("SELECT last_insert_rowid()")).scalar_one())
def _insert_cost(
connection,
period_start: datetime,
meter_id: int,
contract_version_id: int,
sequence: int,
) -> None:
amount = 2.5 + sequence
connection.execute(
text(
"INSERT INTO energy_cost_period "
"(period_start, d1_kwh, d2_kwh, r1_kwh, r2_kwh, import_cost, export_revenue, net_cost, "
"currency, pricing, contract_version_id, meter_id, degraded, computed_at) "
"VALUES (:start, :d1, :d2, :r1, :r2, :import_cost, :export_revenue, :net_cost, "
"'EUR', :pricing, :contract_version_id, :meter, 0, :computed_at)"
),
{
"start": period_start,
"d1": 1.0 + sequence,
"d2": 2.0 + sequence,
"r1": 3.0 + sequence,
"r2": 4.0 + sequence,
"import_cost": amount,
"export_revenue": 0.25 + sequence,
"net_cost": amount - (0.25 + sequence),
"pricing": json.dumps({"historic": True, "sequence": sequence}),
"contract_version_id": contract_version_id,
"meter": meter_id,
"computed_at": period_start + timedelta(seconds=sequence),
},
)
def test_populated_revision_14_adopts_dsmr_history_at_revision_16(tmp_path: Path):
database_url = f"sqlite:///{tmp_path / 'revision_14_history.db'}"
config = _config(database_url)
command.upgrade(config, "20260625_14_meter_uuid")
start = datetime(2026, 8, 1, tzinfo=timezone.utc)
engine = _engine(database_url)
try:
with engine.begin() as connection:
connection.execute(
text("INSERT INTO app_config (key, value, updated_at) VALUES (:key, :value, :at)"),
[
{"key": "DSMR_INGEST_ENABLED", "value": "true", "at": start},
{"key": "DSMR_MQTT_TOPIC", "value": "historic/dsmr", "at": start},
{"key": "DSMR_TARIFF_TOPIC", "value": "historic/tariff", "at": start},
{"key": "DSMR_SAMPLE_INTERVAL_S", "value": "15", "at": start},
{"key": "MQTT_BROKER_HOST", "value": "mqtt.example.invalid", "at": start},
{"key": "MQTT_BROKER_PORT", "value": "1884", "at": start},
{"key": "MQTT_USERNAME", "value": "historic-user", "at": start},
{"key": "MQTT_PASSWORD", "value": "historic-password", "at": start},
{"key": "MQTT_TLS_ENABLED", "value": "true", "at": start},
{"key": "UNRELATED_CONFIG", "value": "untouched", "at": start},
],
)
config_before = dict(connection.execute(text("SELECT key, value FROM app_config")).all())
for offset, telegram_id in ((0, 77), (20, 78), (40, 77)):
connection.execute(
text("INSERT INTO dsmr_reading (recorded_at, source_id, payload) VALUES (:at, :id, :payload)"),
{"at": start + timedelta(minutes=offset), "id": telegram_id,
"payload": json.dumps({"id": telegram_id, "keep": f"payload-{offset}"})},
)
first = _insert_meter(connection, "meterone", start - timedelta(hours=1), start + timedelta(minutes=20))
second = _insert_meter(connection, "metertwo", start + timedelta(minutes=20), start + timedelta(minutes=40))
third = _insert_meter(connection, "meterthree", start + timedelta(minutes=40), None)
_insert_meter(connection, "nodata", start + timedelta(days=1), None)
contract_version = _insert_contract_version(connection, start - timedelta(days=1))
# One normal period per epoch plus a period ending exactly at each
# replacement boundary. Meter/binding intervals are half-open, so
# the latter must remain unbound/degraded.
_insert_cost(connection, start, first, contract_version, 0)
_insert_cost(connection, start + timedelta(minutes=5), first, contract_version, 1)
_insert_cost(connection, start + timedelta(minutes=21), second, contract_version, 2)
_insert_cost(connection, start + timedelta(minutes=25), second, contract_version, 3)
_insert_cost(connection, start + timedelta(minutes=41), third, contract_version, 4)
cost_before = connection.execute(
text(
"SELECT id, period_start, d1_kwh, d2_kwh, r1_kwh, r2_kwh, import_cost, "
"export_revenue, net_cost, currency, pricing, contract_version_id, meter_id, "
"degraded, computed_at FROM energy_cost_period ORDER BY period_start"
)
).mappings().all()
finally:
engine.dispose()
command.upgrade(config, "20260822_16_dsmr_source_adoption")
engine = _engine(database_url)
try:
with engine.connect() as connection:
assert connection.execute(text("SELECT version_num FROM alembic_version")).scalar_one() == (
"20260822_16_dsmr_source_adoption"
)
assert connection.execute(text("SELECT COUNT(*) FROM dsmr_reading")).scalar_one() == 3
source = connection.execute(
text("SELECT id, enabled, config FROM meter_source WHERE kind = 'dsmr_mqtt'")
).one()
assert source.enabled == 1
assert json.loads(source.config) == {
"broker_host": "mqtt.example.invalid", "broker_port": 1884,
"username": "historic-user", "password": "historic-password", "tls_enabled": True,
"topic": "historic/dsmr", "tariff_topic": "historic/tariff", "sample_interval_s": 15,
}
assert connection.execute(text("SELECT value FROM app_config WHERE key = 'DSMR_MQTT_TOPIC'")).scalar_one() == "historic/dsmr"
assert dict(connection.execute(text("SELECT key, value FROM app_config")).all()) == config_before
assert connection.execute(text("SELECT group_concat(telegram_id) FROM dsmr_reading")).scalar_one() == "77,78,77"
assert connection.execute(text("SELECT payload FROM dsmr_reading ORDER BY recorded_at")).scalars().all() == [
json.dumps({"id": 77, "keep": "payload-0"}),
json.dumps({"id": 78, "keep": "payload-20"}),
json.dumps({"id": 77, "keep": "payload-40"}),
]
assert connection.execute(text("SELECT COUNT(*) FROM meter_source_binding")).scalar_one() == 3
assert connection.execute(
text("SELECT COUNT(*) FROM meter_source_binding WHERE meter_id = (SELECT id FROM meter WHERE label = 'nodata')")
).scalar_one() == 0
periods = connection.execute(
text(
"SELECT id, period_start, d1_kwh, d2_kwh, r1_kwh, r2_kwh, import_cost, "
"export_revenue, net_cost, currency, pricing, contract_version_id, meter_id, "
"degraded, computed_at, source_binding_id FROM energy_cost_period ORDER BY period_start"
)
).mappings().all()
assert [
{key: value for key, value in period.items() if key not in {"degraded", "source_binding_id"}}
for period in periods
] == [
{key: value for key, value in period.items() if key != "degraded"}
for period in cost_before
]
assert [(period["degraded"], period["source_binding_id"] is not None) for period in periods] == [
(0, True), (1, False), (0, True), (1, False), (0, True)
]
assert [period["period_start"] for period in periods if period["source_binding_id"] is None] == [
(start + timedelta(minutes=5)).isoformat(sep=" "),
(start + timedelta(minutes=25)).isoformat(sep=" "),
]
bound_meter_ids = connection.execute(
text(
"SELECT binding.meter_id FROM energy_cost_period AS period "
"LEFT JOIN meter_source_binding AS binding "
"ON binding.id = period.source_binding_id ORDER BY period.period_start"
)
).scalars().all()
assert bound_meter_ids == [first, None, second, None, third]
assert connection.execute(text("PRAGMA foreign_key_check")).all() == []
finally:
engine.dispose()
def test_dsmr_source_timestamp_uniqueness_allows_two_sources(tmp_path: Path):
database_url = f"sqlite:///{tmp_path / 'two_sources.db'}"
config = _config(database_url)
command.upgrade(config, "20260822_16_dsmr_source_adoption")
engine = _engine(database_url)
timestamp = datetime(2026, 8, 1, tzinfo=timezone.utc)
try:
with engine.begin() as connection:
first_source = connection.execute(text("SELECT id FROM meter_source WHERE kind = 'dsmr_mqtt'")).scalar_one()
assert connection.execute(
text("SELECT enabled FROM meter_source WHERE id = :id"), {"id": first_source}
).scalar_one() == 0
connection.execute(
text(
"INSERT INTO meter_source (uuid, name, kind, enabled, config, status, created_at, updated_at) "
"VALUES ('22222222-2222-4222-8222-222222222222', 'Second DSMR', 'dsmr_mqtt', 0, '{}', "
"'unknown', :at, :at)"
), {"at": timestamp},
)
second_source = connection.execute(text("SELECT last_insert_rowid()")).scalar_one()
for source_id in (first_source, second_source):
connection.execute(
text(
"INSERT INTO dsmr_reading (recorded_at, telegram_id, meter_source_id, payload) "
"VALUES (:at, 9, :source, '{}')"
), {"at": timestamp, "source": source_id},
)
with pytest.raises(sqlalchemy.exc.IntegrityError):
connection.execute(
text(
"INSERT INTO dsmr_reading (recorded_at, telegram_id, meter_source_id, payload) "
"VALUES (:at, 10, :source, '{}')"
), {"at": timestamp, "source": first_source},
)
finally:
engine.dispose()
inspector = inspect(create_engine(database_url))
assert ("meter_source_id", "recorded_at") in {
tuple(item["column_names"]) for item in inspector.get_unique_constraints("dsmr_reading")
}
def test_revision_15_without_legacy_dsmr_config_creates_unconfigured_source(tmp_path: Path):
database_url = f"sqlite:///{tmp_path / 'revision_15_no_dsmr_config.db'}"
config = _config(database_url)
command.upgrade(config, "20260822_15_meter_sources")
engine = _engine(database_url)
try:
with engine.begin() as connection:
connection.execute(
text(
"INSERT INTO app_config (key, value, updated_at) "
"VALUES ('UNRELATED_CONFIG', 'untouched', :at)"
),
{"at": datetime(2026, 8, 1, tzinfo=timezone.utc)},
)
finally:
engine.dispose()
command.upgrade(config, "20260822_16_dsmr_source_adoption")
engine = _engine(database_url)
try:
with engine.connect() as connection:
source = connection.execute(
text("SELECT enabled, config FROM meter_source WHERE kind = 'dsmr_mqtt'")
).one()
assert source.enabled == 0
assert json.loads(source.config) == {}
assert dict(connection.execute(text("SELECT key, value FROM app_config")).all()) == {
"UNRELATED_CONFIG": "untouched"
}
finally:
engine.dispose()
+159 -129
View File
@@ -1,173 +1,203 @@
"""Tests for restart-free DSMR subscription management (apply_dsmr_subscription).
These verify that toggling DSMR ingest / changing its topic / changing its sample
interval via the config UI is reflected in the live MQTT subscription without an
app restart. A fake MQTT manager is injected so no real broker is touched.
"""
"""DSMR source-driven MQTT subscription reconciliation tests."""
from __future__ import annotations
from unittest.mock import MagicMock
from dataclasses import replace
from types import SimpleNamespace
import pytest
from app.services import dsmr_ingest
from app.services.dsmr_ingest import DsmrSourceSnapshot
class _FakeMqtt:
def __init__(self) -> None:
self.subscribe_calls: list[tuple[str, object]] = []
self.unsubscribe_calls: list[str] = []
self.replace_calls: list[tuple[int, dict[str, object]]] = []
self.remove_calls: list[int] = []
self.handlers: dict[int, dict[str, object]] = {}
def subscribe(self, topic: str, handler) -> None:
self.subscribe_calls.append((topic, handler))
def replace_source(self, source_id: int, **kwargs: object) -> bool:
self.replace_calls.append((source_id, kwargs))
self.handlers[source_id] = kwargs["subscriptions"] # type: ignore[assignment]
return True
def unsubscribe(self, topic: str) -> None:
self.unsubscribe_calls.append(topic)
def remove_source(self, source_id: int) -> None:
self.remove_calls.append(source_id)
self.handlers.pop(source_id, None)
def source_is_active(self, source_id: int) -> bool:
return source_id in self.handlers
def _settings(
*,
enabled: bool = True,
topic: str = "dsmr/json",
interval: int = 10,
def _source(
source_id: int,
topic: str,
tariff_topic: str = "",
):
s = MagicMock()
s.dsmr_ingest_enabled = enabled
s.dsmr_mqtt_topic = topic
s.dsmr_sample_interval_s = interval
s.dsmr_tariff_topic = tariff_topic
return s
interval: int = 10,
**connection: object,
) -> DsmrSourceSnapshot:
return DsmrSourceSnapshot(source_id, topic, tariff_topic, interval, **connection)
@pytest.fixture()
def fake_mqtt(monkeypatch):
fake = _FakeMqtt()
# apply_dsmr_subscription does `from app.integrations.mqtt import mqtt_manager`
# at call time, so patching the module attribute is picked up.
monkeypatch.setattr("app.integrations.mqtt.mqtt_manager", fake)
# Reset (and auto-restore) the module-level "currently subscribed topics".
monkeypatch.setattr(dsmr_ingest, "_current_dsmr_topic", None)
monkeypatch.setattr(dsmr_ingest, "_current_tariff_topic", None)
monkeypatch.setattr(dsmr_ingest, "_subscriptions", {})
monkeypatch.setattr(dsmr_ingest, "_subscription_client_ids", {})
monkeypatch.setattr(dsmr_ingest, "_subscription_tokens", {})
monkeypatch.setattr(dsmr_ingest, "_tariffs", {})
return fake
def test_enabled_subscribes_to_topic(fake_mqtt):
dsmr_ingest.apply_dsmr_subscription(_settings(enabled=True, topic="dsmr/json"))
assert len(fake_mqtt.subscribe_calls) == 1
assert fake_mqtt.subscribe_calls[0][0] == "dsmr/json"
assert fake_mqtt.unsubscribe_calls == []
assert dsmr_ingest._current_dsmr_topic == "dsmr/json"
def test_reconcile_subscribes_each_enabled_source(fake_mqtt, monkeypatch):
monkeypatch.setattr(dsmr_ingest, "_enabled_snapshots", lambda: [_source(1, "a"), _source(2, "b")])
dsmr_ingest.apply_dsmr_subscription()
assert [source_id for source_id, _ in fake_mqtt.replace_calls] == [1, 2]
def test_disabled_after_enabled_unsubscribes(fake_mqtt):
dsmr_ingest.apply_dsmr_subscription(_settings(enabled=True, topic="dsmr/json"))
dsmr_ingest.apply_dsmr_subscription(_settings(enabled=False))
assert fake_mqtt.unsubscribe_calls == ["dsmr/json"]
assert dsmr_ingest._current_dsmr_topic is None
def test_changed_source_replaces_only_its_subscription(fake_mqtt, monkeypatch):
monkeypatch.setattr(dsmr_ingest, "_enabled_snapshots", lambda: [_source(1, "a"), _source(2, "b")])
dsmr_ingest.apply_dsmr_subscription()
monkeypatch.setattr(dsmr_ingest, "_enabled_snapshots", lambda: [_source(1, "changed"), _source(2, "b")])
dsmr_ingest.apply_dsmr_subscription()
assert fake_mqtt.remove_calls == [1]
assert [source_id for source_id, _ in fake_mqtt.replace_calls] == [1, 2, 1]
def test_topic_change_unsubscribes_old_subscribes_new(fake_mqtt):
dsmr_ingest.apply_dsmr_subscription(_settings(topic="dsmr/json"))
dsmr_ingest.apply_dsmr_subscription(_settings(topic="meter/dsmr"))
assert fake_mqtt.unsubscribe_calls == ["dsmr/json"]
assert [t for t, _ in fake_mqtt.subscribe_calls] == ["dsmr/json", "meter/dsmr"]
assert dsmr_ingest._current_dsmr_topic == "meter/dsmr"
def test_disable_unsubscribes_and_clears_source_tariff(fake_mqtt, monkeypatch):
monkeypatch.setattr(dsmr_ingest, "_enabled_snapshots", lambda: [_source(1, "a", "tariff/a")])
dsmr_ingest.apply_dsmr_subscription()
dsmr_ingest.set_current_tariff(1, 2)
monkeypatch.setattr(dsmr_ingest, "_enabled_snapshots", lambda: [])
dsmr_ingest.apply_dsmr_subscription()
assert fake_mqtt.remove_calls == [1]
assert dsmr_ingest.get_current_tariff(1) is None
def test_reapply_same_topic_resubscribes_fresh_handler(fake_mqtt):
# A changed sample interval must take effect — the handler is re-bound to a
# fresh settings snapshot, so re-applying the same topic re-subscribes.
dsmr_ingest.apply_dsmr_subscription(_settings(topic="dsmr/json", interval=10))
dsmr_ingest.apply_dsmr_subscription(_settings(topic="dsmr/json", interval=20))
assert len(fake_mqtt.subscribe_calls) == 2
assert fake_mqtt.unsubscribe_calls == [] # same topic, no churn
handler1 = fake_mqtt.subscribe_calls[0][1]
handler2 = fake_mqtt.subscribe_calls[1][1]
assert handler1 is not handler2 # fresh closure carrying the new settings
def test_same_snapshot_has_no_subscription_churn(fake_mqtt, monkeypatch):
monkeypatch.setattr(dsmr_ingest, "_enabled_snapshots", lambda: [_source(1, "a")])
dsmr_ingest.apply_dsmr_subscription()
dsmr_ingest.apply_dsmr_subscription()
assert len(fake_mqtt.replace_calls) == 1
assert fake_mqtt.remove_calls == []
def test_disabled_when_never_enabled_is_noop(fake_mqtt):
dsmr_ingest.apply_dsmr_subscription(_settings(enabled=False))
def test_changed_base_client_id_replaces_all_enabled_sources(fake_mqtt, monkeypatch):
snapshot = _source(1, "topic", broker_host="broker.test")
monkeypatch.setattr(dsmr_ingest, "_enabled_snapshots", lambda: [snapshot])
assert fake_mqtt.subscribe_calls == []
assert fake_mqtt.unsubscribe_calls == []
assert dsmr_ingest._current_dsmr_topic is None
dsmr_ingest.apply_dsmr_subscription(SimpleNamespace(mqtt_client_id="home-automation"))
dsmr_ingest.apply_dsmr_subscription(SimpleNamespace(mqtt_client_id="home-automation-dev"))
assert fake_mqtt.remove_calls == [1]
assert [source_id for source_id, _ in fake_mqtt.replace_calls] == [1, 1]
assert fake_mqtt.replace_calls[-1][1]["base_client_id"] == "home-automation-dev"
# ---------------------------------------------------------------------------
# Tariff topic subscription management
# ---------------------------------------------------------------------------
def test_enabled_with_tariff_topic_subscribes_both(fake_mqtt):
"""When enabled and tariff_topic is non-empty, both topics must be subscribed."""
dsmr_ingest.apply_dsmr_subscription(
_settings(enabled=True, topic="dsmr/json", tariff_topic="dsmr/meter-stats/electricity_tariff")
def test_same_topic_on_different_brokers_is_allowed(fake_mqtt, monkeypatch):
monkeypatch.setattr(
dsmr_ingest,
"_enabled_snapshots",
lambda: [_source(1, "same", broker_host="one.test"), _source(2, "same", broker_host="two.test")],
)
subscribed_topics = [t for t, _ in fake_mqtt.subscribe_calls]
assert "dsmr/json" in subscribed_topics
assert "dsmr/meter-stats/electricity_tariff" in subscribed_topics
assert len(fake_mqtt.subscribe_calls) == 2
assert dsmr_ingest._current_tariff_topic == "dsmr/meter-stats/electricity_tariff"
dsmr_ingest.apply_dsmr_subscription()
assert [source_id for source_id, _ in fake_mqtt.replace_calls] == [1, 2]
def test_enabled_with_empty_tariff_topic_subscribes_only_main(fake_mqtt):
"""When tariff_topic is empty, only the main DSMR topic is subscribed."""
dsmr_ingest.apply_dsmr_subscription(
_settings(enabled=True, topic="dsmr/json", tariff_topic="")
@pytest.mark.parametrize(
"sources",
[
[_source(1, "shared", "shared")],
],
)
def test_duplicate_telegram_or_tariff_topic_is_rejected(fake_mqtt, monkeypatch, sources):
"""One MQTT topic cannot safely dispatch to more than one source handler."""
monkeypatch.setattr(dsmr_ingest, "_enabled_snapshots", lambda: sources)
dsmr_ingest.apply_dsmr_subscription()
assert fake_mqtt.replace_calls == []
@pytest.mark.parametrize(
"field,value",
[
("broker_host", "changed.test"),
("broker_port", 2883),
("username", "different-user"),
("password", "different-password"),
("tls_enabled", True),
("sample_interval_s", 30),
("tariff_topic", "tariff/changed"),
],
)
def test_each_source_config_change_replaces_only_that_source(fake_mqtt, monkeypatch, field, value):
first = _source(1, "a", "tariff/a", broker_host="one.test", username="one")
second = _source(2, "b", "tariff/b", broker_host="two.test", username="two")
monkeypatch.setattr(dsmr_ingest, "_enabled_snapshots", lambda: [first, second])
dsmr_ingest.apply_dsmr_subscription()
changed = _source(1, "a", "tariff/a", broker_host="one.test", username="one")
changed = replace(changed, **{field: value})
monkeypatch.setattr(dsmr_ingest, "_enabled_snapshots", lambda: [changed, second])
dsmr_ingest.apply_dsmr_subscription()
assert fake_mqtt.remove_calls == [1]
assert [source_id for source_id, _ in fake_mqtt.replace_calls] == [1, 2, 1]
def test_failed_replace_is_not_marked_applied_and_is_retried(fake_mqtt, monkeypatch):
snapshot = _source(1, "a", broker_host="one.test")
monkeypatch.setattr(dsmr_ingest, "_enabled_snapshots", lambda: [snapshot])
original_replace = fake_mqtt.replace_source
outcomes = iter([False, True])
def replace_once_fails(source_id: int, **kwargs: object) -> bool:
original_replace(source_id, **kwargs)
return next(outcomes)
fake_mqtt.replace_source = replace_once_fails # type: ignore[method-assign]
dsmr_ingest.apply_dsmr_subscription()
dsmr_ingest.apply_dsmr_subscription()
assert [source_id for source_id, _ in fake_mqtt.replace_calls] == [1, 1]
def test_inactive_manager_source_is_rebuilt_on_next_reconcile(fake_mqtt, monkeypatch):
snapshot = _source(1, "a", broker_host="one.test")
monkeypatch.setattr(dsmr_ingest, "_enabled_snapshots", lambda: [snapshot])
dsmr_ingest.apply_dsmr_subscription()
fake_mqtt.handlers.clear() # models MqttManager.disconnect() tearing down clients
dsmr_ingest.apply_dsmr_subscription()
assert [source_id for source_id, _ in fake_mqtt.replace_calls] == [1, 1]
def test_same_snapshot_aba_rejects_retained_handler(fake_mqtt, monkeypatch):
"""An equal re-enabled snapshot has a fresh callback identity token."""
snapshot = _source(1, "same", broker_host="one.test")
received: list[tuple[bytes, DsmrSourceSnapshot]] = []
monkeypatch.setattr(dsmr_ingest, "_enabled_snapshots", lambda: [snapshot])
monkeypatch.setattr(
dsmr_ingest, "handle_message", lambda payload, captured: received.append((payload, captured))
)
dsmr_ingest.apply_dsmr_subscription()
old_handler = fake_mqtt.handlers[1]["same"]
monkeypatch.setattr(dsmr_ingest, "_enabled_snapshots", lambda: [])
dsmr_ingest.apply_dsmr_subscription()
monkeypatch.setattr(dsmr_ingest, "_enabled_snapshots", lambda: [snapshot])
dsmr_ingest.apply_dsmr_subscription()
assert len(fake_mqtt.subscribe_calls) == 1
assert fake_mqtt.subscribe_calls[0][0] == "dsmr/json"
assert dsmr_ingest._current_tariff_topic is None
old_handler(b"stale") # type: ignore[operator]
fake_mqtt.handlers[1]["same"](b"fresh") # type: ignore[operator]
assert received == [(b"fresh", snapshot)]
def test_disabled_after_tariff_subscription_unsubscribes_both(fake_mqtt):
"""Disabling ingest must also unsubscribe the tariff topic."""
dsmr_ingest.apply_dsmr_subscription(
_settings(enabled=True, topic="dsmr/json", tariff_topic="dsmr/meter-stats/electricity_tariff")
)
dsmr_ingest.apply_dsmr_subscription(_settings(enabled=False))
def test_source_state_handler_is_generation_scoped(fake_mqtt, monkeypatch):
snapshot = _source(1, "same", broker_host="one.test")
monkeypatch.setattr(dsmr_ingest, "_enabled_snapshots", lambda: [snapshot])
dsmr_ingest.apply_dsmr_subscription()
stale = fake_mqtt.replace_calls[-1][1]["state_handler"]
monkeypatch.setattr(dsmr_ingest, "_enabled_snapshots", lambda: [])
dsmr_ingest.apply_dsmr_subscription()
monkeypatch.setattr(dsmr_ingest, "_enabled_snapshots", lambda: [snapshot])
dsmr_ingest.apply_dsmr_subscription()
assert "dsmr/json" in fake_mqtt.unsubscribe_calls
assert "dsmr/meter-stats/electricity_tariff" in fake_mqtt.unsubscribe_calls
assert dsmr_ingest._current_dsmr_topic is None
assert dsmr_ingest._current_tariff_topic is None
def test_tariff_topic_change_resubscribes(fake_mqtt):
"""Changing the tariff topic must unsubscribe the old one and subscribe the new one."""
old_tariff = "dsmr/meter-stats/electricity_tariff"
new_tariff = "meter/tariff"
dsmr_ingest.apply_dsmr_subscription(
_settings(enabled=True, topic="dsmr/json", tariff_topic=old_tariff)
)
dsmr_ingest.apply_dsmr_subscription(
_settings(enabled=True, topic="dsmr/json", tariff_topic=new_tariff)
)
assert old_tariff in fake_mqtt.unsubscribe_calls
subscribed_topics = [t for t, _ in fake_mqtt.subscribe_calls]
assert new_tariff in subscribed_topics
assert dsmr_ingest._current_tariff_topic == new_tariff
def test_tariff_topic_cleared_unsubscribes(fake_mqtt):
"""Setting tariff_topic to empty after it was subscribed must unsubscribe it."""
dsmr_ingest.apply_dsmr_subscription(
_settings(enabled=True, topic="dsmr/json", tariff_topic="dsmr/meter-stats/electricity_tariff")
)
dsmr_ingest.apply_dsmr_subscription(
_settings(enabled=True, topic="dsmr/json", tariff_topic="")
)
assert "dsmr/meter-stats/electricity_tariff" in fake_mqtt.unsubscribe_calls
assert dsmr_ingest._current_tariff_topic is None
monkeypatch.setattr(dsmr_ingest, "get_session_local", lambda: (_ for _ in ()).throw(AssertionError()))
stale("error") # type: ignore[operator]
+1086 -244
View File
File diff suppressed because it is too large Load Diff
+343 -33
View File
@@ -28,16 +28,18 @@ Coverage:
from __future__ import annotations
from datetime import datetime, timezone
from datetime import datetime, timedelta, timezone
from decimal import Decimal
from pathlib import Path
from typing import Any
from unittest.mock import MagicMock, patch
from zoneinfo import ZoneInfo
import pytest
from alembic import command
from alembic.config import Config
from sqlalchemy import create_engine
from sqlalchemy.orm import Session
from sqlalchemy import create_engine, select
from sqlalchemy.orm import Session, sessionmaker
# ---------------------------------------------------------------------------
@@ -1097,6 +1099,9 @@ def test_import_cost_total_includes_standing_charges(energy_db) -> None:
from app.services import timezone as _tz_mod
now_utc = datetime.now(timezone.utc)
# Keep the dynamic test date, but settle it at a deterministic business time
# beyond the local 01:05 fixed-fee/credit cutoff.
settled_local_now = now_utc.replace(hour=12, minute=0, second=0, microsecond=0)
# D2 anchor = meter.started_at = 10 UTC days ago at midnight
meter_started_at = now_utc.replace(hour=0, minute=0, second=0, microsecond=0) - timedelta(days=10)
effective_from = meter_started_at # contract also starts at the same time
@@ -1123,7 +1128,10 @@ def test_import_cost_total_includes_standing_charges(energy_db) -> None:
with Session(energy_db) as session:
# Pin to UTC so local days = UTC days (deterministic on any CI host).
with patch.object(_tz_mod, "local_tz", return_value=ZoneInfo("UTC")):
with (
patch.object(_tz_mod, "local_tz", return_value=ZoneInfo("UTC")),
patch("app.services.energy_cost.local_now", return_value=settled_local_now),
):
catalog = build_catalog(session)
import_entry = next(
e for e in catalog if e.entity.key == "energy.import_cost_total"
@@ -1163,6 +1171,9 @@ def test_export_revenue_total_includes_tax_credit(energy_db) -> None:
from app.services import timezone as _tz_mod
now_utc = datetime.now(timezone.utc)
# Keep the dynamic test date, but settle it at a deterministic business time
# beyond the local 01:05 fixed-fee/credit cutoff.
settled_local_now = now_utc.replace(hour=12, minute=0, second=0, microsecond=0)
meter_started_at = now_utc.replace(hour=0, minute=0, second=0, microsecond=0) - timedelta(days=4)
effective_from = meter_started_at
@@ -1186,7 +1197,10 @@ def test_export_revenue_total_includes_tax_credit(energy_db) -> None:
session.commit()
with Session(energy_db) as session:
with patch.object(_tz_mod, "local_tz", return_value=ZoneInfo("UTC")):
with (
patch.object(_tz_mod, "local_tz", return_value=ZoneInfo("UTC")),
patch("app.services.energy_cost.local_now", return_value=settled_local_now),
):
catalog = build_catalog(session)
export_entry = next(
e for e in catalog if e.entity.key == "energy.export_revenue_total"
@@ -1412,25 +1426,59 @@ _DUAL_TARIFF_PRICING = {
@pytest.fixture()
def reset_tariff(monkeypatch):
"""Reset dsmr_ingest._current_tariff to None before/after each tariff test."""
def reset_tariff(energy_db, monkeypatch):
"""Isolate legacy and source-scoped DSMR tariff state for each tariff test."""
from app.services import dsmr_ingest as _di
monkeypatch.setattr(_di, "_current_tariff", None)
yield
test_session_local = sessionmaker(bind=energy_db)
monkeypatch.setattr(_di, "get_session_local", lambda: test_session_local)
_di._current_tariff = None
with _di._tariff_lock:
_di._tariffs.clear()
try:
yield
finally:
_di._current_tariff = None
with _di._tariff_lock:
_di._tariffs.clear()
def _insert_manual_period(energy_db) -> None:
"""Insert an active electricity meter + non-degraded manual pricing period into energy_db.
def _bind_migrated_dsmr_source(session: Session, meter: Any, *, now: datetime) -> int:
"""Bind the migration-created DSMR source to an active electricity meter."""
from app.models.meter_source import MeterSource, MeterSourceBinding, MeterSourceChannel
source = session.scalar(select(MeterSource).where(MeterSource.kind == "dsmr_mqtt"))
assert source is not None, "Alembic head must provide the migrated DSMR source"
channel = session.scalar(
select(MeterSourceChannel).where(MeterSourceChannel.source_id == source.id)
)
assert channel is not None, "Alembic head must provide the migrated DSMR channel"
session.add(
MeterSourceBinding(
meter_id=meter.id,
channel_id=channel.id,
started_at=meter.started_at,
ended_at=None,
created_at=now,
updated_at=now,
)
)
session.flush()
return source.id
def _insert_manual_period(energy_db) -> int:
"""Insert a bound active electricity meter and manual pricing period.
FUE-T05: the energy-cost provider requires an active electricity meter, so
this helper inserts one alongside the period so that build_catalog returns
the energy entities.
this helper binds its migration-created DSMR source to that meter. Return
the binding's source id so tariff tests exercise source-scoped state.
"""
t0 = datetime(2026, 3, 1, 10, 0, tzinfo=timezone.utc)
meter_start = datetime(2026, 3, 1, 0, 0, tzinfo=timezone.utc)
with Session(energy_db) as session:
_make_active_meter(session, started_at=meter_start, label="Tariff Test Meter")
meter = _make_active_meter(session, started_at=meter_start, label="Tariff Test Meter")
source_id = _bind_migrated_dsmr_source(session, meter, now=t0)
_make_period(
session,
period_start=t0,
@@ -1441,6 +1489,7 @@ def _insert_manual_period(energy_db) -> None:
degraded=False,
)
session.commit()
return source_id
def test_buy_price_tariff_1_returns_dal(energy_db, reset_tariff) -> None:
@@ -1448,8 +1497,8 @@ def test_buy_price_tariff_1_returns_dal(energy_db, reset_tariff) -> None:
from app.integrations.expose import build_catalog
from app.services.dsmr_ingest import set_current_tariff
_insert_manual_period(energy_db)
set_current_tariff(1)
source_id = _insert_manual_period(energy_db)
set_current_tariff(source_id, 1)
with Session(energy_db) as session:
catalog = build_catalog(session)
@@ -1464,8 +1513,8 @@ def test_buy_price_tariff_2_returns_normal(energy_db, reset_tariff) -> None:
from app.integrations.expose import build_catalog
from app.services.dsmr_ingest import set_current_tariff
_insert_manual_period(energy_db)
set_current_tariff(2)
source_id = _insert_manual_period(energy_db)
set_current_tariff(source_id, 2)
with Session(energy_db) as session:
catalog = build_catalog(session)
@@ -1480,8 +1529,8 @@ def test_buy_price_tariff_none_falls_back_to_normal(energy_db, reset_tariff) ->
from app.integrations.expose import build_catalog
from app.services.dsmr_ingest import set_current_tariff
_insert_manual_period(energy_db)
set_current_tariff(None) # explicitly None (no tariff received)
source_id = _insert_manual_period(energy_db)
set_current_tariff(source_id, None) # explicitly None (no tariff received)
with Session(energy_db) as session:
catalog = build_catalog(session)
@@ -1496,8 +1545,8 @@ def test_sell_price_tariff_1_returns_dal(energy_db, reset_tariff) -> None:
from app.integrations.expose import build_catalog
from app.services.dsmr_ingest import set_current_tariff
_insert_manual_period(energy_db)
set_current_tariff(1)
source_id = _insert_manual_period(energy_db)
set_current_tariff(source_id, 1)
with Session(energy_db) as session:
catalog = build_catalog(session)
@@ -1512,8 +1561,8 @@ def test_sell_price_tariff_2_returns_normal(energy_db, reset_tariff) -> None:
from app.integrations.expose import build_catalog
from app.services.dsmr_ingest import set_current_tariff
_insert_manual_period(energy_db)
set_current_tariff(2)
source_id = _insert_manual_period(energy_db)
set_current_tariff(source_id, 2)
with Session(energy_db) as session:
catalog = build_catalog(session)
@@ -1528,8 +1577,8 @@ def test_sell_price_tariff_none_falls_back_to_normal(energy_db, reset_tariff) ->
from app.integrations.expose import build_catalog
from app.services.dsmr_ingest import set_current_tariff
_insert_manual_period(energy_db)
set_current_tariff(None)
source_id = _insert_manual_period(energy_db)
set_current_tariff(source_id, None)
with Session(energy_db) as session:
catalog = build_catalog(session)
@@ -1559,7 +1608,8 @@ def test_tibber_buy_price_not_affected_by_tariff(energy_db, reset_tariff) -> Non
}
with Session(energy_db) as session:
_make_active_meter(session, started_at=meter_start)
meter = _make_active_meter(session, started_at=meter_start)
source_id = _bind_migrated_dsmr_source(session, meter, now=t0)
_make_period(
session,
period_start=t0,
@@ -1570,7 +1620,7 @@ def test_tibber_buy_price_not_affected_by_tariff(energy_db, reset_tariff) -> Non
# Tibber pricing must return the same value regardless of tariff.
for tariff_val in (1, 2, None):
set_current_tariff(tariff_val)
set_current_tariff(source_id, tariff_val)
with Session(energy_db) as session:
catalog = build_catalog(session)
buy_entry = next(e for e in catalog if e.entity.key == "energy.buy_price_now")
@@ -1600,7 +1650,8 @@ def test_tibber_sell_price_not_affected_by_tariff(energy_db, reset_tariff) -> No
}
with Session(energy_db) as session:
_make_active_meter(session, started_at=meter_start)
meter = _make_active_meter(session, started_at=meter_start)
source_id = _bind_migrated_dsmr_source(session, meter, now=t0)
_make_period(
session,
period_start=t0,
@@ -1610,7 +1661,7 @@ def test_tibber_sell_price_not_affected_by_tariff(energy_db, reset_tariff) -> No
session.commit()
for tariff_val in (1, 2, None):
set_current_tariff(tariff_val)
set_current_tariff(source_id, tariff_val)
with Session(energy_db) as session:
catalog = build_catalog(session)
sell_entry = next(e for e in catalog if e.entity.key == "energy.sell_price_now")
@@ -1945,6 +1996,9 @@ def test_cumulative_anchor_is_meter_started_at(energy_db) -> None:
from app.services import timezone as _tz_mod
now_utc = datetime.now(timezone.utc)
# Keep the dynamic test date, but settle it at a deterministic business time
# beyond the local 01:05 fixed-fee/credit cutoff.
settled_local_now = now_utc.replace(hour=12, minute=0, second=0, microsecond=0)
midnight_today = now_utc.replace(hour=0, minute=0, second=0, microsecond=0)
# Contract starts 180 days ago (far before the meter)
@@ -1966,7 +2020,10 @@ def test_cumulative_anchor_is_meter_started_at(energy_db) -> None:
session.commit()
with Session(energy_db) as session:
with patch.object(_tz_mod, "local_tz", return_value=ZoneInfo("UTC")):
with (
patch.object(_tz_mod, "local_tz", return_value=ZoneInfo("UTC")),
patch("app.services.energy_cost.local_now", return_value=settled_local_now),
):
catalog = build_catalog(session)
import_entry = next(
e for e in catalog if e.entity.key == "energy.import_cost_total"
@@ -2089,6 +2146,9 @@ def test_cumulative_resets_after_meter_swap(energy_db) -> None:
from app.services import timezone as _tz_mod
now_utc = datetime.now(timezone.utc)
# Keep the dynamic test date, but settle it at a deterministic business time
# beyond the local 01:05 fixed-fee/credit cutoff.
settled_local_now = now_utc.replace(hour=12, minute=0, second=0, microsecond=0)
midnight_today = now_utc.replace(hour=0, minute=0, second=0, microsecond=0)
old_meter_start = midnight_today - timedelta(days=30)
@@ -2145,7 +2205,10 @@ def test_cumulative_resets_after_meter_swap(energy_db) -> None:
session.commit()
with Session(energy_db) as session:
with patch.object(_tz_mod, "local_tz", return_value=ZoneInfo("UTC")):
with (
patch.object(_tz_mod, "local_tz", return_value=ZoneInfo("UTC")),
patch("app.services.energy_cost.local_now", return_value=settled_local_now),
):
catalog = build_catalog(session)
import_entry = next(
e for e in catalog if e.entity.key == "energy.import_cost_total"
@@ -2194,6 +2257,9 @@ def test_daily_getters_unaffected_by_d2_meter_anchor(energy_db) -> None:
from app.services import timezone as _tz_mod
now_utc = datetime.now(timezone.utc)
# Keep the dynamic test date, but settle it at a deterministic business time
# beyond the local 01:05 fixed-fee/credit cutoff.
settled_local_now = now_utc.replace(hour=12, minute=0, second=0, microsecond=0)
# Period 1h ago — in today's UTC window
t0 = now_utc.replace(minute=0, second=0, microsecond=0) - timedelta(hours=1)
if t0.date() < now_utc.date():
@@ -2226,7 +2292,10 @@ def test_daily_getters_unaffected_by_d2_meter_anchor(energy_db) -> None:
session.commit()
with Session(energy_db) as session:
with patch.object(_tz_mod, "local_tz", return_value=ZoneInfo("UTC")):
with (
patch.object(_tz_mod, "local_tz", return_value=ZoneInfo("UTC")),
patch("app.services.energy_cost.local_now", return_value=settled_local_now),
):
catalog = build_catalog(session)
import_today_entry = next(
e for e in catalog if e.entity.key == "energy.import_cost_today"
@@ -2685,3 +2754,244 @@ def test_midnight_state_publish_no_raise_when_mqtt_disabled() -> None:
publish_states(sess) # must not raise
mock_mgr.publish.assert_not_called()
# ---------------------------------------------------------------------------
# M8-T17 source / thermal HA catalog
# ---------------------------------------------------------------------------
def _make_thermal_source_and_meter(session: Session, commodity: str, now: datetime) -> Any:
from app.models.energy import Meter
from app.models.meter_source import MeterSource, MeterSourceBinding, MeterSourceChannel
source = MeterSource(
name=f"{commodity} source", kind="warmtelink_serial", enabled=True, config={},
status="online", last_seen_at=now, last_error=None, created_at=now, updated_at=now,
)
session.add(source)
session.flush()
unit = {"electricity": "kWh", "heating": "GJ", "hot_water": ""}[commodity]
channel = MeterSourceChannel(
source_id=source.id, channel_key=f"accepted-{commodity}", label=commodity, unit=unit,
latest_value=Decimal("12.5"), latest_at=now, latest_quality="valid",
created_at=now, updated_at=now,
)
meter = Meter(label=f"{commodity} meter", commodity=commodity, started_at=now - timedelta(days=1),
ended_at=None, reason="initial", note=None, created_at=now)
session.add_all((channel, meter))
session.flush()
session.add(MeterSourceBinding(meter_id=meter.id, channel_id=channel.id,
started_at=meter.started_at, ended_at=None,
created_at=now, updated_at=now))
session.flush()
return source, channel, meter
def test_m8_catalog_has_source_meter_and_thermal_entities_disabled(energy_db) -> None:
"""D13: all new entities are catalogued with safe default toggles and units."""
from app.integrations.expose import build_catalog
now = datetime.now(timezone.utc)
with Session(energy_db) as session:
heating_source, _, heating = _make_thermal_source_and_meter(session, "heating", now)
_water_source, _, water = _make_thermal_source_and_meter(session, "hot_water", now)
_electricity_source, _, electricity = _make_thermal_source_and_meter(session, "electricity", now)
session.commit()
catalog = build_catalog(session)
entries = {entry.entity.key: entry for entry in catalog}
assert entries[f"source.{heating_source.uuid}.online"].entity.device_class == "connectivity"
assert entries[f"meter.{heating.uuid}.total"].entity.unit == "GJ"
assert entries[f"meter.{heating.uuid}.total"].entity.device_class == "energy"
assert entries[f"meter.{water.uuid}.total"].entity.unit == ""
assert entries[f"meter.{water.uuid}.total"].entity.device_class == "volume"
assert entries[f"meter.{electricity.uuid}.total"].entity.unit == "kWh"
assert entries[f"meter.{electricity.uuid}.total"].entity.device_class == "energy"
assert entries[f"meter.{electricity.uuid}.today"].entity.state_class == "total_increasing"
thermal = [entry for key, entry in entries.items() if key.startswith("thermal_cost.")]
assert len(thermal) == 12
assert all(entry.enabled is False for entry in thermal)
assert all(entry.entity.unit == "EUR" and entry.entity.device_class == "monetary" for entry in thermal)
def test_m8_meter_getter_hides_stale_or_offline_source(energy_db) -> None:
from app.integrations.expose import build_catalog
now = datetime.now(timezone.utc)
with Session(energy_db) as session:
source, channel, meter = _make_thermal_source_and_meter(session, "heating", now)
session.commit()
entry = next(item for item in build_catalog(session) if item.entity.key == f"meter.{meter.uuid}.total")
assert entry.entity.value_getter(session) == channel.latest_value
source.status = "error"
assert entry.entity.value_getter(session) is None
source.status = "online"
source.last_seen_at = now - timedelta(minutes=6)
assert entry.entity.value_getter(session) is None
def test_m8_meter_today_uses_current_binding_and_never_invents_zero(energy_db) -> None:
"""A binding is a half-open cumulative epoch, not merely a channel filter."""
from app.integrations.expose import build_catalog
from app.models.meter_source import WarmteLinkReading
from app.services import timezone as tz
now = datetime(2026, 1, 15, 10, tzinfo=timezone.utc)
local_tz = ZoneInfo("Europe/Amsterdam")
with Session(energy_db) as session:
_source, channel, meter = _make_thermal_source_and_meter(session, "heating", now)
binding = meter.source_bindings[0]
binding.started_at = now - timedelta(minutes=20)
session.add_all((
WarmteLinkReading(channel_id=channel.id, recorded_at=now - timedelta(minutes=30),
received_at=now, value=Decimal("100"), unit="GJ", quality="valid",
equipment_fingerprint="test"),
WarmteLinkReading(channel_id=channel.id, recorded_at=now - timedelta(minutes=15),
received_at=now, value=Decimal("110"), unit="GJ", quality="valid",
equipment_fingerprint="test"),
WarmteLinkReading(channel_id=channel.id, recorded_at=now - timedelta(minutes=5),
received_at=now, value=Decimal("115"), unit="GJ", quality="valid",
equipment_fingerprint="test"),
))
session.commit()
with (
patch("app.integrations.expose._utc_now", return_value=now),
patch.object(tz, "local_tz", return_value=local_tz),
patch.object(tz, "local_now", return_value=now.astimezone(local_tz)),
):
entity = next(item.entity for item in build_catalog(session)
if item.entity.key == f"meter.{meter.uuid}.today")
assert entity.value_getter(session) == Decimal("5")
session.query(WarmteLinkReading).filter(WarmteLinkReading.channel_id == channel.id).delete()
session.add(WarmteLinkReading(channel_id=channel.id, recorded_at=now - timedelta(minutes=5),
received_at=now, value=Decimal("115"), unit="GJ", quality="valid",
equipment_fingerprint="test"))
session.flush()
assert entity.value_getter(session) is None
def test_m8_dsmr_electricity_meter_reads_real_telegram_domain(energy_db) -> None:
"""DSMR does not populate channel latest fields: its telegram is authoritative."""
from app.integrations.expose import build_catalog
from app.models.energy import DsmrReading
now = datetime.now(timezone.utc)
with Session(energy_db) as session:
source, channel, meter = _make_thermal_source_and_meter(session, "electricity", now)
source.kind, source.status, source.last_seen_at = "dsmr_mqtt", "unknown", None
channel.latest_at, channel.latest_value, channel.latest_quality = None, None, None
session.add_all((
DsmrReading(meter_source_id=source.id, recorded_at=now - timedelta(minutes=2),
payload={"electricity_delivered_1": "100", "electricity_delivered_2": "20"}),
DsmrReading(meter_source_id=source.id, recorded_at=now - timedelta(minutes=1),
payload={"electricity_delivered_1": "103", "electricity_delivered_2": "22"}),
))
session.commit()
entries = {item.entity.key: item.entity for item in build_catalog(session)}
assert entries[f"source.{source.uuid}.online"].value_getter(session) == "ON"
assert entries[f"meter.{meter.uuid}.total"].value_getter(session) == Decimal("125")
assert entries[f"meter.{meter.uuid}.today"].value_getter(session) == Decimal("5")
def test_m8_warmtelink_future_snapshot_is_offline_but_today_uses_elapsed_points(energy_db) -> None:
"""Future channel snapshots never become current HA state or today's delta."""
from app.integrations.expose import build_catalog
from app.models.meter_source import WarmteLinkReading
from app.services import timezone as tz
now = datetime(2026, 1, 15, 10, tzinfo=timezone.utc)
with Session(energy_db) as session:
source, channel, meter = _make_thermal_source_and_meter(session, "heating", now)
channel.latest_at, channel.latest_value = now + timedelta(minutes=1), Decimal("999")
session.add_all((
WarmteLinkReading(channel_id=channel.id, recorded_at=now - timedelta(minutes=2),
received_at=now, value=Decimal("100"), unit="GJ", quality="valid",
equipment_fingerprint="test"),
WarmteLinkReading(channel_id=channel.id, recorded_at=now,
received_at=now, value=Decimal("110"), unit="GJ", quality="valid",
equipment_fingerprint="test"),
WarmteLinkReading(channel_id=channel.id, recorded_at=now + timedelta(minutes=1),
received_at=now, value=Decimal("999"), unit="GJ", quality="valid",
equipment_fingerprint="test"),
))
session.commit()
with (
patch("app.integrations.expose._utc_now", return_value=now),
patch.object(tz, "local_tz", return_value=ZoneInfo("Europe/Amsterdam")),
patch.object(tz, "local_now", return_value=now.astimezone(ZoneInfo("Europe/Amsterdam"))),
):
entries = {item.entity.key: item.entity for item in build_catalog(session)}
assert entries[f"meter.{meter.uuid}.total"].device.availability_getter(session) is False
assert entries[f"meter.{meter.uuid}.total"].value_getter(session) is None
assert entries[f"meter.{meter.uuid}.today"].value_getter(session) == Decimal("10")
def test_m8_dsmr_future_telegram_is_offline_and_excluded_from_today(energy_db) -> None:
"""DSMR's latest telegram is not fresh when it is in the future."""
from app.integrations.expose import build_catalog
from app.models.energy import DsmrReading
from app.services import timezone as tz
now = datetime(2026, 1, 15, 10, tzinfo=timezone.utc)
local_tz = ZoneInfo("Europe/Amsterdam")
next_midnight = datetime(2026, 1, 15, 23, tzinfo=timezone.utc)
with Session(energy_db) as session:
source, channel, meter = _make_thermal_source_and_meter(session, "electricity", now)
source.kind, source.status, source.last_seen_at = "dsmr_mqtt", "unknown", None
channel.latest_at, channel.latest_value, channel.latest_quality = None, None, None
session.add_all((
DsmrReading(meter_source_id=source.id, recorded_at=now - timedelta(minutes=2),
payload={"electricity_delivered_1": "100", "electricity_delivered_2": "0"}),
DsmrReading(meter_source_id=source.id, recorded_at=now,
payload={"electricity_delivered_1": "110", "electricity_delivered_2": "0"}),
DsmrReading(meter_source_id=source.id, recorded_at=now + timedelta(minutes=1),
payload={"electricity_delivered_1": "999", "electricity_delivered_2": "0"}),
DsmrReading(meter_source_id=source.id, recorded_at=next_midnight,
payload={"electricity_delivered_1": "1000", "electricity_delivered_2": "0"}),
))
session.commit()
with (
patch("app.integrations.expose._utc_now", return_value=now),
patch.object(tz, "local_tz", return_value=local_tz),
patch.object(tz, "local_now", return_value=now.astimezone(local_tz)),
):
entries = {item.entity.key: item.entity for item in build_catalog(session)}
assert entries[f"source.{source.uuid}.online"].value_getter(session) == "OFF"
assert entries[f"meter.{meter.uuid}.total"].value_getter(session) is None
assert entries[f"meter.{meter.uuid}.today"].value_getter(session) == Decimal("10")
def test_m8_thermal_today_summary_ends_at_frozen_now(energy_db) -> None:
"""A running local day must not include thermal cost rows from its future."""
from app.integrations.expose import build_catalog
from app.services import timezone as tz
now = datetime(2026, 1, 15, 10, tzinfo=timezone.utc)
local_tz = ZoneInfo("Europe/Amsterdam")
captured: dict[str, datetime] = {}
result = {
"period_count": 1, "fixed_cost": Decimal("0"), "total_cost": Decimal("0"),
"breakdown": {key: Decimal("0") for key in (
"heating", "hot_water_heating", "hot_water", "hot_water_tax",
)},
}
def summarize_spy(_session: Session, start: datetime, end: datetime, *, now: datetime) -> dict:
captured.update(start=start, end=end, now=now)
return result
with Session(energy_db) as session:
_make_thermal_source_and_meter(session, "heating", now)
_make_thermal_source_and_meter(session, "hot_water", now)
session.commit()
with (
patch("app.integrations.expose._utc_now", return_value=now),
patch.object(tz, "local_tz", return_value=local_tz),
patch.object(tz, "local_now", return_value=now.astimezone(local_tz)),
patch("app.services.meter_cost.summarize", side_effect=summarize_spy),
):
entity = next(item.entity for item in build_catalog(session)
if item.entity.key.endswith(".heating_today"))
assert entity.value_getter(session) == Decimal("0")
assert captured["end"] == now
+193 -16
View File
@@ -21,7 +21,7 @@ import pytest
import sqlalchemy.exc
from alembic import command
from alembic.config import Config
from sqlalchemy import create_engine, event as sa_event, inspect, text
from sqlalchemy import UniqueConstraint, create_engine, event as sa_event, inspect, text
from sqlalchemy.orm import Session
from app.db import Base
@@ -105,37 +105,51 @@ def test_energy_tables_exist_after_upgrade(energy_db):
"energy_contract_version",
"tibber_price",
"energy_cost_period",
"meter_source",
"meter_source_channel",
"meter_source_binding",
}
for table in expected_tables:
assert table in table_names, f"{table!r} missing after upgrade to head"
def test_dsmr_reading_columns(energy_db):
"""dsmr_reading must have id, recorded_at (NOT NULL), source_id (nullable), payload (NOT NULL)."""
"""dsmr_reading stores its telegram id separately from its source identity."""
inspector = inspect(energy_db)
columns = {col["name"]: col for col in inspector.get_columns("dsmr_reading")}
assert "id" in columns and not columns["id"]["nullable"]
assert "recorded_at" in columns and not columns["recorded_at"]["nullable"]
assert "source_id" in columns and columns["source_id"]["nullable"]
assert "telegram_id" in columns and columns["telegram_id"]["nullable"]
assert "meter_source_id" in columns and not columns["meter_source_id"]["nullable"]
assert "payload" in columns and not columns["payload"]["nullable"]
def test_dsmr_reading_recorded_at_unique(energy_db):
"""dsmr_reading.recorded_at is the UNIQUE de-dup key (telegram-id-independent)."""
def test_dsmr_reading_source_timestamp_unique(energy_db):
"""DSMR de-duplication is unique per configured source and timestamp."""
inspector = inspect(energy_db)
unique_constraints = inspector.get_unique_constraints("dsmr_reading")
unique_cols = [col for uc in unique_constraints for col in uc["column_names"]]
assert "recorded_at" in unique_cols, "recorded_at must have a unique constraint"
assert ("meter_source_id", "recorded_at") in {
tuple(uc["column_names"]) for uc in unique_constraints
}
foreign_keys = {
tuple(foreign_key["constrained_columns"]): foreign_key
for foreign_key in inspector.get_foreign_keys("dsmr_reading")
}
assert foreign_keys[("meter_source_id",)]["referred_table"] == "meter_source"
assert foreign_keys[("meter_source_id",)]["options"]["ondelete"] == "RESTRICT"
assert "ix_dsmr_reading_meter_source_id" in {
index["name"] for index in inspector.get_indexes("dsmr_reading")
}
def test_dsmr_reading_source_id_not_unique(energy_db):
"""dsmr_reading.source_id (telegram id) must NOT be unique — it overflows/resets,
def test_dsmr_reading_telegram_id_not_unique(energy_db):
"""dsmr_reading.telegram_id must NOT be unique — it overflows/resets,
so it is kept only as a reference value and never relied on for dedup."""
inspector = inspect(energy_db)
unique_constraints = inspector.get_unique_constraints("dsmr_reading")
unique_cols = [col for uc in unique_constraints for col in uc["column_names"]]
assert "source_id" not in unique_cols, "source_id must NOT have a unique constraint"
assert "telegram_id" not in unique_cols, "telegram_id must NOT have a unique constraint"
def test_energy_contract_columns(energy_db):
@@ -143,11 +157,15 @@ def test_energy_contract_columns(energy_db):
inspector = inspect(energy_db)
columns = {col["name"]: col for col in inspector.get_columns("energy_contract")}
required_non_nullable = {"id", "name", "kind", "active", "currency", "created_at", "updated_at"}
required_non_nullable = {
"id", "name", "kind", "scope", "active", "currency", "created_at", "updated_at"
}
for col_name in required_non_nullable:
assert col_name in columns, f"Missing column: {col_name}"
assert not columns[col_name]["nullable"], f"{col_name} should be NOT NULL"
assert any(index["name"] == "ix_energy_contract_scope" for index in inspector.get_indexes("energy_contract"))
def test_energy_contract_version_columns(energy_db):
"""energy_contract_version must have all required columns with correct nullability."""
@@ -407,12 +425,15 @@ def test_energy_cost_period_meter_id_fk_ondelete_restrict():
)
def test_dsmr_reading_recorded_at_unique_in_metadata():
"""DsmrReading.recorded_at must be the unique de-dup key in ORM metadata,
and source_id must NOT be unique (decoupled from the telegram id)."""
def test_dsmr_reading_source_timestamp_unique_in_metadata():
"""DsmrReading de-duplicates by source/timestamp, never telegram id."""
table = Base.metadata.tables["dsmr_reading"]
assert table.columns["recorded_at"].unique, "recorded_at must be declared unique"
assert not table.columns["source_id"].unique, "source_id must NOT be unique"
assert not table.columns["telegram_id"].unique, "telegram_id must NOT be unique"
assert any(
tuple(constraint.columns.keys()) == ("meter_source_id", "recorded_at")
for constraint in table.constraints
if isinstance(constraint, UniqueConstraint)
)
def test_tibber_price_starts_at_unique_in_metadata():
@@ -520,6 +541,7 @@ def test_energy_contract_insert_and_retrieve(energy_db):
assert fetched is not None
assert fetched.name == "My Manual Contract"
assert fetched.kind == "manual"
assert fetched.scope == "electricity"
assert fetched.active is True
assert fetched.currency == "EUR"
@@ -920,6 +942,7 @@ def test_energy_cost_period_meter_id_nullable(energy_db):
fetched = session.get(EnergyCostPeriod, period_id)
assert fetched is not None
assert fetched.meter_id is None
assert fetched.source_binding_id is None
def test_energy_cost_period_meter_id_fk_enforced(tmp_path: Path):
@@ -1190,3 +1213,157 @@ def test_migration_downgrade_removes_meter_table(tmp_path: Path):
"meter_id must be removed from energy_cost_period after downgrade"
)
engine.dispose()
def test_contract_scope_migration_preserves_historical_contract_audit(tmp_path: Path):
"""A revision-17 fixture upgrades/downgrades without altering contract audit rows."""
db_url = f"sqlite:///{tmp_path / 'contract_scope_history.db'}"
cfg = _make_app_alembic_config(db_url)
command.upgrade(cfg, "20260822_17_warmtelink_readings")
engine = create_engine(db_url, connect_args={"check_same_thread": False})
now = datetime.now(tz=timezone.utc).replace(tzinfo=None)
values = '{"energy":{"buy":{"normal":0.4}}}'
pricing = '{"historic":"unchanged"}'
with engine.begin() as connection:
contract_id = connection.execute(
text(
"INSERT INTO energy_contract (name, kind, active, currency, created_at, updated_at) "
"VALUES ('Historic', 'manual', 1, 'EUR', :now, :now)"
),
{"now": now},
).lastrowid
version_id = connection.execute(
text(
"INSERT INTO energy_contract_version "
"(contract_id, effective_from, effective_to, \"values\", created_at) "
"VALUES (:contract_id, :now, NULL, :values, :now)"
),
{"contract_id": contract_id, "now": now, "values": values},
).lastrowid
connection.execute(
text(
"INSERT INTO energy_cost_period "
"(period_start, d1_kwh, d2_kwh, r1_kwh, r2_kwh, import_cost, export_revenue, "
"net_cost, currency, pricing, contract_version_id, degraded, computed_at, meter_id, "
"source_binding_id) VALUES (:now, 1, 2, 0, 0, 3, 0, 3, 'EUR', :pricing, :version_id, "
"0, :now, NULL, NULL)"
),
{"now": now, "pricing": pricing, "version_id": version_id},
)
command.upgrade(cfg, "20260822_18_contract_scopes")
command.upgrade(cfg, "20260822_18_contract_scopes")
with engine.connect() as connection:
assert connection.execute(text("SELECT version_num FROM alembic_version")).scalar_one() == (
"20260822_18_contract_scopes"
)
assert connection.execute(text("SELECT scope FROM energy_contract")).scalar_one() == "electricity"
assert connection.execute(text("SELECT COUNT(*) FROM energy_contract")).scalar_one() == 1
assert connection.execute(text("SELECT COUNT(*) FROM energy_contract_version")).scalar_one() == 1
assert connection.execute(text("SELECT COUNT(*) FROM energy_cost_period")).scalar_one() == 1
assert connection.execute(text("SELECT \"values\" FROM energy_contract_version")).scalar_one() == values
assert connection.execute(text("SELECT pricing FROM energy_cost_period")).scalar_one() == pricing
assert connection.execute(
text(
"SELECT COUNT(*) FROM energy_contract_version v LEFT JOIN energy_contract c "
"ON c.id = v.contract_id WHERE c.id IS NULL"
)
).scalar_one() == 0
inspector = inspect(connection)
assert any(item["name"] == "ix_energy_contract_scope" for item in inspector.get_indexes("energy_contract"))
command.downgrade(cfg, "20260822_17_warmtelink_readings")
with engine.connect() as connection:
assert "scope" not in {item["name"] for item in inspect(connection).get_columns("energy_contract")}
assert connection.execute(text("SELECT COUNT(*) FROM energy_contract_version")).scalar_one() == 1
assert connection.execute(text("SELECT COUNT(*) FROM energy_cost_period")).scalar_one() == 1
engine.dispose()
def test_contract_scope_migration_audit_failure_restores_revision_17(tmp_path: Path):
"""A post-DDL audit failure leaves no SQLite batch-migration residue."""
db_url = f"sqlite:///{tmp_path / 'contract_scope_audit_failure.db'}"
cfg = _make_app_alembic_config(db_url)
command.upgrade(cfg, "20260822_17_warmtelink_readings")
engine = _engine_with_fk(db_url)
now = datetime.now(tz=timezone.utc).replace(tzinfo=None)
values = '{"energy":{"buy":{"normal":0.4}}}'
pricing = '{"historic":"unchanged"}'
with engine.begin() as connection:
contract_id = connection.execute(
text(
"INSERT INTO energy_contract (name, kind, active, currency, created_at, updated_at) "
"VALUES ('Historic', 'manual', 1, 'EUR', :now, :now)"
),
{"now": now},
).lastrowid
version_id = connection.execute(
text(
"INSERT INTO energy_contract_version "
"(contract_id, effective_from, effective_to, \"values\", created_at) "
"VALUES (:contract_id, :now, NULL, :values, :now)"
),
{"contract_id": contract_id, "now": now, "values": values},
).lastrowid
connection.execute(
text(
"INSERT INTO energy_cost_period "
"(period_start, d1_kwh, d2_kwh, r1_kwh, r2_kwh, import_cost, export_revenue, "
"net_cost, currency, pricing, contract_version_id, degraded, computed_at, meter_id, "
"source_binding_id) VALUES (:now, 1, 2, 0, 0, 3, 0, 3, 'EUR', :pricing, :version_id, "
"0, :now, NULL, NULL)"
),
{"now": now, "pricing": pricing, "version_id": version_id},
)
engine.dispose()
def _raise_after_ddl() -> None:
raise RuntimeError("injected post-DDL audit failure")
cfg.attributes["m8_t12_post_ddl_audit_failure"] = _raise_after_ddl
with pytest.raises(RuntimeError, match="injected post-DDL audit failure"):
command.upgrade(cfg, "head")
engine = _engine_with_fk(db_url)
with engine.connect() as connection:
assert connection.execute(text("SELECT version_num FROM alembic_version")).scalar_one() == (
"20260822_17_warmtelink_readings"
)
assert "scope" not in {item["name"] for item in inspect(connection).get_columns("energy_contract")}
assert not any(
item["name"] == "ix_energy_contract_scope"
for item in inspect(connection).get_indexes("energy_contract")
)
assert connection.execute(text("SELECT COUNT(*) FROM energy_contract")).scalar_one() == 1
assert connection.execute(text("SELECT COUNT(*) FROM energy_contract_version")).scalar_one() == 1
assert connection.execute(text("SELECT COUNT(*) FROM energy_cost_period")).scalar_one() == 1
assert connection.execute(text("SELECT \"values\" FROM energy_contract_version")).scalar_one() == values
assert connection.execute(text("SELECT pricing FROM energy_cost_period")).scalar_one() == pricing
assert connection.execute(
text(
"SELECT COUNT(*) FROM energy_contract_version v LEFT JOIN energy_contract c "
"ON c.id = v.contract_id WHERE c.id IS NULL"
)
).scalar_one() == 0
assert connection.execute(
text(
"SELECT COUNT(*) FROM energy_cost_period p LEFT JOIN energy_contract_version v "
"ON v.id = p.contract_version_id "
"WHERE p.contract_version_id IS NOT NULL AND v.id IS NULL"
)
).scalar_one() == 0
assert connection.execute(text("PRAGMA foreign_key_check")).all() == []
assert connection.execute(
text("SELECT name FROM sqlite_master WHERE name LIKE '_alembic_tmp_%'")
).all() == []
engine.dispose()
del cfg.attributes["m8_t12_post_ddl_audit_failure"]
command.upgrade(cfg, "20260822_18_contract_scopes")
engine = _engine_with_fk(db_url)
with engine.connect() as connection:
assert connection.execute(text("SELECT version_num FROM alembic_version")).scalar_one() == (
"20260822_18_contract_scopes"
)
assert connection.execute(text("SELECT scope FROM energy_contract")).scalar_one() == "electricity"
engine.dispose()
+18 -1
View File
@@ -15,7 +15,7 @@ Coverage:
from __future__ import annotations
from datetime import datetime, timezone
from datetime import datetime, timedelta, timezone
from pathlib import Path
from typing import Any
from unittest.mock import MagicMock, patch
@@ -1361,3 +1361,20 @@ def test_settings_payload_includes_ha_state_topic_prefix() -> None:
assert payload["ha_state_topic_prefix"] == "my_prefix", (
f"Expected 'my_prefix', got {payload['ha_state_topic_prefix']!r}"
)
def test_stale_m8_entities_includes_ended_electricity_meter(disco_db) -> None:
"""An electricity swap clears exactly the old Meter's two discovery configs."""
from app.models.energy import Meter
from app.services.ha_discovery import _stale_m8_entities
now = datetime.now(timezone.utc)
with Session(disco_db) as session:
old = Meter(label="old", commodity="electricity", started_at=now - timedelta(days=1),
ended_at=now, reason="meter_swap", note=None, created_at=now)
current = Meter(label="current", commodity="electricity", started_at=now,
ended_at=None, reason="meter_swap", note=None, created_at=now)
session.add_all((old, current))
session.commit()
keys = {entity.key for entity in _stale_m8_entities(session)}
assert keys == {f"meter.{old.uuid}.total", f"meter.{old.uuid}.today"}
+402
View File
@@ -0,0 +1,402 @@
"""M8-T15 tests for the binding-aware thermal ledger."""
from __future__ import annotations
from datetime import UTC, datetime, timedelta
from decimal import Decimal
from pathlib import Path
from zoneinfo import ZoneInfo
import pytest
from alembic import command
from alembic.config import Config
from sqlalchemy import create_engine, select
from sqlalchemy.orm import Session
from app.models.energy import EnergyContract, EnergyContractVersion, Meter, MeterCostPeriod
from app.models.meter_source import MeterSource, MeterSourceBinding, MeterSourceChannel, WarmteLinkReading
from app.services import timezone as timezone_service
from app.services.meter_cost import compute_closed_periods, compute_period, recompute_range, summarize
@pytest.fixture()
def db(tmp_path: Path) -> Session:
url = f"sqlite:///{tmp_path / 'meter_cost.db'}"
cfg = Config("alembic_app.ini")
cfg.set_main_option("sqlalchemy.url", url)
command.upgrade(cfg, "head")
engine = create_engine(url)
session = Session(engine)
yield session
session.close()
engine.dispose()
T0 = datetime(2026, 6, 23, 10, tzinfo=UTC)
VALUES = {
"variable": {"heating": "20.0", "hot_water_heating": "4.0", "hot_water": "2.0", "hot_water_tax": "1.0"},
"standing": {"heating_network": "365", "metering": "73", "delivery_set": "0", "hot_water_network": "0", "other": "0"},
}
def _contract(session: Session, *, values: dict = VALUES, start: datetime = T0 - timedelta(days=1)) -> EnergyContractVersion:
now = datetime.now(UTC)
contract = EnergyContract(name="thermal", kind="district_heating", scope="thermal", active=True,
currency="EUR", created_at=now, updated_at=now)
session.add(contract)
session.flush()
version = EnergyContractVersion(contract_id=contract.id, effective_from=start, values=values, created_at=now)
session.add(version)
session.flush()
return version
def _domain(session: Session, commodity: str, *, start: datetime = T0 - timedelta(days=1), end: datetime | None = None):
now = datetime.now(UTC)
meter = Meter(label=commodity, commodity=commodity, started_at=start, ended_at=end,
reason="initial", created_at=now)
source = MeterSource(name=f"{commodity}-source", kind="warmtelink_serial", enabled=True, config={},
status="online", created_at=now, updated_at=now)
session.add_all((meter, source))
session.flush()
unit = "GJ" if commodity == "heating" else "m3"
channel = MeterSourceChannel(source_id=source.id, channel_key=commodity, label=commodity,
suggested_commodity=commodity, unit=unit, latest_quality="valid",
created_at=now, updated_at=now)
session.add(channel)
session.flush()
binding = MeterSourceBinding(meter_id=meter.id, channel_id=channel.id, started_at=start, ended_at=end,
created_at=now, updated_at=now)
session.add(binding)
session.flush()
return meter, channel, binding
def _reading(session: Session, channel: MeterSourceChannel, at: datetime, value: str, quality: str = "valid") -> None:
session.add(WarmteLinkReading(channel_id=channel.id, recorded_at=at, received_at=at, value=Decimal(value),
unit=channel.unit, quality=quality, equipment_fingerprint="test"))
def _setup_good(session: Session) -> None:
_contract(session)
_, heating, _ = _domain(session, "heating")
_, water, _ = _domain(session, "hot_water")
_reading(session, heating, T0, "10.000")
_reading(session, heating, T0 + timedelta(minutes=15), "10.050")
_reading(session, water, T0, "20.000")
_reading(session, water, T0 + timedelta(minutes=15), "20.200")
session.commit()
def _row(session: Session, commodity: str) -> MeterCostPeriod:
return session.execute(select(MeterCostPeriod).where(MeterCostPeriod.commodity == commodity)).scalar_one()
def _row_at(session: Session, commodity: str, start: datetime) -> MeterCostPeriod:
return session.execute(select(MeterCostPeriod).where(
MeterCostPeriod.commodity == commodity, MeterCostPeriod.period_start == start
)).scalar_one()
def _version(
session: Session, contract: EnergyContract, *, start: datetime, values: dict
) -> EnergyContractVersion:
version = EnergyContractVersion(
contract_id=contract.id, effective_from=start, values=values, created_at=datetime.now(UTC)
)
session.add(version)
session.flush()
return version
def test_two_commodity_decimal_breakdown_and_scheduler_idempotency(db: Session) -> None:
_setup_good(db)
assert compute_period(db, "heating", T0)
assert compute_period(db, "hot_water", T0)
db.commit()
heating, water = _row(db, "heating"), _row(db, "hot_water")
assert heating.quantity == Decimal("0.050000")
assert heating.cost == Decimal("1.000000000")
assert water.cost == Decimal("1.400000000")
assert {key: Decimal(value) for key, value in water.cost_breakdown.items()} == {
"hot_water_heating": Decimal("0.8"), "hot_water": Decimal("0.4"), "hot_water_tax": Decimal("0.2")
}
assert not compute_period(db, "heating", T0)
@pytest.mark.parametrize("commodity,start,end,reason", [
("heating", "10", "9.99", "negative_delta"),
("heating", "10", "10.101", "delta_limit_exceeded"),
("hot_water", "10", "11.001", "delta_limit_exceeded"),
])
def test_bad_deltas_are_degraded(db: Session, commodity: str, start: str, end: str, reason: str) -> None:
_contract(db)
_, channel, _ = _domain(db, commodity)
_reading(db, channel, T0, start)
_reading(db, channel, T0 + timedelta(minutes=15), end)
db.commit()
compute_period(db, commodity, T0)
db.commit()
row = _row(db, commodity)
assert row.degraded and row.degraded_reason == reason and row.contract_version_id is None
def test_freshness_quality_binding_and_contract_fail_closed(db: Session) -> None:
_contract(db)
meter, channel, binding = _domain(db, "heating")
_reading(db, channel, T0 - timedelta(seconds=121), "1")
_reading(db, channel, T0 + timedelta(minutes=15), "1.01")
db.commit()
compute_period(db, "heating", T0)
assert _row(db, "heating").degraded_reason == "missing_stale_or_invalid_reading"
# Recompute sees invalid quality independently at the end boundary.
db.query(WarmteLinkReading).delete()
_reading(db, channel, T0, "1")
_reading(db, channel, T0 + timedelta(minutes=15), "1.01", "invalid")
db.commit()
recompute_range(db, T0, T0 + timedelta(minutes=15))
assert _row(db, "heating").degraded_reason == "missing_stale_or_invalid_reading"
binding.ended_at = T0 + timedelta(minutes=15)
db.commit()
recompute_range(db, T0, T0 + timedelta(minutes=15))
assert _row(db, "heating").degraded_reason == "missing_or_ambiguous_binding"
assert meter.id is not None
@pytest.mark.parametrize(
("first_quality", "last_quality", "accepted", "period_quality"),
[
("valid", "valid", True, "valid"),
("valid", "unverifiable", True, "unverifiable"),
("unverifiable", "valid", True, "unverifiable"),
("unverifiable", "unverifiable", True, "unverifiable"),
("invalid", "valid", False, "invalid"),
("valid", "invalid", False, "invalid"),
],
)
def test_accepted_reading_quality_is_preserved_without_promotion(
db: Session, first_quality: str, last_quality: str, accepted: bool, period_quality: str
) -> None:
_contract(db)
_, channel, _ = _domain(db, "heating")
_reading(db, channel, T0, "10", first_quality)
_reading(db, channel, T0 + timedelta(minutes=15), "10.05", last_quality)
db.commit()
assert compute_period(db, "heating", T0)
db.commit()
row = _row(db, "heating")
assert row.degraded is not accepted
assert row.quality == period_quality
if not accepted:
assert row.degraded_reason == "missing_stale_or_invalid_reading"
def test_readings_must_be_inside_binding_and_meter_windows(db: Session) -> None:
_contract(db)
meter, channel, binding = _domain(db, "heating", start=T0)
_reading(db, channel, T0 - timedelta(seconds=60), "10")
_reading(db, channel, T0 + timedelta(minutes=15), "10.05")
db.commit()
compute_period(db, "heating", T0)
assert _row(db, "heating").degraded_reason == "missing_stale_or_invalid_reading"
# A candidate after a closed binding is equally outside the cumulative domain.
binding.started_at = T0 - timedelta(days=1)
binding.ended_at = T0 + timedelta(minutes=15, seconds=30)
meter.started_at = T0 - timedelta(days=1)
db.query(WarmteLinkReading).delete()
_reading(db, channel, T0, "10")
_reading(db, channel, T0 + timedelta(minutes=15, seconds=60), "10.05")
db.commit()
recompute_range(db, T0, T0 + timedelta(minutes=15))
assert _row(db, "heating").degraded_reason == "missing_stale_or_invalid_reading"
def test_freshness_is_independent_and_inclusive_with_nearest_candidate(db: Session) -> None:
_contract(db)
_, channel, _ = _domain(db, "heating")
_reading(db, channel, T0 - timedelta(seconds=120), "10")
_reading(db, channel, T0 + timedelta(minutes=15, seconds=120), "10.05")
db.commit()
compute_period(db, "heating", T0)
db.commit()
row = _row(db, "heating")
assert not row.degraded and row.quantity == Decimal("0.050000")
def test_closed_scheduler_retries_degraded_but_not_normal(db: Session) -> None:
_contract(db)
_, channel, _ = _domain(db, "heating")
# The first closed scheduler pass records a degraded row. Supplying the
# missing boundary inputs later must let its next pass repair that row.
db.commit()
assert compute_closed_periods(db, now=T0 + timedelta(minutes=16)) > 0
assert _row_at(db, "heating", T0).degraded
_reading(db, channel, T0, "10")
_reading(db, channel, T0 + timedelta(minutes=15), "10.05")
db.commit()
assert compute_closed_periods(db, now=T0 + timedelta(minutes=16)) > 0
normal = _row_at(db, "heating", T0)
assert not normal.degraded
frozen = (normal.cost, dict(normal.pricing_snapshot), normal.updated_at)
end_reading = db.execute(select(WarmteLinkReading).where(
WarmteLinkReading.channel_id == channel.id,
WarmteLinkReading.recorded_at == T0 + timedelta(minutes=15),
)).scalar_one()
end_reading.value = Decimal("10.090")
version = db.get(EnergyContractVersion, normal.contract_version_id)
assert version is not None
version.values = {**version.values, "variable": {**version.values["variable"], "heating": "99"}}
db.commit()
compute_closed_periods(db, now=T0 + timedelta(minutes=16))
db.expire_all()
unchanged = db.execute(select(MeterCostPeriod).where(MeterCostPeriod.id == normal.id)).scalar_one()
assert (unchanged.cost, unchanged.pricing_snapshot, unchanged.updated_at) == frozen
def test_explicit_recompute_overwrites_and_source_switch_degrades(db: Session) -> None:
_setup_good(db)
compute_period(db, "heating", T0)
db.commit()
row = _row(db, "heating")
old_cost = row.cost
channel = db.get(MeterSourceChannel, row.source_binding.channel_id)
end_reading = db.execute(select(WarmteLinkReading).where(
WarmteLinkReading.channel_id == channel.id,
WarmteLinkReading.recorded_at == T0 + timedelta(minutes=15),
)).scalar_one()
end_reading.value = Decimal("10.080")
db.commit()
assert recompute_range(db, T0, T0 + timedelta(minutes=15)) == 2
assert _row(db, "heating").cost != old_cost
# A hand-off precisely at the right boundary cannot form a single domain.
binding = _row(db, "heating").source_binding
binding.ended_at = T0 + timedelta(minutes=15)
db.commit()
recompute_range(db, T0, T0 + timedelta(minutes=15))
assert _row(db, "heating").degraded_reason == "missing_or_ambiguous_binding"
def test_real_source_binding_handoff_never_crosses_cumulative_registers(db: Session) -> None:
_contract(db)
meter, old_channel, old_binding = _domain(db, "heating")
handoff = T0 + timedelta(minutes=15)
old_binding.ended_at = handoff
new_meter, new_channel, new_binding = _domain(db, "heating", start=handoff, end=handoff)
# Keep the same Meter: this is a source/channel/binding handoff, not a meter swap.
new_binding.meter_id = meter.id
new_binding.ended_at = None
assert new_meter.id != meter.id
_reading(db, old_channel, T0, "1000")
_reading(db, old_channel, handoff - timedelta(seconds=1), "1000.01")
_reading(db, new_channel, handoff, "7")
_reading(db, new_channel, handoff + timedelta(seconds=1), "7.01")
db.commit()
compute_period(db, "heating", T0)
assert _row(db, "heating").degraded_reason == "cross_source_binding"
def test_real_meter_epoch_handoff_never_crosses_cumulative_registers(db: Session) -> None:
_contract(db)
handoff = T0 + timedelta(minutes=15)
_, old_channel, _ = _domain(db, "heating", end=handoff)
_, new_channel, _ = _domain(db, "heating", start=handoff)
_reading(db, old_channel, T0, "1000")
_reading(db, old_channel, handoff - timedelta(seconds=1), "1000.01")
_reading(db, new_channel, handoff, "2")
_reading(db, new_channel, handoff + timedelta(seconds=1), "2.01")
db.commit()
compute_period(db, "heating", T0)
assert _row(db, "heating").degraded_reason == "cross_meter_epoch"
def test_summary_fixed_once_per_contract_day_and_dst(db: Session, monkeypatch: pytest.MonkeyPatch) -> None:
_setup_good(db)
compute_period(db, "heating", T0)
compute_period(db, "hot_water", T0)
db.commit()
monkeypatch.setattr(timezone_service, "local_tz", lambda: ZoneInfo("Europe/Amsterdam"))
result = summarize(db, T0, T0 + timedelta(days=1), now=datetime(2026, 6, 24, 1, 6, tzinfo=UTC))
assert result["variable_cost"] == Decimal("2.400000000")
assert result["fixed_cost"] == Decimal("2.4") # two local days, each charged once not per commodity
# DST local day is charged once too, despite being 23 hours long.
db.execute(select(EnergyContractVersion)).scalar_one().effective_from = datetime(2026, 3, 1, tzinfo=UTC)
db.commit()
dst_start = datetime(2026, 3, 28, 23, tzinfo=UTC)
dst = summarize(db, dst_start, dst_start + timedelta(days=2), now=datetime(2026, 3, 31, tzinfo=UTC))
assert dst["fixed_cost"] == Decimal("3.6")
def test_summary_is_half_open_at_local_midnight_and_settlement(db: Session, monkeypatch: pytest.MonkeyPatch) -> None:
_contract(db)
monkeypatch.setattr(timezone_service, "local_tz", lambda: ZoneInfo("Europe/Amsterdam"))
local = ZoneInfo("Europe/Amsterdam")
start = datetime(2026, 6, 24, 0, tzinfo=local).astimezone(UTC)
end = datetime(2026, 6, 25, 0, tzinfo=local).astimezone(UTC)
assert summarize(db, start, start, now=end + timedelta(hours=2))["fixed_cost"] == Decimal("0")
assert summarize(db, end, start, now=end + timedelta(hours=2))["fixed_cost"] == Decimal("0")
before = datetime(2026, 6, 24, 1, 4, 59, tzinfo=local).astimezone(UTC)
after = datetime(2026, 6, 24, 1, 5, tzinfo=local).astimezone(UTC)
assert summarize(db, start, end, now=before)["fixed_cost"] == Decimal("0")
assert summarize(db, start, end, now=after)["fixed_cost"] == Decimal("1.2")
def test_summary_versions_cover_first_day_intra_day_and_cross_day(db: Session, monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setattr(timezone_service, "local_tz", lambda: ZoneInfo("Europe/Amsterdam"))
local = ZoneInfo("Europe/Amsterdam")
day = datetime(2026, 6, 24, tzinfo=local)
first = _contract(db, start=(day + timedelta(hours=6)).astimezone(UTC))
first.values = {**VALUES, "standing": {**VALUES["standing"], "heating_network": "365"}}
second_values = {**VALUES, "standing": {**VALUES["standing"], "heating_network": "730"}}
switch = (day + timedelta(hours=18)).astimezone(UTC)
first.effective_to = switch
_version(db, first.contract, start=switch, values=second_values)
db.commit()
start = day.astimezone(UTC)
next_day = (day + timedelta(days=1)).astimezone(UTC)
following = (day + timedelta(days=2)).astimezone(UTC)
# Initial-version day only owns 18h: V1 owns 12h, V2 6h; on the following
# day V2 owns the full daily fixed amount.
assert summarize(db, start, next_day, now=following + timedelta(hours=2))["fixed_cost"] == Decimal("1.15")
assert summarize(db, start, following, now=following + timedelta(hours=2))["fixed_cost"] == Decimal("3.35")
def test_normal_row_has_full_audit_snapshot_and_single_commodity_scheduler(db: Session) -> None:
version = _contract(db)
meter, channel, binding = _domain(db, "heating")
_reading(db, channel, T0, "10")
_reading(db, channel, T0 + timedelta(minutes=15), "10.05")
db.commit()
compute_closed_periods(db, now=T0 + timedelta(minutes=16))
row = _row_at(db, "heating", T0)
assert (row.meter_id, row.source_binding_id, row.contract_version_id) == (meter.id, binding.id, version.id)
assert row.pricing_snapshot == VALUES
assert row.currency == "EUR" and {
key: Decimal(value) for key, value in row.cost_breakdown.items()
} == {"heating": Decimal("1.0")}
assert not row.degraded
assert _row_at(db, "hot_water", T0).degraded
@pytest.mark.parametrize(("commodity", "end_value"), [("heating", "10.1"), ("hot_water", "11")])
def test_delta_limit_is_inclusive_at_exact_boundary(db: Session, commodity: str, end_value: str) -> None:
_contract(db)
_, channel, _ = _domain(db, commodity)
_reading(db, channel, T0, "10")
_reading(db, channel, T0 + timedelta(minutes=15), end_value)
db.commit()
compute_period(db, commodity, T0)
assert not _row(db, commodity).degraded
@pytest.mark.parametrize("day", [datetime(2026, 3, 29), datetime(2026, 10, 25)])
def test_summary_dst_local_midnights_charge_one_daily_rate(
db: Session, monkeypatch: pytest.MonkeyPatch, day: datetime
) -> None:
_contract(db, start=datetime(2026, 1, 1, tzinfo=UTC))
monkeypatch.setattr(timezone_service, "local_tz", lambda: ZoneInfo("Europe/Amsterdam"))
local = ZoneInfo("Europe/Amsterdam")
start = day.replace(tzinfo=local).astimezone(UTC)
end = (day + timedelta(days=1)).replace(tzinfo=local).astimezone(UTC)
assert summarize(db, start, end, now=end + timedelta(hours=2))["fixed_cost"] == Decimal("1.2")
+213
View File
@@ -0,0 +1,213 @@
"""API coverage for the thermal meter-cost ledger (M8-T16)."""
from __future__ import annotations
from datetime import UTC, datetime, timedelta
from decimal import Decimal
from unittest.mock import patch
import pytest
from fastapi.testclient import TestClient
from sqlalchemy import create_engine, select
from sqlalchemy.orm import Session
from app.models.energy import EnergyContract, EnergyContractVersion, Meter, MeterCostPeriod
from app.models.meter_source import MeterSource, MeterSourceBinding, MeterSourceChannel
_CSRF = "test-csrf-token"
_T0 = datetime(2026, 6, 23, 10, tzinfo=UTC)
def _login(client: TestClient) -> str:
response = client.post("/api/auth/login", json={"username": "admin", "password": "test-password"})
assert response.status_code == 200
return response.json()["csrf_token"]
@pytest.fixture()
def meter_cost_client(auth_database):
from app.main import create_app
engine = create_engine(auth_database["app_url"], connect_args={"check_same_thread": False})
with TestClient(create_app()) as client:
yield client, engine
engine.dispose()
def _row(start: datetime, *, commodity: str = "heating", degraded: bool = False) -> MeterCostPeriod:
return MeterCostPeriod(
commodity=commodity, period_start=start, period_end=start + timedelta(minutes=15),
meter_id=None, source_binding_id=None, contract_version_id=None,
quantity=Decimal("0"), cost=Decimal("0") if degraded else Decimal("1.250000000"),
currency="EUR", cost_breakdown={} if degraded else {"heating": "1.250000000"},
pricing_snapshot={}, quality="invalid" if degraded else "valid", degraded=degraded,
degraded_reason="missing_contract" if degraded else None, created_at=_T0, updated_at=_T0,
)
def _normal_ids(db: Session, contract_version_id: int) -> tuple[int, int, int]:
now = datetime.now(UTC)
meter = Meter(label="heating", commodity="heating", started_at=_T0 - timedelta(days=1),
reason="initial", created_at=now)
source = MeterSource(name="source", kind="warmtelink_serial", enabled=True, config={}, status="online",
created_at=now, updated_at=now)
db.add_all((meter, source))
db.flush()
channel = MeterSourceChannel(source_id=source.id, channel_key="heating", label="heating",
suggested_commodity="heating", unit="GJ", latest_quality="valid",
created_at=now, updated_at=now)
db.add(channel)
db.flush()
binding = MeterSourceBinding(meter_id=meter.id, channel_id=channel.id, started_at=meter.started_at,
created_at=now, updated_at=now)
db.add(binding)
db.flush()
return meter.id, binding.id, contract_version_id
def test_meter_costs_require_auth_and_paginate_decimal_rows(meter_cost_client) -> None:
client, engine = meter_cost_client
assert client.get("/api/energy/meter-costs?scope=thermal").status_code == 401
with Session(engine) as db:
db.add_all([_row(_T0, degraded=True), _row(_T0 + timedelta(minutes=15), degraded=True)])
db.commit()
_login(client)
response = client.get("/api/energy/meter-costs?scope=thermal&limit=1&offset=1")
assert response.status_code == 200
body = response.json()
assert body["total"] == 2 and len(body["items"]) == 1
assert body["items"][0]["cost"] == "0.000000000"
assert body["items"][0]["degraded_reason"] == "missing_contract"
assert client.get("/api/energy/meter-costs?scope=electricity").status_code == 422
def test_meter_costs_half_open_stable_pagination_and_deep_decimal_audit(meter_cost_client) -> None:
client, engine = meter_cost_client
with Session(engine) as db:
normal = _row(_T0, degraded=True)
normal.quantity = Decimal("0.050000")
normal.cost = Decimal("1.123456789")
normal.cost_breakdown = {"heating": Decimal("1.123456789")}
normal.pricing_snapshot = {"variable": {"heating": Decimal("22.46913578")}}
water = _row(_T0, commodity="hot_water", degraded=True)
later = _row(_T0 + timedelta(minutes=15), degraded=True)
db.add_all((normal, water, later))
db.commit()
_login(client)
base = "/api/energy/meter-costs?scope=thermal&start=2026-06-23T10:00:00Z&end=2026-06-23T10:15:00Z"
response = client.get(base + "&limit=1&offset=0")
assert response.status_code == 200
assert response.json()["total"] == 2
item = response.json()["items"][0]
assert item["commodity"] == "heating"
assert item["quantity"] == "0.050000"
assert item["cost_breakdown"] == {"heating": "1.123456789"}
assert item["pricing_snapshot"]["variable"]["heating"] == "22.46913578"
assert client.get(base + "&commodity=hot_water").json()["items"][0]["commodity"] == "hot_water"
# Tie-breaking includes id, so the second page deterministically returns water.
assert client.get(base + "&limit=1&offset=1").json()["items"][0]["commodity"] == "hot_water"
assert client.get(base + "&offset=2").json()["items"] == []
def test_meter_cost_summary_empty_and_recompute_csrf_window_validation(meter_cost_client) -> None:
client, _engine = meter_cost_client
csrf = _login(client)
summary = client.get(
"/api/energy/meter-costs/summary?scope=thermal&start=2026-06-23T00:00:00Z&end=2026-06-24T00:00:00Z"
)
assert summary.status_code == 200
assert summary.json()["all_in"] == "0"
assert summary.json()["fixed_breakdown"] == {
"heating_network": "0", "metering": "0", "delivery_set": "0",
"hot_water_network": "0", "other": "0",
}
url = "/api/energy/meter-costs/recompute?scope=thermal&start=2026-06-23T10:01:00Z&end=2026-06-23T10:15:00Z"
assert client.post(url).status_code == 403
assert client.post(url, headers={"X-CSRF-Token": _CSRF}).status_code == 422
overlarge = (
"/api/energy/meter-costs/recompute?scope=thermal&start=2026-01-01T00:00:00Z"
"&end=2026-02-02T00:00:00Z"
)
assert client.post(overlarge, headers={"X-CSRF-Token": _CSRF}).status_code == 422
assert client.post(
"/api/energy/meter-costs/recompute?scope=thermal&start=2026-06-23T10:15:00Z&end=2026-06-23T10:00:00Z",
headers={"X-CSRF-Token": csrf},
).status_code == 422
assert client.post(
"/api/energy/meter-costs/recompute?scope=electricity&start=2026-06-23T10:00:00Z&end=2026-06-23T10:15:00Z",
headers={"X-CSRF-Token": csrf},
).status_code == 422
def test_meter_cost_summary_returns_five_fixed_components_and_totals(meter_cost_client) -> None:
client, engine = meter_cost_client
now = datetime.now(UTC)
values = {
"variable": {"heating": "20", "hot_water_heating": "4", "hot_water": "2", "hot_water_tax": "1"},
"standing": {"heating_network": "365", "metering": "73", "delivery_set": "0", "hot_water_network": "0", "other": "0"},
}
with Session(engine) as db:
contract = EnergyContract(name="thermal", kind="district_heating", scope="thermal", active=True,
currency="EUR", created_at=now, updated_at=now)
db.add(contract)
db.flush()
db.add(EnergyContractVersion(contract_id=contract.id, effective_from=_T0 - timedelta(days=2),
values=values, created_at=now))
db.flush() # Allocate the contract-version id before normal ledger fixtures.
meter_id, binding_id, version_id = _normal_ids(db, db.scalars(select(EnergyContractVersion.id)).one())
heating, water = _row(_T0), _row(_T0, commodity="hot_water")
heating.meter_id = water.meter_id = meter_id
heating.source_binding_id = water.source_binding_id = binding_id
heating.contract_version_id = water.contract_version_id = version_id
heating.cost = Decimal("1.000000000")
heating.cost_breakdown = {"heating": "1.000000000"}
water.cost = Decimal("1.400000000")
water.cost_breakdown = {"hot_water_heating": "0.8", "hot_water": "0.4", "hot_water_tax": "0.2"}
db.add_all((heating, water))
db.commit()
_login(client)
with patch("app.api.routes.api.meter_costs.local_now", return_value=datetime(2026, 6, 25, 2, tzinfo=UTC)):
response = client.get("/api/energy/meter-costs/summary?scope=thermal&start=2026-06-23T00:00:00Z&end=2026-06-24T00:00:00Z")
assert response.status_code == 200
body = response.json()
assert body["heating"] == "1.000000000" and body["hot_water_heating"] == "0.8"
assert body["fixed_breakdown"] == {
"heating_network": "2", "metering": "0.4", "delivery_set": "0",
"hot_water_network": "0", "other": "0",
}
assert body["variable_subtotal"] == "2.400000000"
assert body["fixed_subtotal"] == "2.4" and body["all_in"] == "4.800000000"
assert body["period_count"] == 2 and body["degraded_count"] == 0
def test_recompute_commits_once_and_rolls_back_service_or_count_failure(meter_cost_client) -> None:
client, engine = meter_cost_client
csrf = _login(client)
url = "/api/energy/meter-costs/recompute?scope=thermal&start=2026-06-23T10:00:00Z&end=2026-06-23T10:15:00Z"
headers = {"X-CSRF-Token": csrf}
# An exception after a service-side mutation must leave no generated rows.
from app.services import meter_cost as service
original = service.compute_period
calls = 0
def fail_after_first(*args, **kwargs):
nonlocal calls
calls += 1
if calls == 2:
raise RuntimeError("service failure")
return original(*args, **kwargs)
with patch("app.services.meter_cost.compute_period", side_effect=fail_after_first):
with pytest.raises(RuntimeError):
client.post(url, headers=headers)
with Session(engine) as db:
assert db.scalars(select(MeterCostPeriod)).all() == []
# The query after recomputation is part of that same transaction as well.
with patch("app.api.routes.api.meter_costs.select", side_effect=RuntimeError("count failure")):
with pytest.raises(RuntimeError):
client.post(url, headers=headers)
with Session(engine) as db:
assert db.scalars(select(MeterCostPeriod)).all() == []
+465
View File
@@ -0,0 +1,465 @@
"""Migration and ORM contracts for the commodity-scoped meter cost ledger."""
from __future__ import annotations
from datetime import datetime, timedelta, timezone
from decimal import Decimal
from pathlib import Path
import pytest
from alembic import command
from alembic.config import Config
from sqlalchemy import create_engine, event as sa_event, insert, inspect, select, text
from sqlalchemy.exc import IntegrityError, StatementError
from sqlalchemy.orm import Session
from app.models.energy import MeterCostPeriod
from scripts.app_db_adopt import APP_BASELINE_REVISION
REVISION_18 = "20260822_18_contract_scopes"
REVISION_19 = "20260822_19_meter_cost_periods"
UTC = timezone.utc
def _config(database_url: str) -> Config:
config = Config("alembic_app.ini")
config.set_main_option("sqlalchemy.url", database_url)
return config
def _fk_engine(database_url: str):
engine = create_engine(database_url)
@sa_event.listens_for(engine, "connect")
def _enable_foreign_keys(connection, _record) -> None:
connection.execute("PRAGMA foreign_keys = ON")
return engine
def _period(**overrides) -> MeterCostPeriod:
start = datetime(2026, 8, 22, 12, tzinfo=UTC)
values = {
"commodity": "heating",
"period_start": start,
"period_end": start + timedelta(minutes=15),
"quantity": Decimal("999999999.123456"),
"cost": Decimal("999999.123456789"),
"currency": "EUR",
"cost_breakdown": {"heating": "0.012345678"},
"pricing_snapshot": {"heating": "10.000000", "kind": "district_heating"},
"quality": "valid",
"degraded": True,
"degraded_reason": "test_fixture_without_audit_links",
"created_at": start,
"updated_at": start,
}
values.update(overrides)
return MeterCostPeriod(**values)
def test_meter_cost_period_empty_db_upgrade_shape_and_baseline(tmp_path: Path) -> None:
database_url = f"sqlite:///{tmp_path / 'empty.db'}"
config = _config(database_url)
command.upgrade(config, "head")
command.upgrade(config, "head")
engine = create_engine(database_url)
inspector = inspect(engine)
columns = {column["name"]: column for column in inspector.get_columns("meter_cost_period")}
assert APP_BASELINE_REVISION == REVISION_19
assert {"commodity", "period_start", "period_end", "quantity", "cost"} <= columns.keys()
assert columns["meter_id"]["nullable"]
assert columns["source_binding_id"]["nullable"]
assert columns["contract_version_id"]["nullable"]
assert {index["name"] for index in inspector.get_indexes("meter_cost_period")} >= {
"ix_meter_cost_period_commodity_start",
"ix_meter_cost_period_source_binding_id",
}
assert ("commodity", "period_start") in {
tuple(constraint["column_names"])
for constraint in inspector.get_unique_constraints("meter_cost_period")
}
foreign_keys = {
foreign_key["constrained_columns"][0]: foreign_key
for foreign_key in inspector.get_foreign_keys("meter_cost_period")
}
for column, table in {
"meter_id": "meter",
"source_binding_id": "meter_source_binding",
"contract_version_id": "energy_contract_version",
}.items():
assert foreign_keys[column]["referred_table"] == table
assert foreign_keys[column]["options"]["ondelete"] == "RESTRICT"
engine.dispose()
def test_meter_cost_period_revision_18_upgrade_preserves_electricity_rows(tmp_path: Path) -> None:
database_url = f"sqlite:///{tmp_path / 'revision18.db'}"
config = _config(database_url)
command.upgrade(config, REVISION_18)
engine = create_engine(database_url)
stamp = datetime(2026, 8, 22, 12)
with engine.begin() as connection:
connection.execute(
text(
"INSERT INTO energy_cost_period "
"(period_start, d1_kwh, d2_kwh, r1_kwh, r2_kwh, import_cost, "
"export_revenue, net_cost, currency, pricing, contract_version_id, "
"meter_id, source_binding_id, degraded, computed_at) "
"VALUES (:stamp, 1, 2, 3, 4, 5, 6, 7, 'EUR', :pricing, NULL, NULL, NULL, 0, :stamp)"
),
{"stamp": stamp, "pricing": '{"historic":"unchanged"}'},
)
command.upgrade(config, "head")
command.upgrade(config, "head")
with engine.connect() as connection:
assert connection.execute(text("SELECT version_num FROM alembic_version")).scalar_one() == REVISION_19
assert connection.execute(text("SELECT COUNT(*) FROM energy_cost_period")).scalar_one() == 1
assert connection.execute(text("SELECT pricing FROM energy_cost_period")).scalar_one() == (
'{"historic":"unchanged"}'
)
assert connection.execute(text("SELECT COUNT(*) FROM meter_cost_period")).scalar_one() == 0
command.downgrade(config, REVISION_18)
assert "meter_cost_period" not in inspect(engine).get_table_names()
engine.dispose()
def test_meter_cost_period_decimal_json_and_degraded_round_trip(tmp_path: Path) -> None:
database_url = f"sqlite:///{tmp_path / 'roundtrip.db'}"
command.upgrade(_config(database_url), "head")
engine = create_engine(database_url)
with Session(engine) as session:
normal = _period()
degraded = _period(
commodity="hot_water",
meter_id=None,
source_binding_id=None,
contract_version_id=None,
quality="unverifiable",
degraded=True,
degraded_reason="missing_binding",
quantity=Decimal("0.000000"),
cost=Decimal("0.000000000"),
cost_breakdown={"hot_water": "0.000000000"},
pricing_snapshot={"reason": "missing_binding"},
)
session.add_all((normal, degraded))
session.commit()
session.expire_all()
stored = session.get(MeterCostPeriod, normal.id)
assert stored is not None
assert stored.quantity == Decimal("999999999.123456")
assert stored.cost == Decimal("999999.123456789")
assert stored.cost_breakdown == {"heating": "0.012345678"}
assert all(not isinstance(value, float) for value in stored.pricing_snapshot.values())
assert session.get(MeterCostPeriod, degraded.id).degraded_reason == "missing_binding"
engine.dispose()
def test_meter_cost_period_uses_decimal_text_and_normalises_json_snapshots(tmp_path: Path) -> None:
database_url = f"sqlite:///{tmp_path / 'exact.db'}"
command.upgrade(_config(database_url), "head")
engine = create_engine(database_url)
with Session(engine) as session:
row = _period(
cost_breakdown={"nested": [Decimal("2.000000000")]},
pricing_snapshot={"rate": Decimal("2.000000")},
)
session.add(row)
session.commit()
session.expire_all()
stored = session.get(MeterCostPeriod, row.id)
assert stored is not None
assert stored.cost_breakdown == {"nested": ["2.000000000"]}
assert stored.pricing_snapshot == {"rate": "2.000000"}
raw = session.execute(text("SELECT typeof(quantity), typeof(cost) FROM meter_cost_period")).one()
assert raw == ("text", "text")
with pytest.raises(ValueError, match="numeric JSON"):
_period(cost_breakdown={"nested": [1]})
with pytest.raises(ValueError, match="numeric JSON"):
_period(pricing_snapshot={"nested": [1.25]})
engine.dispose()
def test_meter_cost_period_core_json_bind_normalises_decimal_and_rejects_numeric(tmp_path: Path) -> None:
database_url = f"sqlite:///{tmp_path / 'core-json.db'}"
command.upgrade(_config(database_url), "head")
engine = create_engine(database_url)
values = {
"commodity": "heating",
"period_start": datetime(2026, 8, 22, 12, tzinfo=UTC),
"period_end": datetime(2026, 8, 22, 12, 15, tzinfo=UTC),
"quantity": Decimal("0.000000"),
"cost": Decimal("0.000000000"),
"currency": "EUR",
"cost_breakdown": {"nested": [Decimal("2.000000000")]},
"pricing_snapshot": {"rate": Decimal("2.000000")},
"quality": "valid",
"degraded": True,
"degraded_reason": "core fixture",
"created_at": datetime(2026, 8, 22, 12, tzinfo=UTC),
"updated_at": datetime(2026, 8, 22, 12, tzinfo=UTC),
}
with engine.begin() as connection:
connection.execute(insert(MeterCostPeriod.__table__).values(values))
with Session(engine) as session:
stored = session.scalar(select(MeterCostPeriod))
assert stored is not None
assert stored.cost_breakdown == {"nested": ["2.000000000"]}
assert stored.pricing_snapshot == {"rate": "2.000000"}
for field, numeric_value in (("cost_breakdown", {"nested": [1]}), ("pricing_snapshot", {"rate": 1.25})):
with engine.begin() as connection, pytest.raises(StatementError, match="numeric JSON"):
connection.execute(
insert(MeterCostPeriod.__table__).values(
{**values, "commodity": f"invalid-{field}", field: numeric_value}
)
)
engine.dispose()
@pytest.mark.parametrize(
("field", "value", "message"),
[
("quantity", Decimal("1000000000.000000"), "precision"),
("quantity", Decimal("1.1234567"), "scale"),
("cost", Decimal("1000000.000000000"), "precision"),
("cost", Decimal("1.1234567899"), "scale"),
],
)
def test_meter_cost_period_rejects_decimal_precision_and_scale_overflow(field, value, message) -> None:
with pytest.raises(ValueError, match=message):
_period(**{field: value})
def test_meter_cost_period_accepts_signed_decimal_boundaries_without_float_bind(tmp_path: Path) -> None:
database_url = f"sqlite:///{tmp_path / 'boundaries.db'}"
command.upgrade(_config(database_url), "head")
engine = create_engine(database_url)
with Session(engine) as session:
rows = (
_period(quantity=Decimal("999999999.999999"), cost=Decimal("999999.999999999")),
_period(
commodity="hot_water",
quantity=Decimal("-999999999.999999"),
cost=Decimal("-999999.999999999"),
),
)
session.add_all(rows)
session.commit()
session.expire_all()
assert session.get(MeterCostPeriod, rows[0].id).quantity == Decimal("999999999.999999")
assert session.get(MeterCostPeriod, rows[1].id).cost == Decimal("-999999.999999999")
engine.dispose()
def test_meter_cost_period_rejects_duplicate_period_invalid_interval_and_json_float(tmp_path: Path) -> None:
database_url = f"sqlite:///{tmp_path / 'constraints.db'}"
command.upgrade(_config(database_url), "head")
engine = create_engine(database_url)
with Session(engine) as session:
first = _period()
session.add(first)
session.commit()
session.add(_period(cost=Decimal("1.000000000")))
with pytest.raises(IntegrityError):
session.commit()
session.rollback()
session.add(_period(commodity="hot_water", period_end=datetime(2026, 8, 22, 12, tzinfo=UTC)))
with pytest.raises(ValueError, match="after period_start"):
session.flush()
session.rollback()
with pytest.raises(ValueError, match="timezone-aware"):
_period(
commodity="hot_water",
period_start=datetime(2026, 8, 22, 12),
period_end=datetime(2026, 8, 22, 12, 15, tzinfo=UTC),
)
with pytest.raises(ValueError, match="decimal strings"):
_period(pricing_snapshot={"heating": 1.25})
with pytest.raises(ValueError, match="normal meter cost periods"):
session.add(_period(commodity="hot_water", degraded=False, degraded_reason=None))
session.flush()
session.rollback()
engine.dispose()
def test_meter_cost_period_uses_utc_instant_idempotency_and_db_constraints(tmp_path: Path) -> None:
database_url = f"sqlite:///{tmp_path / 'utc.db'}"
command.upgrade(_config(database_url), "head")
engine = create_engine(database_url)
utc_start = datetime(2026, 8, 22, 12, tzinfo=UTC)
offset_start = datetime(2026, 8, 22, 14, tzinfo=timezone(timedelta(hours=2)))
with Session(engine) as session:
first = _period(period_start=utc_start, period_end=utc_start + timedelta(minutes=15))
assert first.period_start == utc_start
session.add(first)
session.commit()
session.add(
_period(
period_start=offset_start,
period_end=offset_start + timedelta(minutes=15),
cost=Decimal("1.000000000"),
)
)
with pytest.raises(IntegrityError):
session.commit()
session.rollback()
with engine.begin() as connection:
values = {
"start": "2026-08-23 00:00:00.000000",
"quantity": "0.000000",
"cost": "0.000000000",
"now": "2026-08-23 00:00:00.000000",
}
for commodity, end in (("equal", values["start"]), ("reversed", "2026-08-22 23:59:59.000000")):
with pytest.raises(IntegrityError):
connection.execute(
text(
"INSERT INTO meter_cost_period "
"(commodity, period_start, period_end, quantity, cost, currency, cost_breakdown, "
"pricing_snapshot, quality, degraded, degraded_reason, created_at, updated_at) "
"VALUES (:commodity, :start, :end, :quantity, :cost, 'EUR', '{}', '{}', 'valid', "
"1, 'core test', :now, :now)"
),
{**values, "commodity": commodity, "end": end},
)
with pytest.raises(IntegrityError):
connection.execute(
text(
"INSERT INTO meter_cost_period "
"(commodity, period_start, period_end, quantity, cost, currency, cost_breakdown, "
"pricing_snapshot, quality, degraded, degraded_reason, created_at, updated_at) "
"VALUES ('missing-reason', :start, '2026-08-23 00:15:00.000000', :quantity, :cost, "
"'EUR', '{}', '{}', 'valid', 1, NULL, :now, :now)"
),
values,
)
engine.dispose()
def test_meter_cost_period_core_uses_utc_instant_idempotency_and_reloads_aware(tmp_path: Path) -> None:
database_url = f"sqlite:///{tmp_path / 'core-utc.db'}"
command.upgrade(_config(database_url), "head")
engine = create_engine(database_url)
utc_start = datetime(2026, 8, 22, 12, tzinfo=UTC)
values = {
"commodity": "heating",
"period_start": utc_start,
"period_end": utc_start + timedelta(minutes=15),
"quantity": Decimal("0.000000"),
"cost": Decimal("0.000000000"),
"currency": "EUR",
"cost_breakdown": {},
"pricing_snapshot": {},
"quality": "valid",
"degraded": True,
"degraded_reason": "core fixture",
"created_at": utc_start,
"updated_at": utc_start,
}
with engine.begin() as connection:
row_id = connection.execute(insert(MeterCostPeriod.__table__).values(values)).inserted_primary_key[0]
with engine.begin() as connection, pytest.raises(IntegrityError):
connection.execute(
insert(MeterCostPeriod.__table__).values(
{
**values,
"period_start": datetime(2026, 8, 22, 14, tzinfo=timezone(timedelta(hours=2))),
"period_end": datetime(2026, 8, 22, 14, 15, tzinfo=timezone(timedelta(hours=2))),
}
)
)
with engine.connect() as connection, pytest.raises(StatementError, match="timezone-aware"):
connection.execute(
insert(MeterCostPeriod.__table__).values(
{**values, "commodity": "naive", "period_start": datetime(2026, 8, 23, 12)}
)
)
with Session(engine) as session:
stored = session.get(MeterCostPeriod, row_id)
assert stored is not None
assert stored.period_start == utc_start
assert stored.period_start.tzinfo is not None
assert stored.period_start.utcoffset() == timedelta(0)
stored.quality = "unverifiable"
session.commit()
engine.dispose()
def test_meter_cost_period_foreign_keys_restrict_deletion(tmp_path: Path) -> None:
database_url = f"sqlite:///{tmp_path / 'foreign_keys.db'}"
command.upgrade(_config(database_url), "head")
engine = _fk_engine(database_url)
timestamp = datetime(2026, 8, 22, 12)
with engine.begin() as connection:
meter_id = connection.execute(
text(
"INSERT INTO meter (uuid, label, commodity, started_at, ended_at, reason, note, created_at) "
"VALUES ('meter-cost-test', 'Test', 'heating', :timestamp, NULL, 'initial', NULL, :timestamp)"
),
{"timestamp": timestamp},
).lastrowid
source_id = connection.execute(
text(
"INSERT INTO meter_source (uuid, name, kind, enabled, config, status, created_at, updated_at) "
"VALUES ('source-cost-test', 'Test', 'warmtelink_serial', 1, '{}', 'online', :timestamp, :timestamp)"
),
{"timestamp": timestamp},
).lastrowid
channel_id = connection.execute(
text(
"INSERT INTO meter_source_channel (uuid, source_id, channel_key, label, unit, created_at, updated_at) "
"VALUES ('channel-cost-test', :source_id, 'heating', 'Heating', 'GJ', :timestamp, :timestamp)"
),
{"source_id": source_id, "timestamp": timestamp},
).lastrowid
binding_id = connection.execute(
text(
"INSERT INTO meter_source_binding "
"(uuid, meter_id, channel_id, started_at, ended_at, created_at, updated_at) "
"VALUES ('binding-cost-test', :meter_id, :channel_id, :timestamp, NULL, :timestamp, :timestamp)"
),
{"meter_id": meter_id, "channel_id": channel_id, "timestamp": timestamp},
).lastrowid
contract_id = connection.execute(
text(
"INSERT INTO energy_contract (name, kind, scope, active, currency, created_at, updated_at) "
"VALUES ('Test', 'district_heating', 'thermal', 1, 'EUR', :timestamp, :timestamp)"
),
{"timestamp": timestamp},
).lastrowid
version_id = connection.execute(
text(
"INSERT INTO energy_contract_version "
"(contract_id, effective_from, effective_to, \"values\", created_at) "
"VALUES (:contract_id, :timestamp, NULL, '{}', :timestamp)"
),
{"contract_id": contract_id, "timestamp": timestamp},
).lastrowid
connection.execute(
text(
"INSERT INTO meter_cost_period "
"(commodity, period_start, period_end, meter_id, source_binding_id, contract_version_id, "
"quantity, cost, currency, cost_breakdown, pricing_snapshot, quality, degraded, degraded_reason, "
"created_at, updated_at) VALUES "
"('heating', :timestamp, :period_end, :meter_id, :binding_id, :version_id, "
"'0.001000', '0.010000000', 'EUR', '{}', '{}', 'valid', 0, NULL, :timestamp, :timestamp)"
),
{
"timestamp": timestamp,
"period_end": timestamp + timedelta(minutes=15),
"meter_id": meter_id,
"binding_id": binding_id,
"version_id": version_id,
},
)
for statement, values in (
("DELETE FROM meter WHERE id = :id", {"id": meter_id}),
("DELETE FROM meter_source_binding WHERE id = :id", {"id": binding_id}),
("DELETE FROM energy_contract_version WHERE id = :id", {"id": version_id}),
):
with pytest.raises(IntegrityError):
connection.execute(text(statement), values)
engine.dispose()
+937
View File
@@ -0,0 +1,937 @@
"""Contract tests for M8 source/channel/binding management routes."""
from __future__ import annotations
from datetime import UTC, datetime, timedelta
from decimal import Decimal
from queue import Empty, Queue
from types import SimpleNamespace
import threading
import time
from unittest.mock import patch
import pytest
from fastapi.testclient import TestClient
from sqlalchemy import create_engine, select
from sqlalchemy.orm import Session
from app.models.config import AppConfigEntry
from app.models.energy import DsmrReading, Meter
from app.models.meter_source import MeterSource, MeterSourceBinding, MeterSourceChannel, WarmteLinkReading
_CSRF = "test-csrf-token"
def _login(client: TestClient) -> None:
assert client.post("/api/auth/login", json={"username": "admin", "password": "test-password"}).status_code == 200
def _client(auth_database):
from app.main import create_app
engine = create_engine(auth_database["app_url"], connect_args={"check_same_thread": False})
return TestClient(create_app()), engine
def _create_source(client: TestClient, *, config: dict | None = None) -> dict:
response = client.post("/api/energy/sources", headers={"X-CSRF-Token": _CSRF}, json={
"name": "Synthetic DSMR", "kind": "dsmr_mqtt", "config": config or {},
})
assert response.status_code == 201
return response.json()
def _add_channel(engine, source_uuid: str, *, key: str = "electricity") -> str:
with Session(engine) as session:
source = session.execute(select(MeterSource).where(MeterSource.uuid == source_uuid)).scalar_one()
now = datetime.now(UTC)
channel = MeterSourceChannel(
source_id=source.id, channel_key=key, label="Electricity", unit="kWh",
created_at=now, updated_at=now,
)
session.add(channel)
session.commit()
return channel.uuid
@pytest.fixture(autouse=True)
def _mock_lifecycle_recompute(monkeypatch):
"""Keep lifecycle API contracts on synthetic DBs; cost engines have their own tests."""
monkeypatch.setattr("app.api.routes.api.meters.recompute_range", lambda *args, **kwargs: 0)
monkeypatch.setattr("app.api.routes.api.meter_sources.electricity_recompute_range", lambda *args, **kwargs: 0)
def test_source_profiles_and_crud_mask_secrets(auth_database):
client, engine = _client(auth_database)
with client:
assert client.get("/api/energy/source-profiles").status_code == 401
_login(client)
profiles = client.get("/api/energy/source-profiles")
assert profiles.status_code == 200
assert {item["kind"] for item in profiles.json()["items"]} == {"dsmr_mqtt", "warmtelink_serial"}
create = client.post("/api/energy/sources", headers={"X-CSRF-Token": _CSRF}, json={
"name": "Synthetic DSMR", "kind": "dsmr_mqtt",
"config": {"username": "user", "password": "not-for-api"},
})
assert create.status_code == 201
source = create.json()
assert source["config"]["username"] == ""
assert source["config"]["password"] == ""
assert "not-for-api" not in str(source)
patch = client.patch(f"/api/energy/sources/{source['uuid']}", headers={"X-CSRF-Token": _CSRF}, json={"config": {"password": ""}})
assert patch.status_code == 200
assert patch.json()["config"]["password"] == ""
with Session(engine) as session:
stored = session.execute(
select(MeterSource).where(MeterSource.uuid == source["uuid"])
).scalar_one()
assert stored.config["password"] == "not-for-api"
assert client.post(f"/api/energy/sources/{source['uuid']}/discover", headers={"X-CSRF-Token": _CSRF}).status_code == 200
assert client.delete(f"/api/energy/sources/{source['uuid']}", headers={"X-CSRF-Token": _CSRF}).status_code == 204
engine.dispose()
def test_serial_source_reconciles_only_after_commit(auth_database, monkeypatch):
"""Creation and mutation trigger the worker manager after a successful commit."""
from app.api.routes.api import meter_sources
calls = []
monkeypatch.setattr(meter_sources.warmtelink_worker_manager, "reconcile", lambda: calls.append(True))
client, engine = _client(auth_database)
with client:
_login(client)
baseline = len(calls)
response = client.post("/api/energy/sources", headers={"X-CSRF-Token": _CSRF}, json={
"name": "Serial", "kind": "warmtelink_serial", "config": {"path": "/dev/ttyUSB0"},
})
assert response.status_code == 201
source_uuid = response.json()["uuid"]
assert len(calls) == baseline + 1
assert client.patch(f"/api/energy/sources/{source_uuid}", headers={"X-CSRF-Token": _CSRF}, json={"enabled": False}).status_code == 200
assert len(calls) == baseline + 2
engine.dispose()
def test_serial_source_commit_is_not_reported_as_failed_when_reconcile_raises(auth_database, monkeypatch):
from app.api.routes.api import meter_sources
client, engine = _client(auth_database)
with client:
_login(client)
monkeypatch.setattr(
meter_sources.warmtelink_worker_manager, "reconcile",
lambda: (_ for _ in ()).throw(RuntimeError()),
)
response = client.post("/api/energy/sources", headers={"X-CSRF-Token": _CSRF}, json={
"name": "Serial", "kind": "warmtelink_serial", "config": {"path": "/dev/ttyUSB0"},
})
assert response.status_code == 201
engine.dispose()
def test_dsmr_crud_reconciles_runtime_and_invalidates_retained_callbacks(auth_database, monkeypatch):
"""CRUD converges real DSMR ownership, not merely a mocked reconcile call."""
from app.api.routes.api import meter_sources
from app.services import dsmr_ingest
class FakeMqttManager:
def __init__(self) -> None:
self.handlers: dict[int, dict[str, object]] = {}
self.replace_calls: list[tuple[int, dict[str, object]]] = []
self.remove_calls: list[int] = []
def replace_source(self, source_id: int, **kwargs: object) -> bool:
self.replace_calls.append((source_id, kwargs))
self.handlers[source_id] = kwargs["subscriptions"] # type: ignore[assignment]
kwargs["state_handler"]("connecting") # type: ignore[operator]
return True
def remove_source(self, source_id: int) -> None:
self.remove_calls.append(source_id)
self.handlers.pop(source_id, None)
def source_is_active(self, source_id: int) -> bool:
return source_id in self.handlers
mqtt = FakeMqttManager()
warmtelink_calls: list[None] = []
monkeypatch.setattr("app.integrations.mqtt.mqtt_manager", mqtt)
monkeypatch.setattr(
meter_sources.warmtelink_worker_manager, "reconcile", lambda: warmtelink_calls.append(None),
)
monkeypatch.setattr(dsmr_ingest, "_subscriptions", {})
monkeypatch.setattr(dsmr_ingest, "_subscription_client_ids", {})
monkeypatch.setattr(dsmr_ingest, "_subscription_tokens", {})
monkeypatch.setattr(dsmr_ingest, "_tariffs", {})
client, engine = _client(auth_database)
with Session(engine) as session:
session.add(AppConfigEntry(
key="MQTT_CLIENT_ID", value="home-automation-api-test", updated_at=datetime.now(UTC),
))
session.commit()
payload = b'{"timestamp":"2030-01-01T00:00:00Z"}'
with client:
_login(client)
baseline_warmtelink_calls = len(warmtelink_calls)
created = client.post("/api/energy/sources", headers={"X-CSRF-Token": _CSRF}, json={
"name": "Managed DSMR", "kind": "dsmr_mqtt",
"config": {"broker_host": "broker.test", "topic": "meter/first"},
})
assert created.status_code == 201
assert created.json()["status"] == "connecting"
source_uuid = created.json()["uuid"]
with Session(engine) as session:
source_id = session.scalar(select(MeterSource.id).where(MeterSource.uuid == source_uuid))
assert source_id is not None
first_handler = mqtt.handlers[source_id]["meter/first"]
assert mqtt.replace_calls[-1][1]["base_client_id"] == "home-automation-api-test"
updated = client.patch(
f"/api/energy/sources/{source_uuid}", headers={"X-CSRF-Token": _CSRF},
json={"config": {"topic": "meter/second"}},
)
assert updated.status_code == 200
assert updated.json()["status"] == "connecting"
assert mqtt.remove_calls == [source_id]
assert "meter/second" in mqtt.handlers[source_id]
first_handler(payload) # type: ignore[operator]
with Session(engine) as session:
assert session.query(DsmrReading).filter_by(meter_source_id=source_id).count() == 0
retained_handler = mqtt.handlers[source_id]["meter/second"]
disabled = client.patch(
f"/api/energy/sources/{source_uuid}", headers={"X-CSRF-Token": _CSRF}, json={"enabled": False},
)
assert disabled.status_code == 200
assert disabled.json()["status"] == "unknown"
assert source_id not in mqtt.handlers
retained_handler(payload) # type: ignore[operator]
with Session(engine) as session:
assert session.query(DsmrReading).filter_by(meter_source_id=source_id).count() == 0
assert client.patch(
f"/api/energy/sources/{source_uuid}", headers={"X-CSRF-Token": _CSRF}, json={"enabled": True},
).status_code == 200
delete_handler = mqtt.handlers[source_id]["meter/second"]
assert client.delete(f"/api/energy/sources/{source_uuid}", headers={"X-CSRF-Token": _CSRF}).status_code == 204
assert source_id not in mqtt.handlers
delete_handler(payload) # type: ignore[operator]
with Session(engine) as session:
assert session.get(MeterSource, source_id) is None
assert session.query(DsmrReading).filter_by(meter_source_id=source_id).count() == 0
assert len(warmtelink_calls) == baseline_warmtelink_calls + 5
engine.dispose()
def test_source_crud_runtime_failures_do_not_hide_commits(auth_database, monkeypatch):
from app.api.routes.api import meter_sources
calls: list[str] = []
client, engine = _client(auth_database)
with client:
_login(client)
monkeypatch.setattr(
meter_sources.warmtelink_worker_manager, "reconcile",
lambda: calls.append("warmtelink") or (_ for _ in ()).throw(RuntimeError()),
)
monkeypatch.setattr(
meter_sources, "apply_dsmr_subscription",
lambda _settings: calls.append("dsmr") or (_ for _ in ()).throw(RuntimeError()),
)
created = client.post("/api/energy/sources", headers={"X-CSRF-Token": _CSRF}, json={
"name": "Durable DSMR", "kind": "dsmr_mqtt", "config": {},
})
assert created.status_code == 201
source_uuid = created.json()["uuid"]
assert calls == ["warmtelink", "dsmr"]
assert client.patch(
f"/api/energy/sources/{source_uuid}", headers={"X-CSRF-Token": _CSRF}, json={"enabled": False},
).status_code == 200
assert client.delete(f"/api/energy/sources/{source_uuid}", headers={"X-CSRF-Token": _CSRF}).status_code == 204
with Session(engine) as session:
assert session.scalar(select(MeterSource.id).where(MeterSource.uuid == source_uuid)) is None
engine.dispose()
def test_binding_routes_and_atomic_meter_declaration(auth_database):
client, engine = _client(auth_database)
with client:
_login(client)
source_response = client.post("/api/energy/sources", headers={"X-CSRF-Token": _CSRF}, json={
"name": "Synthetic DSMR", "kind": "dsmr_mqtt", "config": {},
})
source_uuid = source_response.json()["uuid"]
with Session(engine) as session:
source = session.query(MeterSource).filter_by(uuid=source_uuid).one()
now = datetime.now(UTC)
channel = MeterSourceChannel(
source_id=source.id, channel_key="electricity", label="Electricity", unit="kWh",
created_at=now, updated_at=now,
)
session.add(channel)
session.commit()
channel_uuid = channel.uuid
declaration = {
"label": "Bound meter", "started_at": "2025-01-01T00:00:00Z", "reason": "initial",
"commodity": "electricity", "source_channel_uuid": channel_uuid,
}
created = client.post("/api/energy/meters", headers={"X-CSRF-Token": _CSRF}, json=declaration)
assert created.status_code == 201
assert created.json()["bindings"][0]["source_channel_uuid"] == channel_uuid
meter_id = created.json()["id"]
assert client.get(f"/api/energy/meters/{meter_id}/bindings").json()["total"] == 1
assert client.get(f"/api/energy/sources/{source_uuid}/channels").json()["items"][0]["binding_count"] == 1
invalid = dict(declaration, label="Must roll back", started_at="2025-02-01T00:00:00Z", source_channel_uuid="missing-channel")
assert client.post("/api/energy/meters", headers={"X-CSRF-Token": _CSRF}, json=invalid).status_code == 404
assert client.get("/api/energy/meters").json()["total"] == 1
with Session(engine) as session:
original = session.get(Meter, meter_id)
assert original is not None
assert original.ended_at is None
assert session.execute(select(Meter).where(Meter.label == "Must roll back")).scalar_one_or_none() is None
assert session.execute(select(MeterSourceBinding).where(MeterSourceBinding.meter_id == meter_id)).scalars().all()
engine.dispose()
def test_source_and_binding_error_contracts_csrf_timezone_and_dsmr_compatibility(auth_database, monkeypatch):
"""Exercise the public error boundary without opening serial or MQTT I/O."""
from zoneinfo import ZoneInfo
from app.services import timezone as timezone_service
monkeypatch.setattr(timezone_service, "local_tz", lambda: ZoneInfo("Europe/Amsterdam"))
client, engine = _client(auth_database)
with client:
# All management reads require a session and mutations require CSRF.
assert client.get("/api/energy/commodities").status_code == 401
_login(client)
assert client.post("/api/energy/sources", json={
"name": "No CSRF", "kind": "dsmr_mqtt", "config": {},
}).status_code == 403
assert client.post("/api/energy/sources", headers={"X-CSRF-Token": _CSRF}, json={
"name": "Invalid", "kind": "dsmr_mqtt", "config": {"unexpected": True},
}).status_code == 422
assert client.get("/api/energy/sources/does-not-exist").status_code == 404
source = client.post("/api/energy/sources", headers={"X-CSRF-Token": _CSRF}, json={
"name": "DSMR source", "kind": "dsmr_mqtt", "config": {"password": "stored-secret"},
})
assert source.status_code == 201
source_uuid = source.json()["uuid"]
assert "stored-secret" not in client.get(f"/api/energy/sources/{source_uuid}").text
assert client.get("/api/energy/commodities").json()["items"] == [
{"key": "electricity", "unit": "kWh", "capabilities": ["meter", "binding", "cost"]},
{"key": "heating", "unit": "GJ", "capabilities": ["meter", "binding"]},
{"key": "hot_water", "unit": "", "capabilities": ["meter", "binding"]},
]
with Session(engine) as session:
source_model = session.query(MeterSource).filter_by(uuid=source_uuid).one()
now = datetime.now(UTC)
channel = MeterSourceChannel(
source_id=source_model.id, channel_key="electricity", label="Electricity", unit="kWh",
created_at=now, updated_at=now,
)
session.add(channel)
session.add(DsmrReading(
meter_source_id=source_model.id, telegram_id=7, recorded_at=now,
payload={"compatibility": "latest"},
))
session.commit()
channel_uuid = channel.uuid
# Retained channels prohibit deletion; there is no cascade escape hatch.
assert client.delete(f"/api/energy/sources/{source_uuid}", headers={"X-CSRF-Token": _CSRF}).status_code == 409
assert client.get(f"/api/energy/sources/{source_uuid}/channels/not-a-channel/readings").status_code == 404
readings = client.get(f"/api/energy/sources/{source_uuid}/channels/{channel_uuid}/readings")
assert readings.status_code == 200
assert readings.json()["total"] == 1
assert readings.json()["items"] == [{"recorded_at": now.isoformat().replace("+00:00", ""), "value": None, "quality": None}]
assert "telegram_id" not in readings.text
latest = client.get("/api/energy/dsmr/latest")
assert latest.status_code == 200
assert latest.json()["payload"] == {"compatibility": "latest"}
# This declaration is deliberately retroactive for timezone coverage;
# mock the billing sweep so the API contract test stays bounded.
with patch("app.api.routes.api.meters.recompute_range", return_value=0):
meter = client.post("/api/energy/meters", headers={"X-CSRF-Token": _CSRF}, json={
"label": "Local-time meter", "started_at": "2025-01-02T00:00:00", "reason": "initial",
})
assert meter.status_code == 201
meter_id = meter.json()["id"]
binding = client.post(f"/api/energy/meters/{meter_id}/bindings", headers={"X-CSRF-Token": _CSRF}, json={
"source_channel_uuid": channel_uuid, "started_at": "2025-01-02T00:00:00",
})
assert binding.status_code == 201
binding_body = binding.json()
localized_start = datetime.fromisoformat(binding_body["started_at"].replace("Z", "+00:00"))
# SQLite returns UTC columns without tzinfo; retain the UTC clock instant
# regardless of that transport detail.
assert localized_start.replace(tzinfo=None) == datetime(2025, 1, 1, 23, 0, 0)
assert client.post(f"/api/energy/meters/{meter_id}/bindings", headers={"X-CSRF-Token": _CSRF}, json={
"source_channel_uuid": channel_uuid, "started_at": "2025-01-02T00:00:00",
}).status_code == 422
assert client.patch(f"/api/energy/bindings/{binding_body['uuid']}", headers={"X-CSRF-Token": _CSRF}, json={
"ended_at": "2025-01-02T00:00:00",
}).status_code == 422
assert client.patch(f"/api/energy/bindings/{binding_body['uuid']}", headers={"X-CSRF-Token": _CSRF}, json={
"ended_at": "2025-01-03T00:00:00",
}).status_code == 200
assert client.patch("/api/energy/bindings/not-a-binding", headers={"X-CSRF-Token": _CSRF}, json={
"ended_at": "2025-01-03T00:00:00",
}).status_code == 404
engine.dispose()
def test_management_reads_require_auth_and_mutations_require_csrf(auth_database):
"""Every M8-T06 management route enforces the session/CSRF contract."""
client, engine = _client(auth_database)
with client:
for path in (
"/api/energy/source-profiles",
"/api/energy/commodities",
"/api/energy/sources",
"/api/energy/sources/missing",
"/api/energy/sources/missing/channels",
"/api/energy/sources/missing/channels/missing/readings",
"/api/energy/meters/1/bindings",
):
assert client.get(path).status_code == 401
_login(client)
assert client.post("/api/energy/sources", json={
"name": "No CSRF", "kind": "dsmr_mqtt", "config": {},
}).status_code == 403
source = _create_source(client)
channel_uuid = _add_channel(engine, source["uuid"])
meter = client.post("/api/energy/meters", headers={"X-CSRF-Token": _CSRF}, json={
"label": "CSRF meter", "started_at": "2025-01-01T00:00:00Z", "reason": "initial",
})
assert meter.status_code == 201
meter_id = meter.json()["id"]
binding = client.post(f"/api/energy/meters/{meter_id}/bindings", headers={"X-CSRF-Token": _CSRF}, json={
"source_channel_uuid": channel_uuid, "started_at": "2025-01-01T00:00:00Z",
})
assert binding.status_code == 201
assert client.patch(f"/api/energy/sources/{source['uuid']}", json={"name": "blocked"}).status_code == 403
assert client.delete(f"/api/energy/sources/{source['uuid']}").status_code == 403
assert client.post(f"/api/energy/sources/{source['uuid']}/discover").status_code == 403
assert client.post(f"/api/energy/meters/{meter_id}/bindings", json={
"source_channel_uuid": channel_uuid, "started_at": "2031-01-01T00:00:00Z",
}).status_code == 403
assert client.patch(f"/api/energy/bindings/{binding.json()['uuid']}", json={}).status_code == 403
engine.dispose()
def test_source_channel_binding_response_contract_and_discover_capabilities(auth_database):
client, engine = _client(auth_database)
with client:
_login(client)
source = _create_source(client, config={"username": "private-user", "password": "private-secret"})
channel_uuid = _add_channel(engine, source["uuid"])
with Session(engine) as session:
source_model = session.query(MeterSource).filter_by(uuid=source["uuid"]).one()
source_model.status = "online"
source_model.last_seen_at = datetime.now(UTC)
source_model.last_error = None
session.commit()
listed = client.get("/api/energy/sources")
assert listed.status_code == 200
assert listed.json()["total"] >= 1
source_item = next(item for item in listed.json()["items"] if item["uuid"] == source["uuid"])
assert source_item["uuid"] == source["uuid"]
detail = client.get(f"/api/energy/sources/{source['uuid']}")
assert detail.status_code == 200
assert detail.json()["uuid"] == source["uuid"]
for body in (listed.json(), detail.json()):
rendered = str(body)
assert "private-secret" not in rendered
assert "channel_key" not in rendered
assert "fingerprint" not in rendered
discovered = client.post(f"/api/energy/sources/{source['uuid']}/discover", headers={"X-CSRF-Token": _CSRF})
assert discovered.status_code == 200
assert discovered.json() == {
"requested": False, "supported": True, "status": "managed_by_runtime",
"request_id": None,
"detail": "This source is discovered by its runtime subscription; no connection was opened.",
"channels": [],
}
channels = client.get(f"/api/energy/sources/{source['uuid']}/channels")
assert channels.status_code == 200
channel = channels.json()["items"][0]
assert channel["uuid"] == channel_uuid
assert set(channel) == {
"uuid", "label", "suggested_commodity", "unit", "device_type", "latest_value",
"latest_at", "latest_quality", "binding_count", "bound_meter_ids", "binding_summary",
}
assert channels.json()["source_status"] == "online"
meter = client.post("/api/energy/meters", headers={"X-CSRF-Token": _CSRF}, json={
"label": "Contract meter", "started_at": "2025-01-01T00:00:00Z", "reason": "initial",
})
assert meter.status_code == 201
binding = client.post(f"/api/energy/meters/{meter.json()['id']}/bindings", headers={"X-CSRF-Token": _CSRF}, json={
"source_channel_uuid": channel_uuid, "started_at": "2025-01-01T00:00:00Z",
})
assert binding.status_code == 201
binding_item = client.get(f"/api/energy/meters/{meter.json()['id']}/bindings").json()["items"][0]
assert binding_item["uuid"] == binding.json()["uuid"]
assert binding_item["source_uuid"] == source["uuid"]
assert binding_item["source_channel_uuid"] == channel_uuid
assert set(binding_item) == {
"uuid", "meter_id", "source_channel_uuid", "source_uuid", "started_at", "ended_at",
"created_at", "updated_at",
}
engine.dispose()
def test_binding_patch_omitted_null_and_adjacent_half_open_boundaries(auth_database):
client, engine = _client(auth_database)
with client:
_login(client)
source = _create_source(client)
channel_uuid = _add_channel(engine, source["uuid"])
meter = client.post("/api/energy/meters", headers={"X-CSRF-Token": _CSRF}, json={
"label": "Timeline meter", "started_at": "2025-01-01T00:00:00Z", "reason": "initial",
})
assert meter.status_code == 201
meter_id = meter.json()["id"]
first = client.post(f"/api/energy/meters/{meter_id}/bindings", headers={"X-CSRF-Token": _CSRF}, json={
"source_channel_uuid": channel_uuid, "started_at": "2025-01-01T00:00:00Z",
"ended_at": "2025-02-01T00:00:00Z",
})
assert first.status_code == 201
first_uuid = first.json()["uuid"]
corrected = client.patch(f"/api/energy/bindings/{first_uuid}", headers={"X-CSRF-Token": _CSRF}, json={
"started_at": "2025-01-02T00:00:00Z",
})
assert corrected.status_code == 200
assert corrected.json()["ended_at"] == "2025-02-01T00:00:00"
unchanged = client.patch(f"/api/energy/bindings/{first_uuid}", headers={"X-CSRF-Token": _CSRF}, json={})
assert unchanged.status_code == 200
assert unchanged.json()["ended_at"] == "2025-02-01T00:00:00"
reopened = client.patch(f"/api/energy/bindings/{first_uuid}", headers={"X-CSRF-Token": _CSRF}, json={"ended_at": None})
assert reopened.status_code == 200
assert reopened.json()["ended_at"] is None
reclosed = client.patch(f"/api/energy/bindings/{first_uuid}", headers={"X-CSRF-Token": _CSRF}, json={
"ended_at": "2025-02-01T00:00:00Z",
})
assert reclosed.status_code == 200
adjacent = client.post(f"/api/energy/meters/{meter_id}/bindings", headers={"X-CSRF-Token": _CSRF}, json={
"source_channel_uuid": channel_uuid, "started_at": "2025-02-01T00:00:00Z",
})
assert adjacent.status_code == 201
engine.dispose()
def test_binding_create_and_meter_declare_reject_future_boundaries(auth_database):
client, engine = _client(auth_database)
with client:
_login(client)
source = _create_source(client)
channel_uuid = _add_channel(engine, source["uuid"])
future = (datetime.now(UTC) + timedelta(minutes=5)).isoformat()
assert client.post("/api/energy/meters", headers={"X-CSRF-Token": _CSRF}, json={
"label": "future", "started_at": future, "reason": "initial",
}).status_code == 422
past = (datetime.now(UTC) - timedelta(hours=1)).isoformat()
meter = client.post("/api/energy/meters", headers={"X-CSRF-Token": _CSRF}, json={
"label": "past", "started_at": past, "reason": "initial",
})
assert meter.status_code == 201
assert client.post(f"/api/energy/meters/{meter.json()['id']}/bindings", headers={"X-CSRF-Token": _CSRF}, json={
"source_channel_uuid": channel_uuid, "started_at": future,
}).status_code == 422
engine.dispose()
def test_transfer_recovers_stranded_previous_meter_same_channel(auth_database):
"""A retained open row on a closed predecessor is recoverable in one request."""
client, engine = _client(auth_database)
with client:
_login(client)
source = _create_source(client)
channel_uuid = _add_channel(engine, source["uuid"])
start = datetime.now(UTC) - timedelta(days=2)
boundary = start + timedelta(days=1)
with Session(engine) as session:
channel = session.scalar(select(MeterSourceChannel).where(MeterSourceChannel.uuid == channel_uuid))
assert channel is not None
old = Meter(label="old", commodity="electricity", started_at=start, ended_at=boundary,
reason="meter_swap", created_at=start)
target = Meter(label="target", commodity="electricity", started_at=boundary,
reason="meter_swap", created_at=boundary)
session.add_all([old, target])
session.flush()
stranded = MeterSourceBinding(meter_id=old.id, channel_id=channel.id, started_at=start,
created_at=start, updated_at=start)
session.add(stranded)
session.commit()
target_id, stranded_uuid = target.id, stranded.uuid
response = client.post(f"/api/energy/meters/{target_id}/bindings/transfer", headers={"X-CSRF-Token": _CSRF}, json={
"from_binding_uuid": stranded_uuid, "to_source_channel_uuid": channel_uuid,
"effective_at": (boundary + timedelta(hours=2)).isoformat(),
})
assert response.status_code == 200
assert response.json()["closed_binding"]["ended_at"] is not None
assert response.json()["created_binding"]["started_at"].startswith((boundary + timedelta(hours=2)).isoformat()[:19])
engine.dispose()
def test_transfer_recovers_unique_gapped_predecessor_after_commit(auth_database, monkeypatch):
"""Recovery closes at the old epoch end and recomputes from that earliest boundary."""
from app.api.routes.api import meter_sources
client, engine = _client(auth_database)
calls: list[tuple[datetime, datetime, bool, bool]] = []
published: list[bool] = []
monkeypatch.setattr(
meter_sources, "electricity_recompute_range",
lambda _db, start, end, *, commit, strict: calls.append((start, end, commit, strict)) or 0,
)
monkeypatch.setattr(meter_sources, "_republish_after_commit", lambda _db: published.append(True))
with client:
_login(client)
source = _create_source(client)
channel_uuid = _add_channel(engine, source["uuid"])
start = datetime.now(UTC) - timedelta(days=3)
old_end = start + timedelta(days=1)
target_start = old_end + timedelta(hours=3)
effective_at = target_start + timedelta(hours=1)
with Session(engine) as session:
channel = session.scalar(select(MeterSourceChannel).where(MeterSourceChannel.uuid == channel_uuid))
assert channel is not None
old = Meter(label="old", commodity="electricity", started_at=start, ended_at=old_end,
reason="meter_swap", created_at=start)
target = Meter(label="target", commodity="electricity", started_at=target_start,
reason="initial", created_at=target_start)
session.add_all([old, target])
session.flush()
stranded = MeterSourceBinding(meter_id=old.id, channel_id=channel.id, started_at=start,
created_at=start, updated_at=start)
session.add(stranded)
session.commit()
target_id, stranded_uuid = target.id, stranded.uuid
response = client.post(
f"/api/energy/meters/{target_id}/bindings/transfer", headers={"X-CSRF-Token": _CSRF},
json={"from_binding_uuid": stranded_uuid, "to_source_channel_uuid": channel_uuid,
"effective_at": effective_at.isoformat()},
)
assert response.status_code == 200
assert response.json()["closed_binding"]["ended_at"].startswith(old_end.isoformat()[:19])
assert response.json()["created_binding"]["started_at"].startswith(effective_at.isoformat()[:19])
assert calls and calls[0][0] == old_end and calls[0][2:] == (False, True)
assert published == [True]
engine.dispose()
def test_transfer_rejects_intervening_meter_without_changing_stranded_binding(auth_database):
"""A non-predecessor recovery request is fail-closed and rolls back cleanly."""
client, engine = _client(auth_database)
with client:
_login(client)
source = _create_source(client)
channel_uuid = _add_channel(engine, source["uuid"])
start = datetime.now(UTC) - timedelta(days=4)
old_end = start + timedelta(days=1)
target_start = old_end + timedelta(days=2)
with Session(engine) as session:
channel = session.scalar(select(MeterSourceChannel).where(MeterSourceChannel.uuid == channel_uuid))
assert channel is not None
old = Meter(label="old", commodity="electricity", started_at=start, ended_at=old_end,
reason="meter_swap", created_at=start)
intervening = Meter(label="intervening", commodity="electricity", started_at=old_end,
ended_at=target_start, reason="other", created_at=old_end)
target = Meter(label="target", commodity="electricity", started_at=target_start,
reason="initial", created_at=target_start)
session.add_all([old, intervening, target])
session.flush()
stranded = MeterSourceBinding(meter_id=old.id, channel_id=channel.id, started_at=start,
created_at=start, updated_at=start)
session.add(stranded)
session.commit()
target_id, stranded_uuid, stranded_id = target.id, stranded.uuid, stranded.id
response = client.post(
f"/api/energy/meters/{target_id}/bindings/transfer", headers={"X-CSRF-Token": _CSRF},
json={"from_binding_uuid": stranded_uuid, "to_source_channel_uuid": channel_uuid,
"effective_at": (target_start + timedelta(hours=1)).isoformat()},
)
assert response.status_code == 422
with Session(engine) as observer:
binding = observer.get(MeterSourceBinding, stranded_id)
assert binding is not None and binding.ended_at is None
assert observer.scalars(select(MeterSourceBinding)).all() == [binding]
engine.dispose()
def test_warmtelink_discover_and_minute_history_are_bounded_and_private(auth_database, monkeypatch):
"""Discover delegates to the manager; readings expose accepted minute samples only."""
from app.api.routes.api import meter_sources
requested: list[int] = []
monkeypatch.setattr(
meter_sources.warmtelink_worker_manager, "request_discovery",
lambda source_id: requested.append(source_id) or SimpleNamespace(
status="completed", request_id=1, detail=None, completed=SimpleNamespace(is_set=lambda: False),
),
)
monkeypatch.setattr(meter_sources.warmtelink_worker_manager, "reconcile", lambda: None)
client, engine = _client(auth_database)
with client:
_login(client)
created = client.post("/api/energy/sources", headers={"X-CSRF-Token": _CSRF}, json={
"name": "WarmteLink", "kind": "warmtelink_serial", "config": {"path": "/dev/fake"},
})
assert created.status_code == 201
source_uuid = created.json()["uuid"]
now = datetime(2030, 1, 1, 12, 0, 30, tzinfo=UTC)
with Session(engine) as session:
source = session.execute(select(MeterSource).where(MeterSource.uuid == source_uuid)).scalar_one()
source.status = "online"
channel = MeterSourceChannel(
source_id=source.id, channel_key="heating", label="Heating", unit="GJ",
latest_value=Decimal("7.002"), latest_at=now, latest_quality="unverifiable",
created_at=now, updated_at=now,
)
session.add(channel)
session.flush()
session.add_all([
WarmteLinkReading(
channel_id=channel.id, recorded_at=now - timedelta(minutes=1), received_at=now,
value=Decimal("7.001"), unit="GJ", quality="unverifiable", equipment_fingerprint="masked",
),
WarmteLinkReading(
channel_id=channel.id, recorded_at=now, received_at=now,
value=Decimal("7.002"), unit="GJ", quality="unverifiable", equipment_fingerprint="masked",
),
])
session.commit()
channel_uuid = channel.uuid
discover = client.post(f"/api/energy/sources/{source_uuid}/discover", headers={"X-CSRF-Token": _CSRF})
assert discover.status_code == 200
assert discover.json()["status"] == "completed"
assert requested and "fingerprint" not in discover.text and "channel_key" not in discover.text
assert discover.json()["channels"][0]["latest_quality"] == "unverifiable"
history = client.get(
f"/api/energy/sources/{source_uuid}/channels/{channel_uuid}/readings",
params={"from": "2030-01-01T11:59:00Z", "to": "2030-01-01T12:01:00Z", "limit": 1},
)
assert history.status_code == 200
assert history.json()["total"] == 1
assert history.json()["items"] == [{
"recorded_at": "2030-01-01T11:59:30", "value": "7.001", "quality": "unverifiable",
}]
assert client.get(
f"/api/energy/sources/{source_uuid}/channels/{channel_uuid}/readings",
params={"from": "2030-01-01T12:01:00Z", "to": "2030-01-01T12:00:00Z"},
).status_code == 422
assert client.get(
f"/api/energy/sources/{source_uuid}/channels/{channel_uuid}/readings", params={"limit": 0}
).status_code == 422
engine.dispose()
def test_warmtelink_discover_auth_csrf_disabled_and_source_ownership(auth_database, monkeypatch):
from app.api.routes.api import meter_sources
monkeypatch.setattr(
meter_sources.warmtelink_worker_manager, "request_discovery",
lambda _source_id: SimpleNamespace(
status="pending", request_id=1, detail=None, completed=SimpleNamespace(is_set=lambda: False),
),
)
monkeypatch.setattr(meter_sources.warmtelink_worker_manager, "reconcile", lambda: None)
client, engine = _client(auth_database)
with client:
serial = client.post("/api/energy/sources", headers={"X-CSRF-Token": _CSRF}, json={
"name": "Serial", "kind": "warmtelink_serial", "enabled": False, "config": {"path": "/dev/fake"},
})
assert serial.status_code == 401 # no session yet
_login(client)
serial = client.post("/api/energy/sources", headers={"X-CSRF-Token": _CSRF}, json={
"name": "Serial", "kind": "warmtelink_serial", "enabled": False, "config": {"path": "/dev/fake"},
})
other = _create_source(client)
channel_uuid = _add_channel(engine, other["uuid"])
assert client.post(f"/api/energy/sources/{serial.json()['uuid']}/discover").status_code == 403
disabled = client.post(
f"/api/energy/sources/{serial.json()['uuid']}/discover", headers={"X-CSRF-Token": _CSRF}
)
assert disabled.status_code == 200 and disabled.json()["status"] == "error"
assert client.get(
f"/api/energy/sources/{serial.json()['uuid']}/channels/{channel_uuid}/readings"
).status_code == 404
engine.dispose()
def test_discovery_manager_is_source_scoped_and_never_replaces_a_worker(auth_database):
"""The real manager queues requests on one fake read-only serial owner."""
from app.services.warmtelink_worker import WarmteLinkWorkerManager
engine = create_engine(auth_database["app_url"], connect_args={"check_same_thread": False})
with Session(engine) as session:
now = datetime.now(UTC)
source = MeterSource(
name="Serial", kind="warmtelink_serial", enabled=True, config={"path": "/dev/fake"},
created_at=now, updated_at=now,
)
session.add(source)
session.commit()
source_id = source.id
class FakeReadOnlyWorker:
instances: list["FakeReadOnlyWorker"] = []
def __init__(self, _source_id, _config, **_kwargs):
self.requests = []
self.thread = SimpleNamespace(is_alive=lambda: True)
self.__class__.instances.append(self)
def start(self):
return None
def stop(self):
return None
def join(self, timeout=5):
return True
def request_discovery(self, request):
self.requests.append(request)
manager = WarmteLinkWorkerManager(
session_factory=lambda: Session(engine), worker_factory=FakeReadOnlyWorker,
)
manager.reconcile()
assert manager.worker_count == 1
first = manager.request_discovery(source_id)
assert first.status == "pending"
worker = FakeReadOnlyWorker.instances[0]
assert len(worker.requests) == 1
# A client timing out/cancelling leaves the queued request and worker alone;
# completing it later cannot open another descriptor or create bindings.
worker.requests[0].status = "completed"
worker.requests[0].completed.set()
results = []
threads = [threading.Thread(target=lambda: results.append(manager.request_discovery(source_id))) for _ in range(2)]
for thread in threads:
thread.start()
for thread in threads:
thread.join()
assert manager.worker_count == 1
assert len(FakeReadOnlyWorker.instances) == 1
assert len(worker.requests) == 3
with Session(engine) as session:
assert session.query(Meter).count() == 0
assert session.query(MeterSourceBinding).count() == 0
manager.shutdown()
engine.dispose()
def test_real_warmtelink_discovery_waits_for_admission_and_keeps_its_serial_owner(
auth_database, monkeypatch,
):
"""A rejected candidate is neither a discovery success nor exposed metadata."""
from app.integrations.p1 import dsmr_crc16
from app.services.warmtelink_ingest import WarmteLinkIngestor
from app.services import warmtelink_worker
from app.services.warmtelink_worker import WarmteLinkWorker, WarmteLinkWorkerManager
engine = create_engine(auth_database["app_url"], connect_args={"check_same_thread": False})
received_at = datetime(2026, 8, 22, 10, 0, 30, tzinfo=UTC) # 12:00:30 Europe/Amsterdam.
with Session(engine) as session:
source = MeterSource(
name="Serial", kind="warmtelink_serial", enabled=True, config={"path": "/dev/fake"},
created_at=received_at, updated_at=received_at,
)
session.add(source)
session.commit()
source_id = source.id
def frame(second: int, *, crc: bool = False) -> bytes:
body = (
b"/WARMTE\r\n"
+ f"0-0:1.0.0(2608221200{second:02d}S)\r\n".encode()
+ b"0-0:96.1.1(REDACTED)\r\n"
+ b"0-1:24.1.0(006)\r\n"
+ b"0-1:96.1.0(REDACTED)\r\n"
+ f"0-1:24.2.1(2608221200{second:02d}S)(5.900*m3)\r\n".encode()
+ b"0-2:24.1.0(012)\r\n"
+ b"0-2:96.1.0(REDACTED)\r\n"
+ f"0-2:24.2.1(2608221200{second:02d}S)(0.017*GJ)\r\n".encode()
)
payload = body + b"!"
return payload + (f"{dsmr_crc16(payload):04X}".encode() if crc else b"") + b"\r\n"
class FakeReadOnlySerial:
instances: list["FakeReadOnlySerial"] = []
def __init__(self):
self.frames: Queue[bytes] = Queue()
self.closed = False
self.__class__.instances.append(self)
def read(self, _size: int = 1) -> bytes:
try:
return self.frames.get(timeout=0.01)
except Empty:
return b""
def close(self) -> None:
self.closed = True
monkeypatch.setattr(warmtelink_worker, "_DISCOVERY_TIMEOUT_SECONDS", 0.15)
monkeypatch.setattr(warmtelink_worker, "_DISCOVERY_WAIT_SECONDS", 0.02)
manager = WarmteLinkWorkerManager(
session_factory=lambda: Session(engine), serial_factory=lambda _config: FakeReadOnlySerial(),
worker_factory=lambda source_id, config, **kwargs: WarmteLinkWorker(
source_id,
config,
ingestor=WarmteLinkIngestor(clock=lambda: received_at),
**kwargs,
),
)
manager.reconcile()
serial = FakeReadOnlySerial.instances[0]
request = manager.request_discovery(source_id)
assert request.status == "pending"
serial.frames.put(frame(0)) # First unverifiable candidate is not admitted.
time.sleep(0.04)
assert not request.completed.is_set()
with Session(engine) as session:
assert session.query(MeterSourceChannel).filter_by(source_id=source_id).count() == 0
assert session.query(MeterSourceBinding).count() == 0
serial.frames.put(frame(10)) # Strictly continuous successor admits both channels.
assert request.completed.wait(1)
assert request.status == "completed"
with Session(engine) as session:
assert session.query(MeterSourceChannel).filter_by(source_id=source_id).count() == 2
assert session.query(MeterSourceBinding).count() == 0
rejected = manager.request_discovery(source_id)
assert rejected.status == "pending"
serial.frames.put(b"/malformed!\r\n")
assert rejected.completed.wait(1)
assert rejected.status == "error" and rejected.detail == "Discovery timed out."
assert manager.worker_count == 1 and len(FakeReadOnlySerial.instances) == 1
recovered = manager.request_discovery(source_id)
serial.frames.put(frame(20, crc=True))
assert recovered.completed.wait(1)
assert recovered.status == "completed"
assert manager.worker_count == 1 and len(FakeReadOnlySerial.instances) == 1
manager.shutdown()
assert serial.closed
engine.dispose()
+426
View File
@@ -0,0 +1,426 @@
"""M8-T02 tests for source profiles and transaction-owned binding services."""
from __future__ import annotations
from datetime import UTC, datetime, timedelta
import pytest
from sqlalchemy import create_engine, event
from sqlalchemy.orm import Session
from app.integrations.meter_sources import (
SECRET_MASK,
SourceProfileError,
merge_source_config,
sanitize_source_config,
validate_source_config,
)
from app.models.energy import Meter
from app.models.meter_source import MeterSource, MeterSourceBinding, MeterSourceChannel
from app.services.meter_sources import (
BindingOverlapError,
BindingValidationError,
SourceDeleteRestrictedError,
close_binding,
create_binding,
create_binding_for_meter_swap,
create_source,
delete_source,
transfer_binding,
upsert_discovered_channel,
)
@pytest.mark.parametrize(
("kind", "config", "expected"),
[
("dsmr_mqtt", {}, {"broker_port": 1883, "topic": "dsmr/json", "sample_interval_s": 10}),
(
"warmtelink_serial",
{"path": "/dev/warmtelink"},
{"baudrate": 115200, "data_bits": 7, "parity": "N", "stop_bits": 1},
),
],
)
def test_profiles_fill_defaults(kind, config, expected):
validated = validate_source_config(kind, config)
assert {key: validated[key] for key in expected} == expected
@pytest.mark.parametrize(
("kind", "config"),
[
("warmtelink_serial", {"path": "/tmp/warmtelink"}),
("warmtelink_serial", {"path": "/dev/warmtelink", "data_bits": 8}),
("dsmr_mqtt", {"extra": True}),
],
)
def test_profile_rejects_invalid_or_unknown_config(kind, config):
with pytest.raises(SourceProfileError):
validate_source_config(kind, config)
def test_secret_sanitize_and_mask_merge_keep_old_value():
original = validate_source_config("dsmr_mqtt", {"password": "not-for-response"})
assert sanitize_source_config("dsmr_mqtt", original)["password"] == SECRET_MASK
merged = merge_source_config("dsmr_mqtt", original, {"password": SECRET_MASK, "topic": "new/topic"})
assert merged["password"] == "not-for-response"
assert merged["topic"] == "new/topic"
@pytest.fixture()
def session(tmp_path):
engine = create_engine(f"sqlite:///{tmp_path / 'source_services.db'}")
@event.listens_for(engine, "connect")
def _enable_foreign_keys(dbapi_connection, _connection_record):
dbapi_connection.execute("PRAGMA foreign_keys=ON")
from app.db import Base
Base.metadata.create_all(engine)
with Session(engine) as db_session:
yield db_session
engine.dispose()
def _meter(session: Session, commodity: str, label: str = "Meter") -> Meter:
timestamp = datetime(2026, 8, 22, tzinfo=UTC)
meter = Meter(
label=label,
commodity=commodity,
started_at=timestamp,
reason="initial",
created_at=timestamp,
)
session.add(meter)
session.flush()
return meter
def _source_and_channel(session: Session, kind: str, unit: str) -> tuple[MeterSource, MeterSourceChannel]:
config = {"path": "/dev/warmtelink"} if kind == "warmtelink_serial" else {}
source = create_source(session, name="Source", kind=kind, config=config)
session.flush()
channel = upsert_discovered_channel(
session,
source_id=source.id,
channel_key=f"{unit}-total",
label="Total",
unit=unit,
suggested_commodity="heating",
)
session.flush()
return source, channel
def test_channel_upsert_is_idempotent_and_never_auto_binds(session):
_, channel = _source_and_channel(session, "warmtelink_serial", "GJ")
same = upsert_discovered_channel(
session,
source_id=channel.source_id,
channel_key=channel.channel_key,
label="Renamed total",
unit="GJ",
suggested_commodity="heating",
)
session.flush()
assert same.id == channel.id
assert same.label == "Renamed total"
assert session.query(MeterSourceBinding).count() == 0
def test_channel_upsert_rejects_unit_change_when_channel_is_bound(session):
meter = _meter(session, "heating")
_, channel = _source_and_channel(session, "warmtelink_serial", "GJ")
create_binding(session, meter_id=meter.id, channel_id=channel.id, started_at=meter.started_at)
session.flush()
with pytest.raises(BindingValidationError, match="Cannot change unit of bound channel"):
upsert_discovered_channel(
session,
source_id=channel.source_id,
channel_key=channel.channel_key,
label="Total",
unit="",
suggested_commodity="hot_water",
)
assert channel.unit == "GJ"
def test_binding_checks_both_sides_and_allows_equal_boundaries(session):
meter_one = _meter(session, "heating", "one")
meter_two = _meter(session, "heating", "two")
_, channel_one = _source_and_channel(session, "warmtelink_serial", "GJ")
_, channel_two = _source_and_channel(session, "warmtelink_serial", "GJ")
start = datetime(2026, 8, 22, tzinfo=UTC)
boundary = start + timedelta(hours=1)
create_binding(session, meter_id=meter_one.id, channel_id=channel_one.id, started_at=start, ended_at=boundary)
create_binding(session, meter_id=meter_one.id, channel_id=channel_two.id, started_at=boundary)
create_binding(session, meter_id=meter_two.id, channel_id=channel_one.id, started_at=boundary)
with pytest.raises(BindingOverlapError):
create_binding(session, meter_id=meter_one.id, channel_id=channel_two.id, started_at=start + timedelta(minutes=30))
with pytest.raises(BindingOverlapError):
create_binding(session, meter_id=meter_two.id, channel_id=channel_one.id, started_at=start + timedelta(minutes=30))
def test_binding_rejects_incompatible_unit_and_close_keeps_transaction_open(session):
meter = _meter(session, "electricity")
_, channel = _source_and_channel(session, "warmtelink_serial", "GJ")
with pytest.raises(BindingValidationError):
create_binding(session, meter_id=meter.id, channel_id=channel.id, started_at=meter.started_at)
thermal_meter = _meter(session, "heating", "thermal")
binding = create_binding(session, meter_id=thermal_meter.id, channel_id=channel.id, started_at=meter.started_at)
session.flush()
close_binding(session, binding.id, ended_at=meter.started_at + timedelta(minutes=1))
assert session.in_transaction()
session.rollback()
assert session.get(MeterSourceBinding, binding.id) is None
def test_meter_swap_hands_off_only_the_previous_meter_binding(session):
start = datetime(2026, 8, 22, tzinfo=UTC)
boundary = start + timedelta(days=1)
old_meter = _meter(session, "heating", "old")
old_meter.started_at = start
old_meter.ended_at = None
new_meter = Meter(
label="new",
commodity="heating",
started_at=boundary,
reason="meter_swap",
created_at=boundary,
)
session.add(new_meter)
session.flush()
_, channel = _source_and_channel(session, "warmtelink_serial", "GJ")
old_binding = create_binding(
session, meter_id=old_meter.id, channel_id=channel.id, started_at=start
)
old_meter.ended_at = boundary
session.flush()
new_binding = create_binding_for_meter_swap(
session,
old_meter_id=old_meter.id,
new_meter_id=new_meter.id,
channel_id=channel.id,
started_at=boundary,
)
session.flush()
assert old_binding.ended_at == boundary
assert new_binding.started_at == boundary
assert new_binding.ended_at is None
def test_meter_swap_rejects_channel_owned_by_a_different_meter(session):
start = datetime(2026, 8, 22, tzinfo=UTC)
boundary = start + timedelta(days=1)
old_meter = _meter(session, "heating", "old")
new_meter = _meter(session, "heating", "new")
other_meter = _meter(session, "heating", "other")
old_meter.started_at = start
old_meter.ended_at = boundary
new_meter.started_at = boundary
_, channel = _source_and_channel(session, "warmtelink_serial", "GJ")
create_binding(session, meter_id=other_meter.id, channel_id=channel.id, started_at=start)
with pytest.raises(BindingOverlapError, match="cannot be handed off"):
create_binding_for_meter_swap(
session,
old_meter_id=old_meter.id,
new_meter_id=new_meter.id,
channel_id=channel.id,
started_at=boundary,
)
def test_meter_swap_rejects_ambiguous_channel_without_closing_any_binding(session):
start = datetime(2026, 8, 22, tzinfo=UTC)
boundary = start + timedelta(days=1)
old_meter = _meter(session, "heating", "old")
old_meter.started_at = start
old_meter.ended_at = None
new_meter = Meter(
label="new",
commodity="heating",
started_at=boundary,
reason="meter_swap",
created_at=boundary,
)
other_meter = _meter(session, "heating", "other")
session.add(new_meter)
session.flush()
_, channel = _source_and_channel(session, "warmtelink_serial", "GJ")
old_binding = create_binding(session, meter_id=old_meter.id, channel_id=channel.id, started_at=start)
old_meter.ended_at = boundary
session.add(
MeterSourceBinding(
meter_id=other_meter.id,
channel_id=channel.id,
started_at=start,
created_at=start,
updated_at=start,
)
)
session.flush()
with pytest.raises(BindingOverlapError, match="occupied or has an ambiguous binding"):
create_binding_for_meter_swap(
session,
old_meter_id=old_meter.id,
new_meter_id=new_meter.id,
channel_id=channel.id,
started_at=boundary,
)
assert old_binding.ended_at is None
def test_meter_swap_rejects_incompatible_channel(session):
start = datetime(2026, 8, 22, tzinfo=UTC)
boundary = start + timedelta(days=1)
old_meter = _meter(session, "heating", "old")
old_meter.started_at = start
old_meter.ended_at = boundary
new_meter = Meter(
label="new",
commodity="heating",
started_at=boundary,
reason="meter_swap",
created_at=boundary,
)
session.add(new_meter)
session.flush()
_, channel = _source_and_channel(session, "dsmr_mqtt", "kWh")
with pytest.raises(BindingValidationError, match="requires unit"):
create_binding_for_meter_swap(
session,
old_meter_id=old_meter.id,
new_meter_id=new_meter.id,
channel_id=channel.id,
started_at=boundary,
)
def test_transfer_closes_and_opens_at_shared_boundary(session):
start = datetime(2026, 8, 22, tzinfo=UTC)
meter = _meter(session, "heating")
meter.started_at = start
_, old_channel = _source_and_channel(session, "warmtelink_serial", "GJ")
_, new_channel = _source_and_channel(session, "warmtelink_serial", "GJ")
old = create_binding(session, meter_id=meter.id, channel_id=old_channel.id, started_at=start)
session.flush()
closed, created = transfer_binding(
session, target_meter_id=meter.id, from_binding_id=old.id,
to_channel_id=new_channel.id, effective_at=start + timedelta(hours=1),
)
assert closed.ended_at == created.started_at == start + timedelta(hours=1)
assert created.channel_id == new_channel.id
def test_transfer_rejects_future_without_mutating_source_binding(session):
start = datetime.now(UTC) - timedelta(hours=2)
meter = _meter(session, "heating")
meter.started_at = start
_, old_channel = _source_and_channel(session, "warmtelink_serial", "GJ")
_, new_channel = _source_and_channel(session, "warmtelink_serial", "GJ")
old = create_binding(session, meter_id=meter.id, channel_id=old_channel.id, started_at=start)
session.flush()
with pytest.raises(BindingValidationError, match="future"):
transfer_binding(session, target_meter_id=meter.id, from_binding_id=old.id,
to_channel_id=new_channel.id, effective_at=datetime.now(UTC) + timedelta(minutes=1))
assert old.ended_at is None
def test_cross_meter_transfer_recovers_stranded_same_channel(session):
start = datetime(2026, 8, 20, tzinfo=UTC)
boundary = start + timedelta(days=1)
old = _meter(session, "heating", "old")
old.started_at, old.ended_at = start, None
new = Meter(label="new", commodity="heating", started_at=boundary, reason="meter_swap", created_at=boundary)
session.add(new)
session.flush()
_, channel = _source_and_channel(session, "warmtelink_serial", "GJ")
stranded = create_binding(session, meter_id=old.id, channel_id=channel.id, started_at=start)
old.ended_at = boundary
session.flush()
closed, created = transfer_binding(
session, target_meter_id=new.id, from_binding_id=stranded.id,
to_channel_id=channel.id, effective_at=boundary + timedelta(hours=2),
)
assert closed.ended_at == boundary
assert created.started_at == boundary + timedelta(hours=2)
assert created.channel_id == channel.id
def test_cross_meter_transfer_recovers_unique_gapped_predecessor(session):
"""A deliberate no-meter gap does not make the latest predecessor ambiguous."""
start = datetime(2026, 8, 20, tzinfo=UTC)
old_end = start + timedelta(days=1)
target_start = old_end + timedelta(hours=3)
old = _meter(session, "heating", "old")
old.started_at, old.ended_at = start, old_end
target = Meter(label="target", commodity="heating", started_at=target_start,
reason="initial", created_at=target_start)
session.add(target)
session.flush()
_, channel = _source_and_channel(session, "warmtelink_serial", "GJ")
stranded = MeterSourceBinding(
meter_id=old.id, channel_id=channel.id, started_at=start,
created_at=start, updated_at=start,
)
session.add(stranded)
session.flush()
closed, created = transfer_binding(
session, target_meter_id=target.id, from_binding_id=stranded.id,
to_channel_id=channel.id, effective_at=target_start + timedelta(hours=1),
)
assert closed.ended_at == old_end
assert created.started_at == target_start + timedelta(hours=1)
def test_cross_meter_transfer_rejects_closed_source_and_ambiguous_predecessor(session):
start = datetime(2026, 8, 20, tzinfo=UTC)
boundary = start + timedelta(days=1)
old = _meter(session, "heating", "old")
old.started_at, old.ended_at = start, None
target = Meter(label="target", commodity="heating", started_at=boundary, reason="meter_swap", created_at=boundary)
session.add(target)
session.flush()
_, channel = _source_and_channel(session, "warmtelink_serial", "GJ")
source = create_binding(session, meter_id=old.id, channel_id=channel.id, started_at=start)
old.ended_at = boundary
session.flush()
close_binding(session, source.id, ended_at=boundary - timedelta(hours=1))
with pytest.raises(BindingValidationError, match="open binding"):
transfer_binding(session, target_meter_id=target.id, from_binding_id=source.id,
to_channel_id=channel.id, effective_at=boundary)
assert source.ended_at == boundary - timedelta(hours=1)
source.ended_at = None # synthetic retained bad row, exactly the recovery input.
duplicate = Meter(label="duplicate", commodity="heating", started_at=start + timedelta(hours=1),
ended_at=boundary, reason="other", created_at=start)
session.add(duplicate)
session.flush()
with pytest.raises(BindingValidationError, match="unique immediately preceding"):
transfer_binding(session, target_meter_id=target.id, from_binding_id=source.id,
to_channel_id=channel.id, effective_at=boundary)
assert source.ended_at is None
def test_source_delete_is_restricted_by_discovered_channel(session):
source, _ = _source_and_channel(session, "dsmr_mqtt", "kWh")
with pytest.raises(SourceDeleteRestrictedError):
delete_source(session, source.id)
+386
View File
@@ -0,0 +1,386 @@
"""Schema and model tests for the M8 source/channel/binding identity chain."""
from __future__ import annotations
from datetime import datetime, timedelta, timezone
from pathlib import Path
import pytest
import sqlalchemy.exc
from alembic import command
from alembic.config import Config
from sqlalchemy import UniqueConstraint, create_engine, event as sa_event, inspect, text
from sqlalchemy.orm import Session
from app.db import Base
from app.models.energy import EnergyCostPeriod, Meter
from app.models.meter_source import (
MeterSource,
MeterSourceBinding,
MeterSourceChannel,
half_open_intervals_overlap,
)
def _alembic_config(database_url: str) -> Config:
config = Config("alembic_app.ini")
config.set_main_option("sqlalchemy.url", database_url)
return config
def _engine_with_foreign_keys(database_url: str):
engine = create_engine(database_url, connect_args={"check_same_thread": False})
@sa_event.listens_for(engine, "connect")
def _enable_foreign_keys(dbapi_connection, _connection_record):
cursor = dbapi_connection.cursor()
cursor.execute("PRAGMA foreign_keys=ON")
cursor.close()
return engine
@pytest.fixture()
def source_db(tmp_path: Path):
database_url = f"sqlite:///{tmp_path / 'meter_sources.db'}"
command.upgrade(_alembic_config(database_url), "head")
engine = _engine_with_foreign_keys(database_url)
yield engine
engine.dispose()
def _add_source_chain(session: Session, now: datetime) -> tuple[Meter, MeterSource, MeterSourceChannel]:
meter = Meter(
label="Heating meter",
commodity="heating",
started_at=now,
ended_at=None,
reason="initial",
note=None,
created_at=now,
)
source = MeterSource(
name="WarmteLink",
kind="warmtelink_serial",
created_at=now,
updated_at=now,
)
session.add_all([meter, source])
session.flush()
channel = MeterSourceChannel(
source_id=source.id,
channel_key="heating-total",
label="District heating total",
suggested_commodity="heating",
unit="GJ",
created_at=now,
updated_at=now,
)
session.add(channel)
session.flush()
return meter, source, channel
def test_populated_revision_14_upgrades_to_meter_source_head_with_audit(tmp_path: Path):
database_url = f"sqlite:///{tmp_path / 'revision_14.db'}"
config = _alembic_config(database_url)
command.upgrade(config, "20260625_14_meter_uuid")
historical_at = datetime(2026, 8, 1, tzinfo=timezone.utc)
engine = _engine_with_foreign_keys(database_url)
try:
with engine.begin() as connection:
connection.execute(
text(
"INSERT INTO dsmr_reading (recorded_at, source_id, payload) "
"VALUES (:recorded_at, :source_id, :payload)"
),
{
"recorded_at": historical_at,
"source_id": 17,
"payload": '{"electricity_delivered_1": "100.000"}',
},
)
connection.execute(
text(
"INSERT INTO meter "
"(uuid, label, commodity, started_at, ended_at, reason, note, created_at) "
"VALUES (:uuid, :label, :commodity, :started_at, NULL, :reason, NULL, :created_at)"
),
{
"uuid": "11111111-1111-4111-8111-111111111111",
"label": "Historical meter",
"commodity": "electricity",
"started_at": historical_at,
"reason": "initial",
"created_at": historical_at,
},
)
meter_id = connection.execute(text("SELECT id FROM meter")).scalar_one()
connection.execute(
text(
"INSERT INTO energy_cost_period "
"(period_start, d1_kwh, d2_kwh, r1_kwh, r2_kwh, import_cost, "
"export_revenue, net_cost, currency, pricing, contract_version_id, meter_id, "
"degraded, computed_at) "
"VALUES (:period_start, 1, 2, 0, 0, 0.5, 0, 0.5, 'EUR', '{}', NULL, "
":meter_id, 0, :computed_at)"
),
{
"period_start": historical_at,
"meter_id": meter_id,
"computed_at": historical_at,
},
)
before_counts = {
table_name: engine.connect().execute(text(f"SELECT COUNT(*) FROM {table_name}")).scalar_one()
for table_name in ("dsmr_reading", "meter", "energy_cost_period")
}
finally:
engine.dispose()
# This T01 fixture intentionally audits the schema-only revision 15.
# Revision 16 has its own DSMR-history adoption fixture.
command.upgrade(config, "20260822_15_meter_sources")
engine = _engine_with_foreign_keys(database_url)
try:
inspector = inspect(engine)
assert {"meter_source", "meter_source_channel", "meter_source_binding"} <= set(
inspector.get_table_names()
)
cost_columns = {column["name"] for column in inspector.get_columns("energy_cost_period")}
assert "source_binding_id" in cost_columns
after_counts = {
table_name: engine.connect().execute(text(f"SELECT COUNT(*) FROM {table_name}")).scalar_one()
for table_name in ("dsmr_reading", "meter", "energy_cost_period")
}
assert after_counts == before_counts
assert engine.connect().execute(text("SELECT version_num FROM alembic_version")).scalar_one() == (
"20260822_15_meter_sources"
)
assert engine.connect().execute(text("PRAGMA foreign_key_check")).all() == []
channel_constraints = inspector.get_unique_constraints("meter_source_channel")
assert {tuple(item["column_names"]) for item in channel_constraints} >= {
("uuid",),
("source_id", "channel_key"),
}
for table_name in ("meter_source", "meter_source_channel", "meter_source_binding"):
assert ("uuid",) in {
tuple(item["column_names"])
for item in inspector.get_unique_constraints(table_name)
}
assert {
"ix_meter_source_kind_enabled",
"ix_meter_source_channel_source_id",
"ix_meter_source_binding_meter_id",
"ix_meter_source_binding_channel_id",
} <= {
index["name"]
for table_name in ("meter_source", "meter_source_channel", "meter_source_binding")
for index in inspector.get_indexes(table_name)
}
cost_fks = {
foreign_key["constrained_columns"][0]: foreign_key
for foreign_key in inspector.get_foreign_keys("energy_cost_period")
}
assert cost_fks["meter_id"]["referred_table"] == "meter"
assert cost_fks["contract_version_id"]["referred_table"] == "energy_contract_version"
assert cost_fks["source_binding_id"]["referred_table"] == "meter_source_binding"
command.upgrade(config, "20260822_15_meter_sources")
assert {
table_name: engine.connect().execute(text(f"SELECT COUNT(*) FROM {table_name}")).scalar_one()
for table_name in before_counts
} == before_counts
finally:
engine.dispose()
def test_empty_database_upgrade_and_repeat_are_idempotent(tmp_path: Path):
database_url = f"sqlite:///{tmp_path / 'empty_then_repeat.db'}"
config = _alembic_config(database_url)
command.upgrade(config, "head")
command.upgrade(config, "head")
engine = _engine_with_foreign_keys(database_url)
try:
inspector = inspect(engine)
source_columns = {column["name"]: column for column in inspector.get_columns("meter_source")}
channel_unique = {
tuple(constraint["column_names"])
for constraint in inspector.get_unique_constraints("meter_source_channel")
}
binding_fks = {
tuple(foreign_key["constrained_columns"]): foreign_key
for foreign_key in inspector.get_foreign_keys("meter_source_binding")
}
cost_columns = {
column["name"]: column for column in inspector.get_columns("energy_cost_period")
}
cost_fks = {
tuple(foreign_key["constrained_columns"]): foreign_key
for foreign_key in inspector.get_foreign_keys("energy_cost_period")
}
assert source_columns["uuid"]["nullable"] is False
assert source_columns["config"]["nullable"] is False
assert ("source_id", "channel_key") in channel_unique
assert binding_fks[("meter_id",)]["options"]["ondelete"] == "RESTRICT"
assert binding_fks[("channel_id",)]["options"]["ondelete"] == "RESTRICT"
assert cost_columns["source_binding_id"]["nullable"] is True
assert cost_fks[("source_binding_id",)]["options"]["ondelete"] == "RESTRICT"
finally:
engine.dispose()
def test_meter_source_migration_downgrade_is_schema_reversible(tmp_path: Path):
database_url = f"sqlite:///{tmp_path / 'meter_source_downgrade.db'}"
config = _alembic_config(database_url)
command.upgrade(config, "head")
command.downgrade(config, "20260625_14_meter_uuid")
engine = _engine_with_foreign_keys(database_url)
try:
inspector = inspect(engine)
tables = set(inspector.get_table_names())
assert not {"meter_source", "meter_source_channel", "meter_source_binding"} & tables
assert "source_binding_id" not in {
column["name"] for column in inspector.get_columns("energy_cost_period")
}
finally:
engine.dispose()
def test_source_model_defaults_and_channel_unique_constraint(source_db):
now = datetime.now(timezone.utc)
with Session(source_db) as session:
_, source, channel = _add_source_chain(session, now)
session.commit()
assert source.uuid
assert source.enabled is True
assert source.config == {}
assert source.status == "unknown"
assert channel.uuid
with pytest.raises(sqlalchemy.exc.IntegrityError):
with Session(source_db) as session:
duplicate = MeterSourceChannel(
source_id=source.id,
channel_key="heating-total",
label="Duplicate",
unit="GJ",
created_at=now,
updated_at=now,
)
session.add(duplicate)
session.commit()
def test_foreign_keys_restrict_history_and_no_relationship_delete_cascade(source_db):
now = datetime.now(timezone.utc)
with Session(source_db) as session:
meter, source, channel = _add_source_chain(session, now)
binding = MeterSourceBinding(
meter_id=meter.id,
channel_id=channel.id,
started_at=now,
ended_at=None,
created_at=now,
updated_at=now,
)
session.add(binding)
session.flush()
period = EnergyCostPeriod(
period_start=now,
d1_kwh=0.0,
d2_kwh=0.0,
r1_kwh=0.0,
r2_kwh=0.0,
import_cost=0.0,
export_revenue=0.0,
net_cost=0.0,
currency="EUR",
pricing={},
contract_version_id=None,
meter_id=meter.id,
source_binding_id=binding.id,
degraded=False,
computed_at=now,
)
session.add(period)
session.commit()
source_id, channel_id, binding_id = source.id, channel.id, binding.id
for table_name, row_id in (
("meter_source", source_id),
("meter_source_channel", channel_id),
("meter_source_binding", binding_id),
):
with pytest.raises(sqlalchemy.exc.IntegrityError):
with Session(source_db) as session:
session.execute(text(f"DELETE FROM {table_name} WHERE id = :row_id"), {"row_id": row_id})
session.commit()
with pytest.raises(sqlalchemy.exc.IntegrityError):
with Session(source_db) as session:
binding = session.get(MeterSourceBinding, binding_id)
assert binding is not None
assert len(binding.cost_periods) == 1
session.delete(binding)
session.commit()
with Session(source_db) as session:
binding = session.get(MeterSourceBinding, binding_id)
period = session.execute(
text("SELECT source_binding_id FROM energy_cost_period WHERE id = :period_id"),
{"period_id": 1},
).scalar_one()
assert binding is not None
assert period == binding_id
assert "delete" not in MeterSource.channels.property.cascade
assert "delete" not in MeterSourceChannel.bindings.property.cascade
assert "delete" not in MeterSourceBinding.cost_periods.property.cascade
assert "delete" not in Meter.source_bindings.property.cascade
def test_channel_unique_constraint_metadata_matches_migrated_schema(source_db):
metadata_table = Base.metadata.tables["meter_source_channel"]
metadata_unique_constraints = {
tuple(column.name for column in constraint.columns)
for constraint in metadata_table.constraints
if isinstance(constraint, UniqueConstraint)
}
inspector = inspect(source_db)
schema_unique_constraints = {
tuple(constraint["column_names"])
for constraint in inspector.get_unique_constraints("meter_source_channel")
}
schema_unique_indexes = {
tuple(index["column_names"])
for index in inspector.get_indexes("meter_source_channel")
if index["unique"]
}
assert ("source_id", "channel_key") in metadata_unique_constraints
assert ("source_id", "channel_key") in schema_unique_constraints
assert ("source_id", "channel_key") not in schema_unique_indexes
def test_half_open_binding_interval_helper():
start = datetime(2026, 1, 1, tzinfo=timezone.utc)
boundary = start + timedelta(hours=1)
later = boundary + timedelta(hours=1)
assert half_open_intervals_overlap(start, boundary, boundary, later) is False
assert half_open_intervals_overlap(start, later, boundary, None) is True
assert half_open_intervals_overlap(start, None, boundary, later) is True
def test_model_tables_and_foreign_keys_are_registered():
assert {"meter_source", "meter_source_channel", "meter_source_binding"} <= set(Base.metadata.tables)
binding_fks = Base.metadata.tables["meter_source_binding"].foreign_keys
assert {foreign_key.ondelete for foreign_key in binding_fks} == {"RESTRICT"}
+114
View File
@@ -25,7 +25,10 @@ from sqlalchemy import create_engine, event as sa_event
from sqlalchemy.orm import Session
from app.models.energy import Meter
from app.models.meter_source import MeterSource, MeterSourceBinding, MeterSourceChannel
from app.services.meters import (
_as_utc,
close_meter,
MeterIntervalError,
MeterOverlapError,
declare_meter,
@@ -35,6 +38,62 @@ from app.services.meters import (
)
def test_declare_and_close_reject_future_boundaries(session: Session):
future = datetime.now(UTC) + timedelta(minutes=5)
with pytest.raises(MeterIntervalError, match="future"):
declare_meter(session, label="future", started_at=future, reason="initial")
meter = _make_meter(session, started_at=datetime.now(UTC) - timedelta(hours=1))
with pytest.raises(MeterIntervalError, match="future"):
close_meter(session, meter, ended_at=future)
assert meter.ended_at is None
def test_close_and_declare_reject_boundary_before_retained_closed_binding(session: Session):
"""Lifecycle writes must not silently shorten immutable binding history."""
start = datetime.now(UTC) - timedelta(hours=3)
proposed_end = start + timedelta(hours=1)
meter = _make_meter(session, started_at=start)
binding = _make_binding(session, meter, started_at=start, ended_at=start + timedelta(hours=2))
with pytest.raises(MeterIntervalError, match="Closed binding extends"):
close_meter(session, meter, ended_at=proposed_end)
assert meter.ended_at is None
assert binding.ended_at == start + timedelta(hours=2)
with pytest.raises(MeterIntervalError, match="Closed binding extends"):
declare_meter(session, label="replacement", started_at=proposed_end, reason="meter_swap")
assert meter.ended_at is None
assert session.query(Meter).count() == 1
def test_update_rejects_future_started_at_before_mutating_other_fields(session: Session):
"""A future correction must not leak label/note changes into the Session."""
meter = _make_meter(
session,
started_at=datetime.now(UTC) - timedelta(hours=1),
label="Original",
note="Original note",
)
session.commit()
meter_id = meter.id
with pytest.raises(MeterIntervalError, match="future"):
update_meter(
session,
meter,
label="Changed",
note="Changed note",
started_at=datetime.now(UTC) + timedelta(minutes=5),
)
session.rollback()
with Session(session.bind) as observer:
unchanged = observer.get(Meter, meter_id)
assert unchanged is not None
assert unchanged.label == "Original"
assert unchanged.note == "Original note"
# ---------------------------------------------------------------------------
# Fixtures
# ---------------------------------------------------------------------------
@@ -112,6 +171,28 @@ def _make_meter(
return m
def _make_binding(session: Session, meter: Meter, *, started_at: datetime, ended_at: datetime | None = None):
source = MeterSource(
name=f"source-{meter.id}", kind="dsmr_mqtt", enabled=True, config={}, status="online",
created_at=started_at, updated_at=started_at,
)
session.add(source)
session.flush()
channel = MeterSourceChannel(
source_id=source.id, channel_key=f"channel-{meter.id}", label="Total", unit="kWh",
created_at=started_at, updated_at=started_at,
)
session.add(channel)
session.flush()
binding = MeterSourceBinding(
meter_id=meter.id, channel_id=channel.id, started_at=started_at, ended_at=ended_at,
created_at=started_at, updated_at=started_at,
)
session.add(binding)
session.flush()
return binding
# ---------------------------------------------------------------------------
# 1. meter_at — half-open interval semantics
# ---------------------------------------------------------------------------
@@ -637,3 +718,36 @@ class TestUpdateMeter:
from app.services.meters import _as_utc
fetched = session.get(Meter, m.id)
assert _as_utc(fetched.started_at) == _as_utc(earlier)
@pytest.mark.parametrize("shift", ["later", "earlier"])
def test_update_started_at_rejects_boundary_shift_that_strands_binding(self, session: Session, shift: str):
"""A correction must not create an out-of-epoch binding on either adjacent meter."""
boundary = _T0 + timedelta(days=10)
prev = _make_meter(session, started_at=_T0, ended_at=boundary, label="Prev")
current = _make_meter(session, started_at=boundary, ended_at=None, label="Current")
if shift == "later":
binding = _make_binding(session, current, started_at=boundary)
proposed = boundary + timedelta(days=1)
else:
binding = _make_binding(session, prev, started_at=_T0, ended_at=boundary)
proposed = boundary - timedelta(days=1)
session.commit()
prev_id = prev.id
current_id = current.id
binding_id = binding.id
binding_started_at = binding.started_at
binding_ended_at = binding.ended_at
with pytest.raises(MeterIntervalError, match="Binding"):
update_meter(session, current, started_at=proposed)
session.rollback()
with Session(session.bind) as observer:
assert _as_utc(observer.get(Meter, prev_id).ended_at) == boundary
assert _as_utc(observer.get(Meter, current_id).started_at) == boundary
observed_binding = observer.get(MeterSourceBinding, binding_id)
assert observed_binding is not None
assert _as_utc(observed_binding.started_at) == _as_utc(binding_started_at)
assert (
_as_utc(observed_binding.ended_at) if observed_binding.ended_at is not None else None
) == (_as_utc(binding_ended_at) if binding_ended_at is not None else None)
+45 -1
View File
@@ -9,7 +9,7 @@ from unittest.mock import MagicMock, patch
import pytest
from fastapi.testclient import TestClient
from app.integrations.mqtt import MqttManager
from app.integrations.mqtt import MqttManager, mqtt_source_client_id, mqtt_test_client_id
# ---------------------------------------------------------------------------
@@ -24,6 +24,7 @@ def _make_settings(
mqtt_username: str = "",
mqtt_password: str = "",
mqtt_tls_enabled: bool = False,
mqtt_client_id: str = "home-automation",
ha_discovery_prefix: str = "homeassistant",
):
"""Return a simple namespace that acts like a Settings object for MqttManager tests."""
@@ -34,6 +35,7 @@ def _make_settings(
s.mqtt_username = mqtt_username
s.mqtt_password = mqtt_password
s.mqtt_tls_enabled = mqtt_tls_enabled
s.mqtt_client_id = mqtt_client_id
s.ha_discovery_prefix = ha_discovery_prefix
return s
@@ -120,6 +122,34 @@ def test_connect_creates_paho_client_with_version2() -> None:
)
def test_connect_uses_configured_client_id() -> None:
manager = MqttManager()
settings = _make_settings(mqtt_client_id="home-automation-dev")
mock_client = MagicMock()
with patch("app.integrations.mqtt.mqtt.Client", return_value=mock_client) as mock_cls:
manager.connect(settings)
assert mock_cls.call_args.kwargs["client_id"] == "home-automation-dev"
def test_deployment_client_id_variants_are_distinct() -> None:
production_ids = {
"home-automation",
mqtt_source_client_id("home-automation", 7),
mqtt_test_client_id("home-automation"),
}
development_ids = {
"home-automation-dev",
mqtt_source_client_id("home-automation-dev", 7),
mqtt_test_client_id("home-automation-dev"),
}
assert len(production_ids) == 3
assert len(development_ids) == 3
assert production_ids.isdisjoint(development_ids)
def test_connect_sets_credentials_when_username_provided() -> None:
manager = MqttManager()
settings = _make_settings(mqtt_username="user", mqtt_password="s3cr3t")
@@ -431,3 +461,17 @@ def test_run_mqtt_test_raises_connection_error_on_timeout() -> None:
with patch("threading.Event", side_effect=_make_event):
with pytest.raises(_MqttConnectionError, match="timed out"):
_run_mqtt_test(settings)
def test_run_mqtt_test_uses_deployment_scoped_client_id() -> None:
from app.api.routes.api.config import _run_mqtt_test, _MqttConnectionError
settings = _make_settings(mqtt_client_id="home-automation-dev")
mock_client = MagicMock()
mock_client.connect.side_effect = OSError("Connection refused")
with patch("paho.mqtt.client.Client", return_value=mock_client) as mock_cls:
with pytest.raises(_MqttConnectionError):
_run_mqtt_test(settings)
assert mock_cls.call_args.kwargs["client_id"] == "home-automation-dev-test"
+357
View File
@@ -14,6 +14,7 @@ Covers:
from __future__ import annotations
import threading
from unittest.mock import MagicMock, patch
from app.integrations.mqtt import MqttManager
@@ -231,6 +232,362 @@ def test_on_message_does_not_crash_on_handler_exception_multiple_calls() -> None
assert call_count[0] == 2
def test_replace_source_uses_isolated_client_and_source_credentials() -> None:
"""A DSMR source has its own client; replacing it leaves peers untouched."""
manager = MqttManager()
first_client = MagicMock()
second_client = MagicMock()
third_client = MagicMock()
received: list[tuple[str, bytes]] = []
with patch(
"app.integrations.mqtt.mqtt.Client", side_effect=[first_client, second_client, third_client]
):
manager.replace_source(
1,
host="one.test",
port=1884,
username="one-user",
password="one-secret",
tls_enabled=True,
subscriptions={"one/topic": lambda payload: received.append(("one", payload))},
base_client_id="home-automation-dev",
)
manager.replace_source(
2,
host="two.test",
port=2884,
username="two-user",
password="two-secret",
tls_enabled=False,
subscriptions={"two/topic": lambda payload: received.append(("two", payload))},
)
manager.replace_source(
1,
host="changed.test",
port=1885,
username="changed-user",
password="changed-secret",
tls_enabled=False,
subscriptions={"changed/topic": lambda payload: received.append(("changed", payload))},
)
first_client.tls_set.assert_called_once_with()
first_client.username_pw_set.assert_called_once_with(username="one-user", password="one-secret")
first_client.connect.assert_called_once_with(host="one.test", port=1884, keepalive=60)
first_client.disconnect.assert_called_once_with()
second_client.disconnect.assert_not_called()
second_client.connect.assert_called_once_with(host="two.test", port=2884, keepalive=60)
third_client.connect.assert_called_once_with(host="changed.test", port=1885, keepalive=60)
second_client.on_message(second_client, None, _make_mqtt_message("two/topic", b"two"))
third_client.on_message(third_client, None, _make_mqtt_message("changed/topic", b"changed"))
assert received == [("two", b"two"), ("changed", b"changed")]
def test_source_client_id_is_scoped_by_deployment_and_source() -> None:
manager = MqttManager()
client = MagicMock()
with patch("app.integrations.mqtt.mqtt.Client", return_value=client) as mock_cls:
assert manager.replace_source(
42,
host="broker.test",
port=1883,
username="",
password="",
tls_enabled=False,
subscriptions={"topic": lambda _payload: None},
base_client_id="home-automation-dev",
)
assert mock_cls.call_args.kwargs["client_id"] == "home-automation-dev-dsmr-source-42"
def test_replaced_source_client_callbacks_cannot_reach_new_generation() -> None:
"""A retained old paho client cannot subscribe, mutate state, or dispatch new handlers."""
manager = MqttManager()
old_client = MagicMock()
new_client = MagicMock()
received: list[tuple[str, bytes]] = []
with patch("app.integrations.mqtt.mqtt.Client", side_effect=[old_client, new_client]):
assert manager.replace_source(
7,
host="old.test",
port=1883,
username="",
password="",
tls_enabled=False,
subscriptions={"same/topic": lambda payload: received.append(("old", payload))},
)
assert manager.replace_source(
7,
host="new.test",
port=1883,
username="",
password="",
tls_enabled=False,
subscriptions={"same/topic": lambda payload: received.append(("new", payload))},
)
accepted = MagicMock()
accepted.is_failure = False
old_client.on_connect(old_client, None, MagicMock(), accepted, None)
old_client.on_message(old_client, None, _make_mqtt_message("same/topic", b"stale"))
old_client.on_disconnect(old_client, None, MagicMock(), MagicMock(), None)
old_client.subscribe.assert_not_called()
assert received == []
assert 7 not in manager._source_connected
new_client.on_message(new_client, None, _make_mqtt_message("same/topic", b"fresh"))
assert received == [("new", b"fresh")]
def test_removed_then_reenabled_identical_source_rejects_old_callback() -> None:
manager = MqttManager()
old_client = MagicMock()
reenabled_client = MagicMock()
received: list[bytes] = []
kwargs = {
"host": "broker.test",
"port": 1883,
"username": "",
"password": "",
"tls_enabled": False,
"subscriptions": {"same/topic": lambda payload: received.append(payload)},
}
with patch("app.integrations.mqtt.mqtt.Client", side_effect=[old_client, reenabled_client]):
assert manager.replace_source(7, **kwargs)
manager.remove_source(7)
assert manager.replace_source(7, **kwargs)
old_client.on_message(old_client, None, _make_mqtt_message("same/topic", b"stale"))
reenabled_client.on_message(reenabled_client, None, _make_mqtt_message("same/topic", b"fresh"))
assert received == [b"fresh"]
def test_source_connect_failure_is_not_active_and_can_be_retried() -> None:
manager = MqttManager()
failed_client = MagicMock()
failed_client.connect.side_effect = OSError("broker down")
recovered_client = MagicMock()
with patch("app.integrations.mqtt.mqtt.Client", side_effect=[failed_client, recovered_client]):
assert not manager.replace_source(
1, host="broker.test", port=1883, username="", password="", tls_enabled=False,
subscriptions={"topic": lambda _payload: None},
)
assert not manager.source_is_active(1)
assert manager.replace_source(
1, host="broker.test", port=1883, username="", password="", tls_enabled=False,
subscriptions={"topic": lambda _payload: None},
)
assert manager.source_is_active(1)
failed_client.loop_stop.assert_called_once_with()
def test_source_tls_failure_is_not_active() -> None:
manager = MqttManager()
failed_client = MagicMock()
failed_client.tls_set.side_effect = OSError("bad TLS")
with patch("app.integrations.mqtt.mqtt.Client", return_value=failed_client):
assert not manager.replace_source(
1, host="broker.test", port=1883, username="", password="", tls_enabled=True,
subscriptions={"topic": lambda _payload: None},
)
assert not manager.source_is_active(1)
def test_source_sync_connack_before_connect_returns_subscribes_all_topics() -> None:
"""Ownership is installed before a synchronous CONNACK callback can run."""
manager = MqttManager()
class SyncConnackClient:
def __init__(self) -> None:
self.subscribed: list[str] = []
def loop_start(self) -> None:
pass
def connect(self, **_kwargs: object) -> None:
accepted = MagicMock()
accepted.is_failure = False
self.on_connect(self, None, MagicMock(), accepted, None)
def subscribe(self, topic: str) -> None:
self.subscribed.append(topic)
def disconnect(self) -> None:
pass
def loop_stop(self) -> None:
pass
client = SyncConnackClient()
with patch("app.integrations.mqtt.mqtt.Client", return_value=client):
assert manager.replace_source(
9,
host="broker.test",
port=1883,
username="",
password="",
tls_enabled=False,
subscriptions={"telegram/topic": lambda _payload: None, "tariff/topic": lambda _payload: None},
)
assert manager.source_is_active(9)
assert client.subscribed == ["telegram/topic", "tariff/topic"]
def test_source_health_tracks_connack_disconnect_and_ignores_stale_callbacks() -> None:
"""A source is connecting until CONNACK, and old generations cannot rewrite health."""
manager = MqttManager()
old_client = MagicMock()
new_client = MagicMock()
old_states: list[str] = []
new_states: list[str] = []
kwargs = {
"host": "broker.test",
"port": 1883,
"username": "",
"password": "",
"tls_enabled": False,
"subscriptions": {"topic": lambda _payload: None},
}
with patch("app.integrations.mqtt.mqtt.Client", side_effect=[old_client, new_client]):
assert manager.replace_source(1, **kwargs, state_handler=old_states.append)
assert old_states == ["connecting"]
assert manager.replace_source(1, **kwargs, state_handler=new_states.append)
accepted = MagicMock()
accepted.is_failure = False
old_client.on_connect(old_client, None, MagicMock(), accepted, None)
old_client.on_disconnect(old_client, None, MagicMock(), MagicMock(), None)
assert old_states == ["connecting"]
new_client.on_connect(new_client, None, MagicMock(), accepted, None)
new_client.on_disconnect(new_client, None, MagicMock(), MagicMock(), None)
assert new_states == ["connecting", "online", "error"]
def test_source_health_rejected_connack_reports_error_without_disconnect() -> None:
"""A failed CONNACK is distinct from a later disconnect callback."""
manager = MqttManager()
client = MagicMock()
states: list[str] = []
with patch("app.integrations.mqtt.mqtt.Client", return_value=client):
assert manager.replace_source(
1, host="broker.test", port=1883, username="", password="", tls_enabled=False,
subscriptions={"topic": lambda _payload: None}, state_handler=states.append,
)
accepted = MagicMock()
accepted.is_failure = False
refused = MagicMock()
refused.is_failure = True
client.on_connect(client, None, MagicMock(), accepted, None)
client.on_connect(client, None, MagicMock(), refused, None)
assert states == ["connecting", "online", "error"]
assert 1 not in manager._source_connected
def test_source_health_disconnect_then_reconnect_connack_is_ordered_and_isolated() -> None:
"""One source's reconnect sequence cannot change another source's health."""
manager = MqttManager()
first_client = MagicMock()
second_client = MagicMock()
first_states: list[str] = []
second_states: list[str] = []
kwargs = {
"host": "broker.test",
"port": 1883,
"username": "",
"password": "",
"tls_enabled": False,
"subscriptions": {"topic": lambda _payload: None},
}
with patch("app.integrations.mqtt.mqtt.Client", side_effect=[first_client, second_client]):
assert manager.replace_source(1, **kwargs, state_handler=first_states.append)
assert manager.replace_source(2, **kwargs, state_handler=second_states.append)
accepted = MagicMock()
accepted.is_failure = False
first_client.on_connect(first_client, None, MagicMock(), accepted, None)
first_client.on_disconnect(first_client, None, MagicMock(), MagicMock(), None)
first_client.on_connect(first_client, None, MagicMock(), accepted, None)
assert first_states == ["connecting", "online", "error", "online"]
assert second_states == ["connecting"]
def test_source_teardown_with_joining_loop_stop_waits_for_callback_before_aba() -> None:
"""loop_stop may join a callback that needs the manager lock to finish."""
manager = MqttManager()
class JoiningClient:
def loop_start(self) -> None:
pass
def connect(self, **_kwargs: object) -> None:
pass
def disconnect(self) -> None:
pass
def loop_stop(self) -> None:
self.callback_thread.join()
old_client = JoiningClient()
new_client = MagicMock()
started = threading.Event()
release = threading.Event()
removed = threading.Event()
received: list[bytes] = []
def old_handler(payload: bytes) -> None:
started.set()
release.wait(timeout=2)
received.append(payload)
kwargs = {
"host": "broker.test",
"port": 1883,
"username": "",
"password": "",
"tls_enabled": False,
"subscriptions": {"same/topic": old_handler},
}
with patch("app.integrations.mqtt.mqtt.Client", side_effect=[old_client, new_client]):
assert manager.replace_source(7, **kwargs)
callback_thread = threading.Thread(
target=old_client.on_message,
args=(old_client, None, _make_mqtt_message("same/topic", b"old")),
daemon=True,
)
old_client.callback_thread = callback_thread
callback_thread.start()
assert started.wait(timeout=1)
def remove_source() -> None:
manager.remove_source(7)
removed.set()
teardown_thread = threading.Thread(target=remove_source, daemon=True)
teardown_thread.start()
assert not removed.wait(timeout=0.05)
release.set()
assert removed.wait(timeout=1)
callback_thread.join(timeout=1)
teardown_thread.join(timeout=1)
assert not callback_thread.is_alive()
assert not teardown_thread.is_alive()
assert removed.is_set()
with patch("app.integrations.mqtt.mqtt.Client", return_value=new_client):
assert manager.replace_source(7, **kwargs)
old_client.on_message(old_client, None, _make_mqtt_message("same/topic", b"stale"))
assert received == [b"old"]
# ---------------------------------------------------------------------------
# on_connect re-subscribes registered topics
# ---------------------------------------------------------------------------

Some files were not shown because too many files have changed in this diff Show More