tliu93
0958d9a2e9
The Summary cards labelled `metered_import` / `metered_export` as "(kWh)", but both fields are monetary totals (Σ import_cost / Σ export_revenue). Today's page therefore showed "Import 1.339 kWh" when the meter had actually imported 4.188 kWh — the 1.339 was EUR. Cross-checked against the DSMR cumulative registers and Home Assistant: our energy figures were correct all along, only the label was wrong. summarize() now also aggregates the metered energy, reusing the already-fetched non-degraded rows so no extra query is issued: metered_import_kwh = Σ (d1_kwh + d2_kwh) metered_export_kwh = Σ (r1_kwh + r2_kwh) The Import/Export cards show kWh as the headline figure and keep the monetary equivalent as a sub-line, so the split between energy cost and standing charges/credits behind total_payable stays visible. The `_kwh` suffix is now the only thing separating energy from money in this payload, so the docstrings on both summarize() and SummaryResponse call that out explicitly. app/integrations/expose.py reads only the money keys, so the HA outbound sensors are unaffected by the additive fields.
914 lines
40 KiB
Python
914 lines
40 KiB
Python
"""Billing engine for DSMR 15-minute energy metering periods.
|
||
|
||
This module implements the two-layer billing model described in §3.4 of the
|
||
M6 design document, extended in M7-T03 to be meter-aware:
|
||
|
||
**Layer 1 — per-period metering cost (immutable, price-snapshot)**
|
||
``compute_period(session, t0)`` computes the import cost, export revenue, and
|
||
net cost for the 15-minute period ``[t0, t0+15min)``. The result is written
|
||
to ``energy_cost_period`` with a full pricing snapshot so each row is
|
||
self-contained and auditable. Existing *successful* rows are never overwritten
|
||
by the normal tick path; only an explicit ``recompute_range`` call passes
|
||
``overwrite=True``.
|
||
|
||
**Layer 2 — summary (computed at read time, not stored)**
|
||
``summarize(session, start, end)`` aggregates all non-degraded
|
||
``energy_cost_period`` rows in ``[start, end)``, then adds the daily
|
||
standing charges (network_fee + management_fee, apportioned at EUR/month
|
||
÷ 30 per day) and subtracts the energy-tax credit (heffingskorting,
|
||
apportioned at EUR/year ÷ 365 per day).
|
||
|
||
The summary reports **both** money and energy: ``metered_import`` /
|
||
``metered_export`` are monetary totals (Σ import_cost / Σ export_revenue),
|
||
while ``metered_import_kwh`` / ``metered_export_kwh`` are the corresponding
|
||
metered energy totals in kWh. The ``_kwh`` suffix is the only thing that
|
||
distinguishes them — always check it before labelling a value in a UI.
|
||
|
||
Design notes
|
||
------------
|
||
- **Decimal arithmetic throughout**: all monetary computations use
|
||
``decimal.Decimal`` to avoid float binary rounding errors. Only when
|
||
writing to ``EnergyCostPeriod`` columns (Float) are values converted to
|
||
float. ``summarize`` converts back to Decimal for summation.
|
||
- **UTC quarter-hour grid**: period boundaries are aligned to UTC 00/15/30/45
|
||
minutes (``floor_to_quarter``). NL local time (CET/CEST) is always a whole
|
||
number of hours from UTC, so the quarter-hour grid is the same in both
|
||
timezone representations.
|
||
- **Register keys**: DSMR payload uses JSON strings like ``"20915.154"``
|
||
for cumulative kWh registers. ``register_at`` converts them to Decimal.
|
||
- **Degraded vs skip semantics**:
|
||
- *No meter coverage* (``meter_at`` returns None for t0): write a
|
||
``degraded=True`` row with ``meter_id=None``.
|
||
- *Cross-meter boundary* (m0.id != m1.id for t0/t1): write a ``degraded=True``
|
||
row with ``meter_id=m0.id``; losing this one period at the swap boundary is
|
||
acceptable (D5 decision).
|
||
- *Missing readings* (``register_at`` returns None for start or end
|
||
boundary within the meter window): write a ``degraded=True`` row with
|
||
``meter_id=m0.id`` so the period is tracked and can be retried by
|
||
``compute_closed_periods``.
|
||
- *Negative or excessively large delta* (delta sanity guard D6): write a
|
||
``degraded=True`` row with ``meter_id=m0.id``; prevents negative costs and
|
||
grossly inflated costs from meter resets, DSMR rollover, or data spikes.
|
||
- *Missing Tibber price* (``TibberPriceNotFoundError``): skip entirely (do
|
||
not write a row); the period will be retried once prices arrive.
|
||
- *Missing active contract version*: skip (no contract to compute against).
|
||
- **Meter-aware register lookup**: ``register_at`` now accepts a ``meter``
|
||
parameter and restricts the DSMR reading query to readings within
|
||
``[meter.started_at, meter.ended_at)`` (half-open), preventing old-meter
|
||
readings from leaking into a new-meter epoch.
|
||
- **Lookback window in ``compute_closed_periods``**: to avoid scanning all
|
||
historical DSMR data on every tick, the function looks back at most 7 days
|
||
from the current time. This covers typical short outages (no data / no
|
||
contract) while staying bounded. Periods older than 7 days must be
|
||
recovered via an explicit ``recompute_range`` call.
|
||
|
||
Meter-aware compute_period ordering rationale (M7-T03)
|
||
-------------------------------------------------------
|
||
The order of checks inside ``compute_period`` is:
|
||
|
||
1. **Immutability guard** (existing non-degraded row, overwrite=False) → return False.
|
||
2. **Meter determination** (m0 = meter_at(t0), m1 = meter_at(t1)):
|
||
- No meter (m0 is None) → write degraded, meter_id=None.
|
||
- Cross-meter boundary (m0.id != m1.id) → write degraded, meter_id=m0.id.
|
||
3. **Active contract version check** → skip (no write) if absent.
|
||
4. **Boundary register readings** within m0's window → write degraded if missing.
|
||
5. **Delta sanity guard** → write degraded if any delta < 0 or > _MAX_DELTA_KWH.
|
||
6. **Price strategy** → skip (no write) if Tibber price missing.
|
||
7. **Upsert billing record** with meter_id=m0.id.
|
||
|
||
Why meter before contract? The meter is a *structural* prerequisite: without a
|
||
known meter epoch we cannot trust the delta at all, so we commit a degraded row
|
||
immediately. The contract skip, by contrast, is transient (the period can be
|
||
re-computed once a contract is configured), so it produces no row.
|
||
"""
|
||
|
||
from __future__ import annotations
|
||
|
||
import logging
|
||
from datetime import UTC, datetime, timedelta
|
||
from decimal import Decimal
|
||
from typing import Any
|
||
|
||
from sqlalchemy import select
|
||
from sqlalchemy.orm import Session
|
||
|
||
from app.integrations.pricing.strategies import (
|
||
PeriodDeltas,
|
||
TibberPriceNotFoundError,
|
||
get_strategy,
|
||
)
|
||
from app.models.energy import DsmrReading, EnergyCostPeriod, Meter
|
||
from app.services.contracts import active_contract_version_at, active_contract_versions
|
||
from app.services.meters import meter_at
|
||
from app.services.timezone import local_date, local_now
|
||
|
||
logger = logging.getLogger(__name__)
|
||
|
||
# ---------------------------------------------------------------------------
|
||
# Constants
|
||
# ---------------------------------------------------------------------------
|
||
|
||
_PERIOD_MINUTES = 15
|
||
_LOOKBACK_DAYS = 7 # maximum lookback window for compute_closed_periods
|
||
|
||
# Maximum age a DSMR reading may have relative to the boundary being queried.
|
||
# Under normal operation DSMR readings arrive every ~10 seconds, so a reading
|
||
# more than one period (15 minutes) old at the boundary indicates either a data
|
||
# gap or — critically — a *future* boundary being resolved against the last
|
||
# historical reading. In both cases the reading is considered stale and
|
||
# ``register_at`` returns None, letting the period be marked degraded instead of
|
||
# producing a spurious zero-delta "successful" row.
|
||
_READING_MAX_STALENESS = timedelta(minutes=_PERIOD_MINUTES)
|
||
|
||
# Maximum plausible kWh delta for a single 15-minute period (D6 sanity guard).
|
||
# A typical Dutch household uses well under 5 kWh per quarter hour even under
|
||
# heavy load. 100 kWh per 15 minutes corresponds to ~400 kW — far beyond any
|
||
# residential consumption — but is lenient enough to never fire on legitimate
|
||
# data. Any delta at or above this threshold indicates a meter reset, DSMR
|
||
# rollover, sign error, or other data anomaly, and the period is marked
|
||
# degraded to prevent negative costs or grossly inflated charges.
|
||
_MAX_DELTA_KWH = Decimal("100")
|
||
|
||
# 每日固定费/税补在"本地午夜后多久"才结算入账。延后到 01:05 是为了让累计成本的
|
||
# 整天阶跃落在新一天、且避开 01:00 整点（HA 长期统计的小时桶边界）。
|
||
_SETTLEMENT_OFFSET = timedelta(hours=1, minutes=5)
|
||
|
||
# DSMR payload register keys (cumulative kWh, JSON string values).
|
||
_KEY_D1 = "electricity_delivered_1" # delivered low-tariff (dal / _1)
|
||
_KEY_D2 = "electricity_delivered_2" # delivered high-tariff (normal / _2)
|
||
_KEY_R1 = "electricity_returned_1" # returned low-tariff
|
||
_KEY_R2 = "electricity_returned_2" # returned high-tariff
|
||
|
||
|
||
# ---------------------------------------------------------------------------
|
||
# Internal helpers
|
||
# ---------------------------------------------------------------------------
|
||
|
||
|
||
def floor_to_quarter(dt: datetime) -> datetime:
|
||
"""Return *dt* floored to the nearest UTC quarter-hour boundary.
|
||
|
||
The result always has seconds=0 and microseconds=0, and minutes in
|
||
{0, 15, 30, 45}. Timezone info is preserved if present.
|
||
"""
|
||
floored_minute = (dt.minute // _PERIOD_MINUTES) * _PERIOD_MINUTES
|
||
return dt.replace(minute=floored_minute, second=0, microsecond=0)
|
||
|
||
|
||
def _to_decimal(value: Any) -> Decimal:
|
||
"""Convert *value* to Decimal via str() to avoid float binary rounding."""
|
||
return Decimal(str(value))
|
||
|
||
|
||
def _as_utc(dt: datetime) -> datetime:
|
||
"""Attach UTC tzinfo to a naive datetime (SQLite read-back workaround)."""
|
||
if dt.tzinfo is None:
|
||
return dt.replace(tzinfo=UTC)
|
||
return dt
|
||
|
||
|
||
def _existing_period(session: Session, t0: datetime) -> EnergyCostPeriod | None:
|
||
"""Return the EnergyCostPeriod row for period_start=t0, or None."""
|
||
return session.execute(
|
||
select(EnergyCostPeriod).where(EnergyCostPeriod.period_start == t0)
|
||
).scalar_one_or_none()
|
||
|
||
|
||
# ---------------------------------------------------------------------------
|
||
# register_at — boundary reading lookup (meter-aware)
|
||
# ---------------------------------------------------------------------------
|
||
|
||
|
||
def register_at(
|
||
session: Session,
|
||
boundary: datetime,
|
||
meter: Meter,
|
||
) -> dict[str, Decimal] | None:
|
||
"""Return the four cumulative kWh register values at *boundary*, within *meter*'s window.
|
||
|
||
Queries the most recent ``DsmrReading`` with:
|
||
``recorded_at ≤ boundary``
|
||
AND ``recorded_at ≥ meter.started_at``
|
||
AND (``meter.ended_at IS NULL`` OR ``recorded_at < meter.ended_at``)
|
||
|
||
The meter window constraint (half-open ``[started_at, ended_at)``) ensures
|
||
that readings from a previous meter epoch are never used to anchor a new
|
||
meter's computation. Without this guard, the final reading of the old meter
|
||
would be visible at the start of the new meter's epoch and produce a
|
||
cross-meter delta, defeating the isolation guarantee.
|
||
|
||
Extracts the four energy registers from ``payload``:
|
||
|
||
d1 — electricity_delivered_1 (delivered low-tariff / dal)
|
||
d2 — electricity_delivered_2 (delivered high-tariff / normal)
|
||
r1 — electricity_returned_1 (returned low-tariff)
|
||
r2 — electricity_returned_2 (returned high-tariff)
|
||
|
||
Returns
|
||
-------
|
||
dict[str, Decimal] with keys ``d1``, ``d2``, ``r1``, ``r2``, or ``None``
|
||
when:
|
||
- No ``DsmrReading`` row exists with ``recorded_at ≤ boundary`` within
|
||
*meter*'s epoch window.
|
||
- The most recent such reading is older than ``_READING_MAX_STALENESS``
|
||
relative to *boundary* (freshness guard).
|
||
- Any of the four register keys is absent from the payload.
|
||
- Any of the four register values is ``None`` (null in JSON).
|
||
|
||
SQLite naive datetime note
|
||
--------------------------
|
||
``recorded_at`` is stored as a naive UTC datetime in SQLite. Comparisons
|
||
against *boundary* (always tz-aware UTC) use ``_as_utc()`` for the
|
||
freshness check. The SQL ``WHERE`` clause comparisons work correctly
|
||
because SQLAlchemy's SQLite dialect strips tzinfo when binding parameters
|
||
(leaving the wall-clock UTC value unchanged), consistent with the storage
|
||
format.
|
||
"""
|
||
# Build the meter-window constraints: [started_at, ended_at).
|
||
meter_lower = meter.started_at # DsmrReading.recorded_at >= meter.started_at
|
||
meter_upper = meter.ended_at # DsmrReading.recorded_at < meter.ended_at (if set)
|
||
|
||
stmt = (
|
||
select(DsmrReading)
|
||
.where(
|
||
DsmrReading.recorded_at <= boundary,
|
||
DsmrReading.recorded_at >= meter_lower,
|
||
)
|
||
.order_by(DsmrReading.recorded_at.desc())
|
||
.limit(1)
|
||
)
|
||
# Apply the upper bound only when the meter is closed (ended_at is not None).
|
||
if meter_upper is not None:
|
||
stmt = stmt.where(DsmrReading.recorded_at < meter_upper)
|
||
|
||
row: DsmrReading | None = session.execute(stmt).scalar_one_or_none()
|
||
|
||
if row is None:
|
||
return None
|
||
|
||
# Freshness guard: reject readings that are too old relative to *boundary*.
|
||
# ``recorded_at`` is stored as a naive UTC datetime in SQLite; attach UTC
|
||
# tzinfo before comparing with *boundary* (which is always tz-aware UTC) to
|
||
# avoid an "offset-naive vs offset-aware" TypeError.
|
||
if _as_utc(row.recorded_at) < _as_utc(boundary) - _READING_MAX_STALENESS:
|
||
return None
|
||
|
||
payload = row.payload or {}
|
||
try:
|
||
d1_raw = payload[_KEY_D1]
|
||
d2_raw = payload[_KEY_D2]
|
||
r1_raw = payload[_KEY_R1]
|
||
r2_raw = payload[_KEY_R2]
|
||
except KeyError:
|
||
return None
|
||
|
||
if any(v is None for v in (d1_raw, d2_raw, r1_raw, r2_raw)):
|
||
return None
|
||
|
||
return {
|
||
"d1": _to_decimal(d1_raw),
|
||
"d2": _to_decimal(d2_raw),
|
||
"r1": _to_decimal(r1_raw),
|
||
"r2": _to_decimal(r2_raw),
|
||
}
|
||
|
||
|
||
# ---------------------------------------------------------------------------
|
||
# compute_period — single 15-minute period
|
||
# ---------------------------------------------------------------------------
|
||
|
||
|
||
def compute_period(session: Session, t0: datetime, *, overwrite: bool = False) -> bool:
|
||
"""Compute and upsert the billing record for the period ``[t0, t0+15min)``.
|
||
|
||
Parameters
|
||
----------
|
||
session:
|
||
Active SQLAlchemy session. Caller is responsible for committing.
|
||
t0:
|
||
UTC start of the 15-minute period. **Must** lie on a quarter-hour
|
||
grid boundary (minutes ∈ {0, 15, 30, 45}, seconds=0, microseconds=0).
|
||
overwrite:
|
||
If ``True``, overwrite an existing *successful* row (i.e. re-compute
|
||
even when a non-degraded record already exists). The normal tick path
|
||
always passes ``False``; only ``recompute_range`` passes ``True``.
|
||
|
||
Returns
|
||
-------
|
||
bool
|
||
``True`` if a record was written (inserted or updated), ``False`` if
|
||
the period was skipped (missing contract or missing Tibber price).
|
||
|
||
Side-effects
|
||
------------
|
||
- Inserts or updates an ``EnergyCostPeriod`` row keyed on ``period_start=t0``.
|
||
- If no meter covers t0 (``meter_at`` returns None for t0): inserts/updates
|
||
a degraded row with ``meter_id=None``.
|
||
- If the period spans a meter boundary (``meter_at(t0).id != meter_at(t1).id``):
|
||
inserts/updates a degraded row with ``meter_id=m0.id`` (D5 decision).
|
||
- If readings are missing at either boundary within the meter window:
|
||
inserts/updates a degraded row with ``meter_id=m0.id``.
|
||
- If any delta is negative or exceeds ``_MAX_DELTA_KWH`` (D6 sanity guard):
|
||
inserts/updates a degraded row with ``meter_id=m0.id``.
|
||
- If the active contract version is missing: **skips** (returns False, no write).
|
||
- If the Tibber price is missing (TibberPriceNotFoundError): **skips**
|
||
(returns False, no write).
|
||
"""
|
||
t1 = t0 + timedelta(minutes=_PERIOD_MINUTES)
|
||
now = datetime.now(UTC)
|
||
|
||
# Immutability guard: skip if a successful record already exists and we
|
||
# are not in overwrite mode.
|
||
existing = _existing_period(session, t0)
|
||
if existing is not None and not existing.degraded and not overwrite:
|
||
return False
|
||
|
||
# --- Meter determination (structural prerequisite, checked before contract) ---
|
||
#
|
||
# A missing or cross-boundary meter is a structural problem: we cannot trust
|
||
# the delta at all, so we write a degraded row immediately. This is different
|
||
# from the contract skip (transient, no write): the degraded row ensures the
|
||
# period appears in the history and can be revisited once the meter timeline
|
||
# is corrected and a recompute_range is triggered.
|
||
#
|
||
# Ordering rationale:
|
||
# 1. No meter (m0 is None) → degraded(meter_id=None): no epoch for t0.
|
||
# 2. Cross-meter boundary (m0.id != m1.id) → degraded(meter_id=m0.id): D5.
|
||
# 3. (Single meter, proceed) → contract check → readings → delta guard → price.
|
||
#
|
||
# We place meter before contract so that "cross-table period" is always
|
||
# marked degraded regardless of contract state. If we checked contract
|
||
# first, a missing-contract skip would silently discard the cross-table
|
||
# evidence; once a contract is added and recompute runs, the engine would
|
||
# incorrectly use cross-table reads.
|
||
m0 = meter_at(session, t0)
|
||
m1 = meter_at(session, t1)
|
||
|
||
if m0 is None:
|
||
# No meter epoch covers t0 — degraded with no meter attribution.
|
||
logger.debug(
|
||
"compute_period(%s): no active meter at t0 — writing degraded (meter_id=None).",
|
||
t0.isoformat(),
|
||
)
|
||
_upsert_degraded(session, t0, now, existing, meter_id=None)
|
||
return True
|
||
|
||
if m1 is None or m0.id != m1.id:
|
||
# Period spans a meter boundary or t1 has no meter. Degrade with m0's id
|
||
# (t0's meter attribution): the period's start belongs to m0's epoch.
|
||
logger.debug(
|
||
"compute_period(%s): period crosses meter boundary "
|
||
"(m0.id=%s, m1.id=%s) — writing degraded.",
|
||
t0.isoformat(),
|
||
m0.id,
|
||
m1.id if m1 is not None else None,
|
||
)
|
||
_upsert_degraded(session, t0, now, existing, meter_id=m0.id)
|
||
return True
|
||
|
||
# --- Active contract version at t0 ---
|
||
# If there is no active contract covering t0, skip the period entirely.
|
||
# We do not write a degraded row — there is no meaningful state to recover
|
||
# without a contract (we would not know which strategy to apply once data
|
||
# arrives). The period can be recovered via an explicit recompute_range once
|
||
# a contract is configured and activated.
|
||
version = active_contract_version_at(session, t0)
|
||
if version is None:
|
||
logger.debug("compute_period(%s): no active contract version — skipping.", t0.isoformat())
|
||
return False
|
||
|
||
# --- Boundary readings within m0's meter window ---
|
||
start_regs = register_at(session, t0, m0)
|
||
end_regs = register_at(session, t1, m0)
|
||
|
||
if start_regs is None or end_regs is None:
|
||
# Missing readings within the meter window → degraded with m0 attribution.
|
||
_upsert_degraded(session, t0, now, existing, meter_id=m0.id)
|
||
return True # a record was written (degraded)
|
||
|
||
# --- Compute deltas (end − start) ---
|
||
deltas = PeriodDeltas(
|
||
d1=end_regs["d1"] - start_regs["d1"],
|
||
d2=end_regs["d2"] - start_regs["d2"],
|
||
r1=end_regs["r1"] - start_regs["r1"],
|
||
r2=end_regs["r2"] - start_regs["r2"],
|
||
)
|
||
|
||
# --- Delta sanity guard (D6) ---
|
||
# Any negative delta indicates a meter reset, DSMR rollover, or data error.
|
||
# Any delta exceeding _MAX_DELTA_KWH (100 kWh per 15 min = 400 kW average)
|
||
# is implausible for residential use and indicates an anomaly.
|
||
# Both cases produce a degraded row so no negative or grossly inflated cost
|
||
# is ever written to the billing record.
|
||
all_deltas = (deltas.d1, deltas.d2, deltas.r1, deltas.r2)
|
||
if any(d < Decimal("0") for d in all_deltas) or any(d > _MAX_DELTA_KWH for d in all_deltas):
|
||
logger.debug(
|
||
"compute_period(%s): delta sanity guard triggered "
|
||
"(d1=%s, d2=%s, r1=%s, r2=%s) — writing degraded.",
|
||
t0.isoformat(),
|
||
deltas.d1,
|
||
deltas.d2,
|
||
deltas.r1,
|
||
deltas.r2,
|
||
)
|
||
_upsert_degraded(session, t0, now, existing, meter_id=m0.id)
|
||
return True
|
||
|
||
# --- Price strategy ---
|
||
strategy = get_strategy(version.contract.kind)
|
||
try:
|
||
result = strategy(deltas, t0, version.values, session)
|
||
except TibberPriceNotFoundError:
|
||
# Missing Tibber price → skip the period; it will be retried once the
|
||
# price arrives (e.g. after the next Tibber refresh job runs).
|
||
logger.debug(
|
||
"compute_period(%s): no Tibber price found — skipping.", t0.isoformat()
|
||
)
|
||
return False
|
||
|
||
# --- Upsert the billing record ---
|
||
import_cost: Decimal = result["import_cost"]
|
||
export_revenue: Decimal = result["export_revenue"]
|
||
net_cost: Decimal = result["net_cost"]
|
||
pricing: dict = result["pricing"]
|
||
|
||
if existing is not None:
|
||
# Update in-place (overwrite=True or previous record was degraded).
|
||
existing.d1_kwh = float(deltas.d1)
|
||
existing.d2_kwh = float(deltas.d2)
|
||
existing.r1_kwh = float(deltas.r1)
|
||
existing.r2_kwh = float(deltas.r2)
|
||
existing.import_cost = float(import_cost)
|
||
existing.export_revenue = float(export_revenue)
|
||
existing.net_cost = float(net_cost)
|
||
existing.currency = version.contract.currency
|
||
existing.pricing = pricing
|
||
existing.contract_version_id = version.id
|
||
existing.meter_id = m0.id
|
||
existing.degraded = False
|
||
existing.computed_at = now
|
||
else:
|
||
period = EnergyCostPeriod(
|
||
period_start=t0,
|
||
d1_kwh=float(deltas.d1),
|
||
d2_kwh=float(deltas.d2),
|
||
r1_kwh=float(deltas.r1),
|
||
r2_kwh=float(deltas.r2),
|
||
import_cost=float(import_cost),
|
||
export_revenue=float(export_revenue),
|
||
net_cost=float(net_cost),
|
||
currency=version.contract.currency,
|
||
pricing=pricing,
|
||
contract_version_id=version.id,
|
||
meter_id=m0.id,
|
||
degraded=False,
|
||
computed_at=now,
|
||
)
|
||
session.add(period)
|
||
|
||
return True
|
||
|
||
|
||
def _upsert_degraded(
|
||
session: Session,
|
||
t0: datetime,
|
||
now: datetime,
|
||
existing: EnergyCostPeriod | None,
|
||
*,
|
||
meter_id: int | None,
|
||
) -> None:
|
||
"""Insert or update a degraded placeholder for period *t0*.
|
||
|
||
Parameters
|
||
----------
|
||
session:
|
||
Active SQLAlchemy session.
|
||
t0:
|
||
UTC start of the 15-minute period.
|
||
now:
|
||
Current UTC timestamp for the ``computed_at`` field.
|
||
existing:
|
||
The existing ``EnergyCostPeriod`` row for this period, or ``None``.
|
||
meter_id:
|
||
The meter ID to attribute this degraded period to, or ``None`` when
|
||
no meter epoch covers the period (no-meter degraded case).
|
||
|
||
When *existing* is not None (row was previously written — either degraded
|
||
or successful), the row is explicitly reset to the standard degraded state.
|
||
This is required for the ``recompute_range`` (overwrite=True) path: if the
|
||
row was previously a *successful* computation and the boundary readings have
|
||
since disappeared, the stale non-zero costs must be cleared so the row
|
||
accurately reflects the current "missing readings" state rather than
|
||
masquerading as a valid result.
|
||
|
||
The ``meter_id`` is always updated to reflect the current meter attribution
|
||
judgment (the result of ``meter_at`` at the time of recompute). This
|
||
ensures that a retroactive ``started_at`` change + ``recompute_range`` will
|
||
re-attribute historical degraded periods to the correct meter epoch.
|
||
"""
|
||
if existing is not None:
|
||
# Explicitly reset to degraded state — identical field values to the
|
||
# new-row path below. This covers the recompute-over-successful-row
|
||
# case where old non-zero costs must not survive the downgrade.
|
||
existing.d1_kwh = 0.0
|
||
existing.d2_kwh = 0.0
|
||
existing.r1_kwh = 0.0
|
||
existing.r2_kwh = 0.0
|
||
existing.import_cost = 0.0
|
||
existing.export_revenue = 0.0
|
||
existing.net_cost = 0.0
|
||
existing.pricing = {}
|
||
existing.contract_version_id = None
|
||
existing.meter_id = meter_id
|
||
existing.degraded = True
|
||
existing.computed_at = now
|
||
else:
|
||
period = EnergyCostPeriod(
|
||
period_start=t0,
|
||
d1_kwh=0.0,
|
||
d2_kwh=0.0,
|
||
r1_kwh=0.0,
|
||
r2_kwh=0.0,
|
||
import_cost=0.0,
|
||
export_revenue=0.0,
|
||
net_cost=0.0,
|
||
currency="EUR", # placeholder; real currency known after contract lookup
|
||
pricing={},
|
||
contract_version_id=None,
|
||
meter_id=meter_id,
|
||
degraded=True,
|
||
computed_at=now,
|
||
)
|
||
session.add(period)
|
||
|
||
|
||
# ---------------------------------------------------------------------------
|
||
# compute_closed_periods — periodic tick
|
||
# ---------------------------------------------------------------------------
|
||
|
||
|
||
def compute_closed_periods(session: Session) -> int:
|
||
"""Find and compute all uncalculated closed 15-minute periods.
|
||
|
||
A period ``[t0, t1)`` is *closed* when ``t1 ≤ now``. This function:
|
||
|
||
1. Determines the lookback window: from ``now − LOOKBACK_DAYS`` to ``now``,
|
||
floored to the nearest quarter-hour. This avoids an unbounded full
|
||
historical scan on every tick while still covering the typical recovery
|
||
window (short outages, missing contract, etc.). Periods older than
|
||
``LOOKBACK_DAYS`` must be recovered via an explicit ``recompute_range``.
|
||
2. Iterates over all quarter-hour boundaries in that window where
|
||
``t1 ≤ now`` (i.e. the period has already closed).
|
||
3. For each boundary, calls ``compute_period(overwrite=False)``, which:
|
||
- Skips periods that already have a *successful* (non-degraded) record.
|
||
- Retries periods that have a *degraded* record.
|
||
- Writes degraded rows for periods with no meter or cross-meter boundaries.
|
||
- Skips periods for which no active contract version exists or the
|
||
Tibber price is unavailable (without writing a degraded row).
|
||
|
||
Returns
|
||
-------
|
||
int
|
||
Number of periods for which a record was written (inserted or updated).
|
||
Does not count skipped periods.
|
||
"""
|
||
now = datetime.now(UTC)
|
||
# Current period boundary (the one whose t1 has not yet passed).
|
||
current_t0 = floor_to_quarter(now)
|
||
# Earliest boundary to consider.
|
||
earliest_t0 = floor_to_quarter(now - timedelta(days=_LOOKBACK_DAYS))
|
||
|
||
written = 0
|
||
t0 = earliest_t0
|
||
while t0 < current_t0:
|
||
t1 = t0 + timedelta(minutes=_PERIOD_MINUTES)
|
||
if t1 <= now:
|
||
try:
|
||
did_write = compute_period(session, t0, overwrite=False)
|
||
if did_write:
|
||
written += 1
|
||
except Exception:
|
||
logger.exception(
|
||
"compute_closed_periods: unexpected error for t0=%s — continuing.",
|
||
t0.isoformat(),
|
||
)
|
||
t0 += timedelta(minutes=_PERIOD_MINUTES)
|
||
|
||
if written:
|
||
session.commit()
|
||
logger.info("compute_closed_periods: wrote %d period(s).", written)
|
||
return written
|
||
|
||
|
||
# ---------------------------------------------------------------------------
|
||
# recompute_range — explicit full recompute
|
||
# ---------------------------------------------------------------------------
|
||
|
||
|
||
def recompute_range(session: Session, start: datetime, end: datetime) -> int:
|
||
"""Recompute (overwrite) all 15-minute periods in ``[start, end)``.
|
||
|
||
This is the *explicit opt-in* path for recovering from:
|
||
- Periods where readings or prices arrived late.
|
||
- Price corrections (new contract version retroactively applied).
|
||
- Retroactive meter changes (``update_meter`` with new ``started_at``) —
|
||
re-running this function will re-judge meter attribution and re-compute
|
||
costs using the corrected epoch boundaries.
|
||
- Any other reason to override the immutability guard.
|
||
|
||
The function iterates over every UTC quarter-hour boundary in
|
||
``[floor(start), end)`` and calls ``compute_period(overwrite=True)``.
|
||
Existing rows (including successful ones) are overwritten; their
|
||
``meter_id`` fields will reflect the *current* ``meter_at`` judgment for
|
||
each period's start timestamp, naturally re-attributing periods when meter
|
||
``started_at`` values have been retroactively corrected.
|
||
|
||
Parameters
|
||
----------
|
||
session:
|
||
Active SQLAlchemy session. The function commits after all periods
|
||
have been processed.
|
||
start:
|
||
Inclusive start datetime (floored to the nearest quarter-hour internally).
|
||
end:
|
||
Exclusive end datetime.
|
||
|
||
Returns
|
||
-------
|
||
int
|
||
Number of periods for which a record was written (inserted or updated).
|
||
Periods skipped due to missing contract or missing Tibber price are
|
||
*not* counted.
|
||
"""
|
||
t0 = floor_to_quarter(_as_utc(start))
|
||
end_utc = _as_utc(end)
|
||
now = datetime.now(UTC)
|
||
|
||
written = 0
|
||
while t0 < end_utc:
|
||
t1 = t0 + timedelta(minutes=_PERIOD_MINUTES)
|
||
# Only recompute periods that have already closed (t1 ≤ now). Even
|
||
# when the caller passes a future *end*, we must not write speculative
|
||
# "future" rows: register_at would resolve to the latest historical
|
||
# reading for both boundaries, producing a zero-delta fake-success row
|
||
# that blocks the real computation once the period actually closes.
|
||
if t1 <= now:
|
||
try:
|
||
did_write = compute_period(session, t0, overwrite=True)
|
||
if did_write:
|
||
written += 1
|
||
except Exception:
|
||
logger.exception(
|
||
"recompute_range: unexpected error for t0=%s — continuing.",
|
||
t0.isoformat(),
|
||
)
|
||
t0 += timedelta(minutes=_PERIOD_MINUTES)
|
||
|
||
session.commit()
|
||
logger.info(
|
||
"recompute_range(%s, %s): wrote %d period(s).",
|
||
start.isoformat(),
|
||
end.isoformat(),
|
||
written,
|
||
)
|
||
return written
|
||
|
||
|
||
# ---------------------------------------------------------------------------
|
||
# summarize — layer-2 aggregation (read-time, not stored)
|
||
# ---------------------------------------------------------------------------
|
||
|
||
|
||
def summarize(session: Session, start: datetime, end: datetime) -> dict[str, Any]:
|
||
"""Aggregate billing for the interval ``[start, end)``.
|
||
|
||
Computes the total payable as:
|
||
|
||
total_payable = Σ(net_cost) -- metered electricity
|
||
+ fixed_costs -- per-day standing charges, cross-version
|
||
- credits -- per-day heffingskorting, cross-version
|
||
|
||
**Fixed-cost / credit counting — Principle C (symmetric begin/end)**:
|
||
Both the local calendar day in which *start* falls and the local calendar
|
||
day in which *end* falls are counted as full days. Fixed charges
|
||
(network_fee, management_fee) and the energy-tax credit (heffingskorting)
|
||
are assessed on a "service-is-active" basis — if the meter was online on a
|
||
given calendar day, the full day's charge/credit applies, regardless of
|
||
whether the window starts at midnight or mid-morning.
|
||
|
||
For each local calendar date D in the range
|
||
``[local_date(start), min(local_date(end), today_local)]``:
|
||
- D ≤ today_local (only elapsed / today days count as "whole days").
|
||
- The contract version whose effective_from local-date ≤ D is used.
|
||
- This is cross-version: if V1 is from June 1 and V2 from June 25,
|
||
querying June 1–30 uses V1 for days 1-24 and V2 for day 25.
|
||
|
||
The end-day (last_counted) is included when the end's local midnight falls
|
||
strictly before end_utc; combined with the always-counted start day this
|
||
makes the begin/end handling symmetric. A short same-day window therefore
|
||
counts its single local day. A window contributes 0 days only when the
|
||
counted range is empty (first_counted > last_counted) — e.g. a window lying
|
||
entirely in the future, since last_counted is capped at today_local.
|
||
|
||
Daily getters (``*_today``) use windows exactly aligned to local midnight,
|
||
so their ``first_counted`` is always today — unaffected by this fix.
|
||
|
||
Days that have not yet started in local time (D > today_local) are
|
||
never counted. Switching versions never resets the counter.
|
||
|
||
**Timezone note**: the ``days`` field in the returned dict still represents
|
||
the window length in calendar days (total_seconds / 86400), for backward
|
||
compatibility with existing API consumers. The fixed/credit calculation
|
||
independently counts whole local days as described above.
|
||
|
||
All arithmetic uses Decimal; the returned dict contains Python floats for
|
||
JSON-serialisation convenience.
|
||
|
||
Parameters
|
||
----------
|
||
session:
|
||
Active read-only SQLAlchemy session.
|
||
start:
|
||
Inclusive start of the summary interval (UTC or naive-UTC).
|
||
end:
|
||
Exclusive end of the summary interval (UTC or naive-UTC).
|
||
|
||
Returns
|
||
-------
|
||
dict with keys:
|
||
|
||
currency str ISO 4217 currency (from contract, or "EUR" fallback)
|
||
metered_import float Σ import_cost from non-degraded periods (money)
|
||
metered_export float Σ export_revenue from non-degraded periods (money)
|
||
metered_net float Σ net_cost from non-degraded periods (money)
|
||
metered_import_kwh float Σ (d1_kwh + d2_kwh) from non-degraded periods (energy)
|
||
metered_export_kwh float Σ (r1_kwh + r2_kwh) from non-degraded periods (energy)
|
||
fixed_costs float standing charges for elapsed whole local days
|
||
credits float energy-tax credit for elapsed whole local days
|
||
total_payable float metered_net + fixed_costs − credits
|
||
period_count int number of non-degraded periods in range
|
||
degraded_count int number of degraded periods in range
|
||
days float interval length in days (total_seconds / 86400)
|
||
"""
|
||
from datetime import timedelta as _td, date as _date
|
||
|
||
start_utc = _as_utc(start)
|
||
end_utc = _as_utc(end)
|
||
|
||
# --- Fetch all EnergyCostPeriod rows in [start, end) ---
|
||
rows = session.execute(
|
||
select(EnergyCostPeriod).where(
|
||
EnergyCostPeriod.period_start >= start_utc,
|
||
EnergyCostPeriod.period_start < end_utc,
|
||
)
|
||
).scalars().all()
|
||
|
||
good_rows = [r for r in rows if not r.degraded]
|
||
degraded_rows = [r for r in rows if r.degraded]
|
||
|
||
# Σ monetary amounts (Decimal arithmetic).
|
||
sum_import = sum((_to_decimal(r.import_cost) for r in good_rows), Decimal("0"))
|
||
sum_export = sum((_to_decimal(r.export_revenue) for r in good_rows), Decimal("0"))
|
||
sum_net = sum((_to_decimal(r.net_cost) for r in good_rows), Decimal("0"))
|
||
|
||
# Σ metered energy (kWh), summed across both tariff registers. Reuses the
|
||
# already-fetched ``good_rows`` so no extra query is issued.
|
||
sum_import_kwh = sum(
|
||
(_to_decimal(r.d1_kwh) + _to_decimal(r.d2_kwh) for r in good_rows), Decimal("0")
|
||
)
|
||
sum_export_kwh = sum(
|
||
(_to_decimal(r.r1_kwh) + _to_decimal(r.r2_kwh) for r in good_rows), Decimal("0")
|
||
)
|
||
|
||
# --- Interval length in days (window, not elapsed — kept for API compat) ---
|
||
total_seconds = (end_utc - start_utc).total_seconds()
|
||
days = _to_decimal(str(total_seconds)) / _to_decimal("86400")
|
||
|
||
# --- Fixed costs and credits: Principle C cross-version whole-day counting ---
|
||
_now_local = local_now()
|
||
today_local: _date = _now_local.date()
|
||
|
||
# --- Compute [first_counted, last_counted] local date range (inclusive) ---
|
||
#
|
||
# Both the window-start day and the window-end day are counted as complete
|
||
# local calendar days, regardless of whether the window starts/ends at midnight.
|
||
#
|
||
# Principle C (symmetric begin/end):
|
||
# • first_counted = local calendar date of start_utc (start day always counted)
|
||
# • last_counted = local calendar date of end_utc (end day counted if its
|
||
# local midnight is strictly before end_utc)
|
||
# • Both are then capped at today_local (only elapsed / today days count).
|
||
#
|
||
# Why symmetric? Fixed charges (network_fee, management_fee) and energy-tax credits
|
||
# (heffingskorting) are assessed on a "service-is-active" basis, not on how many
|
||
# hours the service was actually running within that calendar day. If the meter
|
||
# anchor (started_at) falls at 09:18 on June 24, the full June 24 standing charge
|
||
# and credit still apply because the service was online for that day.
|
||
#
|
||
# Previous asymmetric behaviour: a start_utc that was *later* than the local
|
||
# midnight of local_start_date caused first_counted to be bumped to the *next*
|
||
# day, silently dropping the anchor day's charges/credits. This was incorrect
|
||
# for cumulative entities (import_cost_total / export_revenue_total) whose anchor
|
||
# is often an above-midnight started_at. The end-side had always been symmetric
|
||
# (counted if local midnight < end_utc), creating an inconsistency.
|
||
#
|
||
# Daily getters (window = [local today 00:00, local tomorrow 00:00)) are
|
||
# unaffected: local_date(local_midnight_utc(today)) == today, so first_counted
|
||
# is still today regardless of the fix.
|
||
#
|
||
# A short same-day window now counts its single local day (the start day is
|
||
# always counted, symmetric with the end side). A window contributes 0 days
|
||
# only when the counted range is empty (first_counted > last_counted) — e.g. a
|
||
# window lying entirely in the future, where last_counted is capped at
|
||
# today_local while first_counted is later.
|
||
from app.services.timezone import local_midnight_utc as _lmu
|
||
|
||
# Start side: always count the local calendar day in which start_utc falls.
|
||
first_counted: _date = local_date(start_utc)
|
||
|
||
local_end_date: _date = local_date(end_utc)
|
||
# Is the midnight of local_end_date < end_utc? If yes, that day's midnight is in range.
|
||
if _lmu(local_end_date) < end_utc:
|
||
last_counted: _date = local_end_date
|
||
else:
|
||
last_counted = local_end_date - _td(days=1)
|
||
|
||
# Settlement cap: "today" is only counted once the local clock has passed the
|
||
# settlement offset since midnight (01:05). This defers the daily standing-charge
|
||
# step from 00:00 to 01:05, ensuring the cumulative-cost adiabatic jump lands
|
||
# inside the new calendar day and avoids the HA 01:00 hourly-bucket boundary.
|
||
_today_midnight_utc = _lmu(today_local)
|
||
if _now_local >= _today_midnight_utc + _SETTLEMENT_OFFSET:
|
||
settled_cap = today_local
|
||
else:
|
||
settled_cap = today_local - _td(days=1)
|
||
last_counted = min(last_counted, settled_cap)
|
||
|
||
# If the range is empty (first_counted > last_counted), no days are counted.
|
||
|
||
# Fetch all versions of the active contract once (single DB call).
|
||
versions = active_contract_versions(session)
|
||
|
||
fixed_dec = Decimal("0")
|
||
credits_dec = Decimal("0")
|
||
currency = "EUR"
|
||
|
||
if versions:
|
||
# Currency comes from the contract regardless of day count.
|
||
currency = versions[0].contract.currency
|
||
|
||
if versions and first_counted <= last_counted:
|
||
# For each version segment, find its overlap with [first_counted, last_counted]
|
||
# and accumulate whole days.
|
||
version_segments: list[tuple[_date, _date | None, dict]] = []
|
||
for v in versions:
|
||
v_start_local = local_date(_as_utc(v.effective_from))
|
||
v_end_local = local_date(_as_utc(v.effective_to)) if v.effective_to is not None else None
|
||
version_segments.append((v_start_local, v_end_local, v.values or {}))
|
||
|
||
for v_start, v_end_excl, v_values in version_segments:
|
||
# The version covers [v_start, v_end_excl) in local dates,
|
||
# where v_end_excl=None means open-ended (no upper bound).
|
||
# Intersection with [first_counted, last_counted]:
|
||
seg_start = max(first_counted, v_start)
|
||
if v_end_excl is not None:
|
||
seg_end_excl = min(last_counted + _td(days=1), v_end_excl)
|
||
else:
|
||
seg_end_excl = last_counted + _td(days=1)
|
||
|
||
if seg_start >= seg_end_excl:
|
||
continue # no overlap
|
||
|
||
n_days = (seg_end_excl - seg_start).days
|
||
if n_days <= 0:
|
||
continue
|
||
|
||
standing: dict = v_values.get("standing", {})
|
||
creds: dict = v_values.get("credits", {})
|
||
|
||
network_fee = _to_decimal(standing.get("network_fee", 0))
|
||
management_fee = _to_decimal(standing.get("management_fee", 0))
|
||
heffingskorting = _to_decimal(creds.get("heffingskorting", 0))
|
||
|
||
# Standing charges: EUR/month → EUR/day (÷ 30) × n_days.
|
||
fixed_dec += (network_fee + management_fee) / Decimal("30") * Decimal(str(n_days))
|
||
# Energy-tax credit: EUR/year → EUR/day (÷ 365) × n_days.
|
||
credits_dec += heffingskorting / Decimal("365") * Decimal(str(n_days))
|
||
|
||
total_payable = sum_net + fixed_dec - credits_dec
|
||
|
||
return {
|
||
"currency": currency,
|
||
"metered_import": float(sum_import),
|
||
"metered_export": float(sum_export),
|
||
"metered_net": float(sum_net),
|
||
"metered_import_kwh": float(sum_import_kwh),
|
||
"metered_export_kwh": float(sum_export_kwh),
|
||
"fixed_costs": float(fixed_dec),
|
||
"credits": float(credits_dec),
|
||
"total_payable": float(total_payable),
|
||
"period_count": len(good_rows),
|
||
"degraded_count": len(degraded_rows),
|
||
"days": float(days),
|
||
}
|