895 lines
39 KiB
Python
895 lines
39 KiB
Python
"""Billing engine for DSMR 15-minute energy metering periods.
|
||
|
||
This module implements the two-layer billing model described in §3.4 of the
|
||
M6 design document, extended in M7-T03 to be meter-aware:
|
||
|
||
**Layer 1 — per-period metering cost (immutable, price-snapshot)**
|
||
``compute_period(session, t0)`` computes the import cost, export revenue, and
|
||
net cost for the 15-minute period ``[t0, t0+15min)``. The result is written
|
||
to ``energy_cost_period`` with a full pricing snapshot so each row is
|
||
self-contained and auditable. Existing *successful* rows are never overwritten
|
||
by the normal tick path; only an explicit ``recompute_range`` call passes
|
||
``overwrite=True``.
|
||
|
||
**Layer 2 — summary (computed at read time, not stored)**
|
||
``summarize(session, start, end)`` aggregates all non-degraded
|
||
``energy_cost_period`` rows in ``[start, end)``, then adds the daily
|
||
standing charges (network_fee + management_fee, apportioned at EUR/month
|
||
÷ 30 per day) and subtracts the energy-tax credit (heffingskorting,
|
||
apportioned at EUR/year ÷ 365 per day).
|
||
|
||
Design notes
|
||
------------
|
||
- **Decimal arithmetic throughout**: all monetary computations use
|
||
``decimal.Decimal`` to avoid float binary rounding errors. Only when
|
||
writing to ``EnergyCostPeriod`` columns (Float) are values converted to
|
||
float. ``summarize`` converts back to Decimal for summation.
|
||
- **UTC quarter-hour grid**: period boundaries are aligned to UTC 00/15/30/45
|
||
minutes (``floor_to_quarter``). NL local time (CET/CEST) is always a whole
|
||
number of hours from UTC, so the quarter-hour grid is the same in both
|
||
timezone representations.
|
||
- **Register keys**: DSMR payload uses JSON strings like ``"20915.154"``
|
||
for cumulative kWh registers. ``register_at`` converts them to Decimal.
|
||
- **Degraded vs skip semantics**:
|
||
- *No meter coverage* (``meter_at`` returns None for t0): write a
|
||
``degraded=True`` row with ``meter_id=None``.
|
||
- *Cross-meter boundary* (m0.id != m1.id for t0/t1): write a ``degraded=True``
|
||
row with ``meter_id=m0.id``; losing this one period at the swap boundary is
|
||
acceptable (D5 decision).
|
||
- *Missing readings* (``register_at`` returns None for start or end
|
||
boundary within the meter window): write a ``degraded=True`` row with
|
||
``meter_id=m0.id`` so the period is tracked and can be retried by
|
||
``compute_closed_periods``.
|
||
- *Negative or excessively large delta* (delta sanity guard D6): write a
|
||
``degraded=True`` row with ``meter_id=m0.id``; prevents negative costs and
|
||
grossly inflated costs from meter resets, DSMR rollover, or data spikes.
|
||
- *Missing Tibber price* (``TibberPriceNotFoundError``): skip entirely (do
|
||
not write a row); the period will be retried once prices arrive.
|
||
- *Missing active contract version*: skip (no contract to compute against).
|
||
- **Meter-aware register lookup**: ``register_at`` now accepts a ``meter``
|
||
parameter and restricts the DSMR reading query to readings within
|
||
``[meter.started_at, meter.ended_at)`` (half-open), preventing old-meter
|
||
readings from leaking into a new-meter epoch.
|
||
- **Lookback window in ``compute_closed_periods``**: to avoid scanning all
|
||
historical DSMR data on every tick, the function looks back at most 7 days
|
||
from the current time. This covers typical short outages (no data / no
|
||
contract) while staying bounded. Periods older than 7 days must be
|
||
recovered via an explicit ``recompute_range`` call.
|
||
|
||
Meter-aware compute_period ordering rationale (M7-T03)
|
||
-------------------------------------------------------
|
||
The order of checks inside ``compute_period`` is:
|
||
|
||
1. **Immutability guard** (existing non-degraded row, overwrite=False) → return False.
|
||
2. **Meter determination** (m0 = meter_at(t0), m1 = meter_at(t1)):
|
||
- No meter (m0 is None) → write degraded, meter_id=None.
|
||
- Cross-meter boundary (m0.id != m1.id) → write degraded, meter_id=m0.id.
|
||
3. **Active contract version check** → skip (no write) if absent.
|
||
4. **Boundary register readings** within m0's window → write degraded if missing.
|
||
5. **Delta sanity guard** → write degraded if any delta < 0 or > _MAX_DELTA_KWH.
|
||
6. **Price strategy** → skip (no write) if Tibber price missing.
|
||
7. **Upsert billing record** with meter_id=m0.id.
|
||
|
||
Why meter before contract? The meter is a *structural* prerequisite: without a
|
||
known meter epoch we cannot trust the delta at all, so we commit a degraded row
|
||
immediately. The contract skip, by contrast, is transient (the period can be
|
||
re-computed once a contract is configured), so it produces no row.
|
||
"""
|
||
|
||
from __future__ import annotations
|
||
|
||
import logging
|
||
from datetime import UTC, datetime, timedelta
|
||
from decimal import Decimal
|
||
from typing import Any
|
||
|
||
from sqlalchemy import select
|
||
from sqlalchemy.orm import Session
|
||
|
||
from app.integrations.pricing.strategies import (
|
||
PeriodDeltas,
|
||
TibberPriceNotFoundError,
|
||
get_strategy,
|
||
)
|
||
from app.models.energy import DsmrReading, EnergyCostPeriod, Meter
|
||
from app.services.contracts import active_contract_version_at, active_contract_versions
|
||
from app.services.meters import meter_at
|
||
from app.services.timezone import local_date, local_now
|
||
|
||
logger = logging.getLogger(__name__)
|
||
|
||
# ---------------------------------------------------------------------------
|
||
# Constants
|
||
# ---------------------------------------------------------------------------
|
||
|
||
_PERIOD_MINUTES = 15
|
||
_LOOKBACK_DAYS = 7 # maximum lookback window for compute_closed_periods
|
||
|
||
# Maximum age a DSMR reading may have relative to the boundary being queried.
|
||
# Under normal operation DSMR readings arrive every ~10 seconds, so a reading
|
||
# more than one period (15 minutes) old at the boundary indicates either a data
|
||
# gap or — critically — a *future* boundary being resolved against the last
|
||
# historical reading. In both cases the reading is considered stale and
|
||
# ``register_at`` returns None, letting the period be marked degraded instead of
|
||
# producing a spurious zero-delta "successful" row.
|
||
_READING_MAX_STALENESS = timedelta(minutes=_PERIOD_MINUTES)
|
||
|
||
# Maximum plausible kWh delta for a single 15-minute period (D6 sanity guard).
|
||
# A typical Dutch household uses well under 5 kWh per quarter hour even under
|
||
# heavy load. 100 kWh per 15 minutes corresponds to ~400 kW — far beyond any
|
||
# residential consumption — but is lenient enough to never fire on legitimate
|
||
# data. Any delta at or above this threshold indicates a meter reset, DSMR
|
||
# rollover, sign error, or other data anomaly, and the period is marked
|
||
# degraded to prevent negative costs or grossly inflated charges.
|
||
_MAX_DELTA_KWH = Decimal("100")
|
||
|
||
# 每日固定费/税补在"本地午夜后多久"才结算入账。延后到 01:05 是为了让累计成本的
|
||
# 整天阶跃落在新一天、且避开 01:00 整点（HA 长期统计的小时桶边界）。
|
||
_SETTLEMENT_OFFSET = timedelta(hours=1, minutes=5)
|
||
|
||
# DSMR payload register keys (cumulative kWh, JSON string values).
|
||
_KEY_D1 = "electricity_delivered_1" # delivered low-tariff (dal / _1)
|
||
_KEY_D2 = "electricity_delivered_2" # delivered high-tariff (normal / _2)
|
||
_KEY_R1 = "electricity_returned_1" # returned low-tariff
|
||
_KEY_R2 = "electricity_returned_2" # returned high-tariff
|
||
|
||
|
||
# ---------------------------------------------------------------------------
|
||
# Internal helpers
|
||
# ---------------------------------------------------------------------------
|
||
|
||
|
||
def floor_to_quarter(dt: datetime) -> datetime:
|
||
"""Return *dt* floored to the nearest UTC quarter-hour boundary.
|
||
|
||
The result always has seconds=0 and microseconds=0, and minutes in
|
||
{0, 15, 30, 45}. Timezone info is preserved if present.
|
||
"""
|
||
floored_minute = (dt.minute // _PERIOD_MINUTES) * _PERIOD_MINUTES
|
||
return dt.replace(minute=floored_minute, second=0, microsecond=0)
|
||
|
||
|
||
def _to_decimal(value: Any) -> Decimal:
|
||
"""Convert *value* to Decimal via str() to avoid float binary rounding."""
|
||
return Decimal(str(value))
|
||
|
||
|
||
def _as_utc(dt: datetime) -> datetime:
|
||
"""Attach UTC tzinfo to a naive datetime (SQLite read-back workaround)."""
|
||
if dt.tzinfo is None:
|
||
return dt.replace(tzinfo=UTC)
|
||
return dt
|
||
|
||
|
||
def _existing_period(session: Session, t0: datetime) -> EnergyCostPeriod | None:
|
||
"""Return the EnergyCostPeriod row for period_start=t0, or None."""
|
||
return session.execute(
|
||
select(EnergyCostPeriod).where(EnergyCostPeriod.period_start == t0)
|
||
).scalar_one_or_none()
|
||
|
||
|
||
# ---------------------------------------------------------------------------
|
||
# register_at — boundary reading lookup (meter-aware)
|
||
# ---------------------------------------------------------------------------
|
||
|
||
|
||
def register_at(
|
||
session: Session,
|
||
boundary: datetime,
|
||
meter: Meter,
|
||
) -> dict[str, Decimal] | None:
|
||
"""Return the four cumulative kWh register values at *boundary*, within *meter*'s window.
|
||
|
||
Queries the most recent ``DsmrReading`` with:
|
||
``recorded_at ≤ boundary``
|
||
AND ``recorded_at ≥ meter.started_at``
|
||
AND (``meter.ended_at IS NULL`` OR ``recorded_at < meter.ended_at``)
|
||
|
||
The meter window constraint (half-open ``[started_at, ended_at)``) ensures
|
||
that readings from a previous meter epoch are never used to anchor a new
|
||
meter's computation. Without this guard, the final reading of the old meter
|
||
would be visible at the start of the new meter's epoch and produce a
|
||
cross-meter delta, defeating the isolation guarantee.
|
||
|
||
Extracts the four energy registers from ``payload``:
|
||
|
||
d1 — electricity_delivered_1 (delivered low-tariff / dal)
|
||
d2 — electricity_delivered_2 (delivered high-tariff / normal)
|
||
r1 — electricity_returned_1 (returned low-tariff)
|
||
r2 — electricity_returned_2 (returned high-tariff)
|
||
|
||
Returns
|
||
-------
|
||
dict[str, Decimal] with keys ``d1``, ``d2``, ``r1``, ``r2``, or ``None``
|
||
when:
|
||
- No ``DsmrReading`` row exists with ``recorded_at ≤ boundary`` within
|
||
*meter*'s epoch window.
|
||
- The most recent such reading is older than ``_READING_MAX_STALENESS``
|
||
relative to *boundary* (freshness guard).
|
||
- Any of the four register keys is absent from the payload.
|
||
- Any of the four register values is ``None`` (null in JSON).
|
||
|
||
SQLite naive datetime note
|
||
--------------------------
|
||
``recorded_at`` is stored as a naive UTC datetime in SQLite. Comparisons
|
||
against *boundary* (always tz-aware UTC) use ``_as_utc()`` for the
|
||
freshness check. The SQL ``WHERE`` clause comparisons work correctly
|
||
because SQLAlchemy's SQLite dialect strips tzinfo when binding parameters
|
||
(leaving the wall-clock UTC value unchanged), consistent with the storage
|
||
format.
|
||
"""
|
||
# Build the meter-window constraints: [started_at, ended_at).
|
||
meter_lower = meter.started_at # DsmrReading.recorded_at >= meter.started_at
|
||
meter_upper = meter.ended_at # DsmrReading.recorded_at < meter.ended_at (if set)
|
||
|
||
stmt = (
|
||
select(DsmrReading)
|
||
.where(
|
||
DsmrReading.recorded_at <= boundary,
|
||
DsmrReading.recorded_at >= meter_lower,
|
||
)
|
||
.order_by(DsmrReading.recorded_at.desc())
|
||
.limit(1)
|
||
)
|
||
# Apply the upper bound only when the meter is closed (ended_at is not None).
|
||
if meter_upper is not None:
|
||
stmt = stmt.where(DsmrReading.recorded_at < meter_upper)
|
||
|
||
row: DsmrReading | None = session.execute(stmt).scalar_one_or_none()
|
||
|
||
if row is None:
|
||
return None
|
||
|
||
# Freshness guard: reject readings that are too old relative to *boundary*.
|
||
# ``recorded_at`` is stored as a naive UTC datetime in SQLite; attach UTC
|
||
# tzinfo before comparing with *boundary* (which is always tz-aware UTC) to
|
||
# avoid an "offset-naive vs offset-aware" TypeError.
|
||
if _as_utc(row.recorded_at) < _as_utc(boundary) - _READING_MAX_STALENESS:
|
||
return None
|
||
|
||
payload = row.payload or {}
|
||
try:
|
||
d1_raw = payload[_KEY_D1]
|
||
d2_raw = payload[_KEY_D2]
|
||
r1_raw = payload[_KEY_R1]
|
||
r2_raw = payload[_KEY_R2]
|
||
except KeyError:
|
||
return None
|
||
|
||
if any(v is None for v in (d1_raw, d2_raw, r1_raw, r2_raw)):
|
||
return None
|
||
|
||
return {
|
||
"d1": _to_decimal(d1_raw),
|
||
"d2": _to_decimal(d2_raw),
|
||
"r1": _to_decimal(r1_raw),
|
||
"r2": _to_decimal(r2_raw),
|
||
}
|
||
|
||
|
||
# ---------------------------------------------------------------------------
|
||
# compute_period — single 15-minute period
|
||
# ---------------------------------------------------------------------------
|
||
|
||
|
||
def compute_period(session: Session, t0: datetime, *, overwrite: bool = False) -> bool:
|
||
"""Compute and upsert the billing record for the period ``[t0, t0+15min)``.
|
||
|
||
Parameters
|
||
----------
|
||
session:
|
||
Active SQLAlchemy session. Caller is responsible for committing.
|
||
t0:
|
||
UTC start of the 15-minute period. **Must** lie on a quarter-hour
|
||
grid boundary (minutes ∈ {0, 15, 30, 45}, seconds=0, microseconds=0).
|
||
overwrite:
|
||
If ``True``, overwrite an existing *successful* row (i.e. re-compute
|
||
even when a non-degraded record already exists). The normal tick path
|
||
always passes ``False``; only ``recompute_range`` passes ``True``.
|
||
|
||
Returns
|
||
-------
|
||
bool
|
||
``True`` if a record was written (inserted or updated), ``False`` if
|
||
the period was skipped (missing contract or missing Tibber price).
|
||
|
||
Side-effects
|
||
------------
|
||
- Inserts or updates an ``EnergyCostPeriod`` row keyed on ``period_start=t0``.
|
||
- If no meter covers t0 (``meter_at`` returns None for t0): inserts/updates
|
||
a degraded row with ``meter_id=None``.
|
||
- If the period spans a meter boundary (``meter_at(t0).id != meter_at(t1).id``):
|
||
inserts/updates a degraded row with ``meter_id=m0.id`` (D5 decision).
|
||
- If readings are missing at either boundary within the meter window:
|
||
inserts/updates a degraded row with ``meter_id=m0.id``.
|
||
- If any delta is negative or exceeds ``_MAX_DELTA_KWH`` (D6 sanity guard):
|
||
inserts/updates a degraded row with ``meter_id=m0.id``.
|
||
- If the active contract version is missing: **skips** (returns False, no write).
|
||
- If the Tibber price is missing (TibberPriceNotFoundError): **skips**
|
||
(returns False, no write).
|
||
"""
|
||
t1 = t0 + timedelta(minutes=_PERIOD_MINUTES)
|
||
now = datetime.now(UTC)
|
||
|
||
# Immutability guard: skip if a successful record already exists and we
|
||
# are not in overwrite mode.
|
||
existing = _existing_period(session, t0)
|
||
if existing is not None and not existing.degraded and not overwrite:
|
||
return False
|
||
|
||
# --- Meter determination (structural prerequisite, checked before contract) ---
|
||
#
|
||
# A missing or cross-boundary meter is a structural problem: we cannot trust
|
||
# the delta at all, so we write a degraded row immediately. This is different
|
||
# from the contract skip (transient, no write): the degraded row ensures the
|
||
# period appears in the history and can be revisited once the meter timeline
|
||
# is corrected and a recompute_range is triggered.
|
||
#
|
||
# Ordering rationale:
|
||
# 1. No meter (m0 is None) → degraded(meter_id=None): no epoch for t0.
|
||
# 2. Cross-meter boundary (m0.id != m1.id) → degraded(meter_id=m0.id): D5.
|
||
# 3. (Single meter, proceed) → contract check → readings → delta guard → price.
|
||
#
|
||
# We place meter before contract so that "cross-table period" is always
|
||
# marked degraded regardless of contract state. If we checked contract
|
||
# first, a missing-contract skip would silently discard the cross-table
|
||
# evidence; once a contract is added and recompute runs, the engine would
|
||
# incorrectly use cross-table reads.
|
||
m0 = meter_at(session, t0)
|
||
m1 = meter_at(session, t1)
|
||
|
||
if m0 is None:
|
||
# No meter epoch covers t0 — degraded with no meter attribution.
|
||
logger.debug(
|
||
"compute_period(%s): no active meter at t0 — writing degraded (meter_id=None).",
|
||
t0.isoformat(),
|
||
)
|
||
_upsert_degraded(session, t0, now, existing, meter_id=None)
|
||
return True
|
||
|
||
if m1 is None or m0.id != m1.id:
|
||
# Period spans a meter boundary or t1 has no meter. Degrade with m0's id
|
||
# (t0's meter attribution): the period's start belongs to m0's epoch.
|
||
logger.debug(
|
||
"compute_period(%s): period crosses meter boundary "
|
||
"(m0.id=%s, m1.id=%s) — writing degraded.",
|
||
t0.isoformat(),
|
||
m0.id,
|
||
m1.id if m1 is not None else None,
|
||
)
|
||
_upsert_degraded(session, t0, now, existing, meter_id=m0.id)
|
||
return True
|
||
|
||
# --- Active contract version at t0 ---
|
||
# If there is no active contract covering t0, skip the period entirely.
|
||
# We do not write a degraded row — there is no meaningful state to recover
|
||
# without a contract (we would not know which strategy to apply once data
|
||
# arrives). The period can be recovered via an explicit recompute_range once
|
||
# a contract is configured and activated.
|
||
version = active_contract_version_at(session, t0)
|
||
if version is None:
|
||
logger.debug("compute_period(%s): no active contract version — skipping.", t0.isoformat())
|
||
return False
|
||
|
||
# --- Boundary readings within m0's meter window ---
|
||
start_regs = register_at(session, t0, m0)
|
||
end_regs = register_at(session, t1, m0)
|
||
|
||
if start_regs is None or end_regs is None:
|
||
# Missing readings within the meter window → degraded with m0 attribution.
|
||
_upsert_degraded(session, t0, now, existing, meter_id=m0.id)
|
||
return True # a record was written (degraded)
|
||
|
||
# --- Compute deltas (end − start) ---
|
||
deltas = PeriodDeltas(
|
||
d1=end_regs["d1"] - start_regs["d1"],
|
||
d2=end_regs["d2"] - start_regs["d2"],
|
||
r1=end_regs["r1"] - start_regs["r1"],
|
||
r2=end_regs["r2"] - start_regs["r2"],
|
||
)
|
||
|
||
# --- Delta sanity guard (D6) ---
|
||
# Any negative delta indicates a meter reset, DSMR rollover, or data error.
|
||
# Any delta exceeding _MAX_DELTA_KWH (100 kWh per 15 min = 400 kW average)
|
||
# is implausible for residential use and indicates an anomaly.
|
||
# Both cases produce a degraded row so no negative or grossly inflated cost
|
||
# is ever written to the billing record.
|
||
all_deltas = (deltas.d1, deltas.d2, deltas.r1, deltas.r2)
|
||
if any(d < Decimal("0") for d in all_deltas) or any(d > _MAX_DELTA_KWH for d in all_deltas):
|
||
logger.debug(
|
||
"compute_period(%s): delta sanity guard triggered "
|
||
"(d1=%s, d2=%s, r1=%s, r2=%s) — writing degraded.",
|
||
t0.isoformat(),
|
||
deltas.d1,
|
||
deltas.d2,
|
||
deltas.r1,
|
||
deltas.r2,
|
||
)
|
||
_upsert_degraded(session, t0, now, existing, meter_id=m0.id)
|
||
return True
|
||
|
||
# --- Price strategy ---
|
||
strategy = get_strategy(version.contract.kind)
|
||
try:
|
||
result = strategy(deltas, t0, version.values, session)
|
||
except TibberPriceNotFoundError:
|
||
# Missing Tibber price → skip the period; it will be retried once the
|
||
# price arrives (e.g. after the next Tibber refresh job runs).
|
||
logger.debug(
|
||
"compute_period(%s): no Tibber price found — skipping.", t0.isoformat()
|
||
)
|
||
return False
|
||
|
||
# --- Upsert the billing record ---
|
||
import_cost: Decimal = result["import_cost"]
|
||
export_revenue: Decimal = result["export_revenue"]
|
||
net_cost: Decimal = result["net_cost"]
|
||
pricing: dict = result["pricing"]
|
||
|
||
if existing is not None:
|
||
# Update in-place (overwrite=True or previous record was degraded).
|
||
existing.d1_kwh = float(deltas.d1)
|
||
existing.d2_kwh = float(deltas.d2)
|
||
existing.r1_kwh = float(deltas.r1)
|
||
existing.r2_kwh = float(deltas.r2)
|
||
existing.import_cost = float(import_cost)
|
||
existing.export_revenue = float(export_revenue)
|
||
existing.net_cost = float(net_cost)
|
||
existing.currency = version.contract.currency
|
||
existing.pricing = pricing
|
||
existing.contract_version_id = version.id
|
||
existing.meter_id = m0.id
|
||
existing.degraded = False
|
||
existing.computed_at = now
|
||
else:
|
||
period = EnergyCostPeriod(
|
||
period_start=t0,
|
||
d1_kwh=float(deltas.d1),
|
||
d2_kwh=float(deltas.d2),
|
||
r1_kwh=float(deltas.r1),
|
||
r2_kwh=float(deltas.r2),
|
||
import_cost=float(import_cost),
|
||
export_revenue=float(export_revenue),
|
||
net_cost=float(net_cost),
|
||
currency=version.contract.currency,
|
||
pricing=pricing,
|
||
contract_version_id=version.id,
|
||
meter_id=m0.id,
|
||
degraded=False,
|
||
computed_at=now,
|
||
)
|
||
session.add(period)
|
||
|
||
return True
|
||
|
||
|
||
def _upsert_degraded(
|
||
session: Session,
|
||
t0: datetime,
|
||
now: datetime,
|
||
existing: EnergyCostPeriod | None,
|
||
*,
|
||
meter_id: int | None,
|
||
) -> None:
|
||
"""Insert or update a degraded placeholder for period *t0*.
|
||
|
||
Parameters
|
||
----------
|
||
session:
|
||
Active SQLAlchemy session.
|
||
t0:
|
||
UTC start of the 15-minute period.
|
||
now:
|
||
Current UTC timestamp for the ``computed_at`` field.
|
||
existing:
|
||
The existing ``EnergyCostPeriod`` row for this period, or ``None``.
|
||
meter_id:
|
||
The meter ID to attribute this degraded period to, or ``None`` when
|
||
no meter epoch covers the period (no-meter degraded case).
|
||
|
||
When *existing* is not None (row was previously written — either degraded
|
||
or successful), the row is explicitly reset to the standard degraded state.
|
||
This is required for the ``recompute_range`` (overwrite=True) path: if the
|
||
row was previously a *successful* computation and the boundary readings have
|
||
since disappeared, the stale non-zero costs must be cleared so the row
|
||
accurately reflects the current "missing readings" state rather than
|
||
masquerading as a valid result.
|
||
|
||
The ``meter_id`` is always updated to reflect the current meter attribution
|
||
judgment (the result of ``meter_at`` at the time of recompute). This
|
||
ensures that a retroactive ``started_at`` change + ``recompute_range`` will
|
||
re-attribute historical degraded periods to the correct meter epoch.
|
||
"""
|
||
if existing is not None:
|
||
# Explicitly reset to degraded state — identical field values to the
|
||
# new-row path below. This covers the recompute-over-successful-row
|
||
# case where old non-zero costs must not survive the downgrade.
|
||
existing.d1_kwh = 0.0
|
||
existing.d2_kwh = 0.0
|
||
existing.r1_kwh = 0.0
|
||
existing.r2_kwh = 0.0
|
||
existing.import_cost = 0.0
|
||
existing.export_revenue = 0.0
|
||
existing.net_cost = 0.0
|
||
existing.pricing = {}
|
||
existing.contract_version_id = None
|
||
existing.meter_id = meter_id
|
||
existing.degraded = True
|
||
existing.computed_at = now
|
||
else:
|
||
period = EnergyCostPeriod(
|
||
period_start=t0,
|
||
d1_kwh=0.0,
|
||
d2_kwh=0.0,
|
||
r1_kwh=0.0,
|
||
r2_kwh=0.0,
|
||
import_cost=0.0,
|
||
export_revenue=0.0,
|
||
net_cost=0.0,
|
||
currency="EUR", # placeholder; real currency known after contract lookup
|
||
pricing={},
|
||
contract_version_id=None,
|
||
meter_id=meter_id,
|
||
degraded=True,
|
||
computed_at=now,
|
||
)
|
||
session.add(period)
|
||
|
||
|
||
# ---------------------------------------------------------------------------
|
||
# compute_closed_periods — periodic tick
|
||
# ---------------------------------------------------------------------------
|
||
|
||
|
||
def compute_closed_periods(session: Session) -> int:
|
||
"""Find and compute all uncalculated closed 15-minute periods.
|
||
|
||
A period ``[t0, t1)`` is *closed* when ``t1 ≤ now``. This function:
|
||
|
||
1. Determines the lookback window: from ``now − LOOKBACK_DAYS`` to ``now``,
|
||
floored to the nearest quarter-hour. This avoids an unbounded full
|
||
historical scan on every tick while still covering the typical recovery
|
||
window (short outages, missing contract, etc.). Periods older than
|
||
``LOOKBACK_DAYS`` must be recovered via an explicit ``recompute_range``.
|
||
2. Iterates over all quarter-hour boundaries in that window where
|
||
``t1 ≤ now`` (i.e. the period has already closed).
|
||
3. For each boundary, calls ``compute_period(overwrite=False)``, which:
|
||
- Skips periods that already have a *successful* (non-degraded) record.
|
||
- Retries periods that have a *degraded* record.
|
||
- Writes degraded rows for periods with no meter or cross-meter boundaries.
|
||
- Skips periods for which no active contract version exists or the
|
||
Tibber price is unavailable (without writing a degraded row).
|
||
|
||
Returns
|
||
-------
|
||
int
|
||
Number of periods for which a record was written (inserted or updated).
|
||
Does not count skipped periods.
|
||
"""
|
||
now = datetime.now(UTC)
|
||
# Current period boundary (the one whose t1 has not yet passed).
|
||
current_t0 = floor_to_quarter(now)
|
||
# Earliest boundary to consider.
|
||
earliest_t0 = floor_to_quarter(now - timedelta(days=_LOOKBACK_DAYS))
|
||
|
||
written = 0
|
||
t0 = earliest_t0
|
||
while t0 < current_t0:
|
||
t1 = t0 + timedelta(minutes=_PERIOD_MINUTES)
|
||
if t1 <= now:
|
||
try:
|
||
did_write = compute_period(session, t0, overwrite=False)
|
||
if did_write:
|
||
written += 1
|
||
except Exception:
|
||
logger.exception(
|
||
"compute_closed_periods: unexpected error for t0=%s — continuing.",
|
||
t0.isoformat(),
|
||
)
|
||
t0 += timedelta(minutes=_PERIOD_MINUTES)
|
||
|
||
if written:
|
||
session.commit()
|
||
logger.info("compute_closed_periods: wrote %d period(s).", written)
|
||
return written
|
||
|
||
|
||
# ---------------------------------------------------------------------------
|
||
# recompute_range — explicit full recompute
|
||
# ---------------------------------------------------------------------------
|
||
|
||
|
||
def recompute_range(session: Session, start: datetime, end: datetime) -> int:
|
||
"""Recompute (overwrite) all 15-minute periods in ``[start, end)``.
|
||
|
||
This is the *explicit opt-in* path for recovering from:
|
||
- Periods where readings or prices arrived late.
|
||
- Price corrections (new contract version retroactively applied).
|
||
- Retroactive meter changes (``update_meter`` with new ``started_at``) —
|
||
re-running this function will re-judge meter attribution and re-compute
|
||
costs using the corrected epoch boundaries.
|
||
- Any other reason to override the immutability guard.
|
||
|
||
The function iterates over every UTC quarter-hour boundary in
|
||
``[floor(start), end)`` and calls ``compute_period(overwrite=True)``.
|
||
Existing rows (including successful ones) are overwritten; their
|
||
``meter_id`` fields will reflect the *current* ``meter_at`` judgment for
|
||
each period's start timestamp, naturally re-attributing periods when meter
|
||
``started_at`` values have been retroactively corrected.
|
||
|
||
Parameters
|
||
----------
|
||
session:
|
||
Active SQLAlchemy session. The function commits after all periods
|
||
have been processed.
|
||
start:
|
||
Inclusive start datetime (floored to the nearest quarter-hour internally).
|
||
end:
|
||
Exclusive end datetime.
|
||
|
||
Returns
|
||
-------
|
||
int
|
||
Number of periods for which a record was written (inserted or updated).
|
||
Periods skipped due to missing contract or missing Tibber price are
|
||
*not* counted.
|
||
"""
|
||
t0 = floor_to_quarter(_as_utc(start))
|
||
end_utc = _as_utc(end)
|
||
now = datetime.now(UTC)
|
||
|
||
written = 0
|
||
while t0 < end_utc:
|
||
t1 = t0 + timedelta(minutes=_PERIOD_MINUTES)
|
||
# Only recompute periods that have already closed (t1 ≤ now). Even
|
||
# when the caller passes a future *end*, we must not write speculative
|
||
# "future" rows: register_at would resolve to the latest historical
|
||
# reading for both boundaries, producing a zero-delta fake-success row
|
||
# that blocks the real computation once the period actually closes.
|
||
if t1 <= now:
|
||
try:
|
||
did_write = compute_period(session, t0, overwrite=True)
|
||
if did_write:
|
||
written += 1
|
||
except Exception:
|
||
logger.exception(
|
||
"recompute_range: unexpected error for t0=%s — continuing.",
|
||
t0.isoformat(),
|
||
)
|
||
t0 += timedelta(minutes=_PERIOD_MINUTES)
|
||
|
||
session.commit()
|
||
logger.info(
|
||
"recompute_range(%s, %s): wrote %d period(s).",
|
||
start.isoformat(),
|
||
end.isoformat(),
|
||
written,
|
||
)
|
||
return written
|
||
|
||
|
||
# ---------------------------------------------------------------------------
|
||
# summarize — layer-2 aggregation (read-time, not stored)
|
||
# ---------------------------------------------------------------------------
|
||
|
||
|
||
def summarize(session: Session, start: datetime, end: datetime) -> dict[str, Any]:
|
||
"""Aggregate billing for the interval ``[start, end)``.
|
||
|
||
Computes the total payable as:
|
||
|
||
total_payable = Σ(net_cost) -- metered electricity
|
||
+ fixed_costs -- per-day standing charges, cross-version
|
||
- credits -- per-day heffingskorting, cross-version
|
||
|
||
**Fixed-cost / credit counting — Principle C (symmetric begin/end)**:
|
||
Both the local calendar day in which *start* falls and the local calendar
|
||
day in which *end* falls are counted as full days. Fixed charges
|
||
(network_fee, management_fee) and the energy-tax credit (heffingskorting)
|
||
are assessed on a "service-is-active" basis — if the meter was online on a
|
||
given calendar day, the full day's charge/credit applies, regardless of
|
||
whether the window starts at midnight or mid-morning.
|
||
|
||
For each local calendar date D in the range
|
||
``[local_date(start), min(local_date(end), today_local)]``:
|
||
- D ≤ today_local (only elapsed / today days count as "whole days").
|
||
- The contract version whose effective_from local-date ≤ D is used.
|
||
- This is cross-version: if V1 is from June 1 and V2 from June 25,
|
||
querying June 1–30 uses V1 for days 1-24 and V2 for day 25.
|
||
|
||
The end-day (last_counted) is included when the end's local midnight falls
|
||
strictly before end_utc; combined with the always-counted start day this
|
||
makes the begin/end handling symmetric. A short same-day window therefore
|
||
counts its single local day. A window contributes 0 days only when the
|
||
counted range is empty (first_counted > last_counted) — e.g. a window lying
|
||
entirely in the future, since last_counted is capped at today_local.
|
||
|
||
Daily getters (``*_today``) use windows exactly aligned to local midnight,
|
||
so their ``first_counted`` is always today — unaffected by this fix.
|
||
|
||
Days that have not yet started in local time (D > today_local) are
|
||
never counted. Switching versions never resets the counter.
|
||
|
||
**Timezone note**: the ``days`` field in the returned dict still represents
|
||
the window length in calendar days (total_seconds / 86400), for backward
|
||
compatibility with existing API consumers. The fixed/credit calculation
|
||
independently counts whole local days as described above.
|
||
|
||
All arithmetic uses Decimal; the returned dict contains Python floats for
|
||
JSON-serialisation convenience.
|
||
|
||
Parameters
|
||
----------
|
||
session:
|
||
Active read-only SQLAlchemy session.
|
||
start:
|
||
Inclusive start of the summary interval (UTC or naive-UTC).
|
||
end:
|
||
Exclusive end of the summary interval (UTC or naive-UTC).
|
||
|
||
Returns
|
||
-------
|
||
dict with keys:
|
||
|
||
currency str ISO 4217 currency (from contract, or "EUR" fallback)
|
||
metered_import float Σ import_cost from non-degraded periods
|
||
metered_export float Σ export_revenue from non-degraded periods
|
||
metered_net float Σ net_cost from non-degraded periods
|
||
fixed_costs float standing charges for elapsed whole local days
|
||
credits float energy-tax credit for elapsed whole local days
|
||
total_payable float metered_net + fixed_costs − credits
|
||
period_count int number of non-degraded periods in range
|
||
degraded_count int number of degraded periods in range
|
||
days float interval length in days (total_seconds / 86400)
|
||
"""
|
||
from datetime import timedelta as _td, date as _date
|
||
|
||
start_utc = _as_utc(start)
|
||
end_utc = _as_utc(end)
|
||
|
||
# --- Fetch all EnergyCostPeriod rows in [start, end) ---
|
||
rows = session.execute(
|
||
select(EnergyCostPeriod).where(
|
||
EnergyCostPeriod.period_start >= start_utc,
|
||
EnergyCostPeriod.period_start < end_utc,
|
||
)
|
||
).scalars().all()
|
||
|
||
good_rows = [r for r in rows if not r.degraded]
|
||
degraded_rows = [r for r in rows if r.degraded]
|
||
|
||
# Σ monetary amounts (Decimal arithmetic).
|
||
sum_import = sum((_to_decimal(r.import_cost) for r in good_rows), Decimal("0"))
|
||
sum_export = sum((_to_decimal(r.export_revenue) for r in good_rows), Decimal("0"))
|
||
sum_net = sum((_to_decimal(r.net_cost) for r in good_rows), Decimal("0"))
|
||
|
||
# --- Interval length in days (window, not elapsed — kept for API compat) ---
|
||
total_seconds = (end_utc - start_utc).total_seconds()
|
||
days = _to_decimal(str(total_seconds)) / _to_decimal("86400")
|
||
|
||
# --- Fixed costs and credits: Principle C cross-version whole-day counting ---
|
||
_now_local = local_now()
|
||
today_local: _date = _now_local.date()
|
||
|
||
# --- Compute [first_counted, last_counted] local date range (inclusive) ---
|
||
#
|
||
# Both the window-start day and the window-end day are counted as complete
|
||
# local calendar days, regardless of whether the window starts/ends at midnight.
|
||
#
|
||
# Principle C (symmetric begin/end):
|
||
# • first_counted = local calendar date of start_utc (start day always counted)
|
||
# • last_counted = local calendar date of end_utc (end day counted if its
|
||
# local midnight is strictly before end_utc)
|
||
# • Both are then capped at today_local (only elapsed / today days count).
|
||
#
|
||
# Why symmetric? Fixed charges (network_fee, management_fee) and energy-tax credits
|
||
# (heffingskorting) are assessed on a "service-is-active" basis, not on how many
|
||
# hours the service was actually running within that calendar day. If the meter
|
||
# anchor (started_at) falls at 09:18 on June 24, the full June 24 standing charge
|
||
# and credit still apply because the service was online for that day.
|
||
#
|
||
# Previous asymmetric behaviour: a start_utc that was *later* than the local
|
||
# midnight of local_start_date caused first_counted to be bumped to the *next*
|
||
# day, silently dropping the anchor day's charges/credits. This was incorrect
|
||
# for cumulative entities (import_cost_total / export_revenue_total) whose anchor
|
||
# is often an above-midnight started_at. The end-side had always been symmetric
|
||
# (counted if local midnight < end_utc), creating an inconsistency.
|
||
#
|
||
# Daily getters (window = [local today 00:00, local tomorrow 00:00)) are
|
||
# unaffected: local_date(local_midnight_utc(today)) == today, so first_counted
|
||
# is still today regardless of the fix.
|
||
#
|
||
# A short same-day window now counts its single local day (the start day is
|
||
# always counted, symmetric with the end side). A window contributes 0 days
|
||
# only when the counted range is empty (first_counted > last_counted) — e.g. a
|
||
# window lying entirely in the future, where last_counted is capped at
|
||
# today_local while first_counted is later.
|
||
from app.services.timezone import local_midnight_utc as _lmu
|
||
|
||
# Start side: always count the local calendar day in which start_utc falls.
|
||
first_counted: _date = local_date(start_utc)
|
||
|
||
local_end_date: _date = local_date(end_utc)
|
||
# Is the midnight of local_end_date < end_utc? If yes, that day's midnight is in range.
|
||
if _lmu(local_end_date) < end_utc:
|
||
last_counted: _date = local_end_date
|
||
else:
|
||
last_counted = local_end_date - _td(days=1)
|
||
|
||
# Settlement cap: "today" is only counted once the local clock has passed the
|
||
# settlement offset since midnight (01:05). This defers the daily standing-charge
|
||
# step from 00:00 to 01:05, ensuring the cumulative-cost adiabatic jump lands
|
||
# inside the new calendar day and avoids the HA 01:00 hourly-bucket boundary.
|
||
_today_midnight_utc = _lmu(today_local)
|
||
if _now_local >= _today_midnight_utc + _SETTLEMENT_OFFSET:
|
||
settled_cap = today_local
|
||
else:
|
||
settled_cap = today_local - _td(days=1)
|
||
last_counted = min(last_counted, settled_cap)
|
||
|
||
# If the range is empty (first_counted > last_counted), no days are counted.
|
||
|
||
# Fetch all versions of the active contract once (single DB call).
|
||
versions = active_contract_versions(session)
|
||
|
||
fixed_dec = Decimal("0")
|
||
credits_dec = Decimal("0")
|
||
currency = "EUR"
|
||
|
||
if versions:
|
||
# Currency comes from the contract regardless of day count.
|
||
currency = versions[0].contract.currency
|
||
|
||
if versions and first_counted <= last_counted:
|
||
# For each version segment, find its overlap with [first_counted, last_counted]
|
||
# and accumulate whole days.
|
||
version_segments: list[tuple[_date, _date | None, dict]] = []
|
||
for v in versions:
|
||
v_start_local = local_date(_as_utc(v.effective_from))
|
||
v_end_local = local_date(_as_utc(v.effective_to)) if v.effective_to is not None else None
|
||
version_segments.append((v_start_local, v_end_local, v.values or {}))
|
||
|
||
for v_start, v_end_excl, v_values in version_segments:
|
||
# The version covers [v_start, v_end_excl) in local dates,
|
||
# where v_end_excl=None means open-ended (no upper bound).
|
||
# Intersection with [first_counted, last_counted]:
|
||
seg_start = max(first_counted, v_start)
|
||
if v_end_excl is not None:
|
||
seg_end_excl = min(last_counted + _td(days=1), v_end_excl)
|
||
else:
|
||
seg_end_excl = last_counted + _td(days=1)
|
||
|
||
if seg_start >= seg_end_excl:
|
||
continue # no overlap
|
||
|
||
n_days = (seg_end_excl - seg_start).days
|
||
if n_days <= 0:
|
||
continue
|
||
|
||
standing: dict = v_values.get("standing", {})
|
||
creds: dict = v_values.get("credits", {})
|
||
|
||
network_fee = _to_decimal(standing.get("network_fee", 0))
|
||
management_fee = _to_decimal(standing.get("management_fee", 0))
|
||
heffingskorting = _to_decimal(creds.get("heffingskorting", 0))
|
||
|
||
# Standing charges: EUR/month → EUR/day (÷ 30) × n_days.
|
||
fixed_dec += (network_fee + management_fee) / Decimal("30") * Decimal(str(n_days))
|
||
# Energy-tax credit: EUR/year → EUR/day (÷ 365) × n_days.
|
||
credits_dec += heffingskorting / Decimal("365") * Decimal(str(n_days))
|
||
|
||
total_payable = sum_net + fixed_dec - credits_dec
|
||
|
||
return {
|
||
"currency": currency,
|
||
"metered_import": float(sum_import),
|
||
"metered_export": float(sum_export),
|
||
"metered_net": float(sum_net),
|
||
"fixed_costs": float(fixed_dec),
|
||
"credits": float(credits_dec),
|
||
"total_payable": float(total_payable),
|
||
"period_count": len(good_rows),
|
||
"degraded_count": len(degraded_rows),
|
||
"days": float(days),
|
||
}
|