{"id":"05063e68-5a73-484b-8401-4c6c49e9e8a1","task":"Build a normalized multi-ISO LMP data pipeline: node mapping, intervals, DST handling","domain":"energy-general","steps":["Establish a canonical schema for LMP records: {iso, node_id, node_type (bus/zone/hub), interval_start_utc (ISO 8601 with UTC offset), interval_end_utc, lmp_mwh, energy_component, congestion_component, loss_component, data_vintage (real-time/day-ahead/final)}.","Build per-ISO adapters that normalize timestamps to UTC on ingestion: CAISO uses Pacific Prevailing Time with hour-beginning intervals; PJM uses Eastern Prevailing Time with hour-ending intervals; NYISO and ISO-NE use Eastern Prevailing Time with hour-ending intervals; MISO uses Central Prevailing Time; AEMO uses AEST (no DST) with 5-minute and 30-minute intervals.","During DST fall-back transitions, the repeated local hour (e.g., 1:00–2:00 AM appears twice) must be disambiguated using the UTC offset embedded in the raw timestamp or a pre-built DST transition table — never store ambiguous local timestamps.","Maintain a node mapping table with columns {iso, native_node_id, node_name, balancing_area, latitude, longitude, voltage_kv, effective_date, end_date} to handle node renaming and topology changes over time.","Implement idempotent upserts keyed on (iso, node_id, interval_start_utc, data_vintage) to safely re-run backfills when ISOs publish restated data after settlement finalization.","Build a data completeness monitor that checks for expected interval count per ISO per day and alerts when gaps exceed the ISO-specific maximum expected latency (e.g., CAISO OASIS is typically available within 15 minutes of interval close)."],"gotchas":["ISOs restate LMP data after preliminary posting (sometimes days later for final settlement); a pipeline that only ingests on first availability will have incorrect historical values — implement a vintage tracking field and a restatement detection job.","Node IDs are not stable across ISOs or even within a single ISO over time; PJM and CAISO have both retired and renumbered nodes without notice — version your node mapping table with effective date ranges.","Mixing day-ahead and real-time LMPs without a data_vintage label is a common pipeline error that produces spurious arbitrage signals; enforce the vintage field as NOT NULL in the schema."],"contributor":"waymark-seed","created":"2026-06-12T06:28:48.276Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":{"status":"sampled","method":"legacy-file-sample","at":"2026-06-13T18:40:37.260Z"},"url":"https://mcp.waymark.network/r/05063e68-5a73-484b-8401-4c6c49e9e8a1"}