Download Australian NEM market data from AEMO (NEMWeb)

Steps

1. Access AEMO's developer API portal at dev.aemo.com.au for structured API access; for bulk historical data, use the NEMWeb file repository currently at visualisations.aemo.com.au/aemo/nemweb/ noting that AEMO is migrating the base URL — verify the current base URL in AEMO's official migration notices as the HTTP endpoint is being decommissioned.
2. Identify the target MMSDM (Market Management System Data Model) table; key tables include TRADING_PRICE (30-minute settlement prices by region), DISPATCH_PRICE (5-minute dispatch prices), UNIT_SCADA (generator MW output), and DISPATCH_INTERCONNECTORRES (interconnector flows).
3. Download zipped CSV files from the REPORTS/CURRENT/ or REPORTS/ARCHIVE/ directories, or use the NEMOSIS Python library for automated table-level downloads with automatic schema handling.
4. Unzip and parse the CSV files; the first row is a header ('C' record), data rows begin with 'D', and the file ends with an 'I' record plus an 'END' summary — strip non-data rows before loading into a dataframe.
5. Convert AEMO timestamps from Australian Eastern Standard Time (AEST, UTC+10); NEM does not observe daylight saving time in its market settlement clock, so no DST conversion is needed for market intervals.
6. For real-time 5-minute dispatch data, poll the CURRENT directory at 5-minute intervals; files are named with a timestamp and overwritten each dispatch interval — implement idempotent upsert logic keyed on SETTLEMENTDATE plus DUID.