Most utility AI projects stall at the integration layer. The ML model that performed well in the proof-of-concept phase, trained on exported historical data, never makes it to production because connecting it to live SCADA or EMS data turns out to be significantly harder than anyone planned for. Here is what actually matters when you are trying to make that connection work.
The data extraction problem
SCADA and EMS systems were not designed with data export in mind. They were designed to monitor and control grid infrastructure, and their data architectures reflect that. The historian databases that most systems use to store interval data have query interfaces that are functional but not well-suited to the continuous polling pattern that a live ML inference pipeline requires.
The first integration question is always: what is the sustainable way to get interval data out of this system at the cadence the forecast model needs? For most systems, this is a read-only SQL connection to the historian, a REST API if the SCADA vendor provides one, or in older environments, a CSV export via SFTP that gets ingested on a scheduled basis. Each has different latency and reliability characteristics.
Data quality in transit
Live SCADA data is messier than historical exports. Sensor dropouts, communication failures, and meter calibration events produce gaps and anomalies that do not appear in cleaned historical datasets. A production integration needs to handle these gracefully: detecting and flagging bad readings, filling short gaps with interpolation, and degrading to a safe fallback forecast when data quality drops below a reliable threshold.
We have seen integration projects fail because the ML model was validated on clean historical data and then fed with live data that had consistent quality issues the model had never encountered. The model produces erratic outputs, operators lose confidence, and the project stalls. The fix is to build quality monitoring into the ingestion pipeline from the start, not to add it after the model is already in production.
What the 2-week integration actually looks like
Ampgrove's 2-week integration claim is based on a specific set of conditions: read-only access to the historian or EMS database, a clear list of the feeder IDs to monitor, and an IT contact who can authorize the connection. Given those three things, our SCADA connector framework handles the rest.
Week one is data extraction and validation: we establish the connection, extract 12 to 18 months of historical interval data for the target feeders, and run a data quality assessment. We are looking at completeness (what fraction of expected intervals have valid readings), consistency (are readings within physically plausible ranges), and cadence reliability (is the data arriving at the expected interval).
Week two is model training and live feed validation: we train feeder-specific models on the validated historical data, establish the live data pipeline, and run shadow forecasts for 5 to 7 days to confirm that the pipeline is producing outputs that match what we see in the historical validation. If the live forecast accuracy matches the historical validation, we are ready to go live.
Common blockers and how to address them
The most common integration blocker is IT security review. The request to open a read-only database connection from an external system triggers a security review process that can take 4 to 8 weeks at many utilities. Starting that process early, before the pilot engagement begins, is the most effective way to keep the integration timeline on track.
The second most common blocker is historian data gaps. If the historian has significant gaps in historical data for specific feeders, those feeders may not have enough training data for accurate initial model performance. We can often work around this with transfer learning from neighboring feeder data, but the timeline extends.
On-premises versus cloud-hosted integration architecture
For utilities with strict OT network segmentation requirements, connecting a cloud-hosted forecasting platform to the SCADA historian may require an on-premises agent that mediates the data transfer. Ampgrove supports a deployment model where a lightweight agent runs within the utility's OT-adjacent network, reads from the historian via the approved read-only connection, and transmits aggregated interval data to our cloud forecasting infrastructure over an outbound HTTPS connection.
This architecture satisfies most utility OT security policies because the data flow is unidirectional: data leaves the utility network but no inbound connections are made to OT infrastructure from external systems. The agent is stateless and does not retain any data beyond what is being transmitted in the current batch. Its footprint on the OT-adjacent network is minimal: a single process running on a low-spec virtual machine with outbound HTTPS access to our API endpoints.
Maintaining integration health in production
A production integration that was healthy at deployment can degrade over time due to infrastructure changes on the utility's side. SCADA software updates, historian version upgrades, database credential rotations, and network policy changes can all affect the data connection without any visible failure in the utility's own systems. The forecasting platform only learns about these changes when the data flow stops or degrades.
Ampgrove monitors data pipeline health continuously in production. We track data completeness, latency, and format consistency for every active integration, with automated alerts when any metric drops below threshold. When we detect a data quality degradation, we notify the utility's IT contact and provide a specific description of what changed, which usually allows quick diagnosis of the root cause on the utility's side.
This monitoring matters for accuracy guarantee continuity. Our accuracy commitments assume the data pipeline is operating within specification. When data quality degrades, we are explicit about the forecasts produced during degraded periods carrying reduced confidence, and we do not count those periods in our accuracy calculations.
Long-term integration evolution
SCADA systems evolve over their operational lifetimes, and the integration architecture we establish at deployment will not remain static. Utilities that migrate to newer EMS platforms, add substations, or expand service territory will require integration updates. We treat integration maintenance as part of the ongoing subscription, not a separate project that requires new scoping and contracting.
This matters in practice because integration changes at utilities often happen on compressed timelines. A historian upgrade might be scheduled with 30 days of notice. If we needed to scope and contract a new integration project every time the underlying infrastructure changed, the forecasting system would have periodic outages that would undermine operator confidence. Our integration framework is built to adapt to common historian and EMS changes with configuration updates rather than new engineering work, which allows us to maintain production continuity through most infrastructure changes on the utility's side.