Physical Analytics - Renewable Energy, Weather and Climate Forecasting
Description: Accurate forecasts of the atmospheric state have been a particularly challenging problem yet have enormous social and economic benefits. Among many important applications of atmospheric forecasts, the need for the forecast of variable solar and wind energy generation is becoming pressing with increasing penetration of solar and wind energy to the total energy mix. Forecasts of enhanced accuracy are highly desired for reliable grid operation at reduced cost and a more efficient electricity market. Recently, we showed that significant accuracy improvements can be achieved if we combine the individual model forecasts using a machine-learning based approach which takes into account appropriately additional state parameters beyond these need to be explicitly forecasted.
The multi-model blending method can be understood from Figure 1, which hypothesizes the accuracy of various forecasting approaches for different time horizons. The long-term average consists of using a constant climatological value or climate model forecasts. The model approach corresponds to the use of numerical weather prediction models, which rarely achieve good accuracy at short time horizon due to the period they require to achieve numerical stability. A Lagrangian approach provides good accuracy for a few hours ahead via, for example, advecting the most recent upwind observations in space and time to estimate when clouds might reach a forecasting site. Finally, the Eulerian approach uses the persistence of the last observation, which shows highest accuracy at very short forecasting horizons.
Figure 1: Depiction of forecast accuracy of various approaches and models as a function of forecast time horizon.
An key feature - situation-dependent blending - endows this approach high forecast accuracy perfomance and distinguishes it from conventional multi-model ensemble methods. This can be understood by the following equation yielding an optimal forecast (Cblend) for a given parameter (wind, irradiance etc), which can be represented as a linear combination of models (or expert systems),
with τ as the forecast horizon, x the location and spatial extent of the forecast, and s the weather situation defined by a set of environmental parameters E (including forecasted ones). Cm are the forecasts associated with each model and wm the respective machine-learned weightings. wm is a function of forecast horizon, location, and weather situation s. The index m corresponds to each different model, including Eulerian, Lagrangian, model, and climatological inputs.
Figure 2 provides an architectural view of the technology as we applied it to renewable energy forecasting. A “big” data bus provides atmospheric information (such as temperature, wind, cloud properties etc) from various forecasting models. For solar, a radiative transfer model converts these atmospheric information into irradiance. The different forecasts are blended and then converted to power using an irradiance to power (for solar) or a wind to power (for wind) model. A machine learning module provides the blending (weight) coefficients. Initially the system is trained on historical data but as new measurements become available it constantly improves. The training starts with analyzing how the errors of the individual forecast models depend on a large set of atmospheric state parameters using functional analysis of variance. From the analysis, a reduced set of state parameters (E) which has significant impact on the models’ forecast errors is selected. Then, the parameters E, the forecasted and the measured values of the parameter of interest are fed into a machine-learning module to derive the situation-dependent weight coefficients.
Figure 2: Architectural view of situation-dependent, machine learning based multi-model blending.
The situation-dependent model blending method has been successfully applied to a number of forecasting problems, including solar power, wind power, precipitation, and particulate matter pollution etc. in various geographical areas and for hour-, day-, and month-ahead forecasting time horizons. In all cases, substantial accuracy improvements have been observed - typically over 30% improvement with respect to the best individual input models - demonstrating that it is a general framework for enhancing modeling of complex systems. Please refer to the presentations and publications below for more information and forecasting results. You can also find below a demo website containing real-time blended forecasts of contiguous US wide solar irradiance.
Recent presentations:
- 2015 American Geophysical Union Fall Meeting
- 2015 ECC Presentation
- 2016 DoE Solar Forecasting Workshop
Recent publications:
- I. Khabibrakhmanov, S. Lu, H. F. Hamann, K. Warren, “On the usefulness of solar energy forecasting in the presence of asymmetric costs of errors ”, IBM Journal of Research and Development, 7, 1-6(2016).10.1147/JRD.2015.2495001
- W Y. Cheung, J. Zhang, A. Florita, B.-M. Hodge, S. Lu. H. Hamann, Q. Sun, B. Lehman, "Ensemble Solar Forecasting Statistical Quantification and Sensitivity Analysis", Proceeding of the 14th Wind Integration Workshop, in press (2015).
- J. Zhang, B.-M. Hodge, S. Lu, H. F. Hamann, B. Lehman, J. Simmons, E. Campos, V. Banunarayanan, J. Black,J. Tedesco, "Baseline and target values for regional and point PV power forecasts: Toward improved solar forecasting", Solar Energy, 122, 804-819 (2015).doi:10.1016/j.solener.2015.09.047
- H. F. Hamann, S. Lu, “Situation-dependent blending of multiple forecasting models based on machine learning”, SPIE Newsroom, November (2015). DOI: 10.1117/2.1201510.006142
- S. Lu, Y. Hwang, I. Khabibrakhmanov, F. J. Marianno, X. Shao, J. Zhang, B.-M. Hodge, H. F. Hamann, “Machine Learning Based Multi-Physical-Model Blending for Enhancing Renewable Energy Forecast – Improvement via Situation Dependent Error Correction.” Proceeding of European Control Conference 2015 WeB11.3 (2015). 10.1109/ECC.2015.7330558
- J. Zhang, A. Florita, B.-M. Hodge, S. Lu, H. F. Hamann. V. Banunarayanan, Anan M. Brockway, “A suite of metrics for assessing the performance of solar power forecasting”, Solar Energy, 111, 157-175 (2015).doi:10.1016/j.solener.2014.10.016
Recent press:
- LA Times Machine ‘Learners’ Compute Cloud Cover to Balance Power Supplies http://lat.ms/1IWCo5u
- MIT Technology Review Weather Forecasting Enters a New Era http://bit.ly/1KfrhoL
- Mashable Solar and Wind Forecast are New Wave of Weather Reporting http://on.mash.to/1OusOJ1
- ComputerWorld IBM's machine-learning crystal ball can foresee renewable energy availability http://bit.ly/1fasHWJ
- Inside HPC IBM Improves Solar Forecasts with Machine Learning http://bit.ly/1I8xFk3
- Data Center KnowledgeIBM’s Machine Learning Tech Takes on Solar Power’s Flakiness http://bit.ly/1KMQFSS
- R&D Magazine Machine Learning’s Impact on Solar Energy http://bit.ly/1DnQ9Lf
- AltEnergyMagInterview with Hendrik Hamann, Physical Analytics Manager at IBM Research http://bit.ly/1LYO9Ms
- Fierce Energy The Latest in Solar Forecasting: Big Data Meets Cognitive Computing http://bit.ly/1OlOUwr
- Enterprise Times (UK) Renewable energy forecasting improves http://bit.ly/1gFj4Aj
- Solar Industry Machine Learning Helps IBM Boost Accuracy of U.S. Department of Energy Solar Forecasts by up to 30 Percent http://bit.ly/1SriUrZ
- SeeNews Renewables IBM reports 30% more accurate solar, wind forecasts in DOE-backed project http://bit.ly/1HyNnkV
- CleanTechnica 30% Jump in Solar Energy Forecasting Accuracy Gained by Machine Learning http://bit.ly/1I8sluS
- ExecutiveBiz IBM DOE Solar Forecast Accuracy Up 30% With SMT Machine Learning Tool http://bit.ly/1V7oLqr
- Solar Industry IBM Boosts Accuracy Of DOE Solar Forecasts By 30% http://bit.ly/1J2WA7w
- Electronics 360 Machine Learning Helps IBM Boost Accuracy of Solar Forecast http://bit.ly/1IfaDoz
- Energy Matters (Australia) Better Solar & Wind Forecasting http://bit.ly/1JtAgje
- ChannelWorld (India) IBM's machine-learning crystal ball can foresee renewable energy availability http://bit.ly/1DsPByr
- Phys Org Machine learning helps IBM boost accuracy of US Department of Energy solar forecasts by up to 30 percent http://bit.ly/1MeTxL9
- The FINANCIAL Machine Learning Helps IBM Boost Accuracy of U.S. Department of Energy Solar Forecasts by up to 30 Percent http://bit.ly/1I8vTPK
Movies:
- How it Works: https://youtu.be/VHPTfni6j9s
- Chasing the sun: https://youtu.be/fELRzZWVyfI
- Chasing clouds with data: https://youtu.be/fd0F1I33NeE
- Tracking clouds with IBM: https://youtu.be/oHzp9lsAkTM
- Predicting clouds: https://youtu.be/woRpQBaEYbI
- Sky cameras: https://youtu.be/eeLarPDt59c
- Solar panels: https://youtu.be/8tXG6uX7UUk
- Sky camera applications: https://youtu.be/95LYvMztFuE
Demos:
Key patents:
- H. F. Hamann, Y. Hwang, T. G. van Kessel, I. K. Khabibrakhmanov, S. Lu, R. Muralidhar, "Multi-Model Blending", US Patent Application 2015034792, 2015.
- H. F. Hamann, S. Lu, "Multifunctional Sky Camera System for Total Sky Imaging and Spectral Radiance Measurement", US Patent Application 20140320607, 2014.
- H. F. Hamann, S. Lu, "Machine Learning Approach for Analysis and Prediction of Cloud Particle Size and Shape Distribution" US Patent Application 20140324352, 2014.
Partners:
- US Department of Energy SunShot Program
- National Renewable Energy Laboratory
- Northeastern University
- Florida Gulf Coast University and The University of Arizona
- ISO New England
- Green Mountain Power