The complex interactions of the financial, logistical, and geological processes that are involved in the production and delivery of oil and gas to market are driving the industry to real-time business processes.
No longer can decisions be made based on yearly projections, quarterly business plans, monthly royalty allocations, weekly inventory updates, daily prices, hourly blend changes, or minute-by-minute SCADA updates. Processes developed in other industries are showing the way towards this real-time business transition.
Toyota, GE, Boeing, Wal-Mart, Pfizer, Wall Street, and even the Pentagon have developed the real-time planning and operational optimization tools and processes that collectively have become known as lean management.
The appearance of control centers for company-wide coordination of ultradeepwater activity is one example of the coming revolution. Companies variously call these new sensor-driven production systems ‘smart’ or e-fields, and from the first days of operation, surprising new insights into how oil and gas actually move from reservoirs to wellbores have been discovered.
In this new real-time world of decision-making, financial planning is merged with technical and engineering operations. This tenth installment of the lean energy management series is concerned with the algorithms that enable this merger of planning and operations into the new real-time business paradigm.
We call the integration of the processes specific to the energy enterprise that are required for this real-time performance optimization “computer-aided lean energy management” or CALM (see the first 9 parts of this 12-part series (www.ogjonline.com).
Oil and gas business processes are complex not merely because of the physical size of the supply chain required to get the energy to market but also because of the complex array of decision-making uncertainties that can affect the processes along the way.
For example, in the electric power industry, operating engineers and managers are faced with a complex array of variables arising from deregulated markets, uncertain weather conditions, equipment failures, and now the specter of terrorist attacks, in addition to fuel supply costs.
The uncertainties have impacts on business processes that both cross management silos and scale over many orders of magnitude, (e.g., local to global, speed-of-light to weekly and monthly). Further, the impact of each of these variables is dependent on the state of many other associated variables.
As the system becomes more and more complex, it exhibits nonlinear behavior that must be managed by a correspondingly more sophisticated, computer-aided management system that is focused simultaneously on operations, process, risk, and value generation (Fig. 1).
Digital convergence
Conventional oil industry process controllers deal with complexity by fragmenting the decision-making and partitioning it into organizational divisions and hierarchal levels (such as separate regulatory, supervisory, and planning silos). For example, tactical control of the manufacturing process is centered in the office of the COO, whereas operational and capacity planning, which affects the business process on longer time scales, is carried out usually in the office of the CEO.
The business process decisions made at lower levels are often made in isolation and on the basis of static assumptions that are out of date with conditions at other levels of the organization. This fragmented approach leads to gaps and missed synergies that affect the efficiency and security of the business process.
The new CALM real-time control system is made possible by the convergence of several simultaneous and disruptive computer and communications revolutions instead of wireless, cheap plug-and-play sensors (IEEE 1451 standard), web services, distributed processing and storage, and today’s hyperexponential scaling of both the capacity and affordability of memory and processors that is even exceeding Moore’s Law. These new technologies enable the industry to not only drive down the price point of traditional SCADA by an order-of-magnitude, but also to convert to a real-time, two-way control system.
This ‘innervation’ (comparable to the human nervous system) is required before machine learning and adaptive computer intelligence (the brains) can be embedded into the field components (the body) of the decision support system (Fig. 2).
This CALM control system uses web services as its most critical distributed component, with a grid computing system layered on top to host the real options, reinforcement learning, dynamic programming, and machine learning feedback loops that are required for real-time decision support. It uses next-generation storage network technology to distribute the traditionally centralized, IT-department controlled, data-historian functionality of SCADA to the critical field devices themselves via time-series data-tag storage.
Adaptive learning is built into the field sensors so that they become ubiquitous, distributed controllers. In other words, learning is designed into the system at the most fundamental, ‘last-mile’ level. Simultaneously, metrics are kept of all actions of the system so that continuous improvement becomes the norm.
The ‘innervation’ is a key to creating distributed infrastructure security, as well, as marketers like Wal-Mart require RFID tags from all their suppliers to broadcast the location of goods from transport to sale.
How CALM works
In order to better manage the production of oil and gas from individual wells and fields into and through the vast pipeline and transportation system and into refining and petrochemical plants in the most optimal manner, CALM builds a decision support system that balances subsurface reservoir realities with business constraints and contractual requirements to optimally manage this portfolio in real-time.
The business focus shifts to wired processes that connect business decisions to the last-mile of the supply chain, the hydrocarbon reservoirs themselves. This change in focus requires that several challenges must be overcome:
1. The information architecture to integrate and optimize real-time business processes must scale from the very smallest of components to the largest of markets.
2. The information-trail must be ubiquitous and extend from headquarters to every pump jack and wellhead.
3. The decision support system must maximize profit, simultaneously evaluating costs, risks, and benefits from the last mile to the executive board room, and all in real-time.
4. The underlying business logic must be transparent so it can be easily taught to new generations of operators and managers.
CALM’s mission is to eliminate the wish-I-could-have-seen-it-coming statement by computing predictive failure models, cost-benefit analyses, and risk-return optimization to identify solutions to problems quickly and select the most efficient remediation.
All subsequent actions must be tracked to verify not only that the work was done on schedule and on cost but that the outcome was as predicted. CALM’s mission is to improve whole-system performance.
Open architecture
In order to support intelligent controllers, the energy industry must develop an infrastructure to host distributed, remote computing that automatically integrates financial and engineering data into real-time enterprise management.
The integration of all these real-time data provides a challenge to traditional IT departments, in particular, who often feel threatened by loss of control over their domains.
The software architecture for these modern energy control systems requires peer-to-peer computing to arrange embedded and hybrid devices into a grid of simulation agents. Because field devices may be physically insecure and subject to tampering as well as conventional failures and network-based attacks, the remote infrastructure must be self-monitoring and self-healing in a manner similar to that of the internet.
In the face of terrorism, remote detection of erratic behavior indicators must be standard and condition-based maintenance the norm. Reliability and security are particularly challenged by the additional need for an open software framework (e.g., based on web services interfaces).
The open environment is absolutely essential, however, because it assures interoperability among future generations of new devices and inventions that will be attached to the control grid. This open architecture enables:
• Easy updating of embedded systems software that includes: new algorithms; smarter middleware; virtual machines; new component technologies and programming methods for integration.
• Distributed software control that includes: discrete and continuous models for predicting behavior of the system; new concepts in SCADA and process control systems (PCS); and, scalable support for embedded sensors and sensor nets.
• Better interaction with models of the embedded software and systems: new foundations, design, implementation, synthesis, analysis, and certification methods.
• Innovative approaches to failure mode detection, self-testing, and recovery and restitution can be modeled first before deployment.
• Resource optimization: new methods and tools for managing real-time, power-aware, distributed systems; static and dynamic scheduling technology that integrates multiple concerns such as real-time guarantees, power, thermal gain, RF emissions and interference, network bandwidth availability, and component criticality.
Martingale controller
The digital convergence discussed above plays host and precipitator to the second convergence needed for CALM through the enablement of combined financial and engineering optimization and control.
This convergence is based on the realization that a common mathematical framework can be applied to optimization and control under uncertainty for both operational and financial engineering-for all devices in the field.
In the calculus used for CALM’s optimal decision-making under uncertainty, a real options framework is formulated as an optimal stochastic control problem, and integrated with engineering control objective functions to build an anticipatory, adaptive controller based on approximate dynamic programming (also called reinforcement learning).
We term this new system a ‘Martingale’ controller because, as with financial martingales, the controller is always computing ways to keep the system “in-the-money” with respect to both profitability and engineering efficiency.1
The embedded and distributed computational infrastructure of CALM is in the form of a peer-to-peer field network that is aware in real time of changing uncertainties so that it can continuously choose new policies and actions that support the profit-at-all-times mode of the martingale.
Reinforcement learning using dynamic programming has been successfully applied to problems at many organizational levels in other industries, including regulatory control, supervisory control, scheduling, operational planning, capacity planning, portfolio management, and capital budgeting (with real options).
Most importantly, the same dynamic programming algorithm solves both the financial real options and the operational reinforcement learning problems simultaneously.
Innervation, self-organizing, self-healing, reinforcement learning, and the use of evolutionary algorithms all are things nature does routinely within its biosystems. The CALM control system uses machine learning similarly to organize computation in the business environment.
We foresee that real options will be extended to what we call “dynamic micro-options,” i.e., distributed controllers in field devices will be smart enough to compute new real options on their own. This requires real options to be migrated from the strategic planning process, where they exclusively reside now in the energy industry, to the peer-to-peer computing environment in the field.
Real options can then be used to allow each asset to communicate with other relevant field devices and conduct joint modeling simulations of possible outcomes, all on their own. This dream phase of our adaptive control algorithms will use modeling and genetic algorithms to develop new real options to relentlessly drive towards the optimal solution.
Ultradeepwater example
Much technology is being developed for the execution of remote seafloor operations in ultradeepwater basins, including remarkable new drilling, completion, processing, and gathering technologies.
The big picture we have been discussing above must be considered at all times in order to guarantee an adequate return on investment at these water depths (Fig. 3).
CALM’s Martingale controller can be configured to carry out autonomous, remote subsea decisions in real time to affect the form and timing of gas, oil, and water production in the ultradeepwater environment.
As shown in previous parts of this series, integrated reservoir modeling is required. The model must include production constraints based, for example, on real-time environmental assessments of skin damage and water coning, as well as surface facilities and market constraints related to prices and contractual obligations.
The Martingale controller is trained using machine learning to generate flexible production-injection schedules that produce exemplary field production performance based on real option valuations done in real-time by the controller.
There are two critical kinds of flexibility to real options management of ultradeepwater fields.
One is strategic flexibility. The portfolio must have enough flexibility to deal with the volatilities of the market, the uncertainty of reservoir performance, and other externalities that change day to day throughout the life of the assets. Presently, the common option is only to buy or sell properties since production itself is decoupled from market conditions.
This leads to the second, or implementation flexibility. The portfolio must be managed to optimize the shape and mix of the production within fields through operational flexibility, a practice not presently done in oil fields.
A visualization of how real options from the Martingale controller might be used to drive production flexibility of a deepwater oil and gas field is shown in Fig. 4.
The Martingale controller uses reservoir simulations to optimize overall production in the field as prices fluctuate. Fig. 4 is an example of how increased value that can be achieved by flexible control of production that is based upon action and reaction to real-time price signals. If production can be decelerated or accelerated using e-field technology, the CALM controller can adapt production according to these market signals.
Short-term peaks and troughs in production are needed to increase NPV more and more as volatility of prices increases (indicated by the horizontal axis on the left panel of Fig. 4 that shows standard deviations in short-term prices).
Others have perceived this need for more active management of oil and gas fields, as well. Bertimas, et al.2 demonstrated the use of the Martingale technique for a portfolio of stocks, and SPE 82018 showed an application of reservoir simulation for real options evaluation of a single, smart well. SPE 71414 described a concept called “Stochastic Integrated Asset Modeling for Real Options,” and SPE 57472 discussed “Risked-Based Integrated Production Modeling” that also could be incorporated into our simulation-based CALM framework.
CALM control of oil and gas production in two horizontal wells in this field is illustrated on the right. The dotted and solid lines show the production of oil (top) and gas (bottom) from Well 1 (green) and Well 2 (red). Vertical scales for oil production are shown at the left of the graphs.
Gas production (red) uses 0-5,000 Mcf for the vertical scale. CALM suggests adjustments be made in production of oil versus gas at four times relatively late in the production cycle. These changes would represent both acceptable engineering and financial options for increasing oil and-or gas production if the prices are high (dots) and decreasing it if low (solid lines). Oil and gas prices are assumed to be independent over this time frame.
CALM’s value-add increases over the present static production scheme as price volatility increases. Note that the big gains occur only if both acceleration and deceleration in oil and gas production are allowed.
Thus, the use of CALM technologies allows the controller to push finer and finer granularity of automated operational decision-making out to the last mile of remote operations. Proactive shaping of portfolios to produce more companywide stability and immunity to short-term price cyclicity over time is the result. ✦
References
1. Anderson, R.N., and Boulanger, A., “Method and System for Automated Support of Real-Time Business Decisions under Uncertainty Based on Martingale Stochastic Control,” US Letters Patent, applied, 2005.
2. Bertimas, D., Lo, A.W., and Hummel, P., “Optimal control of execution costs for portfolios,” Computing in Science & Engineering,” Vol. 1, No. 6, November-December 1999, pp. 40-53.
Bibliography
Bertsekas, D.P., and Tsitsiklis, J.N., “Neuro-Dynamic Programming,” Athena Scientific, 1996.
Cohena, M.V. Natoli, “Risk and Utility in Portfolio Optimization,” Physica A: Statistical Mechanics and its Applications, Vol. 324, Issues 1-2, 2003, pp. 81-88.
Dietterich, T., “The MAXQ Method for Hierarchical Reinforcement Learning,” Proc. 15th International Conference on Machine Learning, 1998.
Fernandez, F., and Parker, L.E., “Learning in Large Cooperative Multi-Robot Domains,” International Journal of Robotics and Automation, special issue on Computational Intelligence Techniques in Cooperative Robots, Vol. 16, No. 4, 2001, pp. 217-226.
Feurstein, M., and Natter, M., “Neural Networks, Stochastic Dynamic Programming and a Heuristic for Valuing Flexible Manufacturing Systems,” Working Paper 17, Vienna University of Economics and Business Administration, 1998.
Gadaleta, S., and Dangelmayr, G., “Reinforcement learning chaos control using value sensitive vector-quantization,” Proc. International Joint Conference on Neural Networks, July 15-19, 2001, IEEE Catalog Number 0-7803-7044-9, ISSN 1098-7576, Vol. 2, pp. 996-1,001.
Gamba, Andrea, “Real Options Valuation: A Monte Carlo Simulation Approach,” Faculty of Management, University of Calgary WP No. 2002-3; EFA 2002 Berlin Meetings Presented Paper, Mar. 6, 2002.
Kaelbling, L.P., Littman, Michael L., and Moore, Andrew W., “Reinforcement Learning: A survey,” Journal of Artificial Intelligence Research, Vol. 4, 1996, pp. 237-285.
Lau, H.Y.K, Mak, K.L., and Lee, I.S.K., “Adaptive Vector Quantization for Reinforcement Learning,” Proc. 15th World Congress of International Federation of Automatic Control, Barcelona, Spain, July 21-26, 2002.
Littman, M.L., Nguyen, Thu., Hirsh, Haym, Fenson, Eitan M., and Howard, Richard., “Cost-Sensitive Fault Remediation for Autonomic Computing,” Workshop on AI and Autonomic Computing: Developing a Research Agenda for Self-Managing Computer Systems, 2003
Longstaff, F.A., and Schwartz, E.S., “Valuing American options by simulation: a simple least-squares approach,” Rev. Fin., Vol. 14, 2001, pp. 113-147.
Smart, W.D., and Pack Kaelbling, Leslie, “Practical Reinforcement Learning in Continuous Spaces,” Proc. 17th International Conference on Machine Learning, 2000, pp. 903-910.
The authors
Roger N. Anderson ([email protected]) is Doherty Senior Scholar at Lamont-Doherty Earth Observatory, Columbia University, Palisades, NY. He is also director of the Energy and the Environmental Research Center (EERC). His interests include marine geology, 4D seismic, borehole geophysics, portfolio management, real options, and lean management.
Albert Boulanger is senior computational scientist at the EERC at Lamont-Doherty. He has extensive experience in complex systems integration and expertise in providing intelligent reasoning components that interact with humans in large-scale systems. He integrates numerical, intelligent reasoning, human interface, and visualization components into seamless human-oriented systems.
