SPECIAL REPORT: PROCESS CONTROL-1: Best practices improve control system performance

MPC applications typically take one of four paths depending on the level of support and monitoring they receive. The path chosen, conscious or otherwise, affects the application performance tremendously (Fig. 1):

• 25% fail within just months of commissioning.
• 40% fail within about 3 years of commissioning with inadequate support and no monitoring.
• 25% continue to deliver reduced benefits albeit reduced with good support and no monitoring.
• 10% will deliver increased benefits with adequate support and online monitoring.

Poor MPC performance

There are two groups of factors that affect control performance, technology and organizational factors. In any solution to this problem, both must be addressed to sustain long-term benefits from improving the performance of these assets.

Technology factors

Several factors inherently related to the technology used during the deployment of MPC are known contributors to the decline in performance of any MPC application. These include:

• Poorly performing regulatory controls or faulty instrumentation.
• Faulty or poorly performing process analyzers (gas chromatographs, etc.) and online property estimators.
• MPC model error.
• Disturbance problems.
• Constraint issues.
• MPC tuning.

All MPC applications are built on a foundation of regulatory and advanced regulatory controls. Ironically, this hierarchy is often depicted as a pyramid synonymous with enduring strength (Fig. 2). In reality, it can be as fragile as a house of cards with each layer dependant on all of the layers below functioning well.

Each element in the regulatory control loop from the primary sensor to the final control element must perform well in order for the advanced controller to also perform well. This is of course why one of the first steps in any MPC project involves a regulatory and instrumentation layer tune-up. Valves are fixed, transmitters calibrated and controllers are tuned accordingly prior to the implementation of any advanced controller.

Unfortunately, over time the regulatory controller performance also declines. Control element and sensor performance, controller tuning, and controller design are the top causes of poor regulatory control.

Several studies have been done across industries and have documented that anywhere from 40-80% of the regulatory controls in a process plant can be significantly improved.2 4 5

The effect of regulatory controller performance on MPC performance can be significant because poorly performing regulatory controllers affect the accuracy of the MPC models as well as the effectiveness of the MPC control action.

Online analyzers such as gas chromatographs and inferential property measurements are commonly used as CVs in the design and deployment of an MPC controller. Online analyzers are installed to measure important plant quality variables and are usually expensive, complex, and difficult to maintain. To compensate for this, as well as for the sample delay induced by most plant analyzers, online property estimators are used.

These estimators or inferential models use auxiliary measurements to infer the property measure of interest. Gas chromatographs may take more than 15 min to analyze a sample so estimated values are used in the control application between analyzer samples, permitting control action to take place often than once a sample. Estimators also provide a useful means to cross-check analyzer results; a large discrepancy between the estimator and the analyzer often indicate the analyzer result is suspect.

MPC-specific issues

Specific problems with the MPC controller itself of course prevent an application from generating benefits.

The simple fact that MPC controllers optimize the process in the presence of physical constraints allows them to generate the significant financial benefits that justify their implementation. Constraint handling ensures the MPC controller is keeping the plant at or near an economic optimum. The effect of active constraints, however, is to change the effective process matrix and the effective controller gain.

The controller can therefore change behavior significantly as the active set of constraints changes. Bakker3 provides an excellent discussion of the need for monitoring active constraints; this can assist in determining the value of plant debottlenecking projects that would mean relaxing that constraint.

MPC controllers are uniquely able to make use of feedforward information. The models provide an excellent means of predicting the result of an anticipated disturbance, allowing the controller to respond in a coordinated way, which prevents undesired effects. Unfortunately, it is impossible to anticipate every significant disturbance at each operating point that will affect the controller’s performance.

Over time, it is necessary to re-assess the significant disturbances that affect the MPC controller to determine whether they should be added as feedforward variables or if existing feedforward models are adequate. The relatively slow scan rates and detuned nature of most MPC control applications imply that the controller is ill-equipped to deal with fast disturbances.

MPC controllers use models to calculate both the response of the process if no control is taken as well as the optimum control action to take based on current conditions. Model errors cause controllers to over or under-react, react too soon or too late, and worst of all, decrease a controller’s ability to anticipate.

Model degradation can occur with something as simple as heat exchanger fouling. Changes in production rate, pressure profile across the plant, utility allocations can all result in models that are no longer adequate.

Troubleshooting model problems is not always a simple task because a large MPC controller with will have hundreds of models. On a large controller, determining if in fact model performance is the problem can be challenging, let alone isolating the specific model set that is responsible.

Organizational factors

Organizational factors that affect MPC performance are more critical than the technology issues. In the absence of technology, a good organization will always succeed; whereas technology alone without organizational support cannot achieve results.

Organizational factors considered are the organization’s business goals, management support for advanced process control, human resources, and work processes in place at the operating site.

Business goals

On the business side, changing market conditions can dramatically influence the performance of an MPC application that was commissioned under a particular set of operating conditions. Changes in product demand, feedstocks, plant rates, product mix, or energy prices will all affect performance of an MPC controller. If business conditions change significantly or often, the goals that were critical in justifying the controller implementation initially, typically no longer apply.

Consider an MPC application in a plant where the business conditions have changed. Instead of the plant being sold out of product where the primary business objective is to maximize throughput, the plant is now running at lower rates and the business objective has changed to minimizing operating costs.

The new operating objectives create three potential problems with the existing controller:

• Tuning. The effective tuning at any given instant depends on the initial tuning, process dynamics, and current constraint set. The effective controller tuning depends not only on the original tuning, but which manipulated variables are available to move the CVs and which are saturated. Those MVs that are saturated are effectively removed from the control calculation. This can change the effective gain of the controller when the more effective MVs are saturated. A controller that is tuned when the plant is operating at maximum rates may not work effectively when the plant is operating at a different operating point.
• Constraints. The optimum operating point will be at different constraints when minimizing cost as opposed to maximizing rates. New constraints mean that new MVs are saturated and new MVs are available to control the process. Because it is likely that the controller was not commissioned with this set of active constraints, it may perform poorly. Mitigation may require controller recommissioning or even a redesign.
• Models. Because the plant is now running at a different operating point, process dynamics may be significantly different. MPC controllers rely on models to predict the effects of control moves and disturbances. If the models do not reflect the current dynamics, the predictions will be poor resulting in poor control.

Management support

Management personnel are typically sold on the benefits of an APC project before implementation. Because MPC technology is mature, vendors are able to quote typical results in similar facilities and the project is given the go-ahead.

An average crude unit MPC controller implementation, for example, will exceed several hundred thousand dollars in cost, but will return this investment many times during the life of the application, with payback within the first year of operation. On a typical project, at completion of the job, results are typically delivered as promised and everyone is satisfied with the results generated.

For many reasons, several months after the project team has moved on, the application may no longer be in service. Management is faced with the dilemma of why a significant investment in technology, time, and resources has not continued to accrue benefits to the business. Although management has bought and paid for the initial application, they typically have little day-to-day insight into the long-term performance of these applications. They are therefore challenged in attempting to make sound financial decisions regarding the ongoing support and maintenance.

Human resources

MPC controllers are typically commissioned and maintained by well-trained and skilled process control engineers. At the beginning of an MPC project, these resources are typically employed either on a contract basis from a control vendor’s organization, or from within the organization itself.

An assumption often made by management is that when implementation is complete, the project team will maintain the applications over the long term. Often, however, the control engineers that commissioned the controllers move on to new projects often with little or no overlap between implementation and support personnel; documentation from the commissioning team is often inadequate to effectively maintain these systems.

Additionally, many skilled control engineers leave these organizations, often leaving a large number of legacy applications to be maintained by a smaller, less-skilled workforce.

Similarly, maintenance groups have been asked to do more with less. Fewer instrumentation and electrical (I/E) resources are monitoring more regulatory controllers. A ratio of several hundred regulatory control loops per maintenance person is not uncommon.

Because good regulatory control performance is a necessary requirement for good MPC performance, the lack of routine regulatory controller maintenance is a significant contributor to the inadequate performance of an advanced controller.

Operations staff, of course, interact on a day-to-day basis with the control applications implemented in their area. As a general rule, as long as these applications make for a smoother operation and do not make the operators’ job functions more difficult, they will remain in service. Controllers that are pushing constraints that are illogical or not understood by the operator will be adjusted or shut off quickly.

A well-executed advanced control project includes key operations personnel not only to provide valuable process insight, but also to gain acceptance for the new technology and provide critical operator training on the use and troubleshooting of the application.

With adequate training, insight, and skill to identify performance degradation early, operations personnel are invaluable at spotting trouble and providing the process control engineer with valuable diagnostic information. Conversely, if the operators do not understand the application, its purpose, and limitations, or if it causes too much extra work, it will likely be unused.

Despite advances in the ease of use and tools to more effectively deploy and support these applications, knowledgeable well-trained human resources still remain a prerequisite in maintaining the long-term performance of these applications.

Maintenance work processes

Another cause of the decline in MPC performance is the lack of effective work processes and methods to support these applications and underlying regulatory controllers. Work processes are the methods and procedures by which the organization addresses the technical issues described previously.

Most companies we have encountered working with MPC applications do not use rigorous work processes to address both the technical and organizational factors that affect MPC performance. Maintenance on the MPC layer is typically done by control engineering personnel. A single engineer may have several control applications that he is responsible for as well as other responsibilities in the plant.

The most commonly used criteria to determine if maintenance is applied to an application is the “service factor” and operator feedback. Both are good measures of MPC performance, but should not be the only criteria by which decisions are made to spend money maintaining an application.

Diagnosing MPC problems is another matter. It can take days or even weeks to diagnose problems with large MPC applications. Without the proper tools and work processes, the control engineer typically looks at tens or even hundreds of trends to isolate the cause. Obviously this is a time-consuming, laborious process.

Because there are hundreds or even thousands of regulatory control loops in a plant, maintenance is typically done on an ad-hoc basis in the absence of a control performance monitoring system. At best, valves and sensors may receive maintenance on a scheduled basis; but in general, loops causing significant operational issues are addressed in order of priority once they have caused enough trouble to be noticed. Maintenance planners and I/E staff do their best to get to the high-priority problems, leaving most others waiting until they become a higher priority themselves.

Currently, the work process to control maintenance at both the regulatory and MPC layer in most organizations is reactive. Problems occur and control engineers or maintenance personnel respond applying corrective maintenance where it’s believed necessary.

In most organizations, maintenance is done based on priorities set by a combination of operations, maintenance, or engineering personnel. In the absence of objective data of which applications are misbehaving and why, decisions where to apply maintenance is done subjectively rather than based on any evidence of economic payback. In most instances, the “squeaky wheel” receives the most attention.

Best-practices approach

Corporations that are successful in implementing and maintaining MPC applications long term have adopted a best-practices approach to the problem that involves implementing work processes that target both the technical an organizational factors that limit sustained performance of these applications.

In general, the work process will include:

• Documentation and management of change procedures.
• Control monitoring and reporting.
• Effective diagnostic and troubleshooting and maintenance tools.

Documentation, management of change

Successful control applications require rigorous ongoing support of management. Critical to management support, however, is understanding what benefits the applications have delivered to date, and what ongoing benefits are currently generated.

Applications in which ongoing benefits are clearly documented are likely to receive the necessary maintenance funding required to keep them performing optimally. Ongoing monitoring of the benefits accruing from the application, either directly (via an online benefits estimator) or indirectly, by statistically measuring changes to the CVs with the controller in and out of service, is a necessary requirement for ongoing management support.

Brown described perhaps the most rigorous method of documenting MPC benefits.6 Not every application must be audited as rigorously, but it is important to document the actual benefits that accrue from an MPC project.

Rigorous technical documentation of the application is also necessary. Documenting design and implementation details, maintenance records, and ongoing operating details help ensure the application is kept running smoothly. Understanding the performance history of an application as well as the previous maintenance and upgrades help ensure that experienced and new staff are able to effectively maintain it.

A management of change process ensures that upgrades are documented and that they are not made without the knowledge and approval of appropriate individuals in the organization.

Control monitoring, reporting

In our experience, if the performance of these applications is not regularly measured and made visible to those who own the applications in the plant, the application cannot be sustained for the long term.

Because MPC is at, or close to, the top of the control hierarchy (Fig. 2), the lower layers must function properly for the MPC to deliver benefits. Monitoring and maintaining these assets is as important as monitoring the application itself.

Because any one of these layers can be responsible for problems with the MPC application, an effective monitoring application will link MPC performance with performance of the other layers including the estimators, regulatory controls, and instrumentation. Ideally, the application should support an investigative workflow that flows from symptom to cause wherever it happens to be.

Targeted information

Key to addressing many of the organizational issues is providing timely, targeted information to the correct individuals in an organization. Clearly, the requirements of the control room operator for control application health information are different from those of either the control engineer or the plant manager.

Plant management for example, typically would like to know if the application is being used and business benefits being generated by the application are reported in dollars. Conversely, if the controller is off, the cost to the business unit is valuable information.

With this type of management visibility, management obtains a clearer picture of the value of the MPC application. Table 1 highlights the type of information that each owner of a typical MPC application will require.

A monitoring application should address all of the needs of these various stakeholders, not just the control engineer.

Remote or central monitoring

Many organizations have downsized both control engineering and maintenance teams. Corporations are now interested in using control expertise wherever it exists in an organization. Control engineers located centrally in some organizations now support many applications in several remote locations across an organization.

A monitoring solution should be scalable so that it can support many MPC applications on a single site such as a refinery, or physically distributed at small sites with a central server.

Multiple controller technologies

As organizations have grown larger, often through mergers and acquisitions, multiple controller technologies can be found in plants throughout the organization. Some organizations must now support two or more technologies, in some cases on the same site.

It is most effective if a single tool can be used to monitor all control assets and applications within an organization. Otherwise, a patchwork of monitoring systems will result in an inability to compare performance of one application to another, and it becomes difficult to track problems in the MPC controller through the lower levels of the control hierarchy.

Moreover, from a management reporting point of view, it is much simpler to have a single monitoring application report on all of the assets in a business, rather than having to combine reports from several sources.

References

1. Qin, S.J., and Badgwell, T.A., “A Survey of industrial model predictive control technology,” Control Engineering Practice,” July 2003.
2. Treiber, S., Sattelmaier, R., and Starling, M., “Sustaining Benefits of Automation,” presented to the AIChE 2003 Spring National Meeting, New Orleans, Mar. 30-Apr. 3, 2003.
3. den Bakker, K., Seppala, C., and Snoeren, R., “Maintaining the Optimum Advanced Process Control,” Petroleum Technology Quarterly, Spring 2002, pp. 107-15.
4. Bialkowski, W.L., and Elliot, R., “Competency in Process Control - Industry Guidelines,” Pulp and Paper Canada, 1996, pp. 155-60.
5. Shook, D., Khalili, R., Grenier, M., and Lariviere, L., “Achieving business results through IT: Insuring controller performance benefits with web-enabled monitoring at Noranda,” CIM Bulletin, January 2003, pp. 57-59.
6. Brown, R.S., “The Development of a Rigorous Statistical Test to Audit the Advanced Control Benefits on an FCCU,” presented to the NPRA computer conference, Kansas City, Nov. 14-17, 1999.

The author

David Shook is chief technical officer for Matrikon Inc., Edmonton. He has extensive process industry experience and has provided leadership to Matrikon’s product development teams and its research partnerships with the University of Alberta. Shook also works closely with professional services groups and industry boards. He holds a PhD in chemical engineering (process control) from the University of Alberta.