WEB SERVICES TO SOA-1: How BP’s E&P unit navigated the IT vision and chasm

Oct. 8, 2007
Our strategy is to automate BP’s business processes so that the organization can achieve its goals faster, smarter, and more efficiently than the competition.

Our strategy is to automate BP’s business processes so that the organization can achieve its goals faster, smarter, and more efficiently than the competition.

Until now, we have done this by developing and testing dedicated applications each time the business has wanted to create, modify, or improve a process. The time and cost involved in this has meant that, at times, computerization has slowed rather than facilitated BP’s ability to capitalize swiftly on market developments.

This is about to change. A new approach to software development will help us to react much more quickly and cost effectively to new market opportunities, changes in business climate, and new regulation. It is called service-oriented architecture (SOA) and, in simple terms, it breaks applications into a series of “services,” each designed to perform a specific task. By joining these services together-like links in a chain-we create a complete application.

The beauty of this approach is, when the business wants to introduce, change, or improve a process, often we can simply adapt, reconfigure, and resequence our existing services. When we do need to bring in new software, this can be taken off the shelf, making it more cost-effective and faster to implement. So, in a world where companies need to adapt quickly and costs are a constant challenge, SOA offers us an exciting opportunity.

The principles of SOA are not new to all parts of our organization. We are already starting to use this approach in IT projects in our business units. What is new, though, is the move to use SOA more broadly, at an enterprise-wide level. BP is now pursuing a transformational program to realize the potential of SOA and enable BP to achieve new levels of efficiency, speed to market, and, ultimately, to optimize our business, both financially and operationally.

Introduction

IT professionals can’t pick up an industry publication these days without seeing something about web services or service-oriented architectures (SOA).

We have all heard intriguing stories about the vision and value of SOA, and even Hollywood seems to have heard. In the movie “Firewall,” a computer security specialist played by Harrison Ford can be seen reading a journal on SOA and web services. There is a good reason for all this hype.

SOA is an approach to delivering business solutions through services (capabilities) that are linked together by business logic. This approach reflects how a business actually operates compared to conventional application development methods. As a result, the relationship between IT and the business is transformed from consumer/supplier to collaborating partners.

IT groups in many organizations have begun experimenting with web services as the interoperability standard from which to build the future. They may have undertaken a few pilot projects to gain a sense of the challenges, costs, and potential benefits of deploying web services more widely. They may even have begun adapting, or “wrapping,” some of their legacy applications and data stores to use web service interfaces, based on what they’re learning from pilot testing.

In many organizations, enthusiasm is growing about the possibility of moving toward a true SOA-the next-generation IT infrastructure built around web services, and designed to support highly flexible, increasingly competitive and cost-effective business process optimization.

BP started the implementation of web services to solve point-integration problems in the organization in late 2003. Based on early successes, the vision of a fully service-oriented architecture was embraced by the organization, and this article, offered in two parts, describes the approach that BP has adopted to deliver this vision.

Part 1 outlines the vision and key business drivers of SOA, as BP understands them, and introduces a maturity model that can be used to structure the institutionalization of SOA concepts.

Part 2 examines the detailed, multilevel, standards-driven maturity model that will enable BP to plan, track, and measure the evolution of its growing organizational SOA capability.

Edge of the canyon

With both a huge marketing push from the major IT vendors (Microsoft, IBM, Oracle, BEA, etc.) and a realization that SOA is an approach that can deliver real value to the business and not just the “next big thing,” many organizations are preparing to embrace SOA as their architecture of choice.

Just as they’re starting to pick up speed on the path towards SOA, IT groups in many organizations encounter something like the edge of the Grand Canyon. There is an enormous, seemingly unbridgeable gulf between what they are doing at a “micro” or project level and what they realize they need to do at the “macro” corporate or enterprise level in order for SOA to be more than just a tantalizing idea.

All they’ve done so far is take a few baby steps, but the next step will require either a giant, very risky leap of faith (Fig. 1), or they’ll need some extremely pragmatic ideas on how to cross that chasm without falling headlong into an abyss.

In the absence of a strong service-oriented architecture vision and clear roadmap, getting from simple web services projects to a full institutional SOA capability is like taking a giant, very risky leap of faith across the Grand Canyon (Fig. 1). Photo from http://www.the-rocketman.com/Dar-R-gallery.html)
Click here to enlarge image

The problem is there doesn’t seem to be many straightforward, proactive and vendor-neutral roadmaps from here to the other side. There are, of course, numerous technology developers ready to offer an array of shiny new products, and consulting firms poised to insert teams of experts to study the problem and make customized recommendations for every step of the way.

Everyone seems to have an agenda, but it’s never quite clear, despite all the arm-waving and glossy brochures, how anyone will ever achieve the SOA vision without, well, something a bit more definitive.

This is precisely where BP’s Exploration & Production IT organization found itself more than a year ago. The digital communications and technology function had successfully tested web services by adapting the organization’s legacy systems to deliver real-time operational data and were seeing terrific bottom-line potential (see sidebar, p. 43).

This early success provided justification for the next step: the pursuit of a coherent SOA strategy for the whole E&P segment of the business.

The newly formed SOA team read a plethora of industry papers, researched options, and talked with vendors of SOA products and services. That’s when the team nearly drove off the cliff-there were no bridges to the other side.

Building the bridge to SOA

As a result, the team realized that they would have to generate their own blueprints for how to bridge that gap between the small, scattered pilot projects and a global, enterprise-wide SOA capability (beyond even the initial E&P segment).

This two-part article presents a simplified set of those blueprints, which are already in use within as the basis for the organization’s transition to a service-oriented enterprise.

BP believes that three qualities make this model worth considering by other corporations that have arrived at the edge of the cliff:

  1. First, it’s totally vendor-independent. There are no strings attached to any name brands. Any relevant technology component can be adopted, as long as it complies with emerging industry standards.
  2. Second, it’s eminently actionable. BP, one of the largest oil and gas companies, is purposefully implementing the plan now. There’s nothing vague or fuzzy about it.
  3. Third, it’s reassuringly incremental. No “big bang” here. No leap of faith, small or large. Just one, highly measurable step at a time toward a revolutionary new way for IT to collaborate with business. As BP’s Vice-Pres. of Enterprise Architecture Jim Ginsburgh so aptly stated, “Large-scale transformational change is best undertaken through small steps toward a common and compelling vision.”

We offer the following intellectual capital for use by companies and IT groups within organizations in any industry that have been struggling with how to cross the chasm from simple IT experiments to the full organizational SOA vision.

What’s on the other side?

What exactly is a service-oriented architecture?

For starters, while XML-based web services (see sidebar, p. 44) are a critical element of any SOA and have made SOA a reality, successful implementation of web services does not make a service-oriented architecture. We’ve encountered some misunderstanding along these lines from both our development teams and major vendors.

While there has been work done by the standards bodies such as OASIS (www.oasis-open.org), there does not yet seem to be a single definition in the industry that organizations can rally around.

SOA: a working definition

In BP’s view, SOA is a transformational approach to the design, implementation, and management of business solutions and their supporting technical infrastructures.

SOA is more than technology or even a collection of technologies. It’s an architectural style that creates new business applications through the intelligent “orchestration” of discrete, reusable business functions called “services” (Fig. 2), each of which performs a single, well-defined task (such as getting a stock quote or delivering real-time operations data).

Click here to enlarge image

Business solutions, in this new paradigm, are “composite applications” consisting of standard services linked together with business logic and standard web service connections. Unlike traditional monolithic software applications, which reflect current (even outdated) processes, a suite of component services can be rapidly rearranged and-or extended to reflect new business strategies and evolving market conditions.

Ultimately, therefore, an SOA environment will abstract the application engineering process to a higher level, requiring little or no programming as we know it to adapt to changing business requirements.

In a conceptual model of an SOA (Fig. 3), users of a composite business application leverage a common interface layer, which provides access to standard business process modeling and orchestration tools, a common set of generic SOA functions (including security, management, and governance of services), and a repository of specific business services they can work with-including component services provided by external vendors, and legacy internal applications “wrapped” with a standard interface to look and act like any other service.

Click here to enlarge image

Unlike the project-centric approach of simple web services and conventional application solutions, a successful SOA strategy requires corporate investment in common infrastructure components and the definition and operation of relevant governance processes to encourage and reward the development and reuse of services and processes across the organization.

A major point worth underscoring is that SOA will fundamentally alter the historical relationship between IT and the operating units, making the implementation of solutions more of a partnership between business and IT rather than the traditional model where requirements are “thrown over the fence” to be implemented by the technologists.

With this approach, business users who are not programmers can sit down with IT staff and collaboratively develop the composite applications they need to run their business more efficiently and competitively. No more waiting a year or more and spending unpredictable amounts of money for a monolithic, one-off solution that may or may not meet the original need, given the technical constraints, dependencies, and time lag. In an SOA environment, business units take real ownership of the business process and the change management of that process, thereby maintaining direct control of their IT spend.

The IT organization, in turn, provides common services and the relevant infrastructure functions that enable the efficient creation, location, and reuse of these services by authorized internal and external subscribers and partners.

IT, therefore, becomes more of an accelerator of business change, rather than a supplier or, as is often seen, an impediment.

Drivers and benefits

Once a global organization has a sufficient library of services available, almost any business process can be orchestrated without having to write new code.

What’s more, new and better services can be swapped out for old ones without causing a ripple in the business workflow. As such, SOA provides an almost unimaginable degree of business agility. This is one of the top reasons why SOA is gaining so much momentum, and why a growing number of IT industry leaders are actively moving in this direction.

BP believes that the key business drivers and benefits of our SOA vision include:

  • Better aligning IT with the business-especially in decentralized organizations, like BP-enabling more collaborative approaches to incremental change in both business process and technical infrastructure.
  • Responding and adapting to changing business conditions more rapidly by automating and optimizing business processes, lowering both costs and cycle times along the way. According to early industry adopters, SOA should yield time and cost benefits of 50% or more; one company reports that no projects last longer than 6 weeks as they have a full set of services to build upon.
  • Preserving huge investments in legacy applications by exposing their functionality to evolving SOA business processes through standard web service interfaces. Industry pundits claim that a typical organization fully utilizing SOA development techniques will achieve return on investment of more than five to one.
  • Efficiently reusing existing software assets-long the Holy Grail of the IT world-by decomposing hard-wired, traditional applications into “loosely coupled” services. In our experience, the first time a service is reused, it pays for itself. Additionally, there are reports of doubling savings when a formal reuse program (with incentives) is put in place versus just ad hoc reuse of services.
  • Maximizing the value of commodity software, as more vendors adopt SOA for their own product development process and increasingly deliver software in the form of more granular “native” services, rather than monolithic applications that must be adapted to act like services. For example, one major SAP user slashed its annual operating cost by 60% and its application integration time by 75% by using SAP’s new SOA environment.
  • Extending the virtual enterprise, enabling companies to seamlessly integrate services across traditional corporate boundaries as more of their suppliers and partners embrace the same international SOA standards.

Coordinated SOA program

Clearly, SOA is not just about technology.

It can be easy for IT professionals to get sidetracked by all the fascinating technology issues, overlooking the much more challenging communication, culture change, business process, and governance issues essential to achieving the SOA vision.

A coordinated program is necessary, therefore to migrate an organization step by step toward this revolutionary new way of doing business. Basic elements of a corporate SOA program ought to include the following characteristics in some form:

1. A shared vision and understanding of direction.

If SOA is to be a true collaboration between IT and the business, all parties must understand what direction the company is going and why, including the expected benefits and business value to everyone involved. Part 1 of this article is a summary of BP’s SOA vision.

2. A roadmap outlining concrete steps toward the SOA vision.

To plan an organizational transition to SOA, a fairly prescriptive technical architecture and operational infrastructure must be defined and understood internally. The SOA roadmap should outline a natural sequence of activities, best practices, and dependencies-including the use and interpretation of emerging, competing, and established standards-to ensure interoperability, accelerate adoption, and “future-proof” the SOA design. Part 2 of this article, a discussion of BP’s SOA maturity model, describes this in some detail.

3. Definition and support of new organizational roles, processes, and governance.

Traditional organization and governance processes can actually block movement toward SOA. Many aspects of building, operating, and maintaining composite applications based on a library of shared “services” fall outside of traditional IT skill sets, which are usually aimed at developing and supporting client-server applications and-or physical infrastructures. Sharing and reuse of digital assets are essentially alien both to internal organizations as well as to external partners and application vendors. To be clear: SOA is not a natural evolution of current behaviors. Thus, version control, change management, shared services governance, and formal reuse initiatives must be defined, incentivized, and carefully managed to ensure success.

4. Measurement of service quality and value.

Often, IT metrics focus only on the IT aspect of a business solution, such as server availability. SOA standards, on the other hand, enable measurement of actual service quality, which has real meaning to business subscribers as well. Quality of Service (QoS) expectations and appropriate service level agreements (SLAs) need to be explicitly defined for various types of business services, as well as the monitoring and reporting of “new” service metrics such as “number of reuse instance.” Only by tracking such metrics can IT and the business begin to determine the true value of SOA in action.

5. Consistent, multilevel communications.

To ensure that all parties involved actually understand the vision and SOA migration process, communication programs must be created to address both IT and the business, and delivered consistently using a variety of channels.

Blueprint for the bridge

Having established a common and compelling SOA vision, BP set out to define a “maturity model” that is both practical and proactive.

In other words, it doesn’t succumb to the vague descriptions typical of so many transformational change programs (which tend to lose clarity and purpose, especially at the higher levels). In fact, BP’s model describes the detailed standards, tools, and governance processes required at each level of the model and doesn’t just rely on vague statements that a given level is about some undefined goal of “business transformation.”

Neither is the BP model a knee-jerk reaction to what the market may be doing or saying about SOA. Considerable research, thought, discussion, and iteration went into the development of BP’s SOA Maturity Model (Fig. 4), which we’re convinced will enable deployment of SOA in clear, rigorously-defined, incremental steps.

Click here to enlarge image

The five levels defined in BP’s model provide a conservative and comprehensive framework with guidelines for managers to assess the evolving SOA capabilities of staff, projects, a segment of the business, the whole global organization-even partners and vendors.

This model is based on the Carnegie Mellon Software Engineering Institute’s Capability Maturity Model (CMM) (see sidebar, p. 46). The CMM suggests that corporations can achieve progressive improvements in their organizational maturity through “evolutionary steps rather than revolutionary innovations.”

In the BP model, as in the CMM, each maturity level forms a necessary foundation on which the next level is built. Levels, therefore, cannot be skipped without risking failure or unnecessary waste of time and money.

Without systematically defining the processes, implementing the standards, and adopting the common services required at lower levels, individual projects attempting to move up too quickly may have to rip and replace some or all of their work as the broader infrastructure matures.

If, for example, a project tries to jump to Level Three (Process Orchestration) without access to a sufficient library of reusable, standards-based services, they will be forced to go back to Level Two and buy or build them, whether they like it or not.

The SOA Maturity Model is an organizational model. SOA principles must be institutionalized so that all projects in the organization embrace the concepts. Without wide-ranging adoption, the catalogue of available services and the related level of reuse will not reach an optimum level.

BP also believes that the CMM is well suited to adaptation for SOA because of its emphasis on discipline, repeatability, and consistency. If an IT organization lacks rigor and maturity in conventional software engineering, it is unlikely that it can implement SOA very effectively. Our sense is that a group or company that has achieved CMM Level Two will be able to move more quickly to SOA Level Two. However, an organization with poor project management controls will struggle to achieve even basic levels of SOA as the organizational maturity will not support the level of reuse and sharing that a successful SOA requires.

While others have applied the CMM to SOA, two aspects distinguish BP’s approach. First, there is no vendor “slant” put on the model. Second, there is a lot more substance than the usual “high level” SOA roadmap, which may look more like an academic exercise.

Deploying the model

During 2006, BP rolled out SOA Level Two across the E&P segment of the business, with a handful of more advanced projects developing Level Three capabilities.

The RTAP project, described in the sidebar on p. 43, is an example of a Level Two reusable service generating value in the organization today. A significant number of additional services are already in production based on the standards defined for Level Two.

No large company, as far as we know, has fully achieved Levels Three, Four, or Five today. This is mainly because SOA is so young that certain required standards and toolsets are still emerging.

The Maturity Model is built to anticipate that and expects an incremental evolution, not only of the organization’s capabilities, but also of the availability of standards in the industry. For example, the Service Monitoring and Metrics required to achieve Level 4 can be implemented today using proprietary tools, but an industry standard to enable this will only be available in the medium term future-likely after many organizations have managed to develop and embed the first three levels.

Just as individual projects cannot get very far ahead of the organization as a whole, individual corporations cannot get ahead of the IT industry as a whole. This is why it’s critical that innovators, early adopters, IT technology suppliers, and thought leaders-particularly those operating within a given industry vertical-work together toward achieving the SOA vision, and establishing the necessary industry standards to make it happen.

BP is participating in this sharing of expertise by offering both our Maturity Model and specific elements of the SOA standards documents, such as the Interoperability Standards from Level 2, and making them available to other operators and vendors in the E&P industry.

Conclusion

As Albert Einstein said, “We can’t solve problems by using the same kind of thinking we used when we created them.” It has become apparent to many of us that we’ll never solve the problems created by some 40 years of monolithic software engineering by using the same kind of thinking. SOA is a totally different kind of thinking.

The SOA vision is clear. The technologies and standards are evolving rapidly enough to enable enterprise-wide adoption. Perhaps the biggest risk is not in attempting to cross the chasm but in sitting on the wrong side just because it’s more familiar. SOA services and solutions will soon become ubiquitous. If we try to deploy them in the same way we deploy current technology, we’ll never reap the benefits outlined above. We’ll just implement the same chaotic and complex environment with new technologies, without solving the problems that the business sees every day.

SOA will not happen on its own. And it certainly won’t happen in a single leap. Transitioning from an IT environment with, at most, ad hoc web services projects to a fully institutionalized SOA capability requires a coordinated and comprehensive sequence of activities. The definition of the appropriate activities and their logical sequence is the core of the SOA Maturity Model that BP has defined as its roadmap for the next several years.

Next: Part 2, Oct. 15, 2007.

Further reading

In the second part of this article, the discrete maturity levels are discussed in detail. Part 2 examines the standards that must be defined, the potential product types that must be selected and implemented, and the governance processes that must be established for an organization to adopt each of the defined maturity levels of BP’s SOA model.