EIA completes corrections to drilling estimates series

William Trapmann, Phil Shambaugh
Energy Information Administration
Washington, D.C.

The Energy Information Administration (EIA) has published monthly and annual estimates of U.S. oil and gas drilling activity since 1978.

These data are key information for many industry analysts, serving as a leading indicator of trends in the industry and a barometer of general industry status. They are assessed directly for trends, as well as in combination with other measures to assess the productivity and profitability of upstream industry operations. They are major reference points for federal and state policymakers. Private sector users include financial analysts assessing investment opportunities. Firms with upstream operations also rely on these data in apprising their circumstances relative to those of their competitors. EIA uses these data in its own analytical and modeling work.

EIA does not itself collect drilling activity data. Instead, it relies on an external source for data on oil, gas, and dry well completions. These data are provided to EIA monthly on an "as reported" basis.

Due to lags in the reporting of well completions which can (though most do not) range up to several years, EIA must statistically adjust the "as reported" completion data to obtain estimates of the numbers of completions that would have been reported had there been no reporting lags. Essentially, this is done by assuming that the pattern of reporting lags observed in the past holds true for the present and making appropriate upward adjustments to the reported numbers of completions on that basis.

Discrepancies detected

As an integral part of its data gathering function, EIA routinely monitors data quality and periodically conducts work intended to enhance its data systems. During a recent effort to enhance EIA's well completion data system, the detection of unusual patterns in the well completion data as received led to an expanded examination of these data.

Substantial discrepancies between the data as received by EIA and correct record counts since 1987 were identified. For total wells by year, the errors ranged up to more than 2,300 wells, 11% of the 1995 total, and the impact of these errors extended backward in time to at least the early 1980s.

When the magnitude and extent of the "as reported" well completion data problem were confirmed, EIA suspended its publication and distribution of updated drilling data. EIA staff proceeded to acquire replacement files with the "as reported" records and then revise the statistical portion of its drilling data system to reflect the new information.

The replacement files unfortunately also included erroneous data based on the improper allocation of wells between exploration and development. Users of EIA drilling activity data therefore are advised that the drilling activity data that were published or otherwise distributed by EIA prior to August 1998 are not necessarily valid.

Two problems resolved

EIA has now resolved the two data problems and generated revised time series estimates for well completions and footage drilled.

While most industry trends remain consistent with those of the initial, incorrect series, the revised series does exhibit certain differences, chief among which are:

• Total well counts by year in the initial and revised series vary by less than 0.5% until 1996, when the difference is 1.3%.
• Drilling activity did attain its peak level in 1981, but the industry completed an estimated 91,553 wells as opposed to the prior estimate of 90,034.
• The decline in drilling during the mid 1990s was not as steep as previously indicated. The wells in 1995 had been underestimated by 2,384 wells-a difference of 11%.
• Success rates, measured as the share of successful gas and oil wells relative to total wells, are similar in the initial and final drilling series, but the improvement in the mid 1990s was not as great as previously indicated.

The remainder of this report presents background on the drilling activity data: what the records are, how they are collected, and the resulting difficulties in developing timely measures of recent drilling activity. This is followed by a discussion of the nature and extent of errors in the raw data files received by EIA. Last, the revised data are presented along with key differences between the prior and revised series and their implications for understanding industry performance.

Drilling activity data

The most widely cited measures of drilling activity consist of summarized information based on individual well records that describe the completion type and status of each well.

The individual records contain information regarding the well, including the American Petroleum Institute (API) well number, well completion date, well class (exploratory or developmental), well type (oil, gas, or dry), location data, and measures of the footage drilled.

Drilling data traditionally had been compiled and presented as the records are received, a practice which predates EIA's publication and use of these data. (EIA continued this practice as part of its data operation until the mid 1980s.) This approach, however, reflects the reporting activity as measured by the recipient, rather than the industry's activity in the field.

Well completion data by report date ("as-received completions") are not an accurate indicator of actual drilling activity. A preferred measure of drilling activity is a record of well counts and footage drilled by completion date. The completion date marks the point at which the well generally becomes available for production.

Drilling measured by completion date is thus more appropriate for industry analysis purposes than drilling measured by report date. Well counts by report date would match counts by completion date if wells were reported with no delay; however, that is not the case.

In the early 1980s, EIA staff noticed unusual patterns in the as-received completion data, which reflected distortions due to a variable and sometimes very extended reporting lag.

The reporting lag not only creates incorrect magnitudes for peaks, troughs, or changes in drilling activity, it also tends to obscure the timing of these events. The lags are particularly troublesome at times when the drilling trends shift.

The historical data provide some especially clear examples of the distortions that can be caused by recording wells by report date. Wells aggregated by completion date now show that total completions peaked in 1981 at 91,553 and had fallen to 84,397 by 1982 (Fig. 1 [51,714 bytes]). This pattern reflects the effect of oil price levels on drilling activity and is consistent with the data on rotary rigs running, both of which peaked in 1981.

By comparison, total well counts as reported in 1981 and 1982 were 78,538 and 85,795, respectively.1 The higher count in 1982 reflects an influx of reports of drilling activity that occurred in 1981 and earlier years. The reporting lags were sufficient to cause the 1981 count to be off by roughly 13,000 wells, a 14% discrepancy, with the result that the wrong year would be indicated as the peak year for drilling activity.

Another critical example of reporting lag distortion is the measure of drilling in 1986, when the industry suffered a collapse in oil and gas prices. The number of wells actually completed in 1986 was about 40,000, while the number of well completion records received was more than 58,000.

Reliance on the as-received well counts would have significantly masked the serious impact of the price decline on the industry. By that time, however, EIA had taken steps to convert its drilling activity series to an as completed basis, which avoided this problem.

Adjusting for the reporting lag

Concerns about drilling activity data arose at EIA in the early 1980s. These concerns resulted in a plan to develop a procedure that would allow aggregation and reporting of the data on a completion date basis.

As noted earlier, EIA does not collect well completion data directly but relies on a vendor to collect the data, which EIA purchases as a monthly compilation. These are the only such data available, so attempts to convert to a completion date basis must utilize these data.

Further, the quality of the EIA's drilling activity estimates depends directly upon the quality of the vendor records as received.

The conversion to completion date required the development of a statistical model to adjust the as-received completion data for incomplete counts due to the time lags between completion and reporting. The reporting lags result in cumulative as-received well counts that are quite incomplete for most recent months.

For example, records received in 1995 show that only 16% of the wells were reported in the same month that they are completed. More than 13% of the well completion records received were for wells completed more than 12 months prior. In fact, 5.5% of the records were for well completions more than 5 years old (Fig. 2 [38,898 bytes]). These reporting lags affect each of the six subcategories (well type by well class), although the effects vary among the subcategories.

EIA's efforts to convert the well completion statistics from a report date to a completion date basis resulted in the Well Completion Estimation Procedure (WELCOM). WELCOM is a system that summarizes the historical records and produces estimates for drilling activity in the most recent years. Estimates are monthly at a national level for six major categories: oil-exploratory, oil-development, gas-exploratory, gas-development, dry-exploratory, and dry-development.

Until 1982, the files received included few, if any, wells completed more than 5 years earlier. Thus, WELCOM used the cumulative recorded well counts, along with data on rotary rigs running, to construct estimates of the actual numbers of wells completions, and footage drilled, during the most recent 60 months of activity.

EIA has used WELCOM since March 1985 to provide the drilling activity estimates published in the Monthly Energy Review and the Annual Energy Review, as well as other EIA reports. The same drilling data and estimates have also been a part of the foundation underlying numerous analytical efforts including the Annual Energy Outlook, the Short-Term Energy Outlook, Performance Profiles of Major Energy Producers, and Natural Gas Issues and Trends.

Problems in the data

Over the years, EIA's operation of WELCOM has included annual re-estimation of the system coefficients as well as minor adjustments to the overall implementation of the methodology.

The basic system remains essentially the same as that which was developed in 1984. Since then, the patterns and attributes of drilling have evolved as a result of various regulatory reform initiatives and the shift to generally lower oil and gas prices since the mid 1980s.

The collection and processing system for the well records has itself changed. Other changes, such as the fact that the data for active rigs now identifies them by whether they are targeted to oil or gas, have offered opportunities to refine the estimation procedure and enhance the precision of the estimates.

A project to enhance the WELCOM procedure was undertaken by EIA in 1997. Its goal was to take maximum advantage of available data, modify the model specification as appropriate, and test alternate statistical approaches to the estimation of the model's coefficients. The associated data work had an unintended impact, however, when curious patterns in the reported well counts were noticed.

Well counts can be and often are highly variable between months. Reported well counts for some months in 1995, however, were about half the counts of the prior and succeeding months.

Collaborative examination of the data with the vendor verified the existence of errors and omissions in the data files provided to EIA in these periods. Review of other time periods disclosed errors that were pervasive in the data over an extensive period. The data files since 1987 were missing some records and contained duplicates of others, updates to many records were not passed along, and records for recompletions-which were not expected-were present in files for some months but not in others.

A comparison of the well counts as initially received and the replacement files for 1990 to mid-1997 shows only slight discrepancies, with few exceptions, for the years immediately prior to 1995 when a significant number of records were not received (Fig. 3 [39,640 bytes]).

Attempted resolution of this problem required acquisition of replacement files from the vendor, which unfortunately contained data with an improper allocation of wells between exploration and development. The preferred well classification is based on the final designation of wells as either exploratory or developmental. The replacement records were identified based on the initial well class.

The major impact of this slip is that wells initially classified as exploratory should have been classified as developmental, and this occurs more often for successful oil and gas wells than it does for dry holes. The resulting measures reflected exploratory well counts that were too high, developmental wells that were too low, and the oil and gas well counts were distorted more than those for dry wells. Recognition of the second problem occurred after revised well counts had been published for the five months, February to June 1998.

Updated drilling data were not published in July 1998, while EIA staff acquired corrected files from the vendor and re-estimated the WELCOM coefficients. This effort was accomplished in time for EIA to resume publication of drilling activity data in August 1998. The revised series exhibit certain key differences from the prior series, which may alter the perception and understanding of the industry.

Comparative assessment

Differences between the series

The impact of flawed data files is not limited to the period in which they were received. Due to the extensive reporting lag, many of the records report on much earlier months, affecting earlier records also.

For example, even though 1981 is outside the period in which problem data files were received, drilling peaked in that year at 91,553 wells as opposed to the previous measure of 90,034 (Table 1 [283,017 bytes]).

Another significant difference between the two series arises in the drilling pattern of the mid 1990s. The estimates based on files with missing records showed that the decline in oil and gas prices in 1993 to 1995 contributed to annual well completion declines of 17% in 1994, then 11% in 1995 to a level of only 18,655 total completions in 1995.

The 1996 price surge seemed to drive a 22% rise in drilling activity to 22,740 wells. The revised drilling statistics, however, show that the relative worsening of economic conditions resulted in a decline in wells completed from 24,665 to 21,507, followed by a further decline to 21,039 in 1995.

The revised series shows 2,384 more wells were drilled in 1995 than previously estimated. The smaller decline in 1995 wells also results in a smaller relative recovery in response to the rise in both oil and gas prices in 1996, 8% in contrast to the 22% change indicated by the prior series. The 1993-95 decline of 15% in the corrected data is much less than the 26% falloff seen in the initial well series.

Thus, it appears that the initial estimates overstated the industry reaction in both directions. This casts a much different light on the industry's responsiveness to the economics of the time.

In addition to changes in total well counts, the revised data series contains fewer oil and gas exploratory well completions in recent years than previously estimated, distorting the implied success rates.

The shift in recent successful well counts exceeds the corresponding differences in dry holes, so the associated success rates do not increase as much as previously indicated.

According to both series, the 1979 peak success rate of 30% was almost matched by 1993 and was surpassed in the following year. Thereafter, the initial and revised series exhibit very different trends in exploratory success. In the initial series, success rates increased strongly from 1993, reaching 45% in 1996. The final revised series shows improved industry performance in the mid 1990s, but to a lesser degree, with success rates after 1993 ranging between 32 and 35%. Thus, the corrected drilling data provide a different picture of industry performance and the investment incentives for firms engaged in exploratory activities (Fig. 4 [37,492 bytes]).

The trends in success rates in both the initial and final series stand in stark contrast to the trend indicated by the exploratory drilling estimates published by EIA from February through June 1998. This intermediate series exhibited sizable overestimates of the oil and gas exploratory wells for an extended period beginning in the mid 1980s.

The mistakenly high successful well counts caused an excessively high exploratory success rate beginning in 1985 and exceeding the 40% level from 1989 forward. Increases in success rates of such a magnitude and at such an early date suggested by these data would be grossly misleading for any analyst relying on these measures. The surge in the trend coincides with the virtual collapse of oil and gas prices. Interpretation of these trends and conditions would likely distort one's perspective of the industry capability to react to changes in economic conditions.

Comparison of the final corrected drilling activity series with the initial erroneous series shows that the overall trends are comparable, with a few significant exceptions. Most aspects of the drilling series are consistent between the initial and final drilling activity data series. However, the implications of the differences in drilling levels for certain years and the new values for exploratory success rates are fundamental to an understanding of the industry.

Implications of the data shift

The significance of the changes is likely to depend on the particular use of the data. EIA itself uses these data to support analysis integral to two prominent information products: the Annual Energy Outlook (AEO) and the Short-Term Energy Outlook.

The drilling data are specifically used in the National Energy Modeling System (NEMS), which is the tool used to produce the integrated energy market projections that are published in the AEO.

The productivity of exploratory drilling is represented in the NEMS by finding rates, which are measured as the ratio of oil and gas reserve additions to wells completed. The downward shift in the exploratory well counts lowers the finding rate denominator, so the finding rate will shift upward correspondingly.

This productivity increase would raise the available market supplies in future years, all else being equal. However, the corresponding analysis that determines the level of drilling must also be revised, because it is predicated on a high exploratory well count. Thus, the expected drilling response under varying conditions is overestimated in the projection because it is calibrated to an inaccurately high benchmark.

When these functional relations are re-estimated, drilling response should be less in the future than previously expected. Given that the changes in exploratory drilling primarily affect only the last few years of data, the net result of these two offsetting influences on the outlook is likely to be small.

The long-term supply outlook is also influenced by the degree of success in the search for oil and gas. The lower success rates affect the economic attractiveness of exploratory drilling opportunities. Dry holes represent an unavoidable part of the search for oil and gas that add to project cost, lowering the expected profitability of exploratory projects.

Lower average success rates imply that the total drilling costs of an average exploration project should be greater, which suggests that the economically recoverable portion of the technically recoverable resource base may not be as large as previously anticipated. The volumetric impact of this shift should be limited, however, because this change will mostly impact oil and gas accumulations that are marginal in size anyway. The economic merit of massive fields that provide the bulk of expected market supplies should remain largely unaffected.

Maintaining future data quality

Although EIA conducts a wide range of data quality activities to monitor its data systems, the long-lived problems affecting the drilling data series maintained by EIA are a clear indication that further efforts in this regard are necessary.

In a typical EIA data system, ready detection of large discrepancies would be virtually certain; however, the well completion data system has a number of attributes that hinder detection.

First, the work with the raw data and all initial processing are conducted outside EIA. Such an 'arms-length' relationship with the basic data does not facilitate familiarity with the data, and it hampers efforts to investigate concerns about the data.

Additionally, the well completion data are inherently quite erratic, so even sizable fluctuations are not prima facie a cause for concern.

Yet another factor that impedes ready detection of data problems is the statistically-based processing of the data by EIA. The limited numbers of wells reported early after completion are 'inflated' by large factors, which tend to obscure data problems.

Last-and arguably paramount-there is no other timely source of information that can be used to validate the data going into or the set of estimates coming out of WELCOM.

Development of new approaches for quality assurance will proceed along a number of paths including closer monitoring of the as-received well completion data and comparison of estimated drilling data by completion month to other data that are available on a timely basis.

An indication of the difficulty in establishing any definitive set of guidelines can be seen in the data for wells reported each month. While the well counts in the majority of months vary by 20% or less, fluctuations exceeding 30% are common. Even relative shifts beyond 40% are frequent enough to not be necessarily considered a cause for alarm (Fig. 5 [49,003 bytes]).

EIA is working to more closely monitor the drilling data through collaboration with its counterparts at the API. API receives drilling data records from the same vendor as EIA and also generates statistically adjusted estimates for drilling activity by completion date.

EIA and API have been exchanging their latest estimates as soon as available to facilitate comparative analysis. In addition to ongoing communication, API and EIA intend to meet periodically to review the status of data and the estimation process. More frequent working meetings will be held on an as-needed basis.

While EIA and API may often encounter similar pitfalls in their drilling data efforts, they receive the raw drilling records according to their own specifications and on different reporting cycles. The data files as received are not generated by the same process by the vendor, so future data problems may become apparent early as discrepancies in the data files or estimates are noted.

Drilling-related data are used to evaluate the estimates coming from WELCOM. These data include rotary rigs running, seismic crews, and oil and gas prices. EIA staff is also considering the possibility of obtaining relevant data that may exist at the state level in the major producing states to corroborate the EIA estimates. Drilling-related measures from the major states, while incomplete in themselves, could prove to be useful as a benchmark, since most drilling occurs in a limited number of states.

These enhanced monitoring actions will help to avoid future data problems that might require extensive revision of the series.

Conclusions

EIA has inadvertently been publishing erroneous estimates of oil and gas well drilling activity since the late 1980s. Problems in the raw data obtained from a vendor were detected in late 1997 and again in mid 1998 and have since been resolved.

Monthly publication of the corrected EIA drilling activity estimates series resumed with the August 1998 edition of the Monthly Energy Review. The major characteristics of the corrected series are comparable in most regards with the initial erroneous series.

There are, however, a few exceptions which directly bear on fundamental aspects of the industry such as its responses to changing economic conditions and its approach to drilling opportunities.

EIA does not collect the raw data itself, so some data errors may remain extremely difficult to discover, identify, and remedy in a timely manner. However, EIA will make every possible effort to ensure that the same data problem, or a data problem of similar magnitude, does not happen again.

Reference

1. Monthly Energy Review, EIA (DOE/EIA-0035), January 1984.

The Authors