The New World of Data Science and Digital Engineering: the Hype, the Hope, and the Reality

Welcome to SPE’s newest online journal, Data Science and Digital Engineering in Upstream Oil and Gas. My name is Jim Crompton, and I am the director of Reflections Data Consulting and an adjunct teaching faculty member in the petroleum engineering department at the Colorado School of Mines. I am on the editorial board of this new publication and drew the first straw for writing a guest editorial. I chose the title “the Hype, the Hope, and the Reality” to provide my perspective on the exciting new themes of digital transformation of the oil and gas industry (“Digital Oilfield 2.0,” as some experts like to call it).


First the hope: Rising production from North American shale producers (mainly in the Permian Basin of West Texas) as well as new production from the Organization of the Petroleum Exporting Countries and contributions from several basins around the world provide welcome headlines, but unclear demand trends, key infrastructure constraints (e.g., lack of pipeline space and liquefied natural gas export facilities), and geopolitical sanctions are creating an uncertain price future. While the short term looks better, the long term is unclear for today’s oil producers. The answer for many seems to be to count on greater productivity. “I want to grow my production but not grow my staff or increase my operating and capital costs,” is the new mantra for the industry.

“More for less” is not a new theme, but the answer to how to accomplish that business plan lies with the effect of emerging technology, automation, and greater insight from the data we are collecting by using advanced analytical techniques. Many companies have created corporate and divisional centers of analytics, bringing together newly hired data scientists, domain experts, and data-handling support staff. The good news is that we are seeing some interesting developments (at least the ones that companies are willing to share),  including a well designed by artificial intelligence (AI) algorithms in the Permian Basin by Shell as art of its comprehensive iShale program and “pump by exception” techniques, in which an AI algorithm helps improve the productivity of artificial lift in North Dakota, a partnership with Equinor and the AI company Ambyint.

Many companies are using AI and machine learning techniques to deploy predictive maintenance approaches to critical oilfield equipment by taking data from process control systems (supervisory control and data acquisition systems) and getting more useful life and uptime from existing assets and lowering operational expenditures. Other operators are finding ways to visit the wellheads and production pads less often through better logistics (taking a lesson from FedEx) and using data from the field for remote surveillance. Cheaper drilling, more-effective completions, and lower production costs fit right into the “produce more for the same costs” assumptions coming from the C-suite message to the shareholders and bankers.

Now for the hype, and there is plenty of this: First, to take a step back, it is important to recognize the growth of the digital economy, and I am not just talking about Uber and Airbnb. We are getting tired of hearing about that comparison. In a recent article in The Economist, the picture of the effect of high-tech firms falls into perspective.

“Rarely in stock market history have so many investors made so much money from so few shares going up for so long. Some 37% of the rise in the value of all firms in the S&P 500 Index since 2013 is explained by six of its members (Alphabet, Amazon, Apple, Facebook, Microsoft, and Netflix). About 28% of the rise in the Chinese equities over the same period is owing to two firms (Alibaba and Tencent). Managers and investors have bought into a tale of effortless disruption by an elite of firms led by the world’s brainiest people” (Schumpeter 2018).

Our industry executives and tech managers are not immune to this opportunity and promise. Management consultants constantly berate our industry for slow adoption of digital technology. Oil and gas often is rated as a laggard and needs to get on the bandwagon before we are left behind. Hundreds of high-tech vendors are knocking on our doors with new technology and new promises of solving all our problems, even before they know what they are. Trips to Silicon Valley to worship at the feet of Google and others is a mandatory step in the development of business plans and in management development plans. Corporate digital acceleration and innovation programs have been launched. Digital boot camps and specialized training in Python programming are held. New investments and partnership are formed. Pilot and proof-of-concept projects are numerous and expectations are high.

But what is the reality? Most companies really are trying to see how the new digital coat fits. It is not that modeling, simulation, and automation projects are new to the oil field. We have been doing them for years, and seismic acquisition, processing, and interpretation has been a “big data” program since the 1970s. We have had our digital oil fields, integrated operations, smart fields, integrated fields, intelligent fields, or whatever you want to call them for nearly 2 decades now. We have learned quite a bit about work-flow optimization, remote decision support, and real-time operations. Some projects have gone well and created a lot of value. Others we don’t talk about as much.

The barriers are in usage, not in the technology. The wise advice is to think about people, processes, and technology when starting down this path. I would add some consideration around your data foundation as well. The technology is the bright shiny object that gets too much attention. The trouble spots are often in difficult access to relevant data; uncertain data quality, especially when used outside our infamous functional and asset silos; a resistant organizational culture that still values experience over new statistical models; and, often, a lack of digital literacy in the organization.

Our new journal intends to cover the hype, the hope, and the reality of data science and digital engineering. We want to educate SPE members about new emerging digital technologies and data-driven predictive modeling techniques and share use cases from industry leaders. But we also want to talk about the other hard issues that are preventing us from moving faster. We look forward to your comments and suggestions, to your support, you interest, and your criticism (hopefully constructive) as we explore an exciting new era of digital transformation in the oil and gas industry.

Schumpeter. 2018. The Tech Sell-Off. The Economist, 3 November 2018, (accessed 14 January 2019).



Don't miss out on the latest technology delivered to your email monthly.  Sign up for the Data Science and Digital Engineering newsletter.  If you are not logged in, you will receive a confirmation email that you will need to click on to confirm you want to receive the newsletter.