2019: Thoughts on the Key Themes in Oil and Gas Data Management

It’s been a year of accelerating change in the oil and gas industry; some of it very positive, some of it very challenging. These are my thoughts on some of the main themes from the past year, with a focus on data management and digitalization.

Digital Needs Data Needs Data Management

The industry is waking up to the fact that we won’t make any progress around strategic digitalization initiatives without first sorting out our data. And that requires solid data management, not just a bit of data munging to support proof of concepts and pilot projects. Data is now seen as a key business enabler (asset, even), whereas, in the past, it was often seen as a by-product to be stuffed in a cupboard. This growing awareness might also be related to the next theme.

Governance Is Making a Comeback

After a long period of relative dormancy, data governance seems to be back in the spotlight. I believe it’s at the root of many of the challenges we’ve seen in recent years in our efforts to digitally transform. There’s a growing realization that there needs to be a solid framework around data management to make it work. It seems we can’t just dump it all in a data lake and hope for the best. Feel free to dip in to our free resources on the subject, available here.

Stuck at the POC Chasm

I’ve seen many presentations of great looking digital proofs of concept (POCs) but very few examples of real, high-impact operational deployments. This might partly have something to do with a lack of solid underlying data management; it’s relatively easy to bring together a data set to support a POC, but it’s a different matter scaling this up to support the messy reality of operational data. And this failure to translate POCs to operational deployment might explain the next couple of observations.

Dropping Down the Digitalization Hype Curve

There’s a feeling that we’re starting to drop down from the peak of the digitalization hype cycle to something approaching a more realistic view of things. The demand people increasingly make now when presented with funky new digitalization stuff is, “prove it works.” They want to see solid examples of real value, not just a PowerPoint promise. In general, the hyperbole is moderating and expectations are beginning to be dialed back a bit.

Oil and Gas Core Business: Hydrocarbons or Tech?

During the past few years, there seems to have been some sort of trend for oil and gas companies to try to reinvent themselves as tech companies. Some have always dabbled in this space and have done a pretty good job at it. But it now feels like everyone needs a trendy office with beer taps if they’re going to successfully digitalize. I have a feeling that many are just not very good at it. Developing and deploying tech is really hard to do well. And there’s still a real tension between business-as-usual information-technology organizations and the innovation/research-and-development groups popping up and expecting to be supported in an agile way of working. This may also explain why so many digital initiatives stall at the POC stage. It also raises the question: What business are oil and gas companies in? What is their core business? And what should they leave well alone? I suspect part of the driver to “funk it up” a bit might be partly related to the next theme.

Net Zero Sentiment and Oil Shaming

The pace of change in shifting public and industry sentiment this year has been astonishing. The conversation, driven by increasingly visible signs of climate change, has gone from talking about how we support long-term energy transition to how we achieve net zero carbon emissions in less than 20 years. The realities of the scale of the transition challenge seem to be relegated to footnotes in the debate. This makes it an incredibly tough environment to position concepts such as exploration and maximizing economic recovery when looking for investment. And now, oil shaming seems to be a thing, where it’s no longer acceptable to admit you work in oil and gas, which makes the next challenge a whole lot harder.

The Search for Talent

The skills we need in the coming years are in short supply, both in the traditional realms of geoscience and engineering as well as the new roles developing around areas such as digitalization, data science, and artificial intelligence. The problem is we can’t attract young people with these skills to what’s increasingly seen as a dirty, sunset industry. We need to work hard to demonstrate the industry is changing and offer people opportunities to be part of the solution, not the problem.

Fortunately, adversity drives invention.

Collaboration Is Key

From something of a curiosity at the turn of the year, the Open Subsurface Data Universe (OSDU) initiative appears to be getting some real traction. It seems the oil and gas industry is fed up with big vendor lock-in or trying to solve it all themselves. The realization is that value comes not from managing the data but from the use of data. We wait to see if this initiative can go beyond where others have stalled—as always, the devil will be in the detail, but the early signs are positive. The OSDU is just one example of a more collaborative, open environment that is emerging post-downturn. Initiatives such as the release of the National Data Repository in the UK and the reformation of the Oil & Gas UK Information Management Forum point to a brighter future when it comes to cracking the perennial problem of oil and gas data management.


After years of relative stagnation and large service company domination, there has been a veritable explosion in innovation in the past couple years. This has been driven by the downturn (bright people discarded and forming their own companies), new technology (the rapid growth of affordable cloud platforms and data science capability), new opportunity (the growth in renewables and emerging opportunities in areas such as carbon capture and sequestration and clean fuels), and the support of organizations such as the Oil and Gas Technology Centre. We’ve seen a whole host of innovative companies such as Osokey, AEON, Loxodrome, Mapstand, TapeArk, Target, and Infoscience Technologies looking to push the boundaries. There are a host of entrants from outside of oil and gas bringing in their innovative solutions in to the mix, too. Here at Sword Venture, we’re also investing heavily in innovation and digital solutions development to find new ways of tackling data management challenges.

Despite the major challenges, what I’ve seen this year gives me real hope that there is indeed a bright future for our industry, one that balances our need for energy with the needs of the environment. I expect that in the next year or two, we’ll see a real acceleration in innovation and our ability to respond to a rapidly changing landscape.

Gareth Smith is head of consulting at Sword Venture.


Don't miss out on the latest technology delivered to your email monthly.  Sign up for the Data Science and Digital Engineering newsletter.  If you are not logged in, you will receive a confirmation email that you will need to click on to confirm you want to receive the newsletter.