These days it means more to understand business processes and to transform them into data-driven opportunities – using cloud technology when needed
For a lot, but certainly not all enterprises, a typical big data software is Apache Hadoop. Hadoop is the new legacy standard in enterprises when it comes to on-premise tools. Cloud-based big data is still fairly new for enterprises and cloud providers, as the technology has only risen up to become cloud-ready in the last two years or so. Popular tools here are Apache Kafka, Apache Spark or Apache Flink – these are all run best on bare metal servers.
Different strokes for different folks
The pro for Hadoop is the nearly unlimited scalability with an affordable price for hardware. A negative is, however, is high operational cost. Although this can be negated by using commercial distributions for an additional price.
The same counts for Kafka, Spark or Flink. That’s why modern tech giants always operate the largest installations in their own data centres, but enterprises often struggle to attract talent to get that scale into stable productions. In short − it’s great to have the tools on premise, but to operate them can be challenging.
E.ON were confronted with exactly this problem. When I joined, a large Hadoop cluster was running, but on IaaS infrastructures in a public cloud. That had massive implications on stability and performance, and it was extremely costly to let the system breathe. Don’t get me wrong, when I started at E.ON, more than two years ago, the team had already two years of operational experience with Hadoop. That shows how fast an enterprise needs to adopt new big data technologies to get an advantage against competitors.
Given that E.ON’s business units have a growing appetite, we decided to use public cloud big data rather than operate big data tools ourselves. That was the initial departure from traditional infrastructure into cloud technology, using native tools instead of trying to reinvent the wheel for the 10th time. That point in time was an enormous boost to our transformation. We are now able to bring products to customers quicker than ever before.
From new terrain to ROI
Our key ambition is to oversee a fully digitized grid. Having information about the energy infrastructure at any time is imperative to creating transition in the energy industry. To achieve this, we must decarbonize our environment and make it possible to be informed about the production, transport and use of energy over its entire lifecycle.