What technologies are in big data

Successful in the data age - with new standards in performance, flexibility and scalability -Display-

Unstructured data or big data are a curse and a blessing at the same time: From the point of view of the business world, they are a treasure trove of knowledge, from the point of view of IT a huge challenge. The data volume alone forces IT to come up with new types of solutions, and the acquisition of knowledge is developing into the supreme discipline of data analysis.

Evasion seems unlikely. Big data applications are now more widespread than generally assumed. According to a Bitkom survey, 62 percent of all companies are already using big data technologies and they are becoming increasingly important thanks to applications such as cognitive IoT, AI-based fraud detection and predictive maintenance. The core element of all these applications is, in turn, data analysis, especially in combination with artificial intelligence methods, since AI is increasingly becoming a component of the applications themselves.

Deep learning is conquering the business world

The AI ​​methods of modern analytics are primarily algorithms for machine learning (ML) and deep learning (DL), which modify themselves according to the requirements and the knowledge gained. In normal machine learning, learning takes place mainly through human-corrected training of the software, whereas in deep learning neural networks adapt the software independently. Typical applications up to now have been image recognition, natural language processing or complex games such as chess and Go.

Now deep learning is preparing to conquer the business world as well. Experts assume that DL will make the leap into many business applications by next year at the latest. This is justified by the fact that the hardware and software required for this will have made significant progress in terms of speed and accuracy by then. In addition, there is the new learning method of deep learning inference. Here, trained models are used to predict the actual results of an application with high accuracy.

By combining analytics with AI, retailers, for example, can better understand how customers move through sales floors, and manufacturers can more easily identify the causes of defects in their products. Audi, for example, uses real-time analytics in edge applications to drastically improve its quality assurance.

Put all the levers in motion

Machine learning and deep learning are the spearheads of the new analytics worlds, but their prevalence is still relatively low. At the moment, the focus of the CIOs is mostly on a step-by-step expansion of the existing analytics applications, which is mainly due to the fact that the efficient and effective handling of extremely large amounts of data is still in its infancy.

One reason for this is that the technical requirements are often not up to the new tasks. Because many data managers and infrastructure architects have now had to experience that the well-known big data platforms such as Splunk, Cloudera, MongoDB or Elastic quickly push conventional IT systems to the limits of their performance and scalability. The bare metal deployments required can also turn into a management nightmare.

The IT experts are gradually realizing that the problem can only be tackled by taking measures at all levers. This also includes the processor and storage technologies used in the hardware. Optimizing databases that use massively scaled databases requires hardware that can effectively support database and analytics workloads. If these technologies are properly staffed, the many new analysis options can actually be used in a timely and cost-effective manner.

Xeon and Optane for power analytics

These technologies include, for example, the scalable Intel Xeon processors of the latest generation. They are specifically designed to offload database workloads and improve computing power. For this purpose, Intel has made a number of optimizations at the semiconductor level in recent years. This includes, for example, the instruction set extension AVX-512 or VNNI, and in future also TMUL. This means that all common database systems and AI programs can be accelerated significantly.

The new processors are ideally complemented by Intel's persistent Optane memory modules, which can expand the working memory and thus drastically increase processing speed. These provide a performance boost for a range of DBMS platforms and applications, and allow databases in the server's memory to be optimized for advanced data analysis.

And finally, this also includes Intel's Optane SSDs and Intel's NVMe SSDs. These offer a long read-write service life and allow continuous write and read operations with all databases with an excellent service life per drive. In this way, they help to eliminate bottlenecks in data storage in the data center and to cope with larger amounts of data more cost-effectively. Not only can applications be accelerated, but the transaction costs of tasks for which latency is of vital importance can also be significantly reduced. One area of ​​application for this is real-time analytics in edge applications, as used in the quality assurance department at Audi mentioned above.