Predlog arhitekture sistema visokih performansi za generalnu obradu podataka na klasterima za podatke velikog obima

Abstract

In recent years, the application and widespread adoption of Big Data, Internet of Things (IoT), Cloud technologies have increased the use of large-scale data processing systems. These technologies increased significantly and exponentially with the heterogeneous data generated (structured, unstructured, and semi-structured). The processing and analysis of a tremendous amount of data is cumbersome and is gradually moving from the classic "batch" processing - extraction, transformation, loading (ETL) techniques to realtime processing. For example, in the domain of the automobile industry, healthcare, but also in other disciplines. Tracking, data processing, environmental management, timeseries data, and historical data set are crucial to forecasting models not only in these domains. This doctoral dissertation is about the design of a general architecture for processing a large amount of data. The architecture as such enables efficient acquisition of data, their optimal placement, processing of large amounts of data, use of various algorithms for drawing conclusions as well as for displaying data. The doctoral dissertation shows the complete process of modeling and designing architecture, the selection of appropriate software components for its realization. The presented platform met very demanding parameters for meeting the system's performance, including the standard for decision support of the Transaction Processing Council (TPC-H) by following the European Union (EU) legislation and the Czech Republic. Currently, the presented proof of concept (PoC) that has been upgraded to the production environment has united isolated parts of the Czech Republic's healthcare. The reported PoC Big Data Analytics platform, artefacts and concepts can be transferred to health systems in other countries interested in developing or upgrading their national health infrastructure in a costeffective, secure, scalable, and high-performance way.