Revision as of 15:28, 19 October 2017

Big Data Analytics Framework for smart society

BIDAF is a five-year research project financed by the KK-stiftelsen. The project is carried out by researchers at Halmstad University, SICS Swedish ICT AB, and University of Skövde, Sweden. The overall aim of the BIDAF project is to significantly further the research within massive data analysis, by means of machine learning, in response to the increasing demand of retrieving value from data in all of society. This will be done by creating a strong distributed research environment for big data analytics.

The project addresses challenges on several levels:
- Platforms to store and process the data
- Machine learning algorithms to analyze the data
- High level tools to access the results

All of these challenges must be addressed together, in order to enable end users to successfully perform analysis of massive data: (i) the hardware and platform level with the capacity to collect, store, and process the necessary volumes of data in real time, (ii) machine learning algorithms to model and analyse the collected data, and (iii) high level tools and functionality to access the results and to allow exploring and visualizing both the data and the models.

Challenge 1. To develop a computation platform suitable for machine learning of massive streaming and distributed data.

One of the important characteristics of Big Data is that it is often streaming or at least constantly updated. It typically originates from a large number of distributed sources, and is, like most real world data, inherently noisy, vague or uncertain. At the same time, due to sheer size, a scalable framework for efficient processing is needed to adequately take advantage of it. However, today’s Big Data platforms are not well adapted to the specific needs of machine learning algorithms:

- Current platforms lack functionality suitable for analysing real-time, streaming and distributed data. - Machine learning requires storing and updating an internal model of the data. Current platforms lack suitable support for stateful computing. - The advanced processing in machine learning requires a more flexible computational structure than provided within the map-reduce paradigm of big data platforms, for example, iteration.