Big data in public health – key technologies and tools for the public health practitioner

Data analytics is foundational to the practice of public health, but the rapid advances in data storage and processing technologies that have swept through industry are still under-utilized in the public health arena. These big data technologies underpin the dramatic leaps in machine learning capabilities that have enabled such huge advances as the Jeopardy-winning Watson and services such as Google Now.

These new ‘Big Data' technologies are enabling use-cases that were difficult to impossible to achieve and extremely costly to maintain only a few years ago. Within the past few years, new open source tools (HBase, Voldemort, MongoDB) and commercial services (EC2, DynamoDB, Google Prediction API) have emerged that provide the ability to leverage these innovations simply and cheaply, revolutionizing the way industry and academia work with data.

These new technologies require new thinking in how to structure and analyze data, but have the ability to open new frontiers for the discipline of public health informatics. These frontiers revolve around two main dimensions: 1) the ability to efficiently store and query immense data sets of terabytes or even petabytes in size, 2) the ability to perform computationally intense algorithms on this data, including artificial intelligence and machine learning techniques such as cluster analysis, pattern mining, and Bayesian analysis. This talk will focus on exploring the important considerations in using these technologies for novel applications in the public health arena.