Answering Big Questions about the Universe with Big Data

Published Sep-03-18

Breakthrough:
New ways to store, transport, access and explore radio astronomy big data.

Company:
The Netherlands Institute for Radio Astronomy, Netherlands

The Story:

Whether you engage a large crowd or a small collection of individuals with subject matter expertise a hackathon can result in new products and breakthrough solutions in a short space of time. To help ensure that desired outcomes are reached organizers can set various parameters such as the type of experts who can participate and which avenues to explore.

In just 24 or 48 hours or a little longer, you can go from a blank sheet of paper to a working prototype. As brilliant minds get to work, organizers can just sit back and watch innovation happen and stellar results flood in.

Hackathons have been widely embraced by many companies and industry sectors including astronomy. In particular, the field of radio astronomy is evolving and like many other enterprises, it is collecting a vast amount of data. Consequently, there are many new challenges such as how to make the best use of the volumes of information and how best to store and transport them.

Reaching for the Stars

In an attempt to find solutions to some of these challenges a two-day hackathon was organized by ASTRON (the Netherlands Institute for Radio Astronomy).

A total of 34 data experts from academia and industry worked on projects that had been suggested in advance of the open innovation event. They organized themselves into teams of four or five people, thrashing out ideas to solve pressing data challenges for the new era of radio astronomy

According to the hackathon’s website: “We need to solve these data challenges so astronomers can translate this data into ground-breaking science on topics such as the Universe's evolution, the formation of the very first galaxies and the search for extraterrestrial life.”

And there is some urgency because the Square Kilometer Array (SKA) will be coming online within the next five years. This will be the world's largest radio telescope consisting of up to 2,000 parabolic antennas dishes in remote desert locations in South and Australia. Additional dishes could also be built in other countries.

The SKA will be tens of times more sensitive and hundreds of time faster at mapping the sky than the best radio astronomy facilities currently available. And it will generate around 160 TB (1012 bytes) of data per second. That is more than five times the current global internet traffic. Hence the need for solutions.

The projects that participants provided very technical solutions for covered such topics as machine learning classifiers and cloud computing and scalability for astronomy datasets. One of the solutions demonstrated Google Cloud Platform scalability for storing, accessing, analyzing and effectively visualizing radio astronomy data.

First Platform

In addition to the development of promising solutions, the hackathon was significant for another reason. It laid claim to being the first platform to bring together the astronomy community and industry experts to identify and address radio astronomy data issues. More will likely follow in the near future as there is a huge need to extract as much value as possible from the vast quantities of data that will be produced.

Next Story »