Big Data has a reputation for being creepy; the domain of “Big Business” and “Big Government”. At best it’s the driver of relentless advertising, uniquely targeted and eerily reminiscent of our most recent internet searches.

At worst it heralds the Orwellian prophecy: Big Brother is watching you. Recent revelations by Edward Snowden have only fuelled suspicions that our data is being used in ways we don’t really comprehend and by people we don’t even know.

Yet Big Data could be one of the greatest tools we’ve ever had to drive economic, social and environmental sustainability.

How big is Big?

There is no concrete definition or agreed threshold upon which everyday data becomes Big Data.

The term was coined in 1997 by NASA researchers Michael Cox and David Ellsworth to describe the challenge of processing and analysing vast amounts of computer-generated information.

Since then, an explosion in technology including social media, smartphones and sensing devices have seen a corresponding explosion in the amount of data generated. People’s interaction with the internet has changed radically and the amount of personal information they disclose online has skyrocketed.

Mayer-Schonberger and Kukier have offered this definition:

“…big data refers to things one can do at a large scale, that cannot be done at a smaller one, to extract new insights, or create new forms and value, in ways that change markets, organisations, the relationship between governments, citizens and more.”

Let’s just say we’re talking about more zeros and ones than you can possibly imagine; data in different formats, from different types of technology, structured and unstructured.

According to Cisco futurist Dave Evans, by 2015 one Zettabyte of data will flow over the internet. That equals a stack of books from Earth to Pluto 20 times or 116 billion miles.

With our ever-increasing capacity to store data, we’ve moved away from collecting just the relevant bits of data we need to answer the important questions.

Now we can collect everything and see if any interesting questions emerge. It turns traditional research on its head. Rather than proposing a hypothesis and collecting sample information to test it, we scan and analyse information to identify a hypothesis.

Big Data harnesses data analytics, modelling and machine learning to complete infinite complex calculations that are beyond the capabilities of the human brain.

Big Data for sustainable development

In terms of sustainable development, Big Data offers us unparallelled opportunities to study the complex relationships between people, industries, economies and the environment.

It can help us identify previously unknown patterns and relationships and find more sustainable ways to use our resources.

Sense-T, spun out of the University of Tasmania and CSIRO, is analysing data and designing smartphone apps that help food and wine producers to predict pasture growth, minimise chemical use, identify disease risks and schedule irrigation.

Soil sensors can help dairy farmers optimise fertiliser use. UTAS

Importantly the data can be re-used and re-purposed, creating value on many different levels. For example, a dairy farmer can use data from soil sensors to help minimise fertiliser use. The same sensor data could be accessed by an oyster farmer to manage water quality, by science students to study the environment, or by government to help cut red tape.

It’s about sharing data and making it available in a way that helps people do more with the resources they have available.

This is increasingly vital in a world approaching nine billion people, with no more arable land, unprecedented climate change and a responsibility to help lift people out of poverty.

Other examples of Big Data innovations include software to predict bushfire risk, algorithms that give early warnings of disease outbreaks and modelling to demonstrate environmental impacts of climate change.

Big Data can be – and is being – used for good, social and environmental. It is not a silver bullet. It does not have all the answers and does not replace all traditional approaches to sustainable development. But it is one of the most powerful weapons in the arsenal.

Making it less creepy

Unfortunately, Big Data has a creepy reputation because it’s inaccessible to most people and there is limited transparency about how data is used.

Privacy is not the entire issue. There is a lot of information people are happy to share; often very personal information. The success of social media sites is predicated on our overwhelming urge to connect and share with others.

Trust, choice, consent, transparency and accountability are also big issues at play. Legal frameworks have so far struggled to address these issues, unable to adapt to the rapidly advancing world of technology and information.

There are technological and coding measures in development that show promise. For example, Sense-T is working with the Australian Bureau of Statistics to develop a complex combination of standard and unique statistical techniques to safeguard data.

But the real trick will be democratising Big Data in a way that gives people access to the benefits, while being open and transparent about how data is used.

Many people are willing to share their information if it creates value for them and if it creates value for society and the environment we live in.

Organisations that can deliver value on both levels, while respecting people’s privacy and maintaining trust, will have a unique market proposition in the world of Big Data.