The world needs a lot of energy. Global energy demand is expected to increase by 37% percent over the next 25 years, according to the International Energy Agency’s World Energy Outlook 2014. Meeting this demand without significantly increasing carbon emissions requires new energy technologies that involve low or no carbon dioxide emissions.

It is widely recognized that the government plays a critical role in advancing innovation and technology development. State and federal policies have driven deployment of solar and wind power as well as development of hydraulic fracturing, which helped spark a boom in oil and gas drilling in the US.

In the future, many argue that we’ll need many more innovations. For example, the American Energy Innovation Council, a group of high-profile business people, have called for a significant increase in the amount of government funding in research, development and demonstration (RD&D) of energy technologies. The group, which includes Bill Gates, General Electric CEO Jeffrey Immelt and others, also proposed ways to better deploy federal research funds.

The report raises a question vital for the future: what government policies are most effective at accelerating energy innovations? A study of the economic forces in energy and environment, some of which are different than other industries, can help answer this.

Where markets fail

To start, it is important to understand that there are two fundamental market failures in the RD&D of clean energy technologies. A market failure is a situation where the allocation of resources does not happen efficiently. These market failures lead to underinvestment in energy innovation by the private sector and provide justification for government interventions. Separate policy instruments need to be in place to deal with these different market failures.

When one company invests in research, it often leads to technology and knowledge that benefit society overall. That’s good for advancing energy innovation. But because other companies and users can take advantage of that knowledge, individual companies cannot capture the full value of investing in RD&D. A car made by a 3-D printer (really) came out of a government program to develop light-weight and energy-efficient vehicles.US Department of Energy.

As a result, the invisible hand of the market produces too little innovation. The more basic and long-term is the research, the worse the problem tends to be. There are also additional market failures in the adoption and diffusion of innovation, as the cost or value of a new technology to one user may depend on how many other users there are.

Second, greenhouse gas emissions constitute an “externality,” or unintended consequence, because the costs of climate change are borne by parties other than the emitters. Hence, current market prices for fossil fuels don’t reflect the full social cost of their consumption. With the “mispricing" of fossil fuels, the invisible hand of the market suppresses demand for alternative energy technologies that are substitutes for fossil fuel technologies.

To overcome the market failure in knowledge creation, the government needs to fund and invest in basic research and provide subsidies and tax incentives for corporate investment of RD&D in energy innovation.

Moreover, public support of costly and risky early-stage deployment and demonstration of alternative energy technologies is well justified. It is important to disseminate information about the operation, maintenance and incremental improvement of new energy technologies that currently might be more expensive and/or less reliable than existing ones. For example, a research program that achieves a better battery for storing energy on the electric grid could benefit all electricity users. Current US Energy secretary, right, has been advocate of the ARPA-E, a research agency formed to fund breakthrough energy technologies, such as this project to store more natural gas in vehicles.ARPA-E/Department of Energy The government also needs to address the other market failure of the “mispricing” of fossil fuels. It may impose a tax on carbon or a “cap and trade” system that alters prices to reflect more adequately the environmental costs. Or it could enact a specific regulatory requirement (such as emission or performance targets) to correct the failure. Such policies would create a higher demand for low-emission technologies and provide incentives for private investment in climate-friendly technologies.

Insights from other industries

In a intriguing book titled Accelerating Energy Innovation: Insights from Multiple Sectors, experts on the history of innovation offered interesting insights about the roles that the US federal government has historically played in catalyzing innovation and technology development in a variety of industries including agriculture, chemical, life sciences, the computer and semiconductor industry, and the internet. They found there were three broad mechanisms through which government policies have served to accelerate innovation:

1. provision of “substantial, sustained and effectively managed federal funding” for basic research

2. creation of a growing demand for innovation, either through procurement or through the market, and

3. encouragement of extensive competition and entry by newly founded firms.

Note that these industries, unlike energy, don’t face a severe market failure of climate and environmental externalities. Nevertheless, the government still played a critical role in stimulating demand, particularly in the early stages of technology development. Thus, it is all the more important to have policies to address the “mispricing” of fossil fuels to increase the demand for and induce private investment in low-emission technologies.

Moreover, history shows the importance of the government’s role in enabling vigorous competition and entry to stimulate innovation. This is particularly relevant for energy innovation. Combating climate change will involve development and deployment of many different technologies in a diverse array of sectors, varying from the electric grid to transportation and other industries. That’s why it is important for the government to foster rigorous competition, rather than pick “winners” and “losers” among technologies and companies.

Zhen Lei receives funding from National Science Foundation, Department of Energy and National Institutes of Health.