Abraham Lincoln said that those who shape public opinion exert an even greater influence than judges and lawmakers.

In recent years popular sentiment – on issues like marriage equality and legalization of medical marijuana – has molded law and policy in its own image. On the horizon is growing support for physician aid in dying and a deeper personal stake in end of life decisions. Each of these initiatives has been cultivated at the state level, often in the face of rigid federal government rejection of favorable social and scientific evidence.

State and federal conflicts are not new, but rarely has the US government displayed more intransigence than in the arena of marijuana prohibition.

Inventing the marijuana menace

In the 1930s, the first chief of the Federal Bureau of Narcotics Harry J. Anslinger devised a rabid campaign against the cannabis menace. Perceiving marijuana use to be the province of unsavory types like poor blacks, jazz enthusiasts and bohemians of loose morals, he linked it to heroin addiction, madness and mayhem.

In 1944, a report commissioned by New York’s Mayor Fiorello La Guardia determined that cannabis use did not promote opiate dependence, deviant behavior or psychosis in otherwise healthy individuals. Anslinger fiercely condemned the findings and continued business as usual.

In 1973 a report by the federally authorized Shafer Commission proved equally ineffective in deterring President Richard Nixon’s War on Drugs. Concluding that the social and financial cost of criminalizing marijuana use was disproportionate to the drug’s harm, the commission recommended treatment and prevention over rampant prosecution.

Today, Congress and the Drug Enforcement Administration persist, despite evidence to the contrary, in classifying marijuana in the Controlled Substances Act as a Schedule I drug with no medical benefits. At the same time, the US Government continues to obstruct scientific research efforts by restricting access to cannabis supplies - it’s hard to source something that is illegal.

At the same time, fears that legalization will promote underage use have so far, it seems, proved unfounded.

As to the effect of Colorado’s 2012 legalization of personal cannabis use on crime, traffic accidents and educational and health outcomes, the data is still coming in. Evidence of legalization’s impact on traffic accidents is contradictory and its effect on crime rates is anything but conclusive. And, while moderate cannabis use by healthy adults seems to pose minimal risk, the effect of the drug on young, developing brains may prove more problematic.

Yet blanket federal prohibition dictates that a rational assessment of legal marijuana regulation will never see the light of day. Without the reality testing provided by state initiatives, we would be forced to rely on data furnished by federal agencies, entities that have, historically, skewed public perception as to marijuana’s inherent risks.

Even anti-drug activists support the notion of the states as laboratories, if only to see legalization experiments crash and burn.

Two governments, one people: the federalism dilemma

We learned this in high school civics, but we often forget that it matters: every American is a citizen of two governments, the federal and the state.

Congress can only trump our state governments when the US Constitution says so. Very often it does not, and governments in tension are the hallmark of our constitutional order. They have never been at each other’s throats as in the current marijuana debate.

In 2005, the Supreme Court’s Gonzales v. Raich decision held that the Constitution’s “Supremacy Clause” meant that the DEA could enforce the federal law criminalizing all marijuana possession or distribution in states that allowed it. But, at the same time, the Court did not overturn the many state laws that legalize and regulate marijuana.

The result is that marijuana is everywhere illegal and yet legal in nearly half the states. How can this Zen-like status be resolved?

The short answer is that it cannot unless either the federal or state governments give up. A majority of Americans favor the legalization of marijuana, and the movement in the states is growing in strength. Oregon, Alaska, and Washington, DC approved legalization in November, and more ballot initiatives are planned for 2016. So far, however, Congress has repelled every effort to remove marijuana from the Controlled Substances Act.

But it’s not really an evenly matched fight. The enforcement of drug laws in the United States is overwhelmingly a state matter. Federal resources are minuscule in comparison. For the average pot smoker in one of the states that has legalized either medical or recreational marijuana, the fear of arrest is very small. The federal government is responsible for only 2% of all drug arrests nationwide. (In 2013, there were 1,552,432 arrests for drug law violations in the US. That same year, there were 30,688 drug arrests by the DEA. )

What’s more, the federal government is not sure what it wants. The Department of Justice -— the federal agency charged with prosecuting marijuana cases -— has adopted shifting positions in view of the growing revolt by the states. The Attorney General and several US Attorneys have insisted on their right to authorize DEA raids on cannabis dispensaries and to prosecute all marijuana cases, even while issuing pronouncements suggesting accommodation with state efforts to legalize and regulate marijuana.

This wavering suggests an internal struggle within the federal government on how to deal with the marijuana revolution.

In the meantime, the legal situation around the country is chaotic. While medical marijuana is legal in almost half the states, in no state may physicians prescribe this drug without risking loss of their federal license to prescribe narcotics. State laws have tried to bypass that problem by accepting a doctor’s “recommendation.” But even if a doctor could write a cannabis prescription, no pharmacy could fill it without violating federal law.

Marijuana businesses may comply with state law but will run into tax and banking walls erected by the federal government. The Internal Revenue Service has ruled that companies in the business of dispensing marijuana may not deduct their business expenses as other business do —- a staggering financial penalty. And most banks, fearful of federal regulators, have refused to open commercial accounts for cannabis dispensaries, thus relegating these businesses to a hazardous cash-only operation.

Let the states experiment

The federal government should listen to the same popular voices that the states have heard. Marijuana is a drug of choice for millions, which like alcohol may be used moderately or abused. But for those afflicted with the many serious illnesses for which marijuana supplies substantial relief, it is a drug of compassion.

We need a strong central government, but not one that assumes the role of moral guardian at the expense of credibility. Our federal government functions best —- and it has succeeded phenomenally well -— when it keeps its ideals in sight without obscuring the practicalities. On the issue of marijuana’s dangers, the federal government has cried wolf once too often.

You can’t fool all of the people, and all of the states, all of the time.

This piece was co-authored by Ruth C Stern, an independent legal researcher and scholar

J Herbie DiFonzo does not work for, consult to, own shares in or receive funding from any company or organisation that would benefit from this article, and has no relevant affiliations.