Mental illness is a major public health problem in the United States. Suicide alone takes the lives of 38,000 Americans each year, more than double that from homicide. Medical and surgical patients who also have mental illness often experience worse outcomes. And yet, in the face of these glaring challenges, we struggle with an ongoing shortage of psychiatrists.

Psychiatry, as it happens, is not a popular specialty among medical students. Only about 4% of US medical graduates choose psychiatry. As a result, close to half of psychiatry residency positions in the US are filled by international medical graduates, compared to roughly a quarter for general surgery or obstetrics and gynecology. Why is there so little interest in psychiatry?

Psychiatry offers a lower income the relative to other specialties. Our health-care system disproportionately rewards doctors for performing procedures, as opposed to talking with a patient and prescribing a medication or a behavioral intervention. This means that specialties like surgery, dermatology or cardiology can offer higher incomes because they involve more procedures. And as medical student debt has soared in the past few decades, potential income looms large for students deciding on a specialty. But there is a deeper problem.

Stigma and prejudice

Mental illness remains one of the most entrenched areas of stigma in contemporary perceptions of disease. People with mental illness face prejudice in employment and in housing that result in real obstacles to fulfilling the basic goals of a functional life. And many people with mental illness internalize this stigma. They can suffer lower self esteem, lower expectations and a worsened prognosis for their recovery.

The insidiousness of this stigma is such that it even infects health-care providers. Physician attitudes towards mental illness – even after their medical education – reflect the general public’s prejudice. As a result patients with mental illness typicallydo not receive the same quality of care for their medical illnesses compared to their peers without mental illness.

The bias against those with mental illness within the health care setting can seem paradoxical. On one hand, there is the tendency to see those with mental illness as not having “real” diseases. By extension one will hear casual asides to the effect that psychiatrists aren’t “real” doctors.

On the other hand, there is the idea that mental illnesses are particularly frightening and disturbing, and have a potentially dangerous effect on those who work with them. A study by psychiatrists at Columbia University summed up this attitude by quoting one blunt student: “Working with ‘crazies’ will make you ‘crazy'”.

It really shouldn’t surprise us then that the stigma of mental illness has been particularly hard to eradicate. Mental illness attacks the very aspects of a person that differentiate us from other species and make us human: reason, empathy and language. It also attacks those qualities that make us unique as individuals: memory, character, the integrity of the boundaries between self and others, and the sense of a narrative built on choices rooted in authentic feelings. In demonstrating the fragility of these fundamental aspects of our identity, those with mental illness inevitably serve as disturbing and frightening “others”.

In the light of all this, perhaps it is understandable that medical students haven’t been breaking down the doors to sign up. But none of this is reason to despair.

Education and changing attitudes

Attitudes towards some mental illness, particularly depression, do seem to be shifting as a result of education, and the increasing openness of people suffering from the condition. There is plenty of evidence that learning about or meeting someone with mental illness decrease stigma.

During a recent election campaign in my own state of Texas, a candidate was attacked for receiving hospital treatment for depression years before. It backfired, and was seen as a groundless and desperate ploy by his opponent’s camp.

And there is certainly reason to be hopeful that current progress in neuroscience genetics and epigenetics (how our environment effects our genes) will decrease our perception of mental illness as unexplainable. The mysterious nature of mental illnesses historically made these diseases more frightening, and more of a “blank screen” for people to project their most disturbing fears on. Perhaps, as we are able to explain more and more of the causes of mental illness, these conditions will not be so demonized.

If we’re going to make progress in recruiting the psychiatrists that we need, we need to make sure that students have plenty of opportunity to meet patients who are reclaiming their best selves in recovery from mental illness and addiction. We also need to make sure they can work with psychiatrists who are proud to work with people who are living with such disturbing and frightening, but ultimately treatable, illnesses.

Adam Brenner does not work for, consult to, own shares in or receive funding from any company or organisation that would benefit from this article, and has no relevant affiliations.