The tall, rusty haired son of a country doctor, William Potter, 64, has spent most of his life treating mental illness - first as a psychiatrist at the US National Institute of Mental Health and then as a drug developer. A decade ago, he took a job at Lilly's neuroscience labs. There, working on new antidepressants and anti-anxiety meds, he became one of the first researchers to glimpse the approaching storm.

To test products internally, pharmaceutical companies routinely run trials in which a long-established medication and an experimental one compete against each other, as well as against a placebo. As head of Lilly's early-stage psychiatric drug development in the late 90s, Potter saw that even durable warhorses like Prozac were being overtaken by dummy pills in more recent tests. The company's next-generation antidepressants were faring badly, too, doing no better than placebo in seven out of ten trials.

As a psychiatrist, Potter knew that some patients really do seem to get healthier for reasons that have more to do with a doctor's empathy than with the contents of a pill. But it baffled him that drugs he'd been prescribing for years seemed to be struggling to prove their effectiveness. Thinking that something crucial may have been overlooked, Potter tapped an IT geek named David DeBrota to help him comb through the Lilly database of published and unpublished trials - including those that the company had kept secret because of high placebo response. They aggregated the findings from decades of antidepressant trials, trying to see what was changing over time. What they found challenged some of the industry's basic assumptions about its drug-vetting process.

Assumption number one was that if a trial were managed correctly, a medication would perform as well or as badly in a Phoenix hospital as in a Bangalore clinic. Potter discovered, however, that geographic location alone could determine whether a drug beat placebo or crossed the futility boundary. By the late 90s, for example, the classic antianxiety drug diazepam (also known as Valium) was still beating placebo in France and Belgium. But when the drug was tested in the US, it was likely to fail. Conversely, Prozac performed better in the US than it did in western Europe and South Africa. It was an unsettling prospect: regulatory approval could hinge simply on where the company chose to conduct a trial.

Mistaken assumption number two was that the standard tests used to gauge volunteers' improvement in trials yielded consistent results. Potter and his colleagues discovered that ratings by trial observers varied significantly from one testing site to another. It was like finding out that the judges in a tight race each had a different idea about where the finish line was.

Potter and DeBrota's data-mining also revealed that even superbly managed trials were subject to runaway placebo effects. But exactly why any of this was happening remained elusive. "We were able to identify many of the core issues in play," Potter says. "But there was no clear answer to the problem." Convinced that what Lilly was facing was too complex for any one pharmaceutical house to unravel on its own, he came up with a plan to break down the firewalls between researchers across the industry, enabling them to share data in "precompetitive space".

After prodding by Potter and others, the US National Institutes of Health (NIH) focused on the issue in 2000, hosting a conference in Washington. For the first time in medical history, more than 500 drug developers, doctors, academics and trial designers put their heads together to examine the role of the placebo effect in clinical trials and healing in general.

Potter's ambitious plan for a collaborative approach to the problem eventually ran into its own futility boundary: no one would pay for it. And drug companies don't share data, they hoard it. But the NIH conference launched a new wave of placebo research in academic labs in the US and Italy that would make significant progress toward solving the mystery of what was happening in clinical trials.

Visitors to Fabrizio Benedetti's clinic at the University of Turin are asked never to say the P-word around the med students who sign up for his experiments. For all the volunteers know, the trim, soft- spoken neuroscientist is hard at work concocting analgesic skin creams and methods for enhancing athletic performance. One recent afternoon in his lab, a young footballer grimaced with exertion while doing leg curls on a weight machine. Benedetti and his colleagues were exploring the potential of using Pavlovian conditioning to give athletes a competitive edge undetectable by anti-doping authorities. A player would receive doses of a performance- enhancing drug for weeks and then a jolt of placebo just before competition.

Benedetti, 53, first became interested in placebos in the mid-90s, while researching pain. He was surprised that some of the test subjects in his placebo groups seemed to suffer less than those on active drugs. However scientific interest in this phenomenon, and the money to research it, were hard to come by. "The placebo effect was considered little more than a nuisance," he recalls. "Drug companies, physicians and clinicians were not interested in understanding its mechanisms. They were concerned only with figuring out whether their drugs worked better."

Part of the problem was that response to placebo was considered a psychological trait related to neurosis and gullibility rather than a physiological phenomenon that could be scrutinised in the lab and manipulated for therapeutic benefit. But then Benedetti came across a study, done years earlier, that suggested the placebo effect had a neurological foundation. US scientists had found that a drug called naloxone blocks the pain-relieving power of placebo treatments. The brain produces its own analgesic compounds called opioids, released under conditions of stress, and naloxone blocks the action of these natural painkillers and their synthetic analogs. The study gave Benedetti the lead he needed to pursue his own research while running small clinical trials for drug companies.

Now, after 15 years of experimentation, he has succeeded in mapping many of the biochemical reactions responsible for the placebo effect, uncovering a broad repertoire of self-healing responses. Placebo-activated opioids, for example, not only relieve pain; they also modulate heart rate and respiration. The neurotransmitter dopamine, when released by placebo treatment, helps improve motor function in Parkinson's patients. Mechanisms like these can elevate mood, sharpen cognitive ability, alleviate digestive disorders, relieve insomnia and limit the secretion of stress-related hormones such as insulin and cortisol.

In one study, Benedetti found that Alzheimer's patients with impaired cognitive function get less pain relief from analgesic drugs than normal volunteers do. Using advanced methods of EEG analysis, he discovered that the connections between the patients' prefrontal lobes and their opioid systems had been damaged. Healthy volunteers feel the benefit of medication plus a placebo boost. Patients who are unable to formulate ideas about the future because of cortical deficits feel only the effect of the drug itself. The experiment suggests that because Alzheimer's patients don't get the benefits of anticipating the treatment, they require higher doses of painkillers to experience normal levels of relief.

Further research by Benedetti and others showed that the promise of treatment activates areas of the brain involved in weighing the significance of events and the seriousness of threats. "If a fire alarm goes off and you see smoke, you know something bad is going to happen and you get ready to escape," explains Tor Wager, a neuroscientist at Columbia University. "Expectations about pain and pain relief work in a similar way. Placebo treatments tap into this system and orchestrate the responses in your brain and body accordingly."

In other words, one way that placebo aids recovery is by hacking the mind's ability to predict the future. One of the most powerful placebogenic triggers is watching someone else experience the benefits of an alleged drug. Researchers call these social aspects of medicine the therapeutic ritual.

In a study last year, Harvard Medical School researcher Ted Kaptchuk devised a clever strategy for testing his volunteers' response to varying levels of therapeutic ritual. The study focused on irritable bowel syndrome, a painful disorder that costs more than $40 billion a year worldwide to treat. First the volunteers were placed randomly in one of three groups. One group was simply put on a waiting list; researchers know that some patients get better just because they sign up for a trial. Another group received placebo treatment from a clinician who declined to engage in small talk. Volunteers in the third group got the same sham treatment from a clinician who asked them questions about symptoms, outlined the causes of IBS and displayed optimism about their condition.

Not surprisingly, the health of those in the third group improved most. In fact, just by participating in the trial, volunteers in this high-interaction group got as much relief as did people taking the two leading prescription drugs for IBS. And the benefits of their bogus treatment persisted for weeks afterward, contrary to the belief - widespread in the pharmaceutical industry - that the placebo response is short-lived.

Meanwhile, the classic use of placebos in medicine - to boost the confidence of anxious patients - has been employed tacitly for ages. Nearly half of the doctors polled in a 2007 survey in Chicago admitted to prescribing medications they knew were ineffective for a patient's condition - or prescribing effective drugs in doses too low to produce actual benefit - in order to provoke a placebo response.