Every culture has its shamans, oracles and priests who act as intermediaries between the known and the unknown, the seen and the unseen. In secular societies that role is increasingly being filled by scientists working at the frontiers of human understanding. Thanks to the mind-boggling obscurity of quantum theory many of these are physicists—Carlo Rovelli is only the latest to have achieved almost prophetic status.

However, even physicists are denied access to the holiest of holies: human consciousness. The anointed guardians of this sacred space are neuroscientists. Almost everyone now knows that the brain is the organ of thought and feeling, making those who study it the closest people we have to experts on the human soul.

This generates both awe and fear. Would they pluck out the heart of our mystery? Would they sound us from our lowest notes to the top of our compasses? How unworthy a thing they would make of humanity, reducing its noble spirit to the base corporeality of cells and electricity!

But that is precisely not what the most exalted of these high priests do. Like all the most influential religious leaders, for every mystery they solve they pose another. Their guidance takes us deeper into the human mind than ever before but their torches only illuminate a fraction of it. The more we explore anima incognita, the more evident it is just how inadequate our maps of it are.

It would be stretching the clerical analogy to suggest that neuroscience has its sacred texts. But it certainly has revered ones, read by laypeople eagerly seeking a glimpse into their own elusive essence.

The first great writer in this genre was the Russian Alexander Luria whose The Mind of a Mnemonist, published in English in 1968, inspired a young Oliver Sacks, whose own first book Awakenings followed in 1973.

Luria and Sacks established the template of using bizarre case studies, such as the eponymous Man Who Mistook His Wife for a Hat in Sacks’s 1985 work, to explore the strangeness and fragility of human consciousness and sense of self.

When a year earlier Sacks had published A Leg to Stand On, describing how, after an encounter with a bull, he lost awareness of his left leg, he established memoir as a second strand of popular neuroscientific writing.

The problem for those who followed Luria and Sacks was that it was impossible not to walk in their footsteps, but equally impossible to fill their shoes. The clinical neuropsychologist Paul Broks is one of the few who has managed to rise to the challenge. His brilliant 2003 debut Into the Silent Land was in Sacks territory, but Broks, a former Prospect columnist, had his own distinctive voice, marked by an unusual combination of analytic thought and poetic lyricism.

The fact that it has taken 15 years for the follow up to emerge says something about how seriously Broks takes his writing. Sadly, however, that is not the only reason for the hiatus. As The Darker the Night, the Brighter the Stars explains, Broks has had to deal with the illness and subsequent death of his wife, Kate, from cancer.

His book, though, is much more than a memoir of grief. It is as though Kate’s death provided the unwanted opportunity to put his life’s work and thought into context. Far from enabling him to tie everything together, the effect is the opposite. The book is comprised of dozens of short chapters that don’t obviously link with each other.

Some are philosophical essays on the nature of the self and consciousness, some imagined conversations and made-up autobiography, some retellings of Greek myths. The book’s piecemeal structure dramatises the concept of the self Broks endorses: there is no single, stable, permanent essence, only fragments that add up to a reasonably, but never completely, coherent whole.

In lesser hands this would be indulgent and pretentious, but the book is grounded by its author’s unflinching honesty. Reading CS Lewis’s A Grief Observed, Broks found his own experience to be so different “that I began to question whether I had grieved at all.”

He was not depressed, numb or angry, just hit by “overwhelming waves of sadness.” For Broks, “grief is seeing the universe upturned, as if through alien eyes, the stars tumbling like kaleidoscope beads. Stabs of absence; stabs to the brain and heart; an entering of the flesh, a knowing in the flesh that she’s not here anymore.”

This captures Broks’s tone well: unsentimental but not unfeeling. He doesn’t shy away from observing what others might find unsavoury. He identifies what he calls “bereavement envy” in friends who had problems with their own partners. “My relationship with Kate was about to reach its inevitable conclusion, but it would be a clean, final break, unlike the guilt-ridden, tearing-apart they were going through.”

But if neuropsychologists were a priesthood, Broks would be the first to try to get himself defrocked. During his darkest days, he notices “my knowledge of clinical psychology has seemed irrelevant, or if not irrelevant then certainly peripheral to my deepest needs and concerns.”

When he hears the often-repeated claim that science has rejected the soul, self and free will as illusions, he expresses some impatience. “The human brain is a storytelling machine and the self is a yarn it spins. That’s it. Nothing more. The story is all. Blah, blah, blah.”

It’s easy enough to understand, as Broks does, that there is no permanent self; that we are always in flux and internally divided. The difficult task is to know how to live in the light of this knowledge. For this you need the kind of insight that comes from close attention to whole human beings, not from analysis of their brain scans. Broks has this kind of insight in spades.

Despite, or rather because of, his willingness to stare reality in the face, Broks’s book is ultimately uplifting. Without naming it, he seems to capture the spirit of the Japanese concept mono no aware—the bitter-sweet pathos of things.

This fragile, fleeting life is both beautiful and absurd, a source of joy and sorrow. So even at the peak of his mourning, he could be captured by the wondrous thought: “I’m still here. Right here, right now,” a feeling made even more poignant by the fact that Kate is not.

Neurologist Suzanne O’Sullivan’s Brainstorm is a more conventional collection of 12 case studies, or “detective stories” as the subtitle eye-catchingly puts it.

Serial readers of books like this will be able to tick off the inevitable appearance of Phineas Gage—the 19th-century railroad worker whose personality changed dramatically when part of his frontal lobe was destroyed by an iron bar—before we have even reached the end of the introduction.

The famous HM—who was unable to form long-term memories and appears in Sacks’s work—obligingly turns up soon after.

Despite breaking no boundaries, O’Sullivan’s book is a very welcome addition to the literature. Her specialism is epilepsy, which has 50m sufferers worldwide, 600,000 in the UK alone. Of these, 70 per cent can have their symptoms put into remission with medication, but 30 per cent continue to have seizures. Brainstorm is an exercise of much-needed public education about this surprisingly prevalent and often debilitating disease.

O’Sullivan’s humanity and humility whisper gently from every page. Far from presenting herself as a medical hero, she is brutally honest about the limits of what she and science can achieve. “It’s exciting to read about every new innovation, but my optimism is always contained,” she writes.

Both public and professionals can be dazzled by the latest brain scans but O’Sullivan deflates the hype when she describes an experiment that showed an area of a salmon’s brain activated when the fish was asked questions about their emotions.

Even when brain mapping is accurate, there is often little that can be done with the data. “Neurologists are in the business of trying to preserve brain cells because nobody knows how to replace them. We are as bad at curing brain disease as we ever were. Almost.”

There are some happy endings in this book but O’Sullivan refuses to cherry-pick the feel-good stories. One patient, Adrienne (all names are changed) was cured when a small section of her temporal lobes was removed.

But surgery remains a tricky and uncertain option: an operation on Gabriel, another patient, ended his seizures; but it left him more irrational, paranoid and unpredictable. His marriage fell apart and he lost his job.

Others also did not improve under their doctor’s care. Eleanor got worse, having seizures every day. Tim made it to university only to die from Sudep (sudden unexpected death in epilepsy), which kills one in a thousand epileptics annually, 600 a year in the UK.

O’Sullivan finds the “lack of both cure and prevention for problems like multiple sclerosis, epilepsy, Parkinson’s disease, Alzheimer’s, autism, schizophrenia and many more” deeply sobering.

Like Broks, O’Sullivan’s honesty means that when she does offer shafts of light we appreciate their warmth and brightness. Eleanor had to change her life drastically but, says O’Sullivan, “she did not consider herself ill. She accommodated her epilepsy but it didn’t define her.”

Similarly, Maya had epilepsy for 50 years, severely limiting her life choices. But “she lived with it and tolerated it. She seemed to me to be a woman who had had a good life—not perhaps the life she would have had if she had not developed epilepsy, but good nonetheless.”

You can understand O’Sullivan’s claim that she often thinks she is of little help to her patients; but it is clear that her patients are very lucky to have her.

***

Barbara K Lipska’s memoir is a reminder that neuroscientists are not all great sages. The Neuroscientist Who Lost Her Mind is a standard inspirational page-turner, “an incredible journey” full of the usual martial and sporting clichés.

She stays in shape to fight her cancer “like a soldier always ready for battle,” and inspired by cyclist Lance Armstrong—an odd choice you might think now—finds herself “getting ready for the competition of a lifetime.”

Lipska reveals, sometimes unwittingly, just how little her neurological expertise had prepared her to face mortality.

Used to feeling fit, she seems to fall for the myth that by following the right lifestyle, sickness can be kept at bay. “We had ambitious plans for our future,” she writes. “Cancer was not among them.”

It would take a monster not to feel deep sympathy for all Lipska went through. Apart from the physical symptoms caused by the metastasis melanoma in her brain, she suffered two months of mental illness—“a bizarre tailspin” into a very “dark place.” But the unreal perfection of her life and her family as she describes them makes it hard to like her.

This driven successful woman runs marathons and triathlons but after every one has still “returned home tired but beaming with happiness and prepared our dinner.” She and her husband “love to sit in a spacious dining room overlooking the woods enjoying glasses of wine,” at least when they’re not “watching a movie on the huge flat-screen television in our basement turned home theatre.”

Their children, meanwhile, “have beautiful three-storey homes of their own” as well as grandchildren who adore their babcia.

The contrast with Broks is striking. Fearing her death, Lipska finds herself thinking about her husband: “Mirek cannot stay alone. How difficult it would be for him in our house, with everything the same but without me there anymore?”

The late Kate was less sentimental. “You’ll find someone else soon enough when I’m gone,” she tells Broks, adding “A lot of widowers find a new partner within a year or two,” before suggesting candidates.

Towards the end of the book, Lipska thinks: “I am not exactly the same person that I was before my illness. But strangely, I feel completely myself.” Her thought ends just at the point Broks would have taken it up. His book brims with fascinating reflections on the nature of self. At one point, looking through old photographs, he notices that “I’m recognising furniture and wallpaper more than I’m making connection with my childhood self.”

That’s initially surprising but it shouldn’t be. A child changes more in a decade than the home they live in. This observation echoes William James’s idea that a person is in part made up of their possessions, clothes and surroundings. This “material self” is sometimes more stable and constant than the ego—the perception of an enduring “I” that persists in the stream of consciousness.

Nearly 50 years after Luria gave birth to the popular neuroscience book, Broks and O’Sullivan show that its potential is far from exhausted. Nor should we worry that the incessant probing of human subjectivity is going to abolish wonder.

“I have no need to see humanity unravelled,” writes O’Sullivan. “But, of course, I needn’t worry. We are not even close.”