Chekhov (centre, reading) is widely acknowledged as the greatest ever short story writer

Let us begin at a notional beginning. I have an image in my head of a band of Neanderthals (or some similar troupe of humanoids) hunkered round the fire at the cave-mouth as the night is drawing in. One of them says, spontaneously: “You’ll never believe what happened to me today.” Gnawed bones are tossed aside, children are quietened and the tribe gives the storyteller its full attention. The anecdote, the fond reminiscence, the protracted joke, the pointed recollection are surely the genesis of the short stories we write and read today. You could argue that storytelling in one form or other is hardwired into our human discourse as if—as soon as our sense of time past and time future evolved in our awakening consciousnesses—we became aware we could shape the telling of our personal histories and imagine possibilities that would enchant, terrify, enthral, admonish, titillate—and the rest of the gamut of emotions that attend a compelling story.

All this is somewhat fanciful and unproveable, I know, but it strikes me that something of this order must explain the strange power of short fiction. The short form is, conceivably, more natural to us than longer forms: the anecdote that lasts several hours is going to find its listeners drifting away pretty soon. The stories we tell to each other are short, or shortish, and they are shaped. Consider what happens in the telling of a tale: even the most unprofessional anecdotalist will find him or herself having to select some details and omit others, emphasise certain events and ignore the irrelevant or time-consuming, elide, speed up, slow down, describe key characters but not all, in order to head—ideally—towards a denouement of some sort. A whole editing process is engaged, almost unconsciously, of choosing, clarifying, enhancing and inventing. A convincing lie is, in its own way, a tiny, perfect narrative. The well-told story seems to answer something very deep in our nature as if, for the duration of its telling, something special has been created, some essence of our experience extrapolated, some temporary sense has been made of our common, turbulent journey towards the grave and oblivion.

If all this is true then why has it taken so long for the short story, as a literary form, to evolve? After all, the cultural history of the published short story is only a few decades longer than that of film. The answer, of course, is to be found in industrial and demographic processes. The short story had always existed as an informal oral tradition, but until the mass middle-class literacy of the 19th century arrived in the west, and the magazine and periodical market was invented to service the new reading public’s desires and preferences, there had been no real publishing forum for a piece of short fiction in the five to 50-page range. It was this new medium that revealed to writers their capacity to write short fiction. Readers wanted short stories, and writers suddenly discovered they had a new literary form on their hands. The way the short story effectively sprang into being in its full maturity almost proves my point. There were no faltering first steps, no slow centuries of evolution. The fact that in the early to mid-19th century Hawthorne and Poe and Turgenev were capable of writing classic and timeless short stories virtually from the outset signals that the ability had always been dormant within the human imagination. The short story arrived fully fledged in the middle of the 19th century and by its end, in the shape of Anton Chekhov, had reached its apotheosis.

So who wrote and published the first true modern short story? Who was the great precursor? Short narratives and tales had existed for centuries in one form or another: think of Scheherazade, Boccaccio’s Decameron and the Canterbury Tales, let alone the Bible, subplots in plays and novels, satires, pamphlets, sagas, narrative poems, essays, journalism. But what is the first literary text we can point to, classify and declaim with confidence: “This is a modern short story”? It has been argued that the honour goes to Walter Scott’s story “The Two Drovers,” published in Chronicles of the Canongate in 1827. It’s a convenient starting point, if only because the short story’s subsequent rapid development was international and Scott’s influence, huge in its day, was international also—not only inspiring George Eliot and Thomas Hardy at home, but also Balzac in France, Pushkin and Turgenev in Russia and Fenimore Cooper and Hawthorne in America. If one thinks of the influence these writers had in turn on Flaubert and Maupassant, Chekhov, Poe and Melville we can credibly begin to trace the birth lines of the modern short story back to its original source. The only problem is that after Scott’s start, the short story in Britain hardly existed in the mid-19th century, such was the dominance of the novel; writers in France, Russia and America seemed to take more immediately to the form and it’s not until Robert Louis Stevenson in the 1880s that we can see the modern short story beginning to emerge and flourish in Britain once more, with the line extending on from Stevenson through Wells, Bennett, James and Kipling.

Therefore, in many ways the true beginnings of the modern short story are to be found in America. One might posit the publication of Nathaniel Hawthorne’s Twice-Told Tales in 1837 as a starting point. When Edgar Allan Poe read Hawthorne, he made the first real analysis of the difference between the short story and the novel, defining a short story quite simply as a narrative that “can be read at one sitting.” This is not as facile as it may seem at first. What Poe was trying to put his finger on was the short story’s curious singularity of effect, something that he felt very strongly came from its all-in-one-go consumption. Poe continues: “In the whole composition there should be no word written, of which the tendency, direct or indirect, is not to the one pre-established design. And by such means, with such care and skill, a picture is at length painted which leaves in the mind of him who contemplates it with a kindred art, a sense of the fullest satisfaction.”

Poe is perhaps too schematic and prescriptive—wanting only one “pre-established design” as the dominating template of a short story—but he is very acute on the nature of the effect a short story can achieve: “a sense of the fullest satisfaction.” The short story can seem larger, more resonant and memorable than the shortness of the form would appear capable of delivering. One thinks of Poe’s stories—the first detective stories among them—such as “The Fall of the House of Usher” and one realises he was attempting to practise what he preached. However, I would take Poe’s definition a step further and recast it thus: the true, fully functioning short story should achieve a totality of effect that makes it almost impossible to encapsulate or summarise. For it is in this area, it seems to me, that the short story and the novel divide, where the effect of reading a good short story is quite different from the effect of reading a good novel. The great modern short stories possess a quality of mystery and beguiling resonance about them—a complexity of afterthought—that cannot be pinned down or analysed. Bizarrely, in this situation, the whole is undeniably greater than the sum of its component parts. Poe, perhaps inadvertently, achieved this on occasion, but the writer who followed Poe and in whom we see this quality really functioning is Herman Melville.

Melville hated writing stories—he claimed to do so purely for money—but it is in Melville’s stories, published in The Piazza Tales (1856), such as “Benito Cereno” and “Bartleby the Scrivener” that the modern short story comes of age, with remarkable suddenness. In Melville’s stories you can see the first real exemplars of the short story’s strange power. If you understand and relish what Melville is doing in “Benito Cereno” then you can understand and relish what is happening in Stevenson’s “Dr Jekyll and Mr Hyde,” in Conrad’s “The Secret Sharer,” in Chekhov’s “House with the Mezzanine,” Hemingway’s “Hills like White Elephants,” Mansfield’s “Prelude,” Carver’s “Cathedral,” Nabokov’s “Spring at Fialta,” Spark’s “Bang Bang You’re Dead,” Borges’s “Funes the Memorious,” to name a very few. We cannot summarise or paraphrase the totality of effect of these stories, try as we might: something about their unique frisson escapes or defies analysis. It is Melville who establishes the benchmark for what the short story can attain and allows us to set the standards by which all the other great writers of the form can be measured.

Turgenev was also publishing short stories in the 1850s—and one could throw his hat in the ring with Melville’s as the first originator of the modern form—but Turgenev’s great contribution was to start something that Chekhov finished. Why is Anton Chekhov (1860-1904) routinely and correctly described as the greatest short story writer ever? All answers to this question will seem inadequate but, to put it very simply, the fact is that Chekhov, in his mature stories of the 1890s, revolutionised the short story by transforming narrative. Chekhov saw and understood that life is godless, random and absurd, that all history is the history of unintended consequences. He knew, for instance, that being good will not spare you from awful suffering and injustice, that the slothful can flourish effortlessly and that mediocrity is the one great daemonic force. By abandoning the manipulated beginning-middle-and-end plot, by refusing to judge his characters, by not striving for a climax or seeking neat narrative resolution, Chekhov made his stories appear agonisingly, almost unbearably lifelike. Chekhov represents the end of the first phase of the modern short story. From his death onward, his influence is massive and ineluctable: the short story becomes thereafter in the 20th century almost exclusively Chekhovian. Joyce is Chekhovian, Katherine Mansfield almost plagiaristically so, Raymond Carver simply could not exist without him. Perhaps all short stories written after Chekhov are in one way or another in his debt. Only in the last 20 years or thereabouts have writers begun to emerge from his shadow, to middling effect.

But with Chekhov and with the advent of the 20th century, the modern short story entered its golden age. The adjective is very apt: in the early decades of the century you could become rich writing short stories, particularly in America. Magazines proliferated, readers were eager, circulation rose, fees went up and up. In the 1920s, Scott Fitzgerald was paid $4,000 by the Saturday Evening Post for a single short story. You need to multiply by at least 20 to arrive at any idea of the value of the sum in today’s terms. It was about this time, also, as the short story’s popularity grew and was subjected to the pressures and influence of modernism, that the form began to metamorphose somewhat: certain types of short story became distinct from each other and the form’s categories grew.

A couple of years ago I wrote an article in the Guardian (reprinted in my collection Bamboo) in which I proposed a rough taxonomy of the short story and came up with seven basic varieties. Fundamentally, up until the beginning of the 20th century, you have the two great traditions: the event-plot story and Chekhovian story. The event-plot story (the term is William Gerhardie’s) refers to the style of plotted story that flourished pre-Chekhov—before his example of the formless story became pre-eminent. Most short stories, even today, fall into one of these two categories. From them other types emerged over the coming decades. Perhaps the most dominant of these new forms is what I termed the modernist story, in which a deliberate, often baffling obscurity is made a virtue, however limpid the style in which it is written. Hemingway was the great practitioner here (In Our Time being the key volume), and after Chekhov his influence on the 20th-century short story is possibly the greatest.

Next among the other varieties I classified was the cryptic/ludic story. In this form of story there is a meaning to be deciphered that lies beneath the apparently straightforward text. This is also known as “suppressed narrative” and is a more recent development—perhaps the first clear move away from the great Chekhovian model. Mid-20th century writers like Nabokov, Calvino and Borges are representative of this mode of writing, though Rudyard Kipling, in stories such as “Mrs Bathurst” (1904) and “Mary Postgate” (1917), is an early master of suppressed narrative. The mini-novel story is a variety of the event-plot, trying to do in a few pages what the novel does in hundreds. One could see Dickens’s “Christmas Specials” as early examples of this type, though many short story writers turn to it from time to time (including Chekhov). The next category, the poetic/mythic story, is a rarer beast. Dylan Thomas’s and DH Lawrence’s stories are typical and JG Ballard’s bleak voyages into inner space also conform to this set. Here the short story comes as close to lyric poetry as it can—and in so doing most obviously attempts to defy easy summary. Ballard’s tremendous short stories—a haunting body of work that stretches from the 1950s to the present day—will come to be seen as one of the few successful attempts to escape Chekhov. The final category, and one that brings us up to the present day, is what I called the biographical story, a catch-all term to include stories that flirt with the factual or masquerade as non-fiction. Often the impedimenta of the non-fiction book is utilised (footnotes, authorial asides, illustrations, quotations, font changes, statistics, textual gimmickry). This is the most recent transmutation of the short story form and largely originated in America in the 1990s, where it has found particular favour with younger writers: Dave Eggers, David Foster Wallace, William T Vollman are notable exponents. In the hands of less capable writers, this mode can easily degenerate into the whimsical or the twee (almost deserving of its own sub-class). The biographical story also includes stories that introduce real people into fiction or write fictive episodes of real lives. This can be seen as an attempt by fiction, in a world deluged by the advertising media, the documentary, journalism, and 24-hour rolling news, to colonise some of that territory, to invade the world of the real and, as a cannibal will devour the brain of his enemy to make him stronger, to make fiction all the more powerful by blurring the line between hard facts and the invented. It owes little to the Chekhovian example and is potentially the most interesting new direction the short story has recently taken.

As one of the judges for the inaugural National Short Story Prize, I have read dozens of stories over the last few months, allowing me, to some extent, to diagnose the health of the contemporary short story. What is most intriguing is that the two main styles—the event-plot and the Chekhovian—are still hugely dominant, particularly the latter. The other varieties made an appearance but there was scant sign of formal audacity or experiment. It was as if the level of achievement arrived at in the early 20th century represented some sort of comfort zone for 21st-century writers. Our shortlist of five included, according to my makeshift taxonomy, one event-plot story, two Chekhovians, one mini-novel and one cryptic/ludic (see below). But the standard was noticeably high, all the same, and it seemed to indicate that the short story form was in a respectable state. Indeed, even though it has become harder than ever to publish short fiction in this country (let alone elsewhere in Europe), the American market is still large and remunerative, and the increasing, not to say incremental, growth of creative writing courses in the US and also in Britain may be responsible for the next phase in the form’s short history. The short story is both relatively quickly written and the perfect pedagogical tool. I think this explains why more and more young American writers are turning to it, and why more American publishers are publishing collections. Publishers on this side of the Atlantic will doubtless follow suit.

For the taste among readers for short fiction, inculcated over the last century and a half, has never really gone away, despite the vagaries of publishing economics. The reason lies, as I suggested earlier, in some unconscious predisposition in our minds for the short narrative but, perhaps more pertinently, the revolution that Chekhov brought about was to create a type of short story in which narrative changed. For Chekhov, randomness, inexplicability and haphazard elision became the actual form of the short story—shapely design, authorial manipulation, the tailored conclusion were abandoned—and suddenly we had a fictional style that corresponded with the random, haphazard, inexplicable lives we all lead. Virginia Woolf was not a particularly accomplished writer of short stories (which was perhaps why she was so jealous of Katherine Mansfield) but she was an avid and talented amateur photographer. She said of photography, “Isn’t it odd how much more one sees in a photograph than in real life?” This gives us, I think, a clue to the enduring power and appeal of the short story—they are snapshots of the human condition and of human nature, and when they work well, and work on us, we are given the rare chance to see in them more “than in real life.”