It’s impossible not to fall in love with Sidi Bou Said. Originally a place of pilgrimage for visitors to the tomb of the 13th-century Sufi holy man from whom it takes its name, Tunisia’s celebrated blue-and-white village is so enchantingly pretty that it can sometimes seem more painted than real. Seen from a distance, it shimmers under the fierce Mediterranean sun like a giant mosaic. Seen from within, it’s a labyrinth of winding streets and secret places, where crooked flights of steps lead to hidden gardens and wooden gates opening onto flower-filled courtyards.

Everywhere you look in the village, you’ll see its signature colours – dazzling white walls and staircases, with everything else a uniform shade of vivid blue. Doors, window frames, shutters, decorative iron grilles and elaborate latticework window screens (known as moucharabiehs) are all painted the imperial blue of a peacock’s tail. Only the larger doors show some variation, with splashes of chrome yellow, white or red.

Doors are a special feature of the village; ancient, huge and heavy, they’re studded with traditional motifs of crescents, minarets and stars. Some lead into simple shops or mini-garages, others open out into more elaborate complexes, such as the Café Sidi Chabaane, which cascades over the cliff-side in tiered terraces like a waterfall. Others mark the entrance to grand mansions with cool, mosaic-tiled courtyards, fountains and orange trees – rather like Moroccan riads.

A number of doors lead into forges, workshops or artists’ studios. The Italian artist Soro Lo Turco has a studio and gallery here, for instance. It’s a topsy-turvy building up an alleyway, filled with his characteristic smoky ink and pastel paintings. Unsurprisingly, Sidi Bou Said has always been attractive to painters. It became a hideaway for writers and artists visiting from Europe in the late 19th and early 20th centuries. Colette, Simone de Beauvoir and André Gide were among the writers who fell under its spell. Gide described staying there as “bathing in a fluid, mother-of-pearl sedative”.

Visiting artists included Henri Matisse, Michael Foucault and Paul Klee. Klee arrived in 1914, with fellow-artists Gustave-Henri Jossot and August Macke. That year, Macke painted a watercolour of the Café des Nattes, which has become one of the most enduring images of Sidi Bou Said. For Klee, who until then had normally worked within the confines of black and brown graphics, his time in the village marked a decisive turning point in his use of light and colour. “Colour has taken possession of me,” he wrote. “No longer do I have to chase after it, I know that it has hold of me forever.”

By the end of the 19th century, cosmopolitan Tunis had become something of a surrogate Paris for vacationing Europeans. Smaller, prettier, more subtle, and situated 18km from the capital, Sidi Bou Said became Tunis’s equivalent of Montmartre – the “alternative” place to stay, or even to settle, for the more Bohemian cognoscenti.

Nowadays, like Montmartre itself, the village is no longer any sort of a secret. Its population has been swelled by an influx of wealthy Tunisians. A number of politicians and media personalities make daily trips between their luxury homes and the capital in what can look like a cavalcade of chauffeur-driven cars with tinted windows. Likewise, the village has been discovered by mass tourism. As an increasingly popular excursion destination in recent years, its cobbled main street has long been worn to a sheen by the daily tidal waves of visitors. And yet Sidi Bou Said remains enchanting. The secret is to stay for a few days, rather than going on a day trip.

Coach tours apart, most visitors to Sidi Bou Said arrive on the local TGM Métro from Tunis, known locally as the “Blue Train” and operating every 20 minutes or so. Cars are banned from the village’s historic centre. It’s a swift 35-minute journey, and then a 15-minute (uphill) stroll to the old village. At first, you might wonder what all the fuss is about. Downtown Sidi, around Place 7 Novembre, is an unremarkable modern jumble of supermarkets, hardware shops and cheap sandwich bars. The latter are well worth remembering later, if you’re on a budget or in retreat from fellow-tourists. They specialise in brik, deep-fried filo pastries filled with runny yellow egg and either mince or tuna, or takeaway tubs of mechouia, a salad of grilled vegetables.

As soon as you turn the corner for the upward climb, you’ll see immediately what makes the village so special. In front of you, the lovely main street (Rue Dr Habib) is a succession of white sugar-cube buildings, with latticed blue balconies overhanging the cobbles and forming an avenue as you walk.

Pink clouds of bougainvillea festoon railings and trail from the tubs or hanging baskets that are scattered up countless small side-streets and alleyways. Directly ahead is the main mosque. And just below the minaret, up a flight of 22 white-edged steps, is the famous landmark of Macke’s painting: the Café des Nattes (“Café of the Rush u o Mats”). Its interior is extraordinarily weird and wonderful – cool, dark and fabulously decadent, with elaborate Tunisian water pipe apparatus positioned at strategic points.

Customers sit or recline on rush mats spread across vast stone platforms. These are like four-poster beds, with thick pillars set at each corner. Vividly striped with twisted bands of red and green, the pillars support a low ceiling painted in the same startlingly bright colours. A cross between an opium den and a fairground roundabout, it’s like no other café I’ve come across. In summer, though, the café’s clientele stays mostly out of doors. Locals and visitors alike crowd the steps and balconies to drink glasses of traditional thé aux pignons – sweet mint tea with pine-nuts floating on the surface – and to people-watch onto the street below.

By day, Rue Dr Habib is a bazaar – a street hawkers’ paradise, thronged with craft stalls and shops full of jewellery, aromatic oils, inlaid boxes, brassware, ornate bird-cages and leatherwork of all sorts. A bonbalouni stall fries and sells the village’s traditional sugar-coated doughnuts. Vendors with raffia trays weave in and out of the hordes of shoppers, peddling the ubiquitous miniature jasmine bouquets (bound with twine, and known as a mashmoum) that you’ll see many local men wearing tucked behind an ear.

Yet there are still ways to escape the mêlée – even at midday, when the streets are at their busiest. Sidi Bou Said is full of surprises!