The first time I heard The Shepherd's Dog, I vomited into a hat. The record hasn't elicited this reaction since, but the incense-smoke sitars on the spiralling 'White Tooth Man' turned a hungover journey down the M1, all the spiteful juddering and furious pneumatic drills, into a kaleidoscopic netherworld that I couldn't swim up from. The album's technicolor overload caught me unawares, considering Iron and Wine's earlier modus operandi, dusty acoustic songs left largely unadorned. Three years after The Shepherd's Dog, auteur Sam Beam put out a boisterous, tetchy interpretation of 70s FM radio, whose blaring horns and four-minute outros ventured even further from 'Upward over the Mountain' than the album that preceded it. He's not a man to rest on his laurels, is what I'm saying.

On first listen, it's tempting to imagine that Beam's slowing down a little with fifth full-length Ghost on Ghost. Opener 'Caught In The Briars' has all the hallmarks of the friendlier songs on Kiss Each Other Clean – a warm mix, sonorous brass belches, Muscle Shoalsy 'ooh's. Beam prefaced Ghost on Ghost with a statement that attempted to distance the record from its two closest antecedents, saying that there was an 'anxious tension' to The Shepherd's Dog and Kiss Each Other Clean that he wanted to move away from. "Back alleys full of rain, and everything's shining," he sings on 'Caught In The Briars', and it's true, Ghost on Ghost sounds slicked, clean, uncluttered by the claustrophobic miasma of effects that hung over his last two efforts. Lead single 'Grace For Saints And Ramblers' is genuinely jaunty, has a bit of the Paul Simons in its popping bass and syncopated keys. Beam gets his smooth on for 'Desert Babbler' and 'New Mexico's No Breeze', which expertly channel Nixon-era Kurt Wagner and the Lambchop man's love of Memphis soul, lush strings. There's a song called 'Joy', fer Chrissakes, which if not exactly jubilant in tone, at least sounds cautiously optimistic. It's a wonderful tune, by the way, Beam eschewing his usual intoxicating rush of images for a poetic honesty that gets right to the core of the wobbles in a good relationship, all over a twinkling Rhodes, probably the nicest-sounding instrument in the world. I like it a lot, all the more so for Beam's refusal to ham up the sentiment, choosing to clock it in at under three minutes.

So all this white tooth happiness, this expansive warmth, it gets pretty boring, right? Perhaps in the hands of another, it would. But as we soon come to realise, Ghost on Ghost is rammed to the peeling rafters with the kind of vaguely unsettling creepiness that Beam began to flirt with on 'Peace Beneath the City' and perfected around 'Rabbit Will Run', the luxuriant paranoia that gave a sensitive bedroom folkie, that most bland of musical archetypes, such a deliciously unsettling edge. This time, instead of splattering filters over everything or chucking the contents of a school music cupboard at a song, Beam's using the power of jazz (yes, the power of jazz) and all her tributaries to commune with the far-out on Ghost on Ghost. Our first warning comes at the close of 'Caught In The Briars' straight-laced pop, which breaks down into a brief freakout at its close, drunken piano tumbling over what sounds like a shehnai. It's vaguely jarring, but you can chalk this up to anomaly. That is, until Beam mangles that soul imprint into something growling and obsessive for 'Low Light Buddy Of Mine', or the way 'Grass Widows' puts you in mind of some weird, druggy cocktail lounge where someone may or may not be having their toes broken in a back room, or how he manages to make a "quiet line of trees" sound like the scariest thing ever on the taut funk of 'Singers And The Endless Song'. It's 'Lovers Revolution' that tops them all, though, which creeps from gutter-jazz beginnings to an amphetamine-bop breakdown that bypasses brains and goes straight for the hips, a visceral danceability that Beam's barely even hinted at before. He's snarling and angry and bitter through the whole thing, too, and to hear this man of the gossamer whisper bare his teeth like this is completely thrilling. The only bad thing about 'Lovers Revolution' is that it turns closer 'Baby Centre Stage', a crying-lapsteel ballad just made for closing a record, into something unremarkable.

Ghost on Ghost is the sound of a man who's settled, but not complacent. The way 'Winter Prayers' is hidden in the record's hinterland at track nine feels deliberate, as though Beam's reminding us that he can still write something desolate and heartstopping with barely more than his guitar and voice, but that he'd never be so crass as to take the safe option. Ghost on Ghost can feel underwhelming at first, like it doesn't quite claw deep enough after the psychoactive mess of the past two Iron and Wine records, or the unspeakable closeness of Sam Beam's earlier work. But there's a buried edge here, there's rushing torrents of freaky imagery, there's opportunities to groove.

And, yes. The power of jazz.