Roy Edroso may be onto something:

[T]he fact remains: libertarians stand too close to you when they talk, sing along with Frank Zappa songs (even the obscure ones), and smell.

Although I think I prefer Julian Sanchez’s definition slightly better:

[L]ibertarians—I’m including myself here—tend to be an eccentric bunch who, pretty much as a prerequisite of seriously entertaining all the unpopular ideas we do, are skeptical of mainstream consensus and fond of quixotic crusades.

The signal trait of most (semi-sane) libertarians (myself among them) I’ve met has been contrarianism. It’s a reflexive inability to let prevailing wisdom pass without critical comment. This is why libertarians are generally ineffectual as a political force: consensus is almost impossible when everyone refuses to engage in the sort of compromise and nose-holding that coalition building generally requires. And even if a potential coalition appears, the mere fact of its appearance induces spasms of agitation and yelps of counterintuition. The tendency toward self-marginalization, I think, is generally not something that can be helped. (Some paleocons, I suspect, are similarly afflicted.) Politically, the chances for libertarianism are dim. On the other hand, we’re lots of fun at parties.