Well, I don't know about libertarians, but Adam Smith had a pretty clear vision of what government should, and should not, do. From Wealth of Nations, the end of Book IV:
It is thus that every system which endeavours, either, by extraordinary encouragements, to draw towards a particular species of industry a greater share of the capital of the society than what would naturally go to it; or, by extraordinary restraints, to force from a particular species of industry some share of the capital which would otherwise be employed in it; is in reality subversive of the great purpose which it means to promote. It retards, instead of accelerating, the progress of the society towards real wealth and greatness: and diminishes, instead of increasing, the real value of the annual produce of its land and labour.A remarkably concise yet robust statement of political philosophy, if you ask me.
All systems either of preference or of restraint, therefore, being thus completely taken away, the obvious and simple system of natural liberty establishes itself of its own accord. Every man, as long as he does not violate the laws of justice, is left perfectly free to pursue his own interest his own way, and to bring both his industry and capital into competition with those of any other man, or order of men. The sovereign is completely discharged from a duty, in the attempting to perform which he must always be exposed to innumerable delusions, and for the proper performance of which no human wisdom or knowledge could ever be sufficient; the duty of superintending the industry of private people, and of directing it towards the employments most suitable to the interest of the society. According to the system of natural liberty, the sovereign has only three duties to attend to; three duties of great importance, indeed, but plain and intelligible to common understandings; first, the duty of protecting the society from the violence and invasion of other independent societies; secondly, the duty of protecting, as far as possible, every member of the society from the injustice or oppression of every other member of it, or the duty of establishing an exact administration of justice; and, thirdly, the duty of erecting and maintaining certain public works and certain public institutions, which it can never be for the interest of any individual, or small number of individuals, to erect and maintain; because the profit could never repay the expence to any individual or small number of individuals, though it may frequently do much more than repay it to a great society.