TransactionCosts are resources spent in seeking or making a business deal that go to neither the buyer nor the seller; they're costs associated with the deal itself, not with the product or service that's being bought.
Imagine that a city finances its road-maintenance budget on the principle of "pay for what you use". Toll booths are installed at each street corner, and private contractors are responsible for (a) setting the tolls and collecting the revenue therefrom, and (b) maintaining the streets within each group of tollbooths that they collect from. The money spent on maintaining toll booths, finding and catching people who evade the tolls, handling complaints from drivers who say they were charged an incorrect toll, etc., etc., are TransactionCosts; they affect the price a driver pays when going through a tollbooth, but they don't affect the amount of money spent maintaining the road. Likewise, drivers might spend time researching the tolls at various corners to determine the route they want to travel; TransactionCosts include the value of the time they spend on this research, since that value goes to neither the roads nor the contractors maintaining them.
If TransactionCosts are trivial, then this city's road maintenance budget is ParetoOptimal; drivers who want cheap tolls will pick cheaply-maintained streets, drivers who want high-quality pavement will pick high-toll streets, and competition between the contractors will keep efficiency up.
However, if TransactionCosts are significant, then a lot of money and time will be wasted in the infrastructure of the toll system, rather than being spent on road maintenance. In such a case, the traditional system of road maintenance, where the city collects taxes from all its residents and then uses the money to pay for maintenance, would have a lower overall cost to residents than the toll system. (Of course, the dollars that the city spends on tax assessment, collection, and enforcement are also, in their own way, TransactionCosts.) --SethGordon
Suppose I'm a contractor in our city-of-tollbooths scenario and I come up with a technique for making more cost-efficient road improvements, but implementing this technique has a certain up-front cost. If TransactionCosts are zero, I can borrow money (from banks or investors) to cover the upfront costs, reduce tolls through my streets (or provide higher-quality streets for the same tolls) to attract more customers, and pay what I borrowed, and then some, with the extra revenue. Even a marginal improvement in technique (say, one that lets me cut tolls by 5%) can improve my profits.
But if TransactionCosts are significant, then the benefit accruing to me from my road improvement is diluted. I may have customers who are using my competitors' roads, and who could pay less by driving over mine -- but they don't know they could pay less, because they don't want to spend time looking for a new optimum route to work every time the tolls change. So we are no longer in a ParetoOptimal situation. I could advertise my cheaper road system to try to bring the customers in, but if my innovation is sufficiently small, the cost of marketing might overwhelm the benefit from the new customers.
By contrast, if I hire my company out to do road maintenance for a city government, and the city worries about how to collect the money to pay me, then even a marginal improvement in technique can help my bottom line, because even if I can underbid my competitors by 1%, I win the contract and they don't.
(Of course, all this talk about road maintenance is making me think about Boston's "Big Dig", a.k.a. the Central Artery/Tunnel Project, whose gargantuan cost overruns have become a staple of the local news. Maybe I should look for a better analogy. :-) --SethGordon
Gathering taxes in a uniform manner and distributing universally needed services such as defense, road maintenance, law enforcement, disaster relief, etc... seems likely to have lower TransactionCosts than leaving these things to the private sector. Things like economy of scale and the value of standards come into play.