05 October 2006

Open Source

This is a term of art used to refer to software whose code is available for free public distribution. Of course, "open source" may refer to other types of products, but in computer parlance it implies that that intellectual property rights are greatly restricted. The code may be edited and circulated freely, leading to the gradual evolution of software through the contributions of many.

The concept of "Open Source" software is not as immutable or uncontroversial as it might perhaps sound. For example, open source software used to be known as "free software," which implied both the non-pecuniary nature of the software plus its accessibility to editing by users plus a political connotation of freedom from the institutionally structured, command economy of a corporation or government agency. Well, a large volume of software is available "for free" from both; for example, Acrobat Reader is available for free (thereby creating a market for Acrobat Writer); the Army Corps of Engineers circulates lab software for free (mainly through state governments) for reporting environmental lab results. These are, however, designed to remain under the strict control of the developer, so that the software serves its institutional goals.

Another interesting case is the example of IBM, which distributed the source code to its mainframe software for many years. One of these operating systems, VM/MVS, survived long after IBM had withdrawn any sort of support, and became an early prototype of open source software. At that time, software played a comparatively small role in IBM's bottom line, and the company had an interest in helping customers develop applications that could be ported over to other customers.

Acrobat code is not open source, and hence users cannot modify the reader to be a writer; they certainly cannot make it better than the product distributed by Adobe. The software delivered by the Army Corps of Engineers, COELT, is deliberately unpleasant to use; the Army's software developers have no desire to intrude on the market for automated ELIMS.

In the case of user communities receiving "free" assistance from a private sector developer, it's long been observed that the community adds value to the product far in excess of the company's input. The value added to the product is partly captured by the firm by moving the demand curve for its products to the right, and the rest of the economy benefits from this increment in factor productivity. But this is not, in and of itself, the point of the free software movement.
When I first became aware of the existence of free software, it didn't seem terribly interesting to me because it seemed hopelessly idealistic. Sure, I could understand the desire for a college professor, grad student, hobbyist, or corporate employee to circulate her work for general use; what seemed implausible was the idea that the software would work for very long. For example, I was particularly amazed by the survival of native Unix applications, such as Emacs. How could a program survive if everyone could edit it?

Well, the short answer is that typically software must mesh with other systems whose future evolution is unknown. I could conceivably download GNU Emacs, for example, and edit it as much as I liked. By the end of the year I would have something I call JRM Emacs (i.e., a fork). My fork would still have to be compatible with the Unix API or else it would just sit there useless on the hard drive. On the other hand, it would also have to be able to save files that could be opened by other editions of Emacs--otherwise, it would just be a random heap of symbols, not a program. In order for a schism to appear in the Emacs world, in which two anti-Emacs with incompatible formats limp along with too many users for either to vanish, it would actually be necessary for one of two conditions to prevail:
  1. Either there is some very compelling thing my Emacs does, that all the others don't do;
  2. Or else I and a bunch of other folk went off into some isolated place for a long time, and not only re-wrote Emacs, but also the Unix API's, the libraries, a compiler or two, and several of the other programs that Emacs needs to work with
Should neither of these things happen, then my JRM Emacs has got to solve the problem of doing something better, while at the same time being compatible with all of the rest of the Unix community (that it was compatible with before, I mean). In which case, my JRM Emacs is probably going to be a plug-in, not an entirely new software, and no real schism is likely to appear.

Now, readers familiar with the "Unix Wars" may know the split between Bell Lab's System V and BSD was occasioned by the need to develop a version of Unix that used none of the original AT&T source code. As long as there was some trace of the AT&T programming in it, AT&T had a legal claim on the source code. BSD had initially incorporated much of this source code, and its developers had to effectively re-invent the wheel, so that the new wheel was in fact entirely "free." The importance of this accomplishment I shall explain in a moment.

Another important case is the "Browser War," which was really a sort of ambush and massacre, not a real war. The first common web browser was Netscape Navigator, a semi-free software that gained near-universal acceptance in the mid-1990's. At that time, Microsoft had won a monopoly of system & application software for the consumer PC market, and was mildly concerned about the possibility of competition from the Java platform. It had a browser of its own, MS Internet Explorer (MSIE), which was not catching on because it was an almost supernaturally bad browser. Microsoft started off by giving away the browser for free, then integrated it with MS Windows. There was some inconsequential anti-trust action, but most of this occurred after MSIE had driven Netscape Navigator [mostly] out of the market.

MSIE had won a monopoly on a product by reducing its price to nothing. IE was to prove unusually vulnerable to malware, and MS sat on its victory for six years with no innovation whatever. MS introduced non-standard HTML components, then web-authoring tools that excluded rival browsers (Note: MS Expression and Adobe Dreamweaver appear to have stopped this practice). A large number of fairly important websites effectively bar visitors with non-MSIE browsers, sometimes with cruel consequences.

Now, in the first case, the post-1991 BSD Unix did indeed do something prior versions of Unix did not: it was wholly open source software, which allowed huge numbers of grad students, hobbyists, desperate startups, and so forth to further develop. FreeBSD, OpenBSD, and NetBSD are actually "Unix-like," since they don't actually have permission to characterize themselves as Unix. But the development community is out there adapting to the new technologies available, and fixing bugs much faster than was the case during the years of bondage to AT&T.

In the second case, the division came about because Microsoft is so huge that it could make MSIE recognize nonstandard HTML variants, make Frontpage author those nonstandard HTML variants, promote nonstandard versions of Perl and Javascript (or "Jscript"), and incorporate non-standard elements in the web authoring features of MS Office applications. Please note the different theme in these two examples.
In his essay, "The Cathedral and the Bazaar," Unix guru Eric Raymond outlines two basic models of software development communities. One is the cathedral, in which a group of specialists craft standards, and then release those standards at widely-spaced intervals so that the hacking community can then implement them and develop plug-ins. The other is the bazaar, in which there is absolutely no centralizing authority.
Linux overturned much of what I thought I knew. I had been preaching the Unix gospel of small tools, rapid prototyping and evolutionary programming for years. But I also believed there was a certain critical complexity above which a more centralized, a priori approach was required. I believed that the most important software (operating systems and really large tools like the Emacs programming editor) needed to be built like cathedrals, carefully crafted by individual wizards or small bands of mages working in splendid isolation, with no beta to be released before its time.

Linus Torvalds's style of development—release early and often, delegate everything you can, be open to the point of promiscuity—came as a surprise. No quiet, reverent cathedral-building here—rather, the Linux community seemed to resemble a great babbling bazaar of differing agendas and approaches (aptly symbolized by the Linux archive sites, who'd take submissions from anyone) out of which a coherent and stable system could seemingly emerge only by a succession of miracles.

The fact that this bazaar style seemed to work, and work well, came as a distinct shock. As I learned my way around, I worked hard not just at individual projects, but also at trying to understand why the Linux world not only didn't fly apart in confusion but seemed to go from strength to strength at a speed barely imaginable to cathedral-builders.
Raymond's book is influential partly because he is a veteran of so many important open-source software initiatives, such as BSD Unix, and partly because it is actually fairly rare for a technological trend to be interpreted as part of a larger social event (in a reasonably plausible way). Also, Raymond is a talented writer.

A few economists had noticed analogous trends in the past. It was a source of considerable interest to them that highly decentralized groups could organize the planning of future needs and prices more successfully than, say, a benevolent dictator.
As every individual, therefore, endeavours as much as he can both to employ his capital in the support of domestic industry, and so to direct that industry that its produce may be of the greatest value; every individual necessarily labours to render the annual revenue of the society as great as he can. He generally, indeed, neither intends to promote the public interest, nor knows how much he is promoting it. By preferring the support of domestic to that of foreign industry, he intends only his own security; and by directing that industry in such a manner as its produce may be of the greatest value, he intends only his own gain, and he is in this, as in many other cases, led by an invisible hand to promote an end which was no part of his intention.
Wealth of Nations, IV.2.9, Adam Smith
Interestingly enough, a close reading of Smith (particularly in the light of his other great work, The Theory of Moral Sentiments) suggests that his intent was closer to Raymond's surprise dicovery, than the intersecting supply and demand curves attributed to it by classical economists who followed. Smith was interested in the idea of the individual guided by the point of view of an imaginary impartial observer, the "man in the glass." He had also noticed a tendency for humans to use the signals from each other to anticipate and adjust to maintain normality.

However, Smith devoted his work to the direct ways in which individual prudence maintained public balance. Left unresolved was the question of individual providentiality towards the commons, and how that created, or preserved commons. The notion of a true commons is one of the great mysteries of classical liberalism. Usually, when people think of a commons, they are thinking of something like a park or the system of freeways--a thing used in common, benefiting all, allowing free riders, and making exclusion of use difficult. However, the most famous examples are poor ones. They aren't commons so much as owned by a large institution that then collects "protection money." One economist, Ronald Coase, famously argued that the commons could and often was protected without a government "owner." In this case, we are talking about a true commons--something for which there is no owner-protector. We are talking about the creative commons, the commons of intellectual accomplishment.

This leads to the fairly interesting paradox of two admirers of Ayn Rand, Raymond and Rideau, with diametrically opposed ideas about the question of intellectual property rights. Raymond perceives intellectual property rights as an indispensable part of free enterprise; Rideau, as a tool for turning corporations and individuals into state surrogates (as enforcers of monopolies). Rideau, arguably, has assimilated Coase's Theorem and recognizes intellectual accomplishment as a form of commons; Raymond perceives Coase's Theorem as not terribly relevant, and intellectual labor as requiring a sort of accumulated property in order for the market to work. Observing as he does that the creative commons (or "ergosphere") flourishes without property rights, he unwittingly (?) invokes Veblen's observations about the leisure class.
Gift cultures are adaptations not to scarcity but to abundance. They arise in populations that do not have significant material-scarcity problems with survival goods. We can observe gift cultures in action among aboriginal cultures living in ecozones with mild climates and abundant food. We can also observe them in certain strata of our own society, especially in show business and among the very wealthy.
"The Hacker Milieu as Gift Culture"


In these psychological observations we can ground a case that an open-source development group will be substantially more productive (especially over the long term, in which creativity becomes more critical as a productivity multiplier) than an equivalently sized and skilled group of closed-source programmers (de)motivated by scarcity rewards.

This suggests from a slightly different angle one of the speculations in The Cathedral And The Bazaar; that, ultimately, the industrial/factory mode of software production was doomed to be outcompeted from the moment capitalism began to create enough of a wealth surplus that many programmers could live in a post-scarcity gift culture.
"Gift Outcompetes Exchange"
I now return to my cases of MSIE creating a standards schism by being able to alter the standards en bloc, like a perverse cathedral; and of the Unix schism created by a standard that was uniquely competent (in this case, capable of being open source). The latter case could only accomplish anything if it established the conditions of the bazaar, in which the ability of people to introduce innovations and improvements became universal.

At this point, I've run out of things to say; there's a ton of more valuable information, including on Wikipedia, available in the linked sites. I haven't finished reading Ko Kuwabara's and Forrest Cavalier's papers, both of which look really good. However, I want to include two disclaimers here before I sign out: first, I am merely studying this topic out of personal interest, and readers ought to know I'm not an expert; and second, like Sir Thomas More in A Man for all Seasons, I have kept silent on how I feel about the actual topic under discussion. What I have done is put forward what I believe is a body of pertinent positions on open source software, and you don't need to know what my opinion of them is.
ELIMS: electronic laboratory information management system; used to collect data about lab samples, from sample log-in, to extraction, to analysis, to the lab report, thence to archiving or electronic transmission.
ADDITIONAL NOTES & READING: The Cathedral & the Bazaar, Eric Raymond, O'Reilly Press (1998); "Some Implications of Bazaar Size," Forrest J. Cavalier, III, 1998; "The Cathedral Cleric and the Bazaar Preacher," Faré Rideau; "Linux: a Bazaar at the Edge of Chaos," Ko Kuwabara

Labels: , , , ,


Post a Comment

<< Home