24 January 2006

Milestones: Unix (2)

Flavors & "the Unix Wars"

(Part 1)

The huge range of choices IT managers could make with respect to Unix installations and interface meant that a lot of firms saw in Unix an opportunity to add value by making choices about interface and capabilities professionally. Unix can be made to run on so many different machines, while allowing the same programs to run on any of them with almost no modification (meaning, the Unix "flavors" all have consistent API's), that they have become the most popular software for running servers. Finally, there was the critical aspect of applications. Unix provides a comparatively easy programming environment, and has attracted a lot of hobbyists. As a result, a very large number of utilities were developed for Unix first, for free.

The first professional version of Unix was developed by the University of California at Berkeley (UCB) for the Pentagon; it was known as BSD Unix, and contained significant amounts of AT&T code (1980). AT&T, meanwhile, was suddenly able to sell Unix as a product as a result of the 1982 consent decree that broke it up. In '83 it therefore released System V, with the decided advantage that it fully supported the now-essential TCP/IP (internet) protocol. Further BSD releases were compelled to replace a lot of AT&T code with new freeware. AT&T and many of its rivals embraced the proprietary software model, which nearly ruined Unix.

In one sense, the competition and legal struggles of software vendors which became known as "the Unix Wars" was a battle over how to make money from specialized, high-performance software. The large corporate actors gradually realized that they could not sell the software as they had in the past, by licensing copies to paying customers. Instead, they would have to give much of their R&D away to the public domain, so that they could sell specialized hardware or applications (that did not compete with Intel or Microsoft).

In another sense, however, the "Unix Wars" were also about the form that Unix would take. There was a long series of agreements and new standards that eventually restored the unified API's. The resolution of the disputes was made possible by continued refinement and adoption of new approaches not available to original "combatants," who had been limited by smaller memories and slower processors. The landscape of the computer market changed considerably, with minicomputers becoming largely servers in large networks rather than lower-end mainframes. Unix began to dominate servers and workstations, such as Sun Microsystems (founded 1984) and Hewlett-Packard (HP-UX). In 1985, the advent of the Intel 386 chip made it possible to address 4 megabytes of memory with a flat address space. A major impact of this was the massive surge in workstations, which would require the specialized performance available mainly from Unix. Needless to say, the wholly closed development environment of Apple Macintosh and the consumer-oriented market for DOS/Windows were wildly unsuited to specialized graphical systems.

Hardware was an issue for the Unix market. The first Unix machines were the minicomputers, especially the Hewlett Packard 3000 & 9000 series. The first workstations used the same Motorola 68000's used in Macintoshes. Sun introduced 386-based workstations, then its own specialized SPARC architecture. HP developed the Precision Architecture (PA-RISC). A firm called MIPS Computer Systems (later bought by SGI and renamed MIPS Technologies) developed the first RISC architecture microprocessors, which were adopted by Unix workstations such as SGI's IRIS. SGI's products were particularly interesting since they blurred any boundary that had existed between high-end workstations and supercomputers; and they were used largely for the entertainment industry. In the 1990's, the Power PC chip developed by IBM, Motorola, and Apple became prominent for high-end servers.

By the late '90's, however, the industry was once more dominated by Intel-based workstations and servers, often physically indistinguishable from Windows machines. However, many of the innovations, such as RISC architecture and multiple threads, were developed for Unix and revolutionized the computer industry generally. Intel's rivals drifted back into producing application-specific semiconductors (ASICs) such as those used in appliances or videogames.

Linus Torvalds developed a variant of GNU with some Unix components that became known as Linux; this was a huge success. Later, in 2001, Apple released Mac OS X, which was based on a BSD kernal; this would make Macintosh the first mainstream computer to use a Unix kernal.

(Part 3)
ADDITIONAL READING & SOURCES: Wikipedia Unix; Unix System V; Berkeley Software Distribution (BSD); Tru64 UNIX (also known as OSF/1); Unix Wars;

USENIX Association website (fmrly Unix Users' Group); The Open Group: History and Timeline;

Living Internet: Unix; Eric Steven Raymond, The Art of Unix Programming

Labels: ,

22 January 2006

Milestones: Unix (1)

Unix was developed in the mid-1970's to run on a Digital PDP-7; it was eventually "ported" (modified to run on) virtually every computer ever produced. The name "Unix" is a play on the name "Multics," a mainframe operating system that had suffered from excessive size and complexity. It was developed as a hobby by Ken Thompson and Dennis Ritchie and later became one of the more influential products of Bell Labs.

The history of Unix has been told many times (e.g.; here); my interest is in what made Unix a milestone in the development of computer technology. Prior to the 1970's, computer processing was run in a batch format, i.e., without human interaction; typically programs themselves were designed to run like any other industrial machine, with a huge stream of homogeneous inputs subject to a homogeneous process, and with arithmetically predictable outputs, such as the payrolls for a large firm. Multics and Unix were designed for running large numbers of programs on the same processor interactively. This would make Unix compatible with future uses of the computer.

Unix accomplished this through modular architecture; the operating system was to consist of a kernel which was minimal in size, plus a library of utilities. The latter would depend on the computer platform and the peculiar needs of the user. An additional attribute, added in 1973, was the use of high-level programming language, C, for programming applications. This would mean that most of the effort of porting Unix to other machines would hereafter be confined to rewriting the compiler.

One of the results of this approach was that system managers in charge of Unix server or workstation would be allowed [or compelled] to make a very large number of choices about the look and feel of the OS:
X [Windows] strives to provide “mechanism, not policy”, supporting an extremely general set of graphics operations and deferring decisions about toolkits and interface look-and-feel (the policy) up to application level. Unix's other system-level services display similar tendencies; final choices about behavior are pushed as far toward the user as possible. Unix users can choose among multiple shells. Unix programs normally provide many behavior options and sport elaborate preference facilities.

This tendency reflects Unix's heritage as an operating system designed primarily for technical users, and a consequent belief that users know better than operating- system designers what their own needs are.


But the cost of the mechanism-not-policy approach is that when the user can set policy, the user must set policy. Nontechnical end-users frequently find Unix's profusion of options and interface styles overwhelming and retreat to systems that at least pretend to offer them simplicity.

[...] However, it may turn out that this ‘mistake’ confers a critical advantage — because policy tends to have a short lifetime, mechanism a long one. Today's fashion in interface look-and-feel too often becomes tomorrow's evolutionary dead end (as people using obsolete X toolkits will tell you with some feeling!). So the flip side of the flip side is that the “mechanism, not policy” philosophy may enable Unix to renew its relevance long after competitors more tied to one set of policy or interface choices have faded from view.
Eric Steven Raymond, "What Unix Gets Wrong"
This is extremely important for understanding what follows. Even systems managers (and especially their supervisors) like to have most, if not all, of the decisions made for them. Too many options can make decision-making intolerably burdensome. So many engineers, adopting Unix, embraced a particular design policy and thereby created a specific family of Unix systems.

(Part 2)
ADDITIONAL READING & SOURCES: Wikipedia Unix; Unix System V; Berkeley Software Distribution (BSD); Tru64 UNIX (also known as OSF/1); Unix Wars;

USENIX Association website (formerly Unix Users' Group); The Open Group: History and Timeline;

Living Internet: Unix; Eric Steven Raymond, The Art of Unix Programming

Labels: ,

17 January 2006

Virtual Memory

In the early years of mainframe computing, as computers evolved standardized architecture, demand arose for programs that took care of most of the routine system functionality, such as assigning storage capacity to a particular task. At the same time, data storage was a far greater concern; it was no longer acceptable to expect users to walk away with a bunch of freshly-punched cards. If the computer was going to be able to address a task routinely, then it was necessary for that computer to be able to store very large amounts of data in a stable manner.

But the most important concern of all was for the computer to be able to cope with multitasking. Multitasking meant multiple users running the same immense processor at the same time; or it meant one user running multiple programs concurrently—possibly in order to transfer data between the two. A strong incentive for developing multitasking computer software was the need to permit debugging of software. An early form of this was the time-share, in which computers would allocate time to specific tasks much the way they allocate disk space to the storage of specific blocks of data. An alternative form is "pipelining," in which the processor is simply running different steps to different instructions at the same time, much like an assembly line.

While the term "virtual memory" might sound esoteric, it was an immensely important innovation in the early years of computer science. Basically, early applications of virtual memory included the "virtual machine," an imaginary computer simulated by the operating system for allowing many virtual machines on a single tangible machine. For this reason, virtual memory was crucial to allowing engineers to realize the full potential of their hardware. In 1967, IBM developed a system software known as VM, for virtual machine, which was designed to allow multitasking in precisely this way. VM was a fairly rudimentary operating system—it was a software emulator of the hardware it ran on—but it excelled at what it did, so that an IBM 360/67 could run several instances of VM in a VM instance running in another VM instance, in yet another VM instance... with little hit on actual performance.*

VM essentially the task of addressing secondary memory, such as a tape drive, to host some of the data that is theoretically stored in RAM. The job of creating VM to enhance processor functionality is that of the operating system, which designates sectors of a computer storage unit (known as "pages"). Since many applications require access to more code (program instructions) than the computer can store in RAM, much of the less urgent code is stored on secondary memory. The art of VM programming lies in ensuring that this code is readily available when the processor needs it.

This aspect of memory management is made somewhat more tricky by the fact that computer applications, such as MS Word (as opposed to the operating system) have their own memory management, which can "resist" the decisions made by the memory management unit (MMU); as a result, modern operating systems, all of which have VM components, actually keep the identity of memory pages secret; only the MMU knows, and hence the efforts of applications to reside on the RAM whenever the computer is running them, are thwarted.

The introduction of VM to Windows 3.1 in 1990 is credited with giving that software the ability to run viably on the Intel 386 CPU; previous releases of Windows had been a failure.
*VM (the product) later became a component of VM/CMS, which—despite IBM indifference—has actually survived among users of the System/360. VM/CMS was the subject of a highly-acclaimed book-length history, VM and the VM Community: Past, Present, and Future (Melinda Varian).
ADDITIONAL READING & SOURCES: "Multics Virtual Memory - Tutorial and Reflections," by Paul Green. This is very arcane and includes a lot of recondite details such as the fact that the General Electric GE-645 computer at MIT for which Multics was developed used a 36-bit word instead of the 32-bit flaoting point word employed by the IBM System/360.


15 January 2006

Milestones: the IBM System/360

I tend to be fascinated by large industrial organizations, and the effect they have on technology in particular. It seems to me that, in a way, a lot of the major technical breakthroughs that allowed industry to occur did so centuries before the industrial revolution, during a period known as the commercial revolution. The ability of the large industrial organization to take on fundamentally more demanding, complex, and permanent tasks than any prior entity was always a precursor to any major transforming technology.

Much is made of the computer as such a technology, though the word "computer" actually refers very large family technologies. One these technological breakthroughs was creation of a series of processors that possessed a similar architecture and which differed chiefly on basis of size. Prior this time, the term "series" was used to describe computers with little more in common than manufacture by same company. The IBM 700/7000 series, for example, consisted of entirely redesigned machines, although a couple of machines could run programs produced for an earlier version. Part of the problem was, of course, the underlying evolution of electronics was proceeding much too fast to permit stable designs. Another possible challenger for the title, the UNIVAC 1100 series, suffers from the fatal flaw of having the second model in it introduced contemporaneously with IBM's entire S/360 line-up of six machines.

What distinguished the System/360 was that it had been conceived as an architecture before it was designed as a machine. The fleet was to have the same instruction set and numerous other common features, making it possible for the fleet to use (in theory) the same operating system. The memory was standardized and plug-compatible, making it possible to add modular components and replace them with newer models. Interestingly, Gene Amdahl's original paper outlining the design principles of the S/360 (PDF) divulges that his team chose the 4/8 bit word (which hereafter became universal) for coding efficiency in business applications. Numeric data, which was most efficiently expressed in floating point form, was the most common form of data. The 4/6 bit word was compatible for 16-bit instruction field and 32-bit floating point word.

The major design breakthroughs of the S/360 were fairly slow in coming; a uniform operating system for the family was not fully available until 1967 [*]. The OS/360 was obligated to be scalable upward from fairly modest operational capabilities to the largest computers available; in later verions, it would support multiple processors and virtual memory addressing.

This is significant because it stimulated the focus on designing machines varying greatly in capacity or purpose, while ensuring that components were interchangable, and could communicate with each other. It stimulated the faster cross-pollination of technologies across platforms and use of direct machine-machine communication. Finally, in 1975, the 370 was the first computer to be cloned by a competitor: the Amdahl 470, designed by Gene Amdahl himself, was greatly superior to the IBM System/370 and quickly captured 20% of the market for IBM-compatible computers.

Architecture of the IBM System/360, by Gene Amdahl;