the conservatism of open-source software
Jaron Lanier has an intriguing new column explaining his reservations about open-source software. His view is that the history of software development suggests that it’s the closed shops, the companies who make proprietary software, that are truly innovative. Thus:
Why are so many of the more sophisticated examples of code in the online world — like the page-rank algorithms in the top search engines or like Adobe’s Flash — the results of proprietary development? Why did the adored iPhone come out of what many regard as the most closed, tyrannically managed software-development shop on Earth? An honest empiricist must conclude that while the open approach has been able to create lovely, polished copies, it hasn’t been so good at creating notable originals. Even though the open-source movement has a stinging countercultural rhetoric, it has in practice been a conservative force.
It’s curious to think that anarchy can be conservative — I think of Neal Stephenson’s architectural description of the Linux world as “a bunch of RVs, yurts, tepees, and geodesic domes set up in a field and organized by consensus” — but I believe Lanier’s right about this. “The open-source software community is simply too turbulent to focus its tests and maintain its criteria over an extended duration, and that is a prerequisite to evolving highly original things. There is only one iPhone, but there are hundreds of Linux releases” — hundreds of fairly slight variations, he might often added, on a traditional theme: when I look at a Linux desktop I always have a moment when I think I’m looking at Windows 98.
Lanier might also have mentioned that the closed shops actually pay money to software developers, and pay it to the best developers you can find; and even supremely talented people tend to prefer being paid for their work to doing it gratis. But in any case, Lanier’s essay is worthy of your time.
Alan, thank you for this post.
I’ll add only that this (and Lanier would agree) only reinforces the case for institutional diversity, and diversity in ownership/property regimes. We need ‘em both. I tend to think the open-source romantics are right that we’ve moved too far in the one direction, so the romantics offer a useful corrective. But of course Lanier is right to suggest that this can’t calcify into crazy dogmatism, and that true innovators will want elbow room.
This reminds me of Jonathan Zittrain’s book and Tim Lee’s smart interventions on net neutrality: I mean, yes, we want the internet to be a “generative” framework, but efforts to regulate our way to a “generative” framework and to resist proprietary modes and appliancization will stifle innovators who are thinking up stuff we can’t imagine.
So I’m torn. The free culture guys are right about almost everything I care about, but they’re naive about regulatory capture and the eternal truth of Stallmanite nostrums.
— Reihan · Dec 31, 01:57 AM · #
Eric S. Raymond already answered Lanier’s question here:
“the following discriminators push towards open source: * Reliability/stability/scalability are critical. * Correctness of design and implementation cannot readily be verified by means other than independent peer review. * The software is critical to the user’s control of his/her business. * The software establishes or enables a common computing and communications infrastructure. * Key methods (or functional equivalents of them) are part of common engineering knowledge.”
It isn’t hard to see that genuine innovations are unlikely to satisfy most of these criteria, which means their developers are unlikely to publish the source code for them. Therefore what does get published as open source code will be “lovely polished copies” — not due to the turbulent culture of the OSS community, but for simple economic reasons.
— Michael Brazier · Dec 31, 07:20 AM · #
Alan — that’s not necessarily true. Flash for example has gone through many iterative releases and still has many problems today. No one would accuse Microsoft Word or Excel for being innovative, the latter has basically extended Mitch Kapor’s Lotus 1-2-3. The Iphone is not particularly innovative — an Ipod plus a phone, nothing earth shattering.
Many Linux distributions have “safe/canned” default themes and desktops, but you have the freedom to choose your own, with attendant increased resource usage of course. Some people just want to get work done and don’t want eye candy. Nevertheless Linux has more, and more mind-blowing eye candy desktops, than anything Microsoft or even Apple offer.
Many of the innovations are things you can’t see — the Apache Web Server, which has innovations in load handling, fail-over, and logging. Or MySQL, which is both cheaper and faster for web services than commercial software. The GIMP can give Adobe Photoshop a run for it’s money in innovation, particularly web graphics where it’s tightly focused. Plus unlike Photoshop it does not phone home to Adobe periodically.
Content Management systems like Joomla, Wordpress, and others offer innovative ways to get content-driven web sites up and running quickly. The amount of add-ons and plug-ins can be staggering. And presents lots of innovation.
You are wrongly focused on desktop flash, which is fairly trivial for most people’s experience. Even there, Linux out of the box has more innovations. Most modern distributions of Linux will recognize Windows formatted drives/flash devices, Linux formatted, and Mac HFS+ formatted devices. That may not be flashy, but it certainly is innovative. It means your linux box can read your Apple or Windows formatted Ipod with no problem, and write to it as well. Transparently.
It’s worth noting that AJAX code, which powers most of the dynamic page setup on the web (Flash has many problems, not the least of which is working with Google’s content searching and page ranking system) is an open-standard. NOT proprietary.
— Jim Rockford · Dec 31, 07:33 AM · #
Alan:
It’s always great to see software industry economics as a topic on this blog!
A couple of quick comments:
1. Software is a rapidly-maturing business. Michael’s list of the factors relevant for determining when open source makes sense strikes me as consistent with my experience in the industry. If you think about it, these are all most relevant for mature, which is to say commoditized, pieces of a code base. Just as shareholder value in what used to be called the computer industry migrated over decades from hardware to software, it is now migrating away from the creation of something like another OS kernel or lightweight database to either more innovative code that solves some new problem, or the sales, support and other functions required for software companies in the mature code sectors. Rather than open source being a super-cool new method for innovation, it may simply be a marker for where it is no longer economically valuable to construct proprietary code.
2. The impact of Software-as-a-Service (SaaS) on open source is profound. Under a SaaS model, the software company operates the software at a data center and users access it via Internet. If I’m not selling you software, but instead renting you access to software that I operate for you, the GNU open source license doesn’t restrict my right to embed open-source components in my proprietary software (which is the primary method by which open source licenses make software “free”). The bulk of the software industry is moving in this direction – it would be very difficult to get VC backing for a software start-up in the United States today that planned to actually deliver source code to a customer facility.
Both of these ideas support what I see as the basic point of your post: open source is about a group of phenomenal craftsman plying their trade, while the market, as always, changes the rules of value creation around them.
— Jim Manzi · Dec 31, 08:56 AM · #
Yes, yes, great topic.
I wanted to add that Apple, that closed software shop of horrors, has built their web browser (which runs, incidentally, on the iPhone) around a forked open source rendering engine (KHTML/WebKit). Last I heard, most of the nifty software that runs on the iPhone is browser-based (as are the applets that run on your Mac’s Dashboard).
Jim’s point about Apache et al are quite good. I’d add to that languages like Perl, Python, Ruby, and even PHP, which are all open source and are serious languages (and Perl is especially important for researchers).
Also, let’s not short-change the complexity of the algorithms that drive an operating system. It’s not as if the GNU/Linux kernel developers just drop a line of code in here or there when a new version is released—systems are re-architected and re-designed, algorithms are tweaked, etc. There simply isn’t a marketing team shouting these enhancements from the rooftops, so folks that don’t keep up with the Kernel Trap mailing lists or Slashdot wouldn’t know (or care).
— brian · Dec 31, 02:46 PM · #
Jim Manzi: very interesting point about SaaS — but surely Stallmann sees this as a bug in the GNU license to be fixed?
Jim Rockford: I think you mistook my quotations from Lanier for my own point. For example, it’s Lanier who invokes Flash, not me. And both Lanier and I agree that many open-source projects are invaluable: the argument is just that an open-source only world, such as Stallman envisions, would mean less innovation in software development. Let me add a couple of points:
(1) Buggy or not, Flash is an enormously innovative application, and it’s impossible (for economic reasons, as Michael Brazier suggests) to imagine it coming from an open-source community. (For that matter it’s not as trivial as you suggest. I have my browser set to disable all Flash content by default, so that I have to click to start a Flash animation, and I’m continually surprised by how widely it’s used. Also dismayed.)
(2) I use the Gimp myself, but it’s a perfect example of Lanier’s point: it’s an application wholly dependent for its structure and core functionality on another, proprietary, application, Photoshop. It extends Photoshop’s functionality in certain ways, but again, it’s impossible to imagine it having come into existence on its own, without Photoshop as a precedent.
(3) Tangential, but still: the iPhone not innovative? There are a lot of critical things you could say about the iPhone, but not innovative? That visual voicemail thing, enabling people for the first time in human history to see all their voicemail messages and listen and reply to them in any order, not innovative? A screen which flips its orientation automatically in response to the user’s movements, not innovative? Resizing and moving of photos and other objects by finger gestures, not innovative? (Maybe at this point you can tell what I asked for for Christmas but didn’t get, as though I don’t work hard enough to deserve a little treat from time to time, God knows I certainly don’t deserve something like that.)
(4) This is a more debatable point, but the many Linux desktops I’ve seen are essentially just variably colored variations on the Windows 98 basic look. When open-source UI projects get really innovative, they seem to have trouble getting traction in the community, or are require so much developer time that they develop at a glacial pace. Exhibit A: Enlightenment.
— Alan Jacobs · Dec 31, 03:11 PM · #
Back when OSX emerged, a friend voiced the usual complaint that Macs are too expensive, complete with the usual references to stylish European cars vs. cheaper yet functional American ones. I reminded him that now that OSX uses a Unix-based core, it’s like having a BMW for which you can download free exhaust manifolds and shocks, and whose block you can bore out again and again, trying over every time you screw something up.
The development and infrastructure tools that are available for free mean that I can assemble a toolkit on my home computer (or on my server) even nicer than the woodworking outfit my Dad had in the garage (please don’t tell Matt Crawford I made that comparison). The American Scene, for instance, is powered entirely by free tools that are thankfully mature enough for a novice like myself to use.
— Matt Frost · Dec 31, 07:01 PM · #
Alan:
I’m not sure about how easy it will be to modify the GNU license to deal with SaaS. SaaS has been around for a while, and it remains a pretty persistent “bug”. As an example of why this might be the case, consider that Google is a SaaS application, and is supported by advertising revenues with no license fees. Short of saying “you are forbidden to use this code to make money”, it’s hard to see how you stop the market from innovating (in terms of commercial arrangements) around the license.
Alan / Jim:
Much of what the open source community generates is really fantastic, technically-sophisticated work. Much of this is used for very heavy-duty applications. Jim, as per your post, Apache is the best single example of which I am aware. The OS kernel open source teams, in particular, are phenomenal engineers.
I believe that the lack of development of ground-breaking (or “truly innovative”, or whatever label we want to put on it) products, isn’t really driven by a lack of technical capability, but by incentives, and therefore applies only to a certain kind of innovation. As is so often the case, engineers, users and businesspeople are talking past one another on the topic.
In terms of shareholder value creation for the software company, it’s hard to conceive of a more important contemporary innovation than the search engine. Just considering Google in isolation, I bet, would account for more than 100% of net shareholder value creation for the total US software industry over the past 7 years. Every commercially-viable search engine ever built has been constructed by a team of 2 – 4 engineers. I know that there are now open source search engines now, but why wasn’t this done by the open source community back when it was worth an immense amount of money to do so?
Just imagine Brin and Page to be “open source” developers who saw (as denominated by VC investments at a very large pre-money valuation even in the seed round) that by declaring their code to be proprietary they could make a ton of money. That is, the valuation market votes to declare that some code is, on balance, worth more money to its owners by capturing more economic rents by retaining rights to the source code, even if this means forgoing all of the engineering benefits available from open, public peer-review (at least for a while). This is highly analogous to a university research lab jealously guarding the progress of their experiments prior to publication and peer-review.
I think this is why open source is inherently “conservative” in the sense that the linked post indicates – that is, commercially, but not necessarily technically or aesthetically. If an innovation has large commercial value, it will be privatized (at least for a while), but innovations that many smart engineers find “cool” will often be released as open source because it is not profitable to do otherwise. (Although, like academic research labs, I think you will find individual open source engineers will typically develop some new feature “privately” prior to publicly releasing it in order to garner more prestige in their technical community.)
— Jim Manzi · Jan 1, 07:20 PM · #