The Real Effects of Computing Brilliance

Recently, L.E. Modesitt, a noted and highly perceptive (imho) science fiction author with a long background in business, government, education, and the like, wrote a blog post in which he decried the contributions of “brilliant minds, especially at organizations like Google and Facebook” to “society and civilization.” He noted Eric Schmidt’s remarks in a recent interview that young people from colleges were brighter than their predecessors, and argued that they had applied their brilliance in “pursuit of the trivial … [and] of mediocrity.” More specifically, he claimed that their brilliance has helped lead to the undermining of literary copyright and therefore of an important source of ideas, the creation of bubble wealth, the destabilization of the financial system, and potentially the creation of the worst US political deadlock since the Civil War.

Here, however, Mr. Modesitt steps onto my turf as well as his own. I can claim to have been involved with computers and their results since I was in the same freshman class in college as Bill Gates (Junior, thank you). I have had a unique vantage point on those results that combines the mathematical, the technological, the business, and even the political, and have enjoyed an intimate acquaintance with, and given long-running attention to, the field. I am aware of the areas in which this generation (as well as the previous two) has contributed, and their effects. And I see a very different picture.

Let’s start with Mr. Schmidt, who is a very interesting man in his own right. Having been at Sun Microsystems (but not heading it) in its glory days of growth, struggle, and growth (1983-1997), he became CEO of Novell as it continued its slide into oblivion, and then in 2001, a key part of the necessary transition of Google from its founders to successors who can move the company beyond one person’s vision. Only in the last role was he clearly involved with successful technology leadership, as Sun was a master at marketing average technology as visionary, while Novell’s influence on industry innovation by that point was small.

All that being said, Mr. Schmidt is speaking from the viewpoint of having seen all three of computing’s great creative “generations,” waves of college-bred innovators who were influenced by comparable generations of their societies, and who combined that social influence with their own innovations to create the stew that is the Web of today. I would divide those generations as follows:

1. Graduating from college in 1971-1981, the “baby boomer/socially conscious”.
2. Graduating from college in 1982-1995, the “individual empowerment Gen Xer”.
3. Graduating from college in 1996-2007, the “Web is the air we breathe” generation.

Where Mr. Modesitt sees only some of the effects of all three generations from the outside, I believe that Mr. Schmidt sees from within the way that generation 3 incorporates all the technological advances of the previous generations and builds on them. Therefore, where Mr. Modesitt is too negative, I believe that Mr. Schmidt is too positive. Generations 1 and 2 have created a vast substructure – all the technologies of computing and the Internet that allow Generation 3 to move rapidly to respond to, or anticipate, the needs of each segment of a global society. Because the easiest needs to tackle are the most “trivial”, those will be most visible to Mr. Modesitt. Because Generation 3 appears to move from insight to insight faster by building on that infrastructure, Mr. Schmidt sees them as more brilliant.

Let’s get down to cases: the Web, and Google. The years of Generation 3 were the years when computing had a profound effect on the global economy: it gave productivity more than a decade of 2.5-3% growth, a kind of “sunset re-appearance” of the productivity gains of the Industrial Revolution. This effect, however, was not due to Generation 3. Rather, Generation 2’s creation of desktop productivity software, unleashed in the enterprise in the early 1990s, drove personal business productivity higher, and the advent of the business-usable Web browser (if anything, a product of Generation 1’s social consciousness) allowed online sales that cut out “middleman” businesses, cutting product and service costs sharply.

But where Generation 3 has had a profound effect is in our understanding of social networks. Because of its theoretical work, we now understand in practical terms how ideas propagate, where are the key facilitators and bottlenecks, and how to optimize a network. These are emphatically not what we think of when we talk about the value of computing; but they are the necessary foundation for open source product development, new viral and community markets, and the real prospect of products and services that evolve constantly with our needs instead of becoming increasingly irrelevant (see my blog post on Continuous Delivery).

Has that made life better? I would say, that’s like asking if Isaac Newton’s calculus was a good idea, given that it has mainly been used to calculate trajectories for artillery. Ideas can be used for good or ill; and often the inventor or creator has the least control over their long-term use by others.

But let’s play the game, anyway. Undermining copyright? I would blame that more on Generation 2, which used the Web to further its libertarian ideologies that now form a key part of the “Web ethos”. Financial algorithms and bubble wealth? Sorry, that’s a product of the broader society’s reaction to the perceived (but not necessarily real) lawlessness and government “do-gooding” of the Vietnam era, so that law and order and “leave me alone” permitted business deregulation and lack of outside-the-box thinking about the risks of new financial technologies. Political deadlock? Same thing. “Leave me alone” created a split between Vietnam-era social consciousness and previous generations, while removing later generations from the political process except as an amoral or totally self-interested game. In both those cases, no computing Generation could have done very much about it except to enable it.

What are the most profound ideas and impacts of all three Generations’ “brilliant minds”? I would say, first, the idea of a “meta-product” or “meta-thing” (today’s jargon probably would call it a “virtual reality”), an abstraction of a physical good or process or person that allows much more rapid analysis and change of what we need. To a far greater degree than we realize, our products are now infused with computer software that not only creates difficulties in carrying out tasks for poor Mr. Modesitt, but also gives an array of capabilities that can be adjusted, adapted, evolved, and elaborated much more rapidly than ever before. Mr. Modesitt, I know, rightly resents the stupidities of his word processing software; but there is simply no way that his typewriter hardware by now would have the spelling and grammar checking capabilities, much less the rapid publishing capabilities, of that same word processor. It may be more work to achieve the same result; but I can guarantee that, good or bad, many of the results achieved could not have been attained forty years ago. Our computing Generations did that.

The second, I believe, is that of global information. The Web, as imperfect as it is, and as security-ridden as it has become, has meant a sea-change in the amount of important information available to most if not all of us. When I was in college, the information readily available was mostly in textbooks and bookstores, which did not begin to capture the richness of information in the world. The key innovation here was not so much the search engine, which gave a way to sort this information, but the global nature of the Web user interface, the embedding of links between bits of information, and the invention of “crowdsourcing” communities, of which Wikipedia is by far the most successful as a pointer from information area to information area. Say all you like about the lies, omissions, and lack of history behind much of this global information pool; it is still better than a world in which such information appeared not to exist. Our computing Generations did that, too.

The third idea, the one I mentioned above, is understanding of, and ability to affect the design of, social networks. Facebook is only the latest variant on this theme, which has shown up in Internet groups, bulletin boards, open source and special-interest communities, smartphone and text-message interfaces, file sharing services, blogs, job sites, and other “social networks.” The Arab Spring may be an overhyped manifestation of its power; but I think it is fair to say that some of the underlying ideas about mass peaceful regime change could never have been global or effective without computing’s “brilliant minds.” I give Generation 3 most of the credit for that.

So here’s an example of why I disagree with Mr. Modesitt: a couple of years ago, I sat down to understand climate change. At first, the best information came from old-style books, mostly from the local public library. But soon, I was using www.climateprogress.com and neven1.typepad.com to dig deeper, to see the connections between what the books were saying, and then Wikipedia to point me to research articles and concept explanations that allowed me to put it into a coherent whole, adding my own technology, economic, business, political, and government experience and knowledge to fill in the gaps. No set of books, then or now, could possibly have given me enough information to do this. No phone network could possibly have shown me the people and institutions involved. No abacus could have put the statistics together, and no typewriter or TV could have given me such a flexible window to the information. The three great ideas of computing’s three Generations made such a thing possible.

So, no, I don’t believe that much of what this Generation of brilliant minds has been working on is at all trivial or mediocre, despite appearances – not to mention the contributions of the other two Generations – nor has its effect been predominantly bad. Rather, it reminds me of the remark in Dilbert that it is the work of a few smart, unprepossessing drones that moves the world forward, hindered rather than helped by their business or the greater society. Despite society’s – and sometimes their own – best efforts, these Generations’ blind pursuit of “the new new thing”, I say, has produced something that overall is good. In every sense.

Human Math and Arctic Sea Ice

Over the last year and a half, instead of following baseball and football, I have been following the measurements of Arctic sea ice. Will it entirely melt away at minimum? If so, when? Will it continue to melt past that point? Will the Arctic reach the point of being effectively ice-free year-round? If so, when? Will Arctic sea ice set a new record low in extent this year? In area? In volume? By the way, the answers to the last three questions in 2011 are already, by some measures, yes, yes, and yes.

In the process, I have become entirely addicted to the Arctic Sea Ice web site (neven1.typepad.com), the Arctic sea ice equivalent of a fantasy football league. There, I and other scientist and math wannabees can monitor and pit themselves against near-experts, and plunder much of the same data available to scientists to try our own hand at understanding the mechanics of the ebb and flow of this ice, or to do our own predictions. Periodically, scientific research pops up that deepens our understanding in a particular area, or challenges some of our assumptions. Strange weather patterns occur as we near minimum, making the final outcome unpredictable. In fact, this year we still don’t know whether the game is over for the year or not – whether we are going into overtime.

But what disturbs me about the glimpse I am getting into science, this late in life, is that I keep seeing half-veiled glimpses of what I might call “data snow blindness,” not just from us newbies, but from some of the scientific research noted in the site. By that I mean that those who analyze Arctic sea ice data tend to get so wrapped up in elaborating the mathematical details of, say, a decrease in extent at minimum and how short-term and long-term changes in sea-ice albedo, the North Atlantic Dipole Anomaly, cloud cover, and increased Arctic surface temperatures can predict this minimum, that they seem to forget what extent data really conveys and does not convey, and how other factors not yet clearly affecting extent data should play an increasing role in future extent predictions. They keep talking about “tipping points” and “equilibria” and “fundamental changes in the system” that do not appear to be there. And so, what surfaces in the media, with few if any exceptions, gives an entirely misleading picture of what is to come.

So let me lay out my understanding of what the data is really saying, and how people seem to be going wrong. Bearing in mind, of course, that I’m not the greatest mathematician in the world, either.

The Model
I find that the easiest way to visualize what’s going on with Arctic sea ice is to think of a giant glass full of water, with an ice cube floating on top that typically almost but not quite fills the top of the glass. In the summer, the sun shines on the ice and top of the water, adding heat on top; in winter, that heat turns off. In summer and winter, the water below the ice is being heated, although a little less so in winter.

This does not quite complete the picture. You see, water is constantly flowing in from the south on one side and out to the south on another. As it hits the ice, it freezes, and moves throughout the cube until it exits on the other side – an average of about five years, apparently. These currents, too, act to break the ice apart as it is melting in summer, so (especially on the edge) there are lots of little “cubelets”. Where the ice is solidly packed together in the glass, then “area” – the amount of ice we see from above – is the same as “extent” – the region in which there is enough ice to be easily detectable. But where there are cubelets, then extent is much greater than area – as much as 70% more, according to recent figures.

One more important point: the depth of Arctic sea ice is not at all the same everywhere. This was true when the first nuclear sub approached the Pole underwater decades ago, and it is true as the Polarstern measures depth from above this year. The age of the ice, the point in the year in which it first froze, and where it is in relation to the Pole all factor in; but even in small regions, “thickness” varies. Generally, less than half the ice is almost exactly the same thickness as the average, with lots above and lots below. I find it useful to think of a normal curve of sea-ice thickness more or less peaking at the average.

In a state of “equilibrium” such as apparently has existed for perhaps 5-10 million years, the Arctic cube fills at least 70% of the Arctic in summer and just about all of the Arctic in winter. In summer, the sun melts the top and sides of the cube. However, since we only see the top, which only peeks out a little bit from the top of the water, we don’t see the way that the bottom of the cube rises in the water – the “balance point” between “salt” water below and “no-salt” ice above goes up. In the same way, in winter, we don’t see the way that the balance point goes down deeper.

Now suppose the globe starts warming. As it happens, it warms faster in the Arctic than in the equator, and it warms in both the near-surface air and in the water that is flowing through the Arctic underneath the ice, year-round. Estimates are that the air temperatures in summer in or near the Arctic have reached more than 10 degrees higher, and the rate of rise is accelerating; likewise, the ocean temperature is up more than a degree, and the rate of rise is accelerating. What we would expect to happen is that there is less and less ice during the summer – and also less and less ice during the winter, because the water underneath that cube is warmer, and the “balance point” is higher. And the rate we would be losing ice would be accelerating.

The Measures
Now how do we measure what’s going on with the ice? We point our satellites at the Arctic, and we measure in three ways: extent, area, and volume. Extent and area we have explained above; volume is area times average thickness.

To get extent, we divide the Arctic into regions, and then into subregions. If, say, more than 15% of the subregions in a region show ice, then that’s part of the ice extent. The area is the extent times the percent of subregions that show all ice (this is a very simplified description). There are difficulties with this, such as the fact that satellites tend to find it difficult to distinguish between melt ponds at the top of the ice during melting and melting of the ice “all the way down”; but it appears that those cause only minor misestimations of “actual” extent and area. Storms that wash over “cubelet” ice can cause temporary fairly large downward jumps in what the satellite perceives as ice, and hence in area; but again, this goes away as the storm subsides. All in all, the measuring system gives a pretty good picture (by now) of what’s going on with area and extent.

Volume is much more difficult to get at – because it’s very hard to get an accurate picture of thickness from a satellite. Instead, we create a model of how thick each part of the ice should be at any time of the year, and then check it out “on foot”: walking around on the ice (or icebreaking) and testing.

Now here’s what most people either don’t know or don’t think about: that model has been tested against samples almost constantly since it was first developed decades ago, and it is pretty darn accurate. When it has said, this year, that ice is as little as a meter thick near the North Pole, vessels went out and, lo and behold, found a large patch of ice 0.9 meters thick near the North Pole. That’s clearly not something that many scientists were anticipating six years ago as likely – but the model was predicting it.

It’s the Volume, Stupid
All right, let’s get back to the model. Suppose we have equilibrium, and we start an accelerating rise in air and ocean temperatures. What changes would we expect to see in real extent, area, and volume year to year?

Well, think about what’s happening. The water heating is nibbling away at the “balance point” from beneath, and at the edges. The air heating is nibbling away at the top of the ice during the summer, and expanding the length of the summer a bit, and heating the surface water at the edges a bit. So it all depends on the importance of water heating vs. air heating. If water heating has little effect compared to air heating, what happens to volume is not too far from what happens to area and extent: They start plummeting earlier and go farther down, but during the winter, when just about the whole basin gets frozen again, they go back to about where they were – a little less, because the air is more often above freezing at the very edge of the ice even in the depth of winter.

Now suppose water heating has a major role to play. This role shows up mostly as “bottom melt”, it shows up year-round in about the same amounts, and it shows up almost completely, until nearly all ice is melted, in volume. So what you’ll see, when you look at extent, area, and volume, is that volume goes down for quite a while before it’s really clear that area and extent are changing, and then area and extent start going down at minimum but not much if at all at maximum, and then the melt moves the balance point up and up and some of that normal curve starts reaching “negative thickness” as some of that melt reaches the surface of the ice and that means area starts moving down fast, and extent with it, and then in a year or two half of the ice reaches “negative thickness” and it seems like the area is cut in half and the extent by one-third, and the same thing happens the next year, while the rate of volume decrease actually starts slowing, and then in two more years there is effectively less than 1% of the ice area at maximum showing up at minimum, and less than 15% of the extent.

This is where lots of folks seem to fail to understand the difference between a measure and what it’s measuring. If you look at the volume curve, until the last few years before it reaches zero it seems to be accelerating downward – then it starts flattening out. But what’s actually happening is that some parts of the ice have reached the point where they’re melting to zero – others aren’t, because there’s a great variation in ice thickness. If we represented those parts of the ice that have melted as “negative thickness”, then we would continue to see a “volume” curve accelerating downward. Instead, we represent them as “zero thickness”, and the volume delays going to zero for four or five years.

So what are our measures telling us about the relative importance of water and air heating? First of all, the volume curve is going down, and going down at an accelerating rate. From 1980 (start of the model) to 2005, average thickness at minimum area (a good proxy for thickness at ice minimum), went from more than 3 meters to a little more than 2 meters. From 2006 to 2010, it has gone from there to a little more than 1 meter. If it were to follow the same path until 2013 or 2014, then “volume” would go to zero then at minimum, with the model’s measures of volume going to zero perhaps 3 years later, as explained in the last paragraph.

But there’s another key fact about volume: volume at maximum went down by about the same amount (15 million cubic meters) from 1980 to 2010 as volume at minimum (14 million cubic meters). In other words, instead of volume springing back during the winter almost to what it was 30 years ago, it’s almost exactly tracking the loss of volume the rest of the year. The only way this happens is if water melting from the bottom is a major part of the loss of volume from year to year.

Let’s sum up. The samples show that volume is going down at an accelerating rate throughout the year. The accelerating drop in volume shows that bottom melt is driving accelerating loss of Arctic sea ice year-round. Thus, we can predict that area and extent will soon take dramatic drops not obvious from the area and extent data, and that these will approach zero in the next 7 years or so. In other words, the key to understanding what’s going on, and what should happen next, is as far more the measure of volume – rightly understood – than area or extent.

Where’s The Awareness?
And yet, consistently, scientists and wannabes alike seem to show a surprising degree, not just of disagreement with these projections, but also of a seeming lack of awareness that there should be any issue at all. Take some of the models that participants in Arctic Sea Ice are using to predict yearly minimum extent and area. To quote one of them, “volume appears to have no effect.” The resulting model kept being revised down, and down, and down, as previous years that had shorter summers, slower rates of melt for a given weather pattern, and less thin ice, proved bad predictors.

Or, take one of the recent scientific articles that showed – a very interesting result – that if, for example, the Arctic suddenly became ice free right now, it would snap back to “equilibrium” – the scientist’s phrase – within a few years. But why was the scientist talking about “equilibrium” in the first place? Why not “long-term trend”? It is as if the scientist was staring at the volume figures and saying, well, they’re in a model, but they’re not really nailed down, are they, so I’ll just focus on area and extent – where the last IPCC report talked about zero ice in 2100, maybe. So suppose we substituted the volume’s “long-term trend” for “equilibrium”, what would we expect? Why, that a sudden “outlier” dip in volume, area, or extent would return, not to the same level as before, but to where the long-term accelerating downward curve of projected volume would say it should return. And, sure enough, the “outlier” volume dip in 2006-2007 and extent dip in 2007 returned only partway to the 2005 level – in 2009 -- before resuming their decrease at an accelerating rate.

And that brings us to another bizarre tendency: the assumption that any declines in area and extent – and, in the PIOMAS graph, in volume – should be linear. Air and ocean temperatures are rising at an accelerating rate; so an accelerating downward curve of all three measures is more likely than a linear decline. And, sure enough, since 2005, volume has been consistently farther and farther below PIOMAS’ linear-decline curve.

Then there’s “tipping point”: the idea that somehow if Arctic sea ice falls below a certain level, it will inevitably seek some other equilibrium than the one it used to have. Any look at the volume data tells you that there is no such thing as a tipping point there. Moreover, a little more thought tells you that as long as air and ocean temperatures continue their acceleration upwards, there is no such thing as a new equilibrium either, short of no ice year-round – which is a kind of equilibrium, if you don’t consider the additional heat being added to the Arctic afterwards. Sooner or later, the air and ocean temperature at all times in winter reaches above minus 4 degrees C, so that water can’t expel its salt and freeze. And that’s not hypothetical; some projections have that happening by around 2060, if not before.

And, of course, there’s “fundamental change in the system”, which implies that somehow, before or after now, the Arctic climate will act in a new way, which will cause a new equilibrium. No; the long-term trend will cause basic changes in climate, but will not be affected by them in a major way.

Follow-On Failures
The result of this data snow blindness is an inability to see the possibility – and the likelihood -- of far faster additional effects that spread well beyond the Arctic. For instance, the Canadian Prime Minister recently welcomed the idea that the Northwest Passage is opening in late summer, because Canada can skim major revenues off the new summer shipping from now on. By the data I have cited above, it is likely that all the Arctic will be ice-free in part of July, August, and September by 2020, and most of the rest of the year by 2045, and very possibly by 2035. Then shippers simply go in one end of the Arctic via the US or Russia, and the other end via the Danes (Greenland) or Russia, and bypass Canada completely.

And then there’s the Greenland land ice. Recent scientific assessments confirm that net land ice loss has doubled each decade for the last three decades. What the removal of Arctic sea ice means is the loss of a plug that was preventing many of the glaciers from sliding faster into the sea. Add a good couple of degrees of increased global warming from the fact the Arctic sea ice isn’t reflecting light back into space any more, and you have a scenario of doubling net Greenland land ice loss for the next three decades, as well. This, in turn, leads to a global sea ice rise much faster than the 16 feet (or, around the US, maybe 20 feet) rise that today’s most advanced models project by assuming that from now on, Greenland land ice loss will be rising linearly (again, why linearly? Because they aren’t thinking about the reality underlying the data). And, of course, the increased global warming will advance the day when West Antarctica starts contributing too.

The Net-Net: The Burden of Proof

As a result of these instances of data snow blindness, I continually hear an attitude that runs something like this: I’m being scientifically conservative. Show me why I shouldn’t be assuming a possibility of no long-run change, or a new equilibrium short of year-round lack of Arctic ice, or a slow, linear descent such as shows up in the area and extent figures.

What I am saying -- what the data, properly understood, are saying to me – is that, on the contrary, the rapid effects I am projecting are the most likely future outcomes – much more rapid declines in area and extent at minimum in the very near future, and an ice-free Arctic and accelerating Greenland land ice loss over the 20-30 years after. Prima facie, the “conservative” projections are less likely. The burden of proof is not on me; it’s on you, to disprove the null hypothesis.
So what about it, folks? Am I going to hear the same, tired arguments of “the data doesn’t show this yet” and “you have to prove it”, or will you finally start panicking, like those of us that understand what the data seem to be saying? Will you really understand which mathematics applies to the situation, or will you entrance yourselves with cookbook statistics and a fantasy world of “if this goes on”? It may be lonely out here; but it’s real.

The Third Age of Software Development: Andreessen, Thoughtworks, and the Agile World

Recently there passed in front of me two sets of thoughts: one, an article by Marc Andreessen (once a Web software developer, now a VC) in the Wall Street Journal about how software is “eating the world”; one by a small development tools/services company called Thoughtworks about a new technique called Continuous Delivery.

Guess which one I think is more important.

Still, both make important points about changes in computing that have fundamental effects on the global economy. So let’s take them one at a time, and then see why I think Continuous Delivery is more important – indeed, IMHO, far more important. A brief preview: I think that Continuous Delivery is the technique that takes us into the Third Age of Software Development – with effects that could be even more profound than the advent of the Second Age.


To Eat the World, First Add Spices

A very brief and cursory summary of Andreessen’s argument is that companies selling software are not only dominant in the computer industry, but have disrupted (Amazon for books, Netflix for movies) or are “poised to disrupt” (retail marketing, telecom, recruiting, financial services, health care, education, national defense) major additional industries. The result will be a large proportion of the world’s industries and economy in which software companies will be dominant – thus, software will “eat the world”.

My quarrel with this argument is not that it overstates the case, but rather that it understates it. What Andreessen omits, I assume in the interest of conciseness, is that in far more industries and companies, software now is the competitive differentiator between companies, even though the companies’ typical pedigree is primarily “hardware” or “services”, and the spending on software is a small percentage of overall costs.

My favorite examples, of course, are the company providing audio and video for stadiums, which relies on software to synchronize the feeds to the attendees better than the competition, and Boeing, which would not be able to provide the differentiating cost-saving use of composites were it not for the massive amounts of software code coordinating and monitoring the components of the Dreamliner. In health care, handling the medical health record and support for the new concept of the medical home requires differentiating software to glue together existing tools and health care specialists, and allow them to offer new types of services. It isn’t just the solo software company that is eating the world; it’s the spice of software differentiation in the mostly non-software businesses that is eating these companies from within.

So Andreessen’s article signals a growing awareness in the global economy of the importance not just of computing, but of software specifically, and not just as a separate element of the economy, but as a pervasive and thoroughly embedded aspect of the economy whose use is critical to global economic success or failure. How far we have come, indeed, from the point 30-odd years ago when I suggested to Prof. Thurow of MIT that software competitive advantage would allow the US to triumph over the Japanese economic onslaught, and he quite reasonably laughed the idea to scorn.

I do have one nit to pick with Andreessen. He trots out the assertion, which I have heard many, many times in the past 35 years, that companies just can’t find the programming skills they need – and this time, he throws in marketing. Every time people have said this, even at the height of the Internet boom, I have observed that a large body of programmers trained in a previous generation of the technology is being overlooked. That sounds as if it makes sense; but, as anyone should know by now, it is part of the DNA of any decent programmer to constantly acquire new knowledge within and beyond a skill set – good programmers are typically good programmers in any arena, from Web-site design to cloud “program virtualization”.
Meanwhile, I was hearing as late as a year ago about the college pipeline of new programmers being choked off because students saw no future jobs (except in mainframes!), and I can attest that available, technically savvy marketers in the Boston area – a well-known breeding ground – are still thick on the ground, while prospective employers in the Boston area continue to insist that program managers linking marketing and programming have at least five years of experience in that particular job. It just won’t wash, Mr. Andreessen. The fault, as is often the case, is probably not mainly in the lack of labor, but in companies’ ideological blinders.


The Idea of Continuous Delivery

Continuous Delivery, as Thoughtworks presents it, aims to develop, upgrade, and evolve software by constant, incremental bug fixes, changes, and addition of features. The example cited is that of Flickr, the photo sharing site, which is using Continuous Delivery to change its production web site at the rate of ten or more changes per day. Continuous Delivery achieves this rate not only by overlapping development of these changes, but also by modularizing them in small chunks that still “add value” to the end user and by shortening the process from idea to deployment to less than a day in many cases.

Continuous Delivery, therefore, is a logical end point of the whole idea of agile development – and, indeed, agile development processes are the way that Thoughtworks and Flickr choose to achieve this end point. Close, constant interaction with customers/end users is in there; so is the idea of changing directions rapidly, either within each feature’s development process or by a follow-on short process that modifies the original. Operations and development, as well as testing and development, are far more intertwined. The shortness of the process allows such efficiencies as “trunk-based development”, in which the process specifically forbids multi-person parallel development “branches” and thus avoids their inevitable communication and collaboration time, which in a short process turns out to be greater than the time saved by parallelization.

I am scanting the many fascinating details of Continuous Delivery in the real world in order to focus on my main point: it works. In fact, as agile development theory predicts, it appears to work better than even other agile approaches. Specifically, Thoughtworks and Flickr report:


• A much closer fit between user needs and the product over time, because it is being constantly refreshed, with end-user input an integral part of the process;
• Less risk, because the small changes, introduced into production through a process that automatically minimizes risk and involves operations, decrease both the number of operational bugs and the time to identify and fix each; and
• Lower development costs per value added, as loosely measured by the percentage of developed features used frequently as opposed to rarely or never.


The Ages of Software Development

I did my programming time during the First Age of software development, although I didn’t recognize it as such at the time. The dominant idea then was that software development was essentially a bottleneck. While hardware and network performance and price-performance moved briskly ahead according to Moore’s Law, software developer productivity on average barely budged.

That mattered because in order to get actual solutions, end users had to stand in line and wait. They stood in line in front of the data center, while IT took as long as two years to generate requested new reports. They stood in line in front of the relatively few packaged application and infrastructure software vendors, while new features took 9 months to arrive and user wish lists took 1 ½ years to satisfy. They stood in line in front of those places because there was nowhere else to turn, or so it seemed.

I noted the death of the First Age and the arrival of the Second Age in the late ‘90s, when a major Internet vendor (Amazon, iirc) came in and told me that more than half of their software was less than six months old. Almost overnight, it seemed, the old bottlenecks vanished. It wasn’t that IT and vendor schedules were superseded; it was that companies began to recognize that software could be “disposable.” Lots of smaller-scale software could be developed independent of existing software, and its aggregate, made available across the Internet by communities, as well as mechanisms such as open source, meant that bunches of outside software could be added quickly to fill a user need. It was messy, and unpredictable, and much of the new software was “disposable” wasted effort; but it worked. I haven’t heard a serious end user complaint about 2-year IT bottlenecks for almost a decade.

Andreessen’s “software eating the world”, I believe, is a straightforward result of that Second Age. It isn’t just that faster arrival of needed software from the new software development approach allows software to start handling some new complex tasks faster than physical products by themselves – say, tuning fuel mixtures constantly for a car via software rather than waiting for the equivalent set of valves to be created and tested. It is also that the exponential leap in the amount of the resulting software means that for any given product or feature, software to do it is becoming easier to find than people or machines.

However, it appears clear that even in the Second Age, software as well as physical products retain their essential characteristics. Software development fits neatly into new product development. New products and services still operate primarily through more or less lengthy periods of development of successive versions. Software may take over greater and greater chunks of the world, and add spice to it; but it’s still the same world.

The real-world success of Continuous Delivery, I assert, signals a Third Age, in which software development is not only fast in aggregate, but also fast in unitary terms – so fast as to make the process of upgrade of a unitary application by feature additions and changes seem “continuous”. Because of the Second Age, software is now pervasive in products and services. Add the new capabilities, and all software-infused products/services -- all products/services – start changing constantly, to the point where we start viewing continuous product change as natural. Products and services that are fundamentally dynamic, not successions of static versions, are a fundamental, massive change to the global economy.

But it goes even further. These Continuous-Delivery product changes also more closely track changes in end user needs. They also increase the chances of success of introductions of the “new, new thing” in technology that are vital to a thriving, growing global economy, because those introductions are based on an understanding of end user needs at this precise moment in time, not two years ago. According to my definition of agility – rapid, effective reactive and proactive changes – they make products and services truly agile. The new world of Continuous Delivery is not just an almost completely dynamic world. It is an almost Agile World. The only un-agile parts are the rest of the company processes besides software development that continue, behind the scenes of rapidly changing products, to patch up fundamentally un-agile approaches in the same old ways.

And so, I admit, I think Thoughtworks’ news is more important than Andreessen’s. I think that the Third Age of Software Development is more important than the Second Age. I think that changing to an Agile World of products and services is far, far more important than the profound but more superficial changes that the software infused in products and services via the Second Age has caused and will cause.


Final Thoughts and Caveats

Having heralded the advent of a Third Age of Software Development and an Agile World, I must caution that I don’t believe that it will fully arrive any time in the next 2-3 years, and perhaps not for a decade, just as it took the Second Age more than a decade to reach the point of Andreessen’s pronouncements. There is an enormous amount of momentum in the existing system. It took agile development almost a decade, by my count, to reach the critical mass and experience-driven “best practices” it has achieved that made Continuous Delivery even possible to try out. It seems logical that a similar time period will be required to “crowd out” other agile new-product development processes and supersede yet more non-agile ones.

I should also add that, as it stands, it seems to me that Continuous Delivery has a flaw that needs to be worked on, although it does not detract from its superiority as a process. Continuous Delivery encourages breaking down features and changes into smaller chunks that reflect shorter-term thinking. This causes two sub-optimal tendencies: features that “look less far ahead”, and features that are less well integrated. To encapsulate this argument: the virtue of a Steve Jobs is that he has been able to see further ahead of where his customers were, and that he has been able to integrate all the features of a new product together exceptionally well, in the service of one, focused vision rather than many, diffused visions.

Continuous Delivery, as it stands, pushes the bar more towards more present-focused features that are integrated more as a politician aggregates the present views of his or her constituents. Somehow, Continuous Delivery needs to re-infuse the new-product development process with the ability to be guided at appropriate times by the strong views of an individual “design star” like Steve Jobs – else the Agile World will lose some of its ability to deliver the “new, new thing.” And it would be the ultimate irony if agile development, which aims to release programmers from the stultifying, counter-productive constraints of corporate development, wound up drowning their voices in a sea of other (end-user) views.


But who cares? In this latest of bad-news seasons, it’s really nice to look at something that is fundamentally, unreservedly good. And the Agile World brought about by Thoughtworks’ Continuous Delivery’s Third Age of software development, I wholeheartedly believe, is Good News.

Analysis of the War in Libya

I find myself extremely frustrated by the coverage and analysis of the war in Libya. The narrative is of an initial surge of revolt along the coast, with counterattacks by the regime eliminating all western revolt except in Misrata, which barely survived, followed by months of stalemate between east and west, and then suddenly an attack on Tripoli from the interior which has now effectively succeeded. I admit, I am an armchair general, but this seems so clearly wrongheaded to me that even I can do better.

Here is my analysis of the war: the initial revolt was, in the east, so comprehensive as to give the rebels a strong base to west of Benghazi. That was rapidly complemented by control of the borders with Egypt and Sudan. However, to the west and in the rest of the south, the regime was able to counterattack because risings were less strong and uncoordinated. As a result, effectively, the regime was able to control not only that stretch of the seacoast, but, more importantly, the southern borders – which meant that it could buy mercenaries from the starving men of Chad in whatever numbers to supplement existing troops. The counterbalance for the rebels along the coast was NATO bombing, which limited the ability of regime troops to attack within the cities of the coast.

This balance was precarious, for the regime. They needed a guaranteed flow of oil for their tanks. They needed recruits to replace the steady flow of defectors. The rebels had some money worries, but as long as Misrata was in play there would be no effective attacks towards Benghazi, and even if Misrata had fallen it is doubtful that the regime could have pressed home attacks towards Benghazi with the extra men in the face of NATO bombings. No, it was a “stalemate” that in all likelihood could be broken by the rebels but not by the regime.

At this point, a second point of insurrection emerged in the west, on a line from deep in the interior on the Tunisia border towards Gharyan midway along the key Tripoli-to-southern-border road. It was by no means a neat, clean, fully controlled territory; but the playing field was effectively tilted against the regime in that area, since the rebels from Tunisia with local support could muster more people to attack more areas than the regime could send troops to counter them, through a long stretch of tough terrain, in order to drive them back into Tunisia. However, the longer the line from the Tunisian border, the shorter the line to Tripoli, and so there was a reasonable prospect that the line between the two would stabilize before it threatened north-south shipments of oil and mercenaries.

The point when it was clear that this push was breaking the stalemate was well over two months ago, when the rebels managed to start attacking the towns that were the last ones before Gharyan. The moment they could do that, they could threaten the north-south flow, and unless the regime somehow managed to counterattack to send the Nafusa push beyond the range of Gharyan, that threat would remain.

But there was another, equally significant development at about the same time, even further south. The rebels, attacking across a wide swath of desert near the southern border, managed to take two of the towns relatively near that border. As a result, the flow of mercenaries was bound to be badly disrupted.

At this point – well over two months ago – the situation along the coast becomes relevant again. Because the rebels attacking via Nafusa were often only hundreds strong. It may seem unbelievable that regime could not send enough troops to the interior to counterattack effectively; but the fact on the ground is, they did not. Why?

The easy answer is, they were tied down by NATO bombings. However, it is clear that NATO bombings played very little role in the fighting in the interior, neither in the Nafusa region nor near the southern border. Rather, the NATO bombings served only to hinder the regime’s ability to attack, and only along the seacoast.

The correct answer, I believe, is that the regime had neither the personnel to spare nor a way of sending enough of them. The choking off of mercenaries meant that the rebels could replenish their troops after battlefield losses in the interior, while the regime could not. Moreover, the regime could not transfer troops from various areas along the coast, because they were pretty much all tied up facing the rebels along the coast.

It is at this point that the “stalemate” along the coast begins to matter. If the regime had drawn its troops from the east, around Sirte, they would have a long way to travel; they would have to skirt Misrata, with the danger of NATO bombing; and they were fully engaged with Misrata and with Brega, the more or less fluid border between the regime and the eastern rebels in Benghazi. That left the troops in the west, and what the current attacks show is that there were enough regime troops short of Tripoli to repel any attack along the coast, but not enough to counterattack into the interior.

That left Tripoli. However, again, it seems clear that Tripoli troops were fully engaged, in the east toward Misrata, and to some extent in the south to protect the nearby airport and the initial stages of the road and oil pipeline south. In effect, the line of the coast between Tripoli and the Tunisian border could be attacked at any point from the interior, and the regime would be slow to respond and would have little to spare.

Nevertheless, according to reports, the actual attack from the interior was so feeble as to invite disbelief. The number of attackers was about 300; they drove to their target in the equivalent of Jeeps, having little military ordnance; and they were a combination of previously apparently separate smaller groups. It is as if those members of the Harvard Class of 2001 who attended this year’s class reunion had decided to attack Stamford, Connecticut, and had all just grabbed a gun and driven down in their BMWs – and succeeded.

Still, to anyone looking at the situation in the middle of June with a critical eye, it should have been obvious that the rebels had a very good chance of succeeding in whatever they tried. This is because the regime at that point had failed in its counterattack – if it even tried. Reporters didn’t notice it; analysts, preoccupied with the NATO bombings, didn’t notice it; but it was vitally important that the regime not only kick the Nafusa rebels out of the towns they were attacking, but attack the towns beyond them, as well; and the regime didn’t do that.

After mid-June, therefore, the stalemate was definitively over and the end of the regime was only a matter of time. Still, it could have been a matter of a fairly long time. The rebel push could have run out of gas around Gharyan, and it might have been months to a year before the full effects of the loss of mercenaries and oil were felt.

It did not take a year because the rebels at this point, as noted above, took a chance. They basically picked a point on the coast between Tripoli and Tunisia, and the moment they fully took the town before Gharyan, instead of sending their full weight against Gharyan, they immediately sent the bulk of their troops in a headlong dash at Al Zawia, the biggest town on the coast west of Tripoli, and closer to Tripoli than the Tunisian border.

This move reminds one of the remark by Steven Brust, the fantasy author, that battles are won when one side fails to make a mistake at a critical moment. There were certainly endless mistakes by the strategists on either side, and especially the tendency of the rebels to send their troops in to attack positions carefully prepared for maximum casualties by the regime, and the tendency of the regime to expose its troops to NATO bombings unnecessarily, instead of just tying up the rebels inside the towns, where NATO would find it hard to distinguish friend from foe. I believe that once the Nafusa offensive reached a certain point, and the southern border was contested, the rebels were pretty much bound to win; but attacking immediately after the coast was in range was what meant that the war would be won now.

The reason the attack on Al Zawia was successful, and that it was then followed by success on all coastal fronts, was that the attack started a cascade of successes in which regime troops surrendered piecemeal, decreasing the regime’s manpower, and the rebels began to decrease the number of fronts on which they had to engage, allowing that manpower to be funneled into the remaining areas. Thus, the eventual success at Al Zawia meant that troops shipped in from the eastern front, where pressure had eased, could supplement troops from the interior in the attack on Tripoli, and cut off any troops between Al Zawia and the Tunisian border. Before, an attack on the coast over that border had failed. Now, troops have swept along the coast to that border and control almost all of it.

However – and here again commentators seem to me to have entirely missed the mark – the war is by no means over. It is not only that Tripoli is by no means secure; it is that there remain pockets of resistance along the coast (the remaining pockets of resistance in the interior, while huge, represent very little regime military manpower, and certainly not enough to threaten the rebels on the coast in the foreseeable future).

The two key points, right now, are Zuwara and Ben Jawad. Zuwara is pretty much the only holdout along the coast between Tripoli and Tunisia [update: It has now been taken, but attack from the interior is still holding troops in play]. Once that is secure, the rebels will add their manpower there to the troops in Tripoli, tilting the balance in favor of the rebels. Ben Jawad is the only town still contested on the coast to the east of Tripoli. Once that is gone, the rebels from Misrata and those from Benghazi will join hands around Sirte. But based on past experience, both of those events can happen tomorrow, or a month from now; and therefore, the final fall of Tripoli and Sirte could be as much as 2 months down the road, in a worst case scenario.

Nevertheless, I believe that there is just about nothing the regime can do, now, to restore any kind of stalemate. Their troops will keep becoming fewer, the rebels, greater, because of inevitable regime surrenders and lack of any way to add more regime troops. The fact that the regime will run out of supplies is only relevant in that commanders will see that and surrender sooner rather than later.

So let’s list the ways that reporters and analysts would have you think, and the ways I disagree. “The rebels were innocents led by inexperienced and flawed commanders that only won because NATO bombed the hell out of the regime” – flatly wrong. The rebels for the last 6 months have had a steady, smart long-term strategy that was just as important as NATO bombings, and that won the war in the interior, where NATO had very little effect.

“The war was fought and won mostly on the coast.” Wrong again; the stalemate was ended decisively in the interior. “There was a stalemate from March until early August”; nope, there was an increasingly tilted semi-stalemate until mid-June, when the rebels clearly established that they would win. “The war is over”. No; as Yogi Berra once said, “it ain’t over until it’s over”; and the news reports clearly establish that there is still plenty of fighting going on.

This experience has taught me one thing, very clearly; most of the analysts I see and hear are pretty much useless in certain cases, reporters or not. There were analysts and statesmen still talking about the importance of NATO bombing civilians and of the stalemate in the region as little as three weeks ago, while members of Congress questioned the President’s power to bomb Libya as if it mattered any more. Meanwhile, Wikipedia was pretty much the only source I needed to see that things were different. Folks, what’s wrong with this picture? How could it be that, almost universally, the authorities we depend on screwed this up?

I’m only an armchair general, and not a very good one, so I don’t know the answer to that question. But I can tell you, the next time there’s a conflict out there, I’m going to Wikipedia first.

Giving the Devil His Due -- Then Naming Him Evil

Careful readers of one of my previous blog posts will note that I said nothing about the validity of the complaint of scientific misconduct against the “Arctic scientist muzzled.” I had assumed that better commentators would cover that subject quite well. However, thinking about it, it occurs to me that I have a unique perspective on the specifics of the case. So I’d like to lay out what I think happened – and what I conclude it means.

Setting the Stage

Monnett, the scientist, is an employee of the Mining and Materials Service (recently renamed) in the Interior Dept. As it happened, in the summer of 1969 I was an intern at the Office of the Under Secretary of Interior in Washington, so I got a birds’-eye view of Interior, later supplemented by a senior thesis on its workings.
Back then, Interior had two types of Services: “foxes” (the Army Corps of Engineers and MMS) and “chickens” (Fish and Wildlife Service). In very broad terms, foxes would tend to favor the interest of business, while chickens would tend to favor the interest of environmentalists. At the time, new environmental departments were being formed, and the thought was that placing these in Interior would tilt the department more towards environmentalism, both in the department and in its representation in Congress, so that the overall result was a stronger US environmental policy.

As it happened, Nixon chose to place EPA and NOAA in a separate agency to cut down Interior’s power, because he felt Secretary Hickel was being politically disloyal (Hickel himself confirmed this to me). It appears, therefore, that Interior remains today much the same as it was then – with fox and chicken departments, and MMS being one of the foxes. Monnett, his investigators, and his bosses and co-workers all seem to be employees of the MMS.

Monnett had an odd function at the MMS. As part of an earlier compromise between Alaskan native communities and oil companies seeking permits to drill offshore, he was to monitor animal activities vital to these natives – specifically, the bowhead whale migration – to see if oil drilling had any effect on them. It was important to the oil companies and to MMS that drilling not be interrupted for no reason; it was important to the natives that their means of livelihood not be disrupted, and therefore Monnett’s primary responsibility was supposed to be to determine if it was. At the same time, Monnett as a scientist was supposed to provide and analyze “just the facts”.

The Complaint

What I am saying here is pure conjecture, but is my most positive take as to what the person(s) who generated the complaint against Monnett was thinking.

In the years shortly before Monnett’s paper, oil company engineers and/or geologists were probably making complaints about Monnett. He appeared to be changing the statistics for determining problems with bowhead whale migration in the middle of the game, in favor of finding problems. There was friction with his superiors, who were questioning his conclusions. The engineers and/or geologists were credible complainers, who could be expected to understand his math and his experiments. And then the paper arrived.

To the complainer, looking at the paper, it appeared odd. For one thing, it appeared that most of the data in the paper was not collected as part of a designed survey, but rather as the “other” category in a survey on whale migration. Over the last 10 years Monnett himself collected the majority of the data, and the categories in which the data was typically recorded were unclear. In order to get records from 20-10 years ago, Monnett appeared to simply ask his predecessor what he recalled.

Then there were the “statistics” of the paper. In the current year’s data (the only one that seemed to have independent verification) there were only 4 swimming polar bears in the first day’s survey, and 3 dead polar bears in the second day’s survey. Traditionally, this is usually far too small a sample to draw any statistical conclusions at all.

“Scientific misconduct” to me typically means misrepresenting the real data. Switching to legal terms for a moment, Monnett apparently had the means, the motive, and the opportunity to do so. But to show scientific misconduct, Monnett had to actually misrepresent the data. Was there something in the paper that seemed to show data misrepresentation? Well, yes. It might have seemed to the complainer (as an example) that Monnett was claiming that “death of polar bears caused by their attempts to swim to receding ice” was conclusively proven. In other words, Monnett may or may not have falsified the data; but at least he seemed to be claiming that the data proved something that the clearly valid data he had could not possibly support.

Given this, the complainer may have felt he had the equivalent of a “prima facie” case: a case in which the evidence tended to show misconduct, and the burden was on Monnett to show it was untrue.

The Game

In reading the transcript of the interview, it is helpful to think of it as partially a legal “game.” This is the way it apparently worked. The investigators arrived and were surprised to find that Monnett was represented by legal counsel, and they were being recorded. This meant that all of the actors in the interview had specific aims and tasks. The easiest role was that of the lawyer. As a good lawyer, he could be counted on to be quite aggressive in making claims about his client’s innocence, and noting instances where he could claim the investigators themselves were breaking the law.

Monnett himself was walking a careful line. On the one hand, the best way to put himself in a strong legal position was to overwhelm the investigators with details of the things that he had done right. On the other hand, the more he talked, the more possible mistakes he might reveal. Thus, throughout the interview, Monnett is providing massive amounts of detail, trying on the fly to shore up things that might be used to convict him, and trying to avoid being drawn into conclusions beyond what he had already said in the paper.

Meanwhile, the investigators split their role. Gary went through the details of the complainer’s allegation; thus, he focused on data collection, whether the data or conclusions in the paper had been reviewed adequately, and whether the data justified the conclusions. He was, in effect, trying to build the bricks to solidify and support the prima facie case. Lynn was his protector; when he seemed to be saying things that might expose the investigators or the department to legal attack, she would cut him off. Therefore, the investigators as a whole spent great effort in avoiding laying out what the details of the complaint were: these details might alert Monnett to areas of vulnerability and expose the department to counter-actions.

My Verdict – Not Just Not Proven, Innocent

Having heard the prima facie case, let’s analyze the data and the paper based on the information in the interview. What did it show?

First, let’s assume that the data is valid. Can we draw any statistical conclusions at all from it? Actually, yes. 4 and 3 may seem like small deviations from zero, but they follow a string of 20 zeros. The only plausible model I can see in the real world for a null hypothesis, that more than 3 swimming or dead polar bears is “the luck of the draw,” is a lambda probability function, in which the most probable occurrence is 0, the next most probable 1, and so on. Given 20 years of zeros, the only lambda function that would fit would have a probability of less than 1% of seeing 1 swimming/dead polar bear in year 21, and less than .1% of seeing 3 or more. It seems very clear to me that, if the data are valid, it is overwhelmingly likely that something new has happened to cause the swimming and dead polar bears.

All right, how about the validity of the data? Well, the three dead polar bears are clearly valid: there were attested by several other people besides Monnett. The four swimming polar bears are very likely to be valid, both because no one else in the helicopter, hearing the unusual sighting, found anything wrong, and because the death of the polar bears had to be caused by something, and that something was in the context of things highly likely to have been swimming. That leaves the question of whether Monnett omitted previous sightings of swimming and/or dead polar bears.

Actually, that one’s pretty easy to dispose of without recourse to statistics or the likelihood of personal actions. As Monnett makes clear, this was about the first year in which ice had clearly drawn off from the land. In most if not all previous years, Monnett or his predecessor could not possibly have seen swimming polar bears, since there was no water to swim in, just ice. And as for dead polar bears, again, it is overwhelmingly likely that 2 or 3 would have been noted in previous years in any case, since the researchers were trying to note any anomaly and the natives would care about this one, and no one has advanced a clear alternative cause (in previous years) for three to die.

Did Monnett overclaim based on this data? It appears that he said: (1) The numbers of swimming and dead polar bears was unusual, (2) It appears that the deaths were caused by the polar bears’ need to swim to the ice, (3) The 4 and 3 bears in 11% of his total area can be “extrapolated” to 36 and 27 across his entire area and a 25% survival rate (of bears that swam). We’ve already shown that statistically (whether or not he did the statistics), conclusion (1) is completely sound.

As for (2), this is not an overclaim. Something caused the deaths, and the most obvious explanation is the most obvious new factor that should cause an increase in deaths: withdrawal of ice from the land. That’s the equivalent of what Monnett said.
And then there’s (3). As I noted in my previous blog post, “extrapolation” is not an indication that something is highly likely. It’s an indication that if your sample from a population turns up this result, it’s the most likely frequency in the entire population, but not very likely in and of itself – and the frequency in the population is just as likely to be greater than that number as to be less than that number.

Were I a reviewer of such a paper, I would understand its import to be just as Monnett indicated in the interview: an establishment of a result that cannot be explained by the hypothesis of “just random chance”, a proposed explanation for the result that is a logically and physically appropriate model, and an invitation to other scientists to do work to replicate or propose alternative new explanations for the results. I would regard this as an entirely appropriate use of necessarily limited data.

My verdict: the prima facie case is entirely destroyed. Yes, in an ideal case the experiment could have been done better and better statistical tests might have been applied; but they were not needed. Monnett’s paper shows no scientific misconduct and is entirely appropriate to the data.

Why the Devil is in the Follow-on

Having, as I have said, given the “devil” (the complainer) his due, I maintain that it seems very likely that whoever has suspended Monnett has, in fact, been guilty of legal and scientific misconduct himself (or herself). Why? Because of what happened after the interview.

I have shown, I hope, that Monnett established, at the least, a strong prima facie case during the interview that he did not misrepresent the data. From then until now, there have been no public communications, nor communications with Monnett, about the case.

However, a week ago, Monnett was placed on “administrative leave”, and the stated cause was an investigation by the same investigator of allegations that Monnett had mishandled administration of a contract with U. of Alberta as subcontracted researchers into polar bear migration. Moreover, the same paper trail of permissions for his work is apparently available for the contract administration. That means that (1) investigators failed to investigate Monnett’s allegations of research distortion; (2) Monnett was suspended without even double-checking a “prima facie case”, if there was one – since investigators already knew that he typically had a paper trail; (3) Monnett was never cleared of the original charge by the investigators, although that was supposedly their only concern. What seems very likely is that this suspension is the result of proceedings covered by legal requirements that deliberately failed to follow them, both with regard to the paper and the administration. Whatever the relevant law, that suggests to me misconduct according to the law.

It also suggests scientific misconduct. The lack of contact with Monnett about his paper makes it clear that they failed to follow up with him his allegation that his research report had been distorted. Misrepresenting the data in a research report should also be considered scientific misconduct, whether the report is distributed beyond the department or not. And failure to correct a report that you should have known is distorted means that you share in the scientific misconduct, whether you are a scientist or not.

In the abstract, this type of misconduct seems isolated and unimportant. But it is also clear that these issues matter a great deal to everyone. Had the oil companies had a legitimate complaint, and if additional oil had mattered a great deal to the world’s economy and well-being, a Monnett prevention of drilling would have had major negative effects. If a valid Monnett scientific finding is not weighed and responded to appropriately and fossil fuels are indeed causing a global warming disaster, then this suspension and similar actions are equally if not more damaging to the world.
It appears from the only available evidence that those who suspended Monnett have thereby put themselves beyond excuse. I have tried to give the devil his due – is there any way I can avoid now naming him evil?

The Final Question -- and Cupcakes

Once upon a time, there was a math professor that gave an exam. When the students entered the testing room, they were given a mathematical framework and a list of problems to solve, each of which built on the solution to the previous one. The professor told them that they must solve all the problems in the time allotted, and that whoever did so, would pass; whoever did not, would fail.

The professor also told them that they should take each proposed problem solution to a machine in the corner. If the solution was correct, the machine would give them a cupcake.

Well, on the first problem some of the students who found the solution first got a bit rowdy and started waving their cupcakes in the faces of those who were slower and teasing them. The professor told them to settle down and stop it, or the professor would kick them out and they would fail.

At that that point, one of the students said to several others: “Look, this test is hard, and I think it’s so hard that none of us is going to pass unless the professor lets us. So the best way to do that is to show the professor that we appreciate all that he has done to teach us, and that we’re trying hard to please him, and we’re good people – not like those rowdy ones. So let’s all work together on each problem, and when we get our cupcakes we’ll keep giving some of them to the professor. And if we give him enough cupcakes, even though we will never answer all the questions, we’ll still pass.”

Many of the students, looking at the nasty problems coming up, were convinced that this was the only way they were going to pass, and so they agreed. But the student did not convince everybody. Some students rejected the idea that the test was impossible, and that passing the exam depended in any way on the professor.

Then there ensued a strange scene. On each problem, the students working together in a group solved the problem faster, and so should have given the machine the solution ahead of the few who were working solo. But many of them were concerned that they didn’t have enough cupcakes to give, and so they were over trying to convince the ones who were doing it alone that they should try to join the group. And sometimes they would succeed, as one of the loners felt that the test was becoming too hard; and sometimes they would lose a member, as some members of the group felt their point of view wasn’t being listened to. And then sometimes one of the group would get hungry and eat a cupcake instead of contributing it; so they had to kick that member out of the group until the member said he or she was sorry. And then they had to spend some time gathering the cupcakes and giving them to the professor. So as it worked out, the loners were solving the problems just about as fast as the group.

And then they got to the problem right before the last one. And it was a really nasty problem. Moreover, because they had been working in a group, the members of the group hadn’t been paying close attention to all the details of the previous solution, so none of them could put it all together and come up with the last step in this solution. And they were really worried that they hadn’t gotten enough cupcakes yet.

So finally one of the loners came up with the solution. And this caused an even bigger argument among the group, but finally one part of the group went over and looked at the loner’s answer, and got the solution – although they didn’t really understand that last step. And now there were two groups. The first group said, we haven’t got enough cupcakes, but if everyone comes back and helps us solve the next-to-last problem, and gives us their cupcakes, we’ll have enough. And the second group said, yes, we solved the next-to-last problem, but if we don’t get the cupcakes for the next-to-last problem from everybody, the first group and the loners, and give them to the professor, the professor will decide that we can’t work together and just aren’t nice enough. And the loners just kept being stubbornly convinced that it was all about solving the final problem.

And now there was frantic activity all over the room. The first group was still trying to solve the next-to-last problem, and badgering the second group and the loners for cupcakes. The second group had given up on the final problem (partly because they still thought it was impossible, and partly because they couldn’t solve it anyway, since they didn’t understand the solution to the next-to-last problem) and was badgering the first group and the loners for cupcakes. And the loners were trying to work on the final problem, and telling both groups that if they weren’t going to help solve the final problem, at least they should go away and stop bothering them. And as they did this, the clock kept ticking; and the end of the exam drew closer; and closer; and closer.

The moral of the story is – no, I’m not going to tell you the moral of the story. Consider this the final question of your exam: what is the moral of this story? If you get it right, you pass; if you get it wrong, you fail.

You can send me some cupcakes, if you think that will help.

Trashing JRR Tolkien

I note in the news that here in the US, a Republican Senator has criticized Republican House members by calling them naïve “hobbits”, and a Republican Representative has riposted by saying he’d rather be a hobbit than a troll. This is just the latest instance of Tolkien’s Lord of the Rings being referenced by US politicians, especially Republicans.

Enough, enough of this. I may be alone in saying this, but I suspect that Tolkien would be sick at heart at what his message has become.

It is clear from his published Letters that Tolkien distrusted Americans – which, at that time (1950s and early 1960s), meant American men. He was, for example, very wary of having his books made into movies by Americans – his stated reason was that the books’ words were meant to be mythical, and didn’t sound right when spoken. As it turns out, the makers of the movie found that in many cases the words of the book sounded better than a modern rewrite, so in this case he was unnecessarily modest. Also, a Ballantine version, while popularizing the book in America, infringed his copyright, so that at first he was deprived of badly needed royalties. But there was more than that at work; what else caused him to distrust American men, is not clear.

However, I find increasing evidence since that time that he was right to do so. I first read the books in the early 1960s, so I have watched the phenomenon unfold since the beginning. The divergence began then, I think.

One of the notable features about The Lord of the Rings that I have noted is that it has appealed to an unusual number of women – not only men. This was unusual for its time: fantasy then in America was focused on Robert Howard’s over-muscled superheroes, with the occasional female jock who somehow lacked the basic protective armor except in strategic but minimal places – not the kind of thing that many women then or now find interesting, except as an obvious example of men’s incomprehensible, irritating taste. Moreover, as female norms have changed over here, Tolkien’s focus on men characters would normally seem dated.

The key, I think, lies in a surprising amount of “rethinking” underlying the text. Not that Tolkien was a proto-feminist. However, there is a very interesting story called “Aldarion and Eldaris” set in the same world that he wrote at about that time, that clearly enters into his thinking. In it, he sets out at great length and with some sympathy the objections of the partner whom the hero-king on his quest has left. Moreover, in her own way Eldaris sets up a counter-kingdom excluding men, showing clear leadership qualities. We see traces of this not in Arwen, as the movie would have it, but in Eowyn, who is able to make the switch from killing things to being an equal ruler who grows things.

This, in turn, allows women who read it for the first time to feel connected to the story, and then to appreciate Tolkien’s focus on relationships and beauty. It was noteworthy that when the movie came out, the one scholar who focused on Tolkien’s vision of a nature that was so alive it glowed with an inner light, was the one woman scholar quoted.

However, right from the start, many American men have read these same sections with impatience, even scorn at Tolkien’s style. Early on, Gary Gygax, creator of the Dungeons and Dragons game that was in many ways a straightforward elaboration of Tolkien’s cast of creatures, was quoted as saying that he could write better, because he would go straight to the action. Peter Jackson (yes, I know he’s from New Zealand, but he was clearly tailoring the movie to American tastes) did his very best to keep the sense of dread and action constant, and dropped the much of the first “book” in consequence. Moreover, his experience in horror movies tailored to American tastes, I think, led him to the understandable decision to give a “horror” or “action” tinge to all the scenes, especially the fight scenes. This movie, in turn, reinforced a new generation of American men who may never have read the book in the belief that it’s about plucky little and big superheroes who go out and save the world from evil by winning wars.

This is so far from Tolkien’s apparent thinking that it’s hard to know where to begin. His experience in the trenches of WW I left him with what today might be described as Post Traumatic Stress Disorder, and he was reluctant to have his son serve in WW II, because he felt that it was being used by the government to demonize the enemy. Lord of the Rings reflects that thinking: war is a horrible experience (“it was Sam’s first experience of a battle, and he didn’t like it much”), and a last resort, and those who wage it must be very clear-eyed about why it is necessary and how to minimize its effects. The Lord of Gondor, who believes the struggle is all about Gondor and only he can save it, is just as destructive as the counselor in Rohan who betrays the kingdom in order to “get” the woman he lusts after.

The second key point is that the story is about people changing – or not. Above all, it’s a story about Frodo changing. Perhaps the sentence that is most charged with meaning in the entire story is this: “I wanted to save the world, and it was saved – but not for me.” The entire first book is filled with meetings in which Frodo’s (and our) vision of the world is stretched, and stretched again, until he understands much of what is valuable in the world from its beginning until now. Then – and this point cannot be emphasized enough – he sets out to help save that world, and he fails. The result of that failure – the failure to resist the Ring enough to cast it in the Fire – and of his wounds, the equivalent of PTSD, is that he can’t stop missing the Ring, and he can’t enjoy the world that is saved, and especially because many of the things he wanted to save are vanishing anyway. I repeat: this is not a story about superheroes winning wars; it is about people enlarging their vision so they understand what’s at stake, instead of thinking it’s all about them and only they know the right answer. “Even Sauron was not evil in the beginning”; and that was the key mistake Sauron made.

The third key point is that in Tolkien’s world, not only the people are connected parts of an integral whole, but so is nature. The weather, the trees, all affect people’s moods and thinking, and are in turn fostered and destroyed by people. This is no abstract “the hero and his buddies go on a quest” folktale. It is a story in which your success is connected to the success of Brand and Dain halfway around the world, and your failure at Weathertop to the north is connected to the success of the Haradrim at Pelargir in the uttermost south. It is a story in which you would not succeed unless you helped save the trees of Fangorn Forest, and unless the wind from the South dissipated the storm of Mordor. If you lose that connection, the world is “blasted beyond repair, unless the healing hand of the Sea should cover the land in merciful oblivion.”

OK, so back to my quotes. It’s about plucky hobbits and evil trolls and fighting the enemy – yes, that sounds like an American man; and is the opposite of Tolkien. It’s time to cut down the evil government, we know how to do it, and we don’t believe the alarmists who say the cure is worse than the disease – yes, that sounds like an American man; and is the opposite of Tolkien. Let’s cut back on environmental regulation to save money, and drill more oil for our energy needs, as the rich suggest; well, let’s see what Tolkien says: “He started shipping food down south. People didn’t like it, what with winter coming on … But lately, he’s been pouring filth out of that mill for no reason, fouling the water …” When there’s a clear connection between present increases in fossil fuels and future injury of most people’s habitat, Tolkien would very clearly vote for better environmental protection and less drilling. Whether you think he’s right or not, he very likely would regard the US House efforts in this regard as clearly evil.

There’s a wonderful quote I have never been able to track down, that runs something like this: “When a man first commits murder, it is to be expected that he will then pass to assault and battery; he may indeed go on to wanton destruction of personal property, and in some cases from thence to defacement of public property; indeed, as hard as it is to believe, even violation of the Sabbath may not be beyond his reach.” I realize that in the scale of things, compared to risking the full faith and credit of the US government and threatening the global economy, trashing JRR Tolkien’s deep personal beliefs may not be of the same measure. But still, you American men, can you not at least spare those few of us who appreciate the full measure of his work the act of trampling on his grave? At long last, sirs, have you no shame? No shame at all?

Hello?