Recently there passed in front of me two sets of thoughts: one, an article by Marc Andreessen (once a Web software developer, now a VC) in the Wall Street Journal about how software is “eating the world”; one by a small development tools/services company called Thoughtworks about a new technique called Continuous Delivery.
Guess which one I think is more important.
Still, both make important points about changes in computing that have fundamental effects on the global economy. So let’s take them one at a time, and then see why I think Continuous Delivery is more important – indeed, IMHO, far more important. A brief preview: I think that Continuous Delivery is the technique that takes us into the Third Age of Software Development – with effects that could be even more profound than the advent of the Second Age.
To Eat the World, First Add Spices
A very brief and cursory summary of Andreessen’s argument is that companies selling software are not only dominant in the computer industry, but have disrupted (Amazon for books, Netflix for movies) or are “poised to disrupt” (retail marketing, telecom, recruiting, financial services, health care, education, national defense) major additional industries. The result will be a large proportion of the world’s industries and economy in which software companies will be dominant – thus, software will “eat the world”.
My quarrel with this argument is not that it overstates the case, but rather that it understates it. What Andreessen omits, I assume in the interest of conciseness, is that in far more industries and companies, software now is the competitive differentiator between companies, even though the companies’ typical pedigree is primarily “hardware” or “services”, and the spending on software is a small percentage of overall costs.
My favorite examples, of course, are the company providing audio and video for stadiums, which relies on software to synchronize the feeds to the attendees better than the competition, and Boeing, which would not be able to provide the differentiating cost-saving use of composites were it not for the massive amounts of software code coordinating and monitoring the components of the Dreamliner. In health care, handling the medical health record and support for the new concept of the medical home requires differentiating software to glue together existing tools and health care specialists, and allow them to offer new types of services. It isn’t just the solo software company that is eating the world; it’s the spice of software differentiation in the mostly non-software businesses that is eating these companies from within.
So Andreessen’s article signals a growing awareness in the global economy of the importance not just of computing, but of software specifically, and not just as a separate element of the economy, but as a pervasive and thoroughly embedded aspect of the economy whose use is critical to global economic success or failure. How far we have come, indeed, from the point 30-odd years ago when I suggested to Prof. Thurow of MIT that software competitive advantage would allow the US to triumph over the Japanese economic onslaught, and he quite reasonably laughed the idea to scorn.
I do have one nit to pick with Andreessen. He trots out the assertion, which I have heard many, many times in the past 35 years, that companies just can’t find the programming skills they need – and this time, he throws in marketing. Every time people have said this, even at the height of the Internet boom, I have observed that a large body of programmers trained in a previous generation of the technology is being overlooked. That sounds as if it makes sense; but, as anyone should know by now, it is part of the DNA of any decent programmer to constantly acquire new knowledge within and beyond a skill set – good programmers are typically good programmers in any arena, from Web-site design to cloud “program virtualization”.
Meanwhile, I was hearing as late as a year ago about the college pipeline of new programmers being choked off because students saw no future jobs (except in mainframes!), and I can attest that available, technically savvy marketers in the Boston area – a well-known breeding ground – are still thick on the ground, while prospective employers in the Boston area continue to insist that program managers linking marketing and programming have at least five years of experience in that particular job. It just won’t wash, Mr. Andreessen. The fault, as is often the case, is probably not mainly in the lack of labor, but in companies’ ideological blinders.
The Idea of Continuous Delivery
Continuous Delivery, as Thoughtworks presents it, aims to develop, upgrade, and evolve software by constant, incremental bug fixes, changes, and addition of features. The example cited is that of Flickr, the photo sharing site, which is using Continuous Delivery to change its production web site at the rate of ten or more changes per day. Continuous Delivery achieves this rate not only by overlapping development of these changes, but also by modularizing them in small chunks that still “add value” to the end user and by shortening the process from idea to deployment to less than a day in many cases.
Continuous Delivery, therefore, is a logical end point of the whole idea of agile development – and, indeed, agile development processes are the way that Thoughtworks and Flickr choose to achieve this end point. Close, constant interaction with customers/end users is in there; so is the idea of changing directions rapidly, either within each feature’s development process or by a follow-on short process that modifies the original. Operations and development, as well as testing and development, are far more intertwined. The shortness of the process allows such efficiencies as “trunk-based development”, in which the process specifically forbids multi-person parallel development “branches” and thus avoids their inevitable communication and collaboration time, which in a short process turns out to be greater than the time saved by parallelization.
I am scanting the many fascinating details of Continuous Delivery in the real world in order to focus on my main point: it works. In fact, as agile development theory predicts, it appears to work better than even other agile approaches. Specifically, Thoughtworks and Flickr report:
• A much closer fit between user needs and the product over time, because it is being constantly refreshed, with end-user input an integral part of the process;
• Less risk, because the small changes, introduced into production through a process that automatically minimizes risk and involves operations, decrease both the number of operational bugs and the time to identify and fix each; and
• Lower development costs per value added, as loosely measured by the percentage of developed features used frequently as opposed to rarely or never.
The Ages of Software Development
I did my programming time during the First Age of software development, although I didn’t recognize it as such at the time. The dominant idea then was that software development was essentially a bottleneck. While hardware and network performance and price-performance moved briskly ahead according to Moore’s Law, software developer productivity on average barely budged.
That mattered because in order to get actual solutions, end users had to stand in line and wait. They stood in line in front of the data center, while IT took as long as two years to generate requested new reports. They stood in line in front of the relatively few packaged application and infrastructure software vendors, while new features took 9 months to arrive and user wish lists took 1 ½ years to satisfy. They stood in line in front of those places because there was nowhere else to turn, or so it seemed.
I noted the death of the First Age and the arrival of the Second Age in the late ‘90s, when a major Internet vendor (Amazon, iirc) came in and told me that more than half of their software was less than six months old. Almost overnight, it seemed, the old bottlenecks vanished. It wasn’t that IT and vendor schedules were superseded; it was that companies began to recognize that software could be “disposable.” Lots of smaller-scale software could be developed independent of existing software, and its aggregate, made available across the Internet by communities, as well as mechanisms such as open source, meant that bunches of outside software could be added quickly to fill a user need. It was messy, and unpredictable, and much of the new software was “disposable” wasted effort; but it worked. I haven’t heard a serious end user complaint about 2-year IT bottlenecks for almost a decade.
Andreessen’s “software eating the world”, I believe, is a straightforward result of that Second Age. It isn’t just that faster arrival of needed software from the new software development approach allows software to start handling some new complex tasks faster than physical products by themselves – say, tuning fuel mixtures constantly for a car via software rather than waiting for the equivalent set of valves to be created and tested. It is also that the exponential leap in the amount of the resulting software means that for any given product or feature, software to do it is becoming easier to find than people or machines.
However, it appears clear that even in the Second Age, software as well as physical products retain their essential characteristics. Software development fits neatly into new product development. New products and services still operate primarily through more or less lengthy periods of development of successive versions. Software may take over greater and greater chunks of the world, and add spice to it; but it’s still the same world.
The real-world success of Continuous Delivery, I assert, signals a Third Age, in which software development is not only fast in aggregate, but also fast in unitary terms – so fast as to make the process of upgrade of a unitary application by feature additions and changes seem “continuous”. Because of the Second Age, software is now pervasive in products and services. Add the new capabilities, and all software-infused products/services -- all products/services – start changing constantly, to the point where we start viewing continuous product change as natural. Products and services that are fundamentally dynamic, not successions of static versions, are a fundamental, massive change to the global economy.
But it goes even further. These Continuous-Delivery product changes also more closely track changes in end user needs. They also increase the chances of success of introductions of the “new, new thing” in technology that are vital to a thriving, growing global economy, because those introductions are based on an understanding of end user needs at this precise moment in time, not two years ago. According to my definition of agility – rapid, effective reactive and proactive changes – they make products and services truly agile. The new world of Continuous Delivery is not just an almost completely dynamic world. It is an almost Agile World. The only un-agile parts are the rest of the company processes besides software development that continue, behind the scenes of rapidly changing products, to patch up fundamentally un-agile approaches in the same old ways.
And so, I admit, I think Thoughtworks’ news is more important than Andreessen’s. I think that the Third Age of Software Development is more important than the Second Age. I think that changing to an Agile World of products and services is far, far more important than the profound but more superficial changes that the software infused in products and services via the Second Age has caused and will cause.
Final Thoughts and Caveats
Having heralded the advent of a Third Age of Software Development and an Agile World, I must caution that I don’t believe that it will fully arrive any time in the next 2-3 years, and perhaps not for a decade, just as it took the Second Age more than a decade to reach the point of Andreessen’s pronouncements. There is an enormous amount of momentum in the existing system. It took agile development almost a decade, by my count, to reach the critical mass and experience-driven “best practices” it has achieved that made Continuous Delivery even possible to try out. It seems logical that a similar time period will be required to “crowd out” other agile new-product development processes and supersede yet more non-agile ones.
I should also add that, as it stands, it seems to me that Continuous Delivery has a flaw that needs to be worked on, although it does not detract from its superiority as a process. Continuous Delivery encourages breaking down features and changes into smaller chunks that reflect shorter-term thinking. This causes two sub-optimal tendencies: features that “look less far ahead”, and features that are less well integrated. To encapsulate this argument: the virtue of a Steve Jobs is that he has been able to see further ahead of where his customers were, and that he has been able to integrate all the features of a new product together exceptionally well, in the service of one, focused vision rather than many, diffused visions.
Continuous Delivery, as it stands, pushes the bar more towards more present-focused features that are integrated more as a politician aggregates the present views of his or her constituents. Somehow, Continuous Delivery needs to re-infuse the new-product development process with the ability to be guided at appropriate times by the strong views of an individual “design star” like Steve Jobs – else the Agile World will lose some of its ability to deliver the “new, new thing.” And it would be the ultimate irony if agile development, which aims to release programmers from the stultifying, counter-productive constraints of corporate development, wound up drowning their voices in a sea of other (end-user) views.
But who cares? In this latest of bad-news seasons, it’s really nice to look at something that is fundamentally, unreservedly good. And the Agile World brought about by Thoughtworks’ Continuous Delivery’s Third Age of software development, I wholeheartedly believe, is Good News.
Wednesday, September 7, 2011
Friday, August 26, 2011
Analysis of the War in Libya
I find myself extremely frustrated by the coverage and analysis of the war in Libya. The narrative is of an initial surge of revolt along the coast, with counterattacks by the regime eliminating all western revolt except in Misrata, which barely survived, followed by months of stalemate between east and west, and then suddenly an attack on Tripoli from the interior which has now effectively succeeded. I admit, I am an armchair general, but this seems so clearly wrongheaded to me that even I can do better.
Here is my analysis of the war: the initial revolt was, in the east, so comprehensive as to give the rebels a strong base to west of Benghazi. That was rapidly complemented by control of the borders with Egypt and Sudan. However, to the west and in the rest of the south, the regime was able to counterattack because risings were less strong and uncoordinated. As a result, effectively, the regime was able to control not only that stretch of the seacoast, but, more importantly, the southern borders – which meant that it could buy mercenaries from the starving men of Chad in whatever numbers to supplement existing troops. The counterbalance for the rebels along the coast was NATO bombing, which limited the ability of regime troops to attack within the cities of the coast.
This balance was precarious, for the regime. They needed a guaranteed flow of oil for their tanks. They needed recruits to replace the steady flow of defectors. The rebels had some money worries, but as long as Misrata was in play there would be no effective attacks towards Benghazi, and even if Misrata had fallen it is doubtful that the regime could have pressed home attacks towards Benghazi with the extra men in the face of NATO bombings. No, it was a “stalemate” that in all likelihood could be broken by the rebels but not by the regime.
At this point, a second point of insurrection emerged in the west, on a line from deep in the interior on the Tunisia border towards Gharyan midway along the key Tripoli-to-southern-border road. It was by no means a neat, clean, fully controlled territory; but the playing field was effectively tilted against the regime in that area, since the rebels from Tunisia with local support could muster more people to attack more areas than the regime could send troops to counter them, through a long stretch of tough terrain, in order to drive them back into Tunisia. However, the longer the line from the Tunisian border, the shorter the line to Tripoli, and so there was a reasonable prospect that the line between the two would stabilize before it threatened north-south shipments of oil and mercenaries.
The point when it was clear that this push was breaking the stalemate was well over two months ago, when the rebels managed to start attacking the towns that were the last ones before Gharyan. The moment they could do that, they could threaten the north-south flow, and unless the regime somehow managed to counterattack to send the Nafusa push beyond the range of Gharyan, that threat would remain.
But there was another, equally significant development at about the same time, even further south. The rebels, attacking across a wide swath of desert near the southern border, managed to take two of the towns relatively near that border. As a result, the flow of mercenaries was bound to be badly disrupted.
At this point – well over two months ago – the situation along the coast becomes relevant again. Because the rebels attacking via Nafusa were often only hundreds strong. It may seem unbelievable that regime could not send enough troops to the interior to counterattack effectively; but the fact on the ground is, they did not. Why?
The easy answer is, they were tied down by NATO bombings. However, it is clear that NATO bombings played very little role in the fighting in the interior, neither in the Nafusa region nor near the southern border. Rather, the NATO bombings served only to hinder the regime’s ability to attack, and only along the seacoast.
The correct answer, I believe, is that the regime had neither the personnel to spare nor a way of sending enough of them. The choking off of mercenaries meant that the rebels could replenish their troops after battlefield losses in the interior, while the regime could not. Moreover, the regime could not transfer troops from various areas along the coast, because they were pretty much all tied up facing the rebels along the coast.
It is at this point that the “stalemate” along the coast begins to matter. If the regime had drawn its troops from the east, around Sirte, they would have a long way to travel; they would have to skirt Misrata, with the danger of NATO bombing; and they were fully engaged with Misrata and with Brega, the more or less fluid border between the regime and the eastern rebels in Benghazi. That left the troops in the west, and what the current attacks show is that there were enough regime troops short of Tripoli to repel any attack along the coast, but not enough to counterattack into the interior.
That left Tripoli. However, again, it seems clear that Tripoli troops were fully engaged, in the east toward Misrata, and to some extent in the south to protect the nearby airport and the initial stages of the road and oil pipeline south. In effect, the line of the coast between Tripoli and the Tunisian border could be attacked at any point from the interior, and the regime would be slow to respond and would have little to spare.
Nevertheless, according to reports, the actual attack from the interior was so feeble as to invite disbelief. The number of attackers was about 300; they drove to their target in the equivalent of Jeeps, having little military ordnance; and they were a combination of previously apparently separate smaller groups. It is as if those members of the Harvard Class of 2001 who attended this year’s class reunion had decided to attack Stamford, Connecticut, and had all just grabbed a gun and driven down in their BMWs – and succeeded.
Still, to anyone looking at the situation in the middle of June with a critical eye, it should have been obvious that the rebels had a very good chance of succeeding in whatever they tried. This is because the regime at that point had failed in its counterattack – if it even tried. Reporters didn’t notice it; analysts, preoccupied with the NATO bombings, didn’t notice it; but it was vitally important that the regime not only kick the Nafusa rebels out of the towns they were attacking, but attack the towns beyond them, as well; and the regime didn’t do that.
After mid-June, therefore, the stalemate was definitively over and the end of the regime was only a matter of time. Still, it could have been a matter of a fairly long time. The rebel push could have run out of gas around Gharyan, and it might have been months to a year before the full effects of the loss of mercenaries and oil were felt.
It did not take a year because the rebels at this point, as noted above, took a chance. They basically picked a point on the coast between Tripoli and Tunisia, and the moment they fully took the town before Gharyan, instead of sending their full weight against Gharyan, they immediately sent the bulk of their troops in a headlong dash at Al Zawia, the biggest town on the coast west of Tripoli, and closer to Tripoli than the Tunisian border.
This move reminds one of the remark by Steven Brust, the fantasy author, that battles are won when one side fails to make a mistake at a critical moment. There were certainly endless mistakes by the strategists on either side, and especially the tendency of the rebels to send their troops in to attack positions carefully prepared for maximum casualties by the regime, and the tendency of the regime to expose its troops to NATO bombings unnecessarily, instead of just tying up the rebels inside the towns, where NATO would find it hard to distinguish friend from foe. I believe that once the Nafusa offensive reached a certain point, and the southern border was contested, the rebels were pretty much bound to win; but attacking immediately after the coast was in range was what meant that the war would be won now.
The reason the attack on Al Zawia was successful, and that it was then followed by success on all coastal fronts, was that the attack started a cascade of successes in which regime troops surrendered piecemeal, decreasing the regime’s manpower, and the rebels began to decrease the number of fronts on which they had to engage, allowing that manpower to be funneled into the remaining areas. Thus, the eventual success at Al Zawia meant that troops shipped in from the eastern front, where pressure had eased, could supplement troops from the interior in the attack on Tripoli, and cut off any troops between Al Zawia and the Tunisian border. Before, an attack on the coast over that border had failed. Now, troops have swept along the coast to that border and control almost all of it.
However – and here again commentators seem to me to have entirely missed the mark – the war is by no means over. It is not only that Tripoli is by no means secure; it is that there remain pockets of resistance along the coast (the remaining pockets of resistance in the interior, while huge, represent very little regime military manpower, and certainly not enough to threaten the rebels on the coast in the foreseeable future).
The two key points, right now, are Zuwara and Ben Jawad. Zuwara is pretty much the only holdout along the coast between Tripoli and Tunisia [update: It has now been taken, but attack from the interior is still holding troops in play]. Once that is secure, the rebels will add their manpower there to the troops in Tripoli, tilting the balance in favor of the rebels. Ben Jawad is the only town still contested on the coast to the east of Tripoli. Once that is gone, the rebels from Misrata and those from Benghazi will join hands around Sirte. But based on past experience, both of those events can happen tomorrow, or a month from now; and therefore, the final fall of Tripoli and Sirte could be as much as 2 months down the road, in a worst case scenario.
Nevertheless, I believe that there is just about nothing the regime can do, now, to restore any kind of stalemate. Their troops will keep becoming fewer, the rebels, greater, because of inevitable regime surrenders and lack of any way to add more regime troops. The fact that the regime will run out of supplies is only relevant in that commanders will see that and surrender sooner rather than later.
So let’s list the ways that reporters and analysts would have you think, and the ways I disagree. “The rebels were innocents led by inexperienced and flawed commanders that only won because NATO bombed the hell out of the regime” – flatly wrong. The rebels for the last 6 months have had a steady, smart long-term strategy that was just as important as NATO bombings, and that won the war in the interior, where NATO had very little effect.
“The war was fought and won mostly on the coast.” Wrong again; the stalemate was ended decisively in the interior. “There was a stalemate from March until early August”; nope, there was an increasingly tilted semi-stalemate until mid-June, when the rebels clearly established that they would win. “The war is over”. No; as Yogi Berra once said, “it ain’t over until it’s over”; and the news reports clearly establish that there is still plenty of fighting going on.
This experience has taught me one thing, very clearly; most of the analysts I see and hear are pretty much useless in certain cases, reporters or not. There were analysts and statesmen still talking about the importance of NATO bombing civilians and of the stalemate in the region as little as three weeks ago, while members of Congress questioned the President’s power to bomb Libya as if it mattered any more. Meanwhile, Wikipedia was pretty much the only source I needed to see that things were different. Folks, what’s wrong with this picture? How could it be that, almost universally, the authorities we depend on screwed this up?
I’m only an armchair general, and not a very good one, so I don’t know the answer to that question. But I can tell you, the next time there’s a conflict out there, I’m going to Wikipedia first.
Here is my analysis of the war: the initial revolt was, in the east, so comprehensive as to give the rebels a strong base to west of Benghazi. That was rapidly complemented by control of the borders with Egypt and Sudan. However, to the west and in the rest of the south, the regime was able to counterattack because risings were less strong and uncoordinated. As a result, effectively, the regime was able to control not only that stretch of the seacoast, but, more importantly, the southern borders – which meant that it could buy mercenaries from the starving men of Chad in whatever numbers to supplement existing troops. The counterbalance for the rebels along the coast was NATO bombing, which limited the ability of regime troops to attack within the cities of the coast.
This balance was precarious, for the regime. They needed a guaranteed flow of oil for their tanks. They needed recruits to replace the steady flow of defectors. The rebels had some money worries, but as long as Misrata was in play there would be no effective attacks towards Benghazi, and even if Misrata had fallen it is doubtful that the regime could have pressed home attacks towards Benghazi with the extra men in the face of NATO bombings. No, it was a “stalemate” that in all likelihood could be broken by the rebels but not by the regime.
At this point, a second point of insurrection emerged in the west, on a line from deep in the interior on the Tunisia border towards Gharyan midway along the key Tripoli-to-southern-border road. It was by no means a neat, clean, fully controlled territory; but the playing field was effectively tilted against the regime in that area, since the rebels from Tunisia with local support could muster more people to attack more areas than the regime could send troops to counter them, through a long stretch of tough terrain, in order to drive them back into Tunisia. However, the longer the line from the Tunisian border, the shorter the line to Tripoli, and so there was a reasonable prospect that the line between the two would stabilize before it threatened north-south shipments of oil and mercenaries.
The point when it was clear that this push was breaking the stalemate was well over two months ago, when the rebels managed to start attacking the towns that were the last ones before Gharyan. The moment they could do that, they could threaten the north-south flow, and unless the regime somehow managed to counterattack to send the Nafusa push beyond the range of Gharyan, that threat would remain.
But there was another, equally significant development at about the same time, even further south. The rebels, attacking across a wide swath of desert near the southern border, managed to take two of the towns relatively near that border. As a result, the flow of mercenaries was bound to be badly disrupted.
At this point – well over two months ago – the situation along the coast becomes relevant again. Because the rebels attacking via Nafusa were often only hundreds strong. It may seem unbelievable that regime could not send enough troops to the interior to counterattack effectively; but the fact on the ground is, they did not. Why?
The easy answer is, they were tied down by NATO bombings. However, it is clear that NATO bombings played very little role in the fighting in the interior, neither in the Nafusa region nor near the southern border. Rather, the NATO bombings served only to hinder the regime’s ability to attack, and only along the seacoast.
The correct answer, I believe, is that the regime had neither the personnel to spare nor a way of sending enough of them. The choking off of mercenaries meant that the rebels could replenish their troops after battlefield losses in the interior, while the regime could not. Moreover, the regime could not transfer troops from various areas along the coast, because they were pretty much all tied up facing the rebels along the coast.
It is at this point that the “stalemate” along the coast begins to matter. If the regime had drawn its troops from the east, around Sirte, they would have a long way to travel; they would have to skirt Misrata, with the danger of NATO bombing; and they were fully engaged with Misrata and with Brega, the more or less fluid border between the regime and the eastern rebels in Benghazi. That left the troops in the west, and what the current attacks show is that there were enough regime troops short of Tripoli to repel any attack along the coast, but not enough to counterattack into the interior.
That left Tripoli. However, again, it seems clear that Tripoli troops were fully engaged, in the east toward Misrata, and to some extent in the south to protect the nearby airport and the initial stages of the road and oil pipeline south. In effect, the line of the coast between Tripoli and the Tunisian border could be attacked at any point from the interior, and the regime would be slow to respond and would have little to spare.
Nevertheless, according to reports, the actual attack from the interior was so feeble as to invite disbelief. The number of attackers was about 300; they drove to their target in the equivalent of Jeeps, having little military ordnance; and they were a combination of previously apparently separate smaller groups. It is as if those members of the Harvard Class of 2001 who attended this year’s class reunion had decided to attack Stamford, Connecticut, and had all just grabbed a gun and driven down in their BMWs – and succeeded.
Still, to anyone looking at the situation in the middle of June with a critical eye, it should have been obvious that the rebels had a very good chance of succeeding in whatever they tried. This is because the regime at that point had failed in its counterattack – if it even tried. Reporters didn’t notice it; analysts, preoccupied with the NATO bombings, didn’t notice it; but it was vitally important that the regime not only kick the Nafusa rebels out of the towns they were attacking, but attack the towns beyond them, as well; and the regime didn’t do that.
After mid-June, therefore, the stalemate was definitively over and the end of the regime was only a matter of time. Still, it could have been a matter of a fairly long time. The rebel push could have run out of gas around Gharyan, and it might have been months to a year before the full effects of the loss of mercenaries and oil were felt.
It did not take a year because the rebels at this point, as noted above, took a chance. They basically picked a point on the coast between Tripoli and Tunisia, and the moment they fully took the town before Gharyan, instead of sending their full weight against Gharyan, they immediately sent the bulk of their troops in a headlong dash at Al Zawia, the biggest town on the coast west of Tripoli, and closer to Tripoli than the Tunisian border.
This move reminds one of the remark by Steven Brust, the fantasy author, that battles are won when one side fails to make a mistake at a critical moment. There were certainly endless mistakes by the strategists on either side, and especially the tendency of the rebels to send their troops in to attack positions carefully prepared for maximum casualties by the regime, and the tendency of the regime to expose its troops to NATO bombings unnecessarily, instead of just tying up the rebels inside the towns, where NATO would find it hard to distinguish friend from foe. I believe that once the Nafusa offensive reached a certain point, and the southern border was contested, the rebels were pretty much bound to win; but attacking immediately after the coast was in range was what meant that the war would be won now.
The reason the attack on Al Zawia was successful, and that it was then followed by success on all coastal fronts, was that the attack started a cascade of successes in which regime troops surrendered piecemeal, decreasing the regime’s manpower, and the rebels began to decrease the number of fronts on which they had to engage, allowing that manpower to be funneled into the remaining areas. Thus, the eventual success at Al Zawia meant that troops shipped in from the eastern front, where pressure had eased, could supplement troops from the interior in the attack on Tripoli, and cut off any troops between Al Zawia and the Tunisian border. Before, an attack on the coast over that border had failed. Now, troops have swept along the coast to that border and control almost all of it.
However – and here again commentators seem to me to have entirely missed the mark – the war is by no means over. It is not only that Tripoli is by no means secure; it is that there remain pockets of resistance along the coast (the remaining pockets of resistance in the interior, while huge, represent very little regime military manpower, and certainly not enough to threaten the rebels on the coast in the foreseeable future).
The two key points, right now, are Zuwara and Ben Jawad. Zuwara is pretty much the only holdout along the coast between Tripoli and Tunisia [update: It has now been taken, but attack from the interior is still holding troops in play]. Once that is secure, the rebels will add their manpower there to the troops in Tripoli, tilting the balance in favor of the rebels. Ben Jawad is the only town still contested on the coast to the east of Tripoli. Once that is gone, the rebels from Misrata and those from Benghazi will join hands around Sirte. But based on past experience, both of those events can happen tomorrow, or a month from now; and therefore, the final fall of Tripoli and Sirte could be as much as 2 months down the road, in a worst case scenario.
Nevertheless, I believe that there is just about nothing the regime can do, now, to restore any kind of stalemate. Their troops will keep becoming fewer, the rebels, greater, because of inevitable regime surrenders and lack of any way to add more regime troops. The fact that the regime will run out of supplies is only relevant in that commanders will see that and surrender sooner rather than later.
So let’s list the ways that reporters and analysts would have you think, and the ways I disagree. “The rebels were innocents led by inexperienced and flawed commanders that only won because NATO bombed the hell out of the regime” – flatly wrong. The rebels for the last 6 months have had a steady, smart long-term strategy that was just as important as NATO bombings, and that won the war in the interior, where NATO had very little effect.
“The war was fought and won mostly on the coast.” Wrong again; the stalemate was ended decisively in the interior. “There was a stalemate from March until early August”; nope, there was an increasingly tilted semi-stalemate until mid-June, when the rebels clearly established that they would win. “The war is over”. No; as Yogi Berra once said, “it ain’t over until it’s over”; and the news reports clearly establish that there is still plenty of fighting going on.
This experience has taught me one thing, very clearly; most of the analysts I see and hear are pretty much useless in certain cases, reporters or not. There were analysts and statesmen still talking about the importance of NATO bombing civilians and of the stalemate in the region as little as three weeks ago, while members of Congress questioned the President’s power to bomb Libya as if it mattered any more. Meanwhile, Wikipedia was pretty much the only source I needed to see that things were different. Folks, what’s wrong with this picture? How could it be that, almost universally, the authorities we depend on screwed this up?
I’m only an armchair general, and not a very good one, so I don’t know the answer to that question. But I can tell you, the next time there’s a conflict out there, I’m going to Wikipedia first.
Wednesday, August 3, 2011
Giving the Devil His Due -- Then Naming Him Evil
Careful readers of one of my previous blog posts will note that I said nothing about the validity of the complaint of scientific misconduct against the “Arctic scientist muzzled.” I had assumed that better commentators would cover that subject quite well. However, thinking about it, it occurs to me that I have a unique perspective on the specifics of the case. So I’d like to lay out what I think happened – and what I conclude it means.
Setting the Stage
Monnett, the scientist, is an employee of the Mining and Materials Service (recently renamed) in the Interior Dept. As it happened, in the summer of 1969 I was an intern at the Office of the Under Secretary of Interior in Washington, so I got a birds’-eye view of Interior, later supplemented by a senior thesis on its workings.
Back then, Interior had two types of Services: “foxes” (the Army Corps of Engineers and MMS) and “chickens” (Fish and Wildlife Service). In very broad terms, foxes would tend to favor the interest of business, while chickens would tend to favor the interest of environmentalists. At the time, new environmental departments were being formed, and the thought was that placing these in Interior would tilt the department more towards environmentalism, both in the department and in its representation in Congress, so that the overall result was a stronger US environmental policy.
As it happened, Nixon chose to place EPA and NOAA in a separate agency to cut down Interior’s power, because he felt Secretary Hickel was being politically disloyal (Hickel himself confirmed this to me). It appears, therefore, that Interior remains today much the same as it was then – with fox and chicken departments, and MMS being one of the foxes. Monnett, his investigators, and his bosses and co-workers all seem to be employees of the MMS.
Monnett had an odd function at the MMS. As part of an earlier compromise between Alaskan native communities and oil companies seeking permits to drill offshore, he was to monitor animal activities vital to these natives – specifically, the bowhead whale migration – to see if oil drilling had any effect on them. It was important to the oil companies and to MMS that drilling not be interrupted for no reason; it was important to the natives that their means of livelihood not be disrupted, and therefore Monnett’s primary responsibility was supposed to be to determine if it was. At the same time, Monnett as a scientist was supposed to provide and analyze “just the facts”.
The Complaint
What I am saying here is pure conjecture, but is my most positive take as to what the person(s) who generated the complaint against Monnett was thinking.
In the years shortly before Monnett’s paper, oil company engineers and/or geologists were probably making complaints about Monnett. He appeared to be changing the statistics for determining problems with bowhead whale migration in the middle of the game, in favor of finding problems. There was friction with his superiors, who were questioning his conclusions. The engineers and/or geologists were credible complainers, who could be expected to understand his math and his experiments. And then the paper arrived.
To the complainer, looking at the paper, it appeared odd. For one thing, it appeared that most of the data in the paper was not collected as part of a designed survey, but rather as the “other” category in a survey on whale migration. Over the last 10 years Monnett himself collected the majority of the data, and the categories in which the data was typically recorded were unclear. In order to get records from 20-10 years ago, Monnett appeared to simply ask his predecessor what he recalled.
Then there were the “statistics” of the paper. In the current year’s data (the only one that seemed to have independent verification) there were only 4 swimming polar bears in the first day’s survey, and 3 dead polar bears in the second day’s survey. Traditionally, this is usually far too small a sample to draw any statistical conclusions at all.
“Scientific misconduct” to me typically means misrepresenting the real data. Switching to legal terms for a moment, Monnett apparently had the means, the motive, and the opportunity to do so. But to show scientific misconduct, Monnett had to actually misrepresent the data. Was there something in the paper that seemed to show data misrepresentation? Well, yes. It might have seemed to the complainer (as an example) that Monnett was claiming that “death of polar bears caused by their attempts to swim to receding ice” was conclusively proven. In other words, Monnett may or may not have falsified the data; but at least he seemed to be claiming that the data proved something that the clearly valid data he had could not possibly support.
Given this, the complainer may have felt he had the equivalent of a “prima facie” case: a case in which the evidence tended to show misconduct, and the burden was on Monnett to show it was untrue.
The Game
In reading the transcript of the interview, it is helpful to think of it as partially a legal “game.” This is the way it apparently worked. The investigators arrived and were surprised to find that Monnett was represented by legal counsel, and they were being recorded. This meant that all of the actors in the interview had specific aims and tasks. The easiest role was that of the lawyer. As a good lawyer, he could be counted on to be quite aggressive in making claims about his client’s innocence, and noting instances where he could claim the investigators themselves were breaking the law.
Monnett himself was walking a careful line. On the one hand, the best way to put himself in a strong legal position was to overwhelm the investigators with details of the things that he had done right. On the other hand, the more he talked, the more possible mistakes he might reveal. Thus, throughout the interview, Monnett is providing massive amounts of detail, trying on the fly to shore up things that might be used to convict him, and trying to avoid being drawn into conclusions beyond what he had already said in the paper.
Meanwhile, the investigators split their role. Gary went through the details of the complainer’s allegation; thus, he focused on data collection, whether the data or conclusions in the paper had been reviewed adequately, and whether the data justified the conclusions. He was, in effect, trying to build the bricks to solidify and support the prima facie case. Lynn was his protector; when he seemed to be saying things that might expose the investigators or the department to legal attack, she would cut him off. Therefore, the investigators as a whole spent great effort in avoiding laying out what the details of the complaint were: these details might alert Monnett to areas of vulnerability and expose the department to counter-actions.
My Verdict – Not Just Not Proven, Innocent
Having heard the prima facie case, let’s analyze the data and the paper based on the information in the interview. What did it show?
First, let’s assume that the data is valid. Can we draw any statistical conclusions at all from it? Actually, yes. 4 and 3 may seem like small deviations from zero, but they follow a string of 20 zeros. The only plausible model I can see in the real world for a null hypothesis, that more than 3 swimming or dead polar bears is “the luck of the draw,” is a lambda probability function, in which the most probable occurrence is 0, the next most probable 1, and so on. Given 20 years of zeros, the only lambda function that would fit would have a probability of less than 1% of seeing 1 swimming/dead polar bear in year 21, and less than .1% of seeing 3 or more. It seems very clear to me that, if the data are valid, it is overwhelmingly likely that something new has happened to cause the swimming and dead polar bears.
All right, how about the validity of the data? Well, the three dead polar bears are clearly valid: there were attested by several other people besides Monnett. The four swimming polar bears are very likely to be valid, both because no one else in the helicopter, hearing the unusual sighting, found anything wrong, and because the death of the polar bears had to be caused by something, and that something was in the context of things highly likely to have been swimming. That leaves the question of whether Monnett omitted previous sightings of swimming and/or dead polar bears.
Actually, that one’s pretty easy to dispose of without recourse to statistics or the likelihood of personal actions. As Monnett makes clear, this was about the first year in which ice had clearly drawn off from the land. In most if not all previous years, Monnett or his predecessor could not possibly have seen swimming polar bears, since there was no water to swim in, just ice. And as for dead polar bears, again, it is overwhelmingly likely that 2 or 3 would have been noted in previous years in any case, since the researchers were trying to note any anomaly and the natives would care about this one, and no one has advanced a clear alternative cause (in previous years) for three to die.
Did Monnett overclaim based on this data? It appears that he said: (1) The numbers of swimming and dead polar bears was unusual, (2) It appears that the deaths were caused by the polar bears’ need to swim to the ice, (3) The 4 and 3 bears in 11% of his total area can be “extrapolated” to 36 and 27 across his entire area and a 25% survival rate (of bears that swam). We’ve already shown that statistically (whether or not he did the statistics), conclusion (1) is completely sound.
As for (2), this is not an overclaim. Something caused the deaths, and the most obvious explanation is the most obvious new factor that should cause an increase in deaths: withdrawal of ice from the land. That’s the equivalent of what Monnett said.
And then there’s (3). As I noted in my previous blog post, “extrapolation” is not an indication that something is highly likely. It’s an indication that if your sample from a population turns up this result, it’s the most likely frequency in the entire population, but not very likely in and of itself – and the frequency in the population is just as likely to be greater than that number as to be less than that number.
Were I a reviewer of such a paper, I would understand its import to be just as Monnett indicated in the interview: an establishment of a result that cannot be explained by the hypothesis of “just random chance”, a proposed explanation for the result that is a logically and physically appropriate model, and an invitation to other scientists to do work to replicate or propose alternative new explanations for the results. I would regard this as an entirely appropriate use of necessarily limited data.
My verdict: the prima facie case is entirely destroyed. Yes, in an ideal case the experiment could have been done better and better statistical tests might have been applied; but they were not needed. Monnett’s paper shows no scientific misconduct and is entirely appropriate to the data.
Why the Devil is in the Follow-on
Having, as I have said, given the “devil” (the complainer) his due, I maintain that it seems very likely that whoever has suspended Monnett has, in fact, been guilty of legal and scientific misconduct himself (or herself). Why? Because of what happened after the interview.
I have shown, I hope, that Monnett established, at the least, a strong prima facie case during the interview that he did not misrepresent the data. From then until now, there have been no public communications, nor communications with Monnett, about the case.
However, a week ago, Monnett was placed on “administrative leave”, and the stated cause was an investigation by the same investigator of allegations that Monnett had mishandled administration of a contract with U. of Alberta as subcontracted researchers into polar bear migration. Moreover, the same paper trail of permissions for his work is apparently available for the contract administration. That means that (1) investigators failed to investigate Monnett’s allegations of research distortion; (2) Monnett was suspended without even double-checking a “prima facie case”, if there was one – since investigators already knew that he typically had a paper trail; (3) Monnett was never cleared of the original charge by the investigators, although that was supposedly their only concern. What seems very likely is that this suspension is the result of proceedings covered by legal requirements that deliberately failed to follow them, both with regard to the paper and the administration. Whatever the relevant law, that suggests to me misconduct according to the law.
It also suggests scientific misconduct. The lack of contact with Monnett about his paper makes it clear that they failed to follow up with him his allegation that his research report had been distorted. Misrepresenting the data in a research report should also be considered scientific misconduct, whether the report is distributed beyond the department or not. And failure to correct a report that you should have known is distorted means that you share in the scientific misconduct, whether you are a scientist or not.
In the abstract, this type of misconduct seems isolated and unimportant. But it is also clear that these issues matter a great deal to everyone. Had the oil companies had a legitimate complaint, and if additional oil had mattered a great deal to the world’s economy and well-being, a Monnett prevention of drilling would have had major negative effects. If a valid Monnett scientific finding is not weighed and responded to appropriately and fossil fuels are indeed causing a global warming disaster, then this suspension and similar actions are equally if not more damaging to the world.
It appears from the only available evidence that those who suspended Monnett have thereby put themselves beyond excuse. I have tried to give the devil his due – is there any way I can avoid now naming him evil?
Setting the Stage
Monnett, the scientist, is an employee of the Mining and Materials Service (recently renamed) in the Interior Dept. As it happened, in the summer of 1969 I was an intern at the Office of the Under Secretary of Interior in Washington, so I got a birds’-eye view of Interior, later supplemented by a senior thesis on its workings.
Back then, Interior had two types of Services: “foxes” (the Army Corps of Engineers and MMS) and “chickens” (Fish and Wildlife Service). In very broad terms, foxes would tend to favor the interest of business, while chickens would tend to favor the interest of environmentalists. At the time, new environmental departments were being formed, and the thought was that placing these in Interior would tilt the department more towards environmentalism, both in the department and in its representation in Congress, so that the overall result was a stronger US environmental policy.
As it happened, Nixon chose to place EPA and NOAA in a separate agency to cut down Interior’s power, because he felt Secretary Hickel was being politically disloyal (Hickel himself confirmed this to me). It appears, therefore, that Interior remains today much the same as it was then – with fox and chicken departments, and MMS being one of the foxes. Monnett, his investigators, and his bosses and co-workers all seem to be employees of the MMS.
Monnett had an odd function at the MMS. As part of an earlier compromise between Alaskan native communities and oil companies seeking permits to drill offshore, he was to monitor animal activities vital to these natives – specifically, the bowhead whale migration – to see if oil drilling had any effect on them. It was important to the oil companies and to MMS that drilling not be interrupted for no reason; it was important to the natives that their means of livelihood not be disrupted, and therefore Monnett’s primary responsibility was supposed to be to determine if it was. At the same time, Monnett as a scientist was supposed to provide and analyze “just the facts”.
The Complaint
What I am saying here is pure conjecture, but is my most positive take as to what the person(s) who generated the complaint against Monnett was thinking.
In the years shortly before Monnett’s paper, oil company engineers and/or geologists were probably making complaints about Monnett. He appeared to be changing the statistics for determining problems with bowhead whale migration in the middle of the game, in favor of finding problems. There was friction with his superiors, who were questioning his conclusions. The engineers and/or geologists were credible complainers, who could be expected to understand his math and his experiments. And then the paper arrived.
To the complainer, looking at the paper, it appeared odd. For one thing, it appeared that most of the data in the paper was not collected as part of a designed survey, but rather as the “other” category in a survey on whale migration. Over the last 10 years Monnett himself collected the majority of the data, and the categories in which the data was typically recorded were unclear. In order to get records from 20-10 years ago, Monnett appeared to simply ask his predecessor what he recalled.
Then there were the “statistics” of the paper. In the current year’s data (the only one that seemed to have independent verification) there were only 4 swimming polar bears in the first day’s survey, and 3 dead polar bears in the second day’s survey. Traditionally, this is usually far too small a sample to draw any statistical conclusions at all.
“Scientific misconduct” to me typically means misrepresenting the real data. Switching to legal terms for a moment, Monnett apparently had the means, the motive, and the opportunity to do so. But to show scientific misconduct, Monnett had to actually misrepresent the data. Was there something in the paper that seemed to show data misrepresentation? Well, yes. It might have seemed to the complainer (as an example) that Monnett was claiming that “death of polar bears caused by their attempts to swim to receding ice” was conclusively proven. In other words, Monnett may or may not have falsified the data; but at least he seemed to be claiming that the data proved something that the clearly valid data he had could not possibly support.
Given this, the complainer may have felt he had the equivalent of a “prima facie” case: a case in which the evidence tended to show misconduct, and the burden was on Monnett to show it was untrue.
The Game
In reading the transcript of the interview, it is helpful to think of it as partially a legal “game.” This is the way it apparently worked. The investigators arrived and were surprised to find that Monnett was represented by legal counsel, and they were being recorded. This meant that all of the actors in the interview had specific aims and tasks. The easiest role was that of the lawyer. As a good lawyer, he could be counted on to be quite aggressive in making claims about his client’s innocence, and noting instances where he could claim the investigators themselves were breaking the law.
Monnett himself was walking a careful line. On the one hand, the best way to put himself in a strong legal position was to overwhelm the investigators with details of the things that he had done right. On the other hand, the more he talked, the more possible mistakes he might reveal. Thus, throughout the interview, Monnett is providing massive amounts of detail, trying on the fly to shore up things that might be used to convict him, and trying to avoid being drawn into conclusions beyond what he had already said in the paper.
Meanwhile, the investigators split their role. Gary went through the details of the complainer’s allegation; thus, he focused on data collection, whether the data or conclusions in the paper had been reviewed adequately, and whether the data justified the conclusions. He was, in effect, trying to build the bricks to solidify and support the prima facie case. Lynn was his protector; when he seemed to be saying things that might expose the investigators or the department to legal attack, she would cut him off. Therefore, the investigators as a whole spent great effort in avoiding laying out what the details of the complaint were: these details might alert Monnett to areas of vulnerability and expose the department to counter-actions.
My Verdict – Not Just Not Proven, Innocent
Having heard the prima facie case, let’s analyze the data and the paper based on the information in the interview. What did it show?
First, let’s assume that the data is valid. Can we draw any statistical conclusions at all from it? Actually, yes. 4 and 3 may seem like small deviations from zero, but they follow a string of 20 zeros. The only plausible model I can see in the real world for a null hypothesis, that more than 3 swimming or dead polar bears is “the luck of the draw,” is a lambda probability function, in which the most probable occurrence is 0, the next most probable 1, and so on. Given 20 years of zeros, the only lambda function that would fit would have a probability of less than 1% of seeing 1 swimming/dead polar bear in year 21, and less than .1% of seeing 3 or more. It seems very clear to me that, if the data are valid, it is overwhelmingly likely that something new has happened to cause the swimming and dead polar bears.
All right, how about the validity of the data? Well, the three dead polar bears are clearly valid: there were attested by several other people besides Monnett. The four swimming polar bears are very likely to be valid, both because no one else in the helicopter, hearing the unusual sighting, found anything wrong, and because the death of the polar bears had to be caused by something, and that something was in the context of things highly likely to have been swimming. That leaves the question of whether Monnett omitted previous sightings of swimming and/or dead polar bears.
Actually, that one’s pretty easy to dispose of without recourse to statistics or the likelihood of personal actions. As Monnett makes clear, this was about the first year in which ice had clearly drawn off from the land. In most if not all previous years, Monnett or his predecessor could not possibly have seen swimming polar bears, since there was no water to swim in, just ice. And as for dead polar bears, again, it is overwhelmingly likely that 2 or 3 would have been noted in previous years in any case, since the researchers were trying to note any anomaly and the natives would care about this one, and no one has advanced a clear alternative cause (in previous years) for three to die.
Did Monnett overclaim based on this data? It appears that he said: (1) The numbers of swimming and dead polar bears was unusual, (2) It appears that the deaths were caused by the polar bears’ need to swim to the ice, (3) The 4 and 3 bears in 11% of his total area can be “extrapolated” to 36 and 27 across his entire area and a 25% survival rate (of bears that swam). We’ve already shown that statistically (whether or not he did the statistics), conclusion (1) is completely sound.
As for (2), this is not an overclaim. Something caused the deaths, and the most obvious explanation is the most obvious new factor that should cause an increase in deaths: withdrawal of ice from the land. That’s the equivalent of what Monnett said.
And then there’s (3). As I noted in my previous blog post, “extrapolation” is not an indication that something is highly likely. It’s an indication that if your sample from a population turns up this result, it’s the most likely frequency in the entire population, but not very likely in and of itself – and the frequency in the population is just as likely to be greater than that number as to be less than that number.
Were I a reviewer of such a paper, I would understand its import to be just as Monnett indicated in the interview: an establishment of a result that cannot be explained by the hypothesis of “just random chance”, a proposed explanation for the result that is a logically and physically appropriate model, and an invitation to other scientists to do work to replicate or propose alternative new explanations for the results. I would regard this as an entirely appropriate use of necessarily limited data.
My verdict: the prima facie case is entirely destroyed. Yes, in an ideal case the experiment could have been done better and better statistical tests might have been applied; but they were not needed. Monnett’s paper shows no scientific misconduct and is entirely appropriate to the data.
Why the Devil is in the Follow-on
Having, as I have said, given the “devil” (the complainer) his due, I maintain that it seems very likely that whoever has suspended Monnett has, in fact, been guilty of legal and scientific misconduct himself (or herself). Why? Because of what happened after the interview.
I have shown, I hope, that Monnett established, at the least, a strong prima facie case during the interview that he did not misrepresent the data. From then until now, there have been no public communications, nor communications with Monnett, about the case.
However, a week ago, Monnett was placed on “administrative leave”, and the stated cause was an investigation by the same investigator of allegations that Monnett had mishandled administration of a contract with U. of Alberta as subcontracted researchers into polar bear migration. Moreover, the same paper trail of permissions for his work is apparently available for the contract administration. That means that (1) investigators failed to investigate Monnett’s allegations of research distortion; (2) Monnett was suspended without even double-checking a “prima facie case”, if there was one – since investigators already knew that he typically had a paper trail; (3) Monnett was never cleared of the original charge by the investigators, although that was supposedly their only concern. What seems very likely is that this suspension is the result of proceedings covered by legal requirements that deliberately failed to follow them, both with regard to the paper and the administration. Whatever the relevant law, that suggests to me misconduct according to the law.
It also suggests scientific misconduct. The lack of contact with Monnett about his paper makes it clear that they failed to follow up with him his allegation that his research report had been distorted. Misrepresenting the data in a research report should also be considered scientific misconduct, whether the report is distributed beyond the department or not. And failure to correct a report that you should have known is distorted means that you share in the scientific misconduct, whether you are a scientist or not.
In the abstract, this type of misconduct seems isolated and unimportant. But it is also clear that these issues matter a great deal to everyone. Had the oil companies had a legitimate complaint, and if additional oil had mattered a great deal to the world’s economy and well-being, a Monnett prevention of drilling would have had major negative effects. If a valid Monnett scientific finding is not weighed and responded to appropriately and fossil fuels are indeed causing a global warming disaster, then this suspension and similar actions are equally if not more damaging to the world.
It appears from the only available evidence that those who suspended Monnett have thereby put themselves beyond excuse. I have tried to give the devil his due – is there any way I can avoid now naming him evil?
Monday, August 1, 2011
The Final Question -- and Cupcakes
Once upon a time, there was a math professor that gave an exam. When the students entered the testing room, they were given a mathematical framework and a list of problems to solve, each of which built on the solution to the previous one. The professor told them that they must solve all the problems in the time allotted, and that whoever did so, would pass; whoever did not, would fail.
The professor also told them that they should take each proposed problem solution to a machine in the corner. If the solution was correct, the machine would give them a cupcake.
Well, on the first problem some of the students who found the solution first got a bit rowdy and started waving their cupcakes in the faces of those who were slower and teasing them. The professor told them to settle down and stop it, or the professor would kick them out and they would fail.
At that that point, one of the students said to several others: “Look, this test is hard, and I think it’s so hard that none of us is going to pass unless the professor lets us. So the best way to do that is to show the professor that we appreciate all that he has done to teach us, and that we’re trying hard to please him, and we’re good people – not like those rowdy ones. So let’s all work together on each problem, and when we get our cupcakes we’ll keep giving some of them to the professor. And if we give him enough cupcakes, even though we will never answer all the questions, we’ll still pass.”
Many of the students, looking at the nasty problems coming up, were convinced that this was the only way they were going to pass, and so they agreed. But the student did not convince everybody. Some students rejected the idea that the test was impossible, and that passing the exam depended in any way on the professor.
Then there ensued a strange scene. On each problem, the students working together in a group solved the problem faster, and so should have given the machine the solution ahead of the few who were working solo. But many of them were concerned that they didn’t have enough cupcakes to give, and so they were over trying to convince the ones who were doing it alone that they should try to join the group. And sometimes they would succeed, as one of the loners felt that the test was becoming too hard; and sometimes they would lose a member, as some members of the group felt their point of view wasn’t being listened to. And then sometimes one of the group would get hungry and eat a cupcake instead of contributing it; so they had to kick that member out of the group until the member said he or she was sorry. And then they had to spend some time gathering the cupcakes and giving them to the professor. So as it worked out, the loners were solving the problems just about as fast as the group.
And then they got to the problem right before the last one. And it was a really nasty problem. Moreover, because they had been working in a group, the members of the group hadn’t been paying close attention to all the details of the previous solution, so none of them could put it all together and come up with the last step in this solution. And they were really worried that they hadn’t gotten enough cupcakes yet.
So finally one of the loners came up with the solution. And this caused an even bigger argument among the group, but finally one part of the group went over and looked at the loner’s answer, and got the solution – although they didn’t really understand that last step. And now there were two groups. The first group said, we haven’t got enough cupcakes, but if everyone comes back and helps us solve the next-to-last problem, and gives us their cupcakes, we’ll have enough. And the second group said, yes, we solved the next-to-last problem, but if we don’t get the cupcakes for the next-to-last problem from everybody, the first group and the loners, and give them to the professor, the professor will decide that we can’t work together and just aren’t nice enough. And the loners just kept being stubbornly convinced that it was all about solving the final problem.
And now there was frantic activity all over the room. The first group was still trying to solve the next-to-last problem, and badgering the second group and the loners for cupcakes. The second group had given up on the final problem (partly because they still thought it was impossible, and partly because they couldn’t solve it anyway, since they didn’t understand the solution to the next-to-last problem) and was badgering the first group and the loners for cupcakes. And the loners were trying to work on the final problem, and telling both groups that if they weren’t going to help solve the final problem, at least they should go away and stop bothering them. And as they did this, the clock kept ticking; and the end of the exam drew closer; and closer; and closer.
The moral of the story is – no, I’m not going to tell you the moral of the story. Consider this the final question of your exam: what is the moral of this story? If you get it right, you pass; if you get it wrong, you fail.
You can send me some cupcakes, if you think that will help.
The professor also told them that they should take each proposed problem solution to a machine in the corner. If the solution was correct, the machine would give them a cupcake.
Well, on the first problem some of the students who found the solution first got a bit rowdy and started waving their cupcakes in the faces of those who were slower and teasing them. The professor told them to settle down and stop it, or the professor would kick them out and they would fail.
At that that point, one of the students said to several others: “Look, this test is hard, and I think it’s so hard that none of us is going to pass unless the professor lets us. So the best way to do that is to show the professor that we appreciate all that he has done to teach us, and that we’re trying hard to please him, and we’re good people – not like those rowdy ones. So let’s all work together on each problem, and when we get our cupcakes we’ll keep giving some of them to the professor. And if we give him enough cupcakes, even though we will never answer all the questions, we’ll still pass.”
Many of the students, looking at the nasty problems coming up, were convinced that this was the only way they were going to pass, and so they agreed. But the student did not convince everybody. Some students rejected the idea that the test was impossible, and that passing the exam depended in any way on the professor.
Then there ensued a strange scene. On each problem, the students working together in a group solved the problem faster, and so should have given the machine the solution ahead of the few who were working solo. But many of them were concerned that they didn’t have enough cupcakes to give, and so they were over trying to convince the ones who were doing it alone that they should try to join the group. And sometimes they would succeed, as one of the loners felt that the test was becoming too hard; and sometimes they would lose a member, as some members of the group felt their point of view wasn’t being listened to. And then sometimes one of the group would get hungry and eat a cupcake instead of contributing it; so they had to kick that member out of the group until the member said he or she was sorry. And then they had to spend some time gathering the cupcakes and giving them to the professor. So as it worked out, the loners were solving the problems just about as fast as the group.
And then they got to the problem right before the last one. And it was a really nasty problem. Moreover, because they had been working in a group, the members of the group hadn’t been paying close attention to all the details of the previous solution, so none of them could put it all together and come up with the last step in this solution. And they were really worried that they hadn’t gotten enough cupcakes yet.
So finally one of the loners came up with the solution. And this caused an even bigger argument among the group, but finally one part of the group went over and looked at the loner’s answer, and got the solution – although they didn’t really understand that last step. And now there were two groups. The first group said, we haven’t got enough cupcakes, but if everyone comes back and helps us solve the next-to-last problem, and gives us their cupcakes, we’ll have enough. And the second group said, yes, we solved the next-to-last problem, but if we don’t get the cupcakes for the next-to-last problem from everybody, the first group and the loners, and give them to the professor, the professor will decide that we can’t work together and just aren’t nice enough. And the loners just kept being stubbornly convinced that it was all about solving the final problem.
And now there was frantic activity all over the room. The first group was still trying to solve the next-to-last problem, and badgering the second group and the loners for cupcakes. The second group had given up on the final problem (partly because they still thought it was impossible, and partly because they couldn’t solve it anyway, since they didn’t understand the solution to the next-to-last problem) and was badgering the first group and the loners for cupcakes. And the loners were trying to work on the final problem, and telling both groups that if they weren’t going to help solve the final problem, at least they should go away and stop bothering them. And as they did this, the clock kept ticking; and the end of the exam drew closer; and closer; and closer.
The moral of the story is – no, I’m not going to tell you the moral of the story. Consider this the final question of your exam: what is the moral of this story? If you get it right, you pass; if you get it wrong, you fail.
You can send me some cupcakes, if you think that will help.
Saturday, July 30, 2011
Trashing JRR Tolkien
I note in the news that here in the US, a Republican Senator has criticized Republican House members by calling them naïve “hobbits”, and a Republican Representative has riposted by saying he’d rather be a hobbit than a troll. This is just the latest instance of Tolkien’s Lord of the Rings being referenced by US politicians, especially Republicans.
Enough, enough of this. I may be alone in saying this, but I suspect that Tolkien would be sick at heart at what his message has become.
It is clear from his published Letters that Tolkien distrusted Americans – which, at that time (1950s and early 1960s), meant American men. He was, for example, very wary of having his books made into movies by Americans – his stated reason was that the books’ words were meant to be mythical, and didn’t sound right when spoken. As it turns out, the makers of the movie found that in many cases the words of the book sounded better than a modern rewrite, so in this case he was unnecessarily modest. Also, a Ballantine version, while popularizing the book in America, infringed his copyright, so that at first he was deprived of badly needed royalties. But there was more than that at work; what else caused him to distrust American men, is not clear.
However, I find increasing evidence since that time that he was right to do so. I first read the books in the early 1960s, so I have watched the phenomenon unfold since the beginning. The divergence began then, I think.
One of the notable features about The Lord of the Rings that I have noted is that it has appealed to an unusual number of women – not only men. This was unusual for its time: fantasy then in America was focused on Robert Howard’s over-muscled superheroes, with the occasional female jock who somehow lacked the basic protective armor except in strategic but minimal places – not the kind of thing that many women then or now find interesting, except as an obvious example of men’s incomprehensible, irritating taste. Moreover, as female norms have changed over here, Tolkien’s focus on men characters would normally seem dated.
The key, I think, lies in a surprising amount of “rethinking” underlying the text. Not that Tolkien was a proto-feminist. However, there is a very interesting story called “Aldarion and Eldaris” set in the same world that he wrote at about that time, that clearly enters into his thinking. In it, he sets out at great length and with some sympathy the objections of the partner whom the hero-king on his quest has left. Moreover, in her own way Eldaris sets up a counter-kingdom excluding men, showing clear leadership qualities. We see traces of this not in Arwen, as the movie would have it, but in Eowyn, who is able to make the switch from killing things to being an equal ruler who grows things.
This, in turn, allows women who read it for the first time to feel connected to the story, and then to appreciate Tolkien’s focus on relationships and beauty. It was noteworthy that when the movie came out, the one scholar who focused on Tolkien’s vision of a nature that was so alive it glowed with an inner light, was the one woman scholar quoted.
However, right from the start, many American men have read these same sections with impatience, even scorn at Tolkien’s style. Early on, Gary Gygax, creator of the Dungeons and Dragons game that was in many ways a straightforward elaboration of Tolkien’s cast of creatures, was quoted as saying that he could write better, because he would go straight to the action. Peter Jackson (yes, I know he’s from New Zealand, but he was clearly tailoring the movie to American tastes) did his very best to keep the sense of dread and action constant, and dropped the much of the first “book” in consequence. Moreover, his experience in horror movies tailored to American tastes, I think, led him to the understandable decision to give a “horror” or “action” tinge to all the scenes, especially the fight scenes. This movie, in turn, reinforced a new generation of American men who may never have read the book in the belief that it’s about plucky little and big superheroes who go out and save the world from evil by winning wars.
This is so far from Tolkien’s apparent thinking that it’s hard to know where to begin. His experience in the trenches of WW I left him with what today might be described as Post Traumatic Stress Disorder, and he was reluctant to have his son serve in WW II, because he felt that it was being used by the government to demonize the enemy. Lord of the Rings reflects that thinking: war is a horrible experience (“it was Sam’s first experience of a battle, and he didn’t like it much”), and a last resort, and those who wage it must be very clear-eyed about why it is necessary and how to minimize its effects. The Lord of Gondor, who believes the struggle is all about Gondor and only he can save it, is just as destructive as the counselor in Rohan who betrays the kingdom in order to “get” the woman he lusts after.
The second key point is that the story is about people changing – or not. Above all, it’s a story about Frodo changing. Perhaps the sentence that is most charged with meaning in the entire story is this: “I wanted to save the world, and it was saved – but not for me.” The entire first book is filled with meetings in which Frodo’s (and our) vision of the world is stretched, and stretched again, until he understands much of what is valuable in the world from its beginning until now. Then – and this point cannot be emphasized enough – he sets out to help save that world, and he fails. The result of that failure – the failure to resist the Ring enough to cast it in the Fire – and of his wounds, the equivalent of PTSD, is that he can’t stop missing the Ring, and he can’t enjoy the world that is saved, and especially because many of the things he wanted to save are vanishing anyway. I repeat: this is not a story about superheroes winning wars; it is about people enlarging their vision so they understand what’s at stake, instead of thinking it’s all about them and only they know the right answer. “Even Sauron was not evil in the beginning”; and that was the key mistake Sauron made.
The third key point is that in Tolkien’s world, not only the people are connected parts of an integral whole, but so is nature. The weather, the trees, all affect people’s moods and thinking, and are in turn fostered and destroyed by people. This is no abstract “the hero and his buddies go on a quest” folktale. It is a story in which your success is connected to the success of Brand and Dain halfway around the world, and your failure at Weathertop to the north is connected to the success of the Haradrim at Pelargir in the uttermost south. It is a story in which you would not succeed unless you helped save the trees of Fangorn Forest, and unless the wind from the South dissipated the storm of Mordor. If you lose that connection, the world is “blasted beyond repair, unless the healing hand of the Sea should cover the land in merciful oblivion.”
OK, so back to my quotes. It’s about plucky hobbits and evil trolls and fighting the enemy – yes, that sounds like an American man; and is the opposite of Tolkien. It’s time to cut down the evil government, we know how to do it, and we don’t believe the alarmists who say the cure is worse than the disease – yes, that sounds like an American man; and is the opposite of Tolkien. Let’s cut back on environmental regulation to save money, and drill more oil for our energy needs, as the rich suggest; well, let’s see what Tolkien says: “He started shipping food down south. People didn’t like it, what with winter coming on … But lately, he’s been pouring filth out of that mill for no reason, fouling the water …” When there’s a clear connection between present increases in fossil fuels and future injury of most people’s habitat, Tolkien would very clearly vote for better environmental protection and less drilling. Whether you think he’s right or not, he very likely would regard the US House efforts in this regard as clearly evil.
There’s a wonderful quote I have never been able to track down, that runs something like this: “When a man first commits murder, it is to be expected that he will then pass to assault and battery; he may indeed go on to wanton destruction of personal property, and in some cases from thence to defacement of public property; indeed, as hard as it is to believe, even violation of the Sabbath may not be beyond his reach.” I realize that in the scale of things, compared to risking the full faith and credit of the US government and threatening the global economy, trashing JRR Tolkien’s deep personal beliefs may not be of the same measure. But still, you American men, can you not at least spare those few of us who appreciate the full measure of his work the act of trampling on his grave? At long last, sirs, have you no shame? No shame at all?
Hello?
Enough, enough of this. I may be alone in saying this, but I suspect that Tolkien would be sick at heart at what his message has become.
It is clear from his published Letters that Tolkien distrusted Americans – which, at that time (1950s and early 1960s), meant American men. He was, for example, very wary of having his books made into movies by Americans – his stated reason was that the books’ words were meant to be mythical, and didn’t sound right when spoken. As it turns out, the makers of the movie found that in many cases the words of the book sounded better than a modern rewrite, so in this case he was unnecessarily modest. Also, a Ballantine version, while popularizing the book in America, infringed his copyright, so that at first he was deprived of badly needed royalties. But there was more than that at work; what else caused him to distrust American men, is not clear.
However, I find increasing evidence since that time that he was right to do so. I first read the books in the early 1960s, so I have watched the phenomenon unfold since the beginning. The divergence began then, I think.
One of the notable features about The Lord of the Rings that I have noted is that it has appealed to an unusual number of women – not only men. This was unusual for its time: fantasy then in America was focused on Robert Howard’s over-muscled superheroes, with the occasional female jock who somehow lacked the basic protective armor except in strategic but minimal places – not the kind of thing that many women then or now find interesting, except as an obvious example of men’s incomprehensible, irritating taste. Moreover, as female norms have changed over here, Tolkien’s focus on men characters would normally seem dated.
The key, I think, lies in a surprising amount of “rethinking” underlying the text. Not that Tolkien was a proto-feminist. However, there is a very interesting story called “Aldarion and Eldaris” set in the same world that he wrote at about that time, that clearly enters into his thinking. In it, he sets out at great length and with some sympathy the objections of the partner whom the hero-king on his quest has left. Moreover, in her own way Eldaris sets up a counter-kingdom excluding men, showing clear leadership qualities. We see traces of this not in Arwen, as the movie would have it, but in Eowyn, who is able to make the switch from killing things to being an equal ruler who grows things.
This, in turn, allows women who read it for the first time to feel connected to the story, and then to appreciate Tolkien’s focus on relationships and beauty. It was noteworthy that when the movie came out, the one scholar who focused on Tolkien’s vision of a nature that was so alive it glowed with an inner light, was the one woman scholar quoted.
However, right from the start, many American men have read these same sections with impatience, even scorn at Tolkien’s style. Early on, Gary Gygax, creator of the Dungeons and Dragons game that was in many ways a straightforward elaboration of Tolkien’s cast of creatures, was quoted as saying that he could write better, because he would go straight to the action. Peter Jackson (yes, I know he’s from New Zealand, but he was clearly tailoring the movie to American tastes) did his very best to keep the sense of dread and action constant, and dropped the much of the first “book” in consequence. Moreover, his experience in horror movies tailored to American tastes, I think, led him to the understandable decision to give a “horror” or “action” tinge to all the scenes, especially the fight scenes. This movie, in turn, reinforced a new generation of American men who may never have read the book in the belief that it’s about plucky little and big superheroes who go out and save the world from evil by winning wars.
This is so far from Tolkien’s apparent thinking that it’s hard to know where to begin. His experience in the trenches of WW I left him with what today might be described as Post Traumatic Stress Disorder, and he was reluctant to have his son serve in WW II, because he felt that it was being used by the government to demonize the enemy. Lord of the Rings reflects that thinking: war is a horrible experience (“it was Sam’s first experience of a battle, and he didn’t like it much”), and a last resort, and those who wage it must be very clear-eyed about why it is necessary and how to minimize its effects. The Lord of Gondor, who believes the struggle is all about Gondor and only he can save it, is just as destructive as the counselor in Rohan who betrays the kingdom in order to “get” the woman he lusts after.
The second key point is that the story is about people changing – or not. Above all, it’s a story about Frodo changing. Perhaps the sentence that is most charged with meaning in the entire story is this: “I wanted to save the world, and it was saved – but not for me.” The entire first book is filled with meetings in which Frodo’s (and our) vision of the world is stretched, and stretched again, until he understands much of what is valuable in the world from its beginning until now. Then – and this point cannot be emphasized enough – he sets out to help save that world, and he fails. The result of that failure – the failure to resist the Ring enough to cast it in the Fire – and of his wounds, the equivalent of PTSD, is that he can’t stop missing the Ring, and he can’t enjoy the world that is saved, and especially because many of the things he wanted to save are vanishing anyway. I repeat: this is not a story about superheroes winning wars; it is about people enlarging their vision so they understand what’s at stake, instead of thinking it’s all about them and only they know the right answer. “Even Sauron was not evil in the beginning”; and that was the key mistake Sauron made.
The third key point is that in Tolkien’s world, not only the people are connected parts of an integral whole, but so is nature. The weather, the trees, all affect people’s moods and thinking, and are in turn fostered and destroyed by people. This is no abstract “the hero and his buddies go on a quest” folktale. It is a story in which your success is connected to the success of Brand and Dain halfway around the world, and your failure at Weathertop to the north is connected to the success of the Haradrim at Pelargir in the uttermost south. It is a story in which you would not succeed unless you helped save the trees of Fangorn Forest, and unless the wind from the South dissipated the storm of Mordor. If you lose that connection, the world is “blasted beyond repair, unless the healing hand of the Sea should cover the land in merciful oblivion.”
OK, so back to my quotes. It’s about plucky hobbits and evil trolls and fighting the enemy – yes, that sounds like an American man; and is the opposite of Tolkien. It’s time to cut down the evil government, we know how to do it, and we don’t believe the alarmists who say the cure is worse than the disease – yes, that sounds like an American man; and is the opposite of Tolkien. Let’s cut back on environmental regulation to save money, and drill more oil for our energy needs, as the rich suggest; well, let’s see what Tolkien says: “He started shipping food down south. People didn’t like it, what with winter coming on … But lately, he’s been pouring filth out of that mill for no reason, fouling the water …” When there’s a clear connection between present increases in fossil fuels and future injury of most people’s habitat, Tolkien would very clearly vote for better environmental protection and less drilling. Whether you think he’s right or not, he very likely would regard the US House efforts in this regard as clearly evil.
There’s a wonderful quote I have never been able to track down, that runs something like this: “When a man first commits murder, it is to be expected that he will then pass to assault and battery; he may indeed go on to wanton destruction of personal property, and in some cases from thence to defacement of public property; indeed, as hard as it is to believe, even violation of the Sabbath may not be beyond his reach.” I realize that in the scale of things, compared to risking the full faith and credit of the US government and threatening the global economy, trashing JRR Tolkien’s deep personal beliefs may not be of the same measure. But still, you American men, can you not at least spare those few of us who appreciate the full measure of his work the act of trampling on his grave? At long last, sirs, have you no shame? No shame at all?
Hello?
Friday, July 29, 2011
OMG This Math Is Depressing
I just had the dubious pleasure of reading a transcript of an interview related to the “Arctic scientist muzzled for paper about polar bears.” It is incredibly depressing, because of the lack of knowledge of absolutely basic math it reveals – and not by the scientist, who does just fine.
Here is my summary of the interview. Let’s see if you can do better than the interviewers. I have even simplified the numbers ever so slightly.
Two Interviewers: Hi. We’re here from the investigative branch of the department to investigate allegations of scientific misconduct in a paper you wrote about a sudden apparent increase in deaths of polar bears.
Scientist: Do you have the scientific background to understand the paper?
Interviewers: No.
Scientist: OK, I’ll do my best.
We had been doing surveys of whales up here off the Alaska coast for 20 years, noting all other creatures out there as well. Each sweep covers (randomly) 10% of the total area we watch over. One year, for the first time, the ice moved well away from the land. On our next sweep that year, we saw four polar bears swimming. On the sweep after that, we saw three dead polar bears. Now, we couldn’t ever remember seeing such a thing, so I went and checked the notes and checked the memory and notes of the guy who had been doing this before me, since the beginning, and we’d never seen such a thing. So we wrote up a paper about it, passed it by everyone at the agency, had it anonymously peer reviewed by three people, and it was published by Polar Biology.
Interviewers: OK, so what you’re saying is, you saw 7 polar bears. How can you say there were 30 dead polar bears out there?
Scientist: What?! I didn’t say there were 7 dead polar bears – I said there were three.
Interviewers: No, in your paper, you say 4 polar bears on one sweep, and 3 in the next. Four plus three equals seven.
Scientist: But …
Interviewers: Also, why didn’t you say 7 dead polar bears instead of 30, since that’s all you saw?
Scientist: Look, in the first place I only swept 10% of the area, so I multiplied the number I saw by 10 …
Interviewers: Why would you multiply by 10?
Scientist: Excuse me, but have you ever taken any fifth grade math?
Interviewer: And even if it was OK to multiply by 10, that would mean you were claiming you saw 70 dead polar bears.
Scientist: No, I’m claiming I saw 3 dead polar bears, and that the best guess for the total in my area was 30 dead polar bears.
Interviewers: Ah, so let me read back to you what you have said. ‘I am claiming in my paper that it is likely that there are 30 dead polar bears out there.’ Is that correct?
Scientist: No. Have you ever taken any statistics? Even just a little? It is not “likely” that there are 30 dead polar bears out there. It’s just the most likely number, and there is an almost 50% chance of a number less than that, and an almost 50% chance of a number grea - …
Interviewers: Well, I think we have all we need.
Scientist: In that case, on the record, let me tell you what’s really going on. First, the purpose of the paper was not to establish a final determination of what was going on, but to say that something odd was going on. Second, my hypothesis – that there are increased polar bear deaths because of ice withdrawal from land – has been amply proven since by scientific research, anonymously peer reviewed. Third, this department has persistently attempted to prevent me and others from publishing any research that might support global warming, even though this research is a clear part of my job as a scientist and a clear part of my task in this department. Fourth, I am supposed to be checking out anything that might affect the natives here, not just whales. They want to know about my research, whales and otherwise. The only people who don’t are the oil companies whose permits might be affected and the political appointees in this department who seem to be doing their bidding. Instead of investigating me, why don’t you investigate them for “scientific misconduct”? I can certainly document attempts to distort my research, to the point where I took my name off the product …
Interviewers: Goodbye.
Here is my summary of the interview. Let’s see if you can do better than the interviewers. I have even simplified the numbers ever so slightly.
Two Interviewers: Hi. We’re here from the investigative branch of the department to investigate allegations of scientific misconduct in a paper you wrote about a sudden apparent increase in deaths of polar bears.
Scientist: Do you have the scientific background to understand the paper?
Interviewers: No.
Scientist: OK, I’ll do my best.
We had been doing surveys of whales up here off the Alaska coast for 20 years, noting all other creatures out there as well. Each sweep covers (randomly) 10% of the total area we watch over. One year, for the first time, the ice moved well away from the land. On our next sweep that year, we saw four polar bears swimming. On the sweep after that, we saw three dead polar bears. Now, we couldn’t ever remember seeing such a thing, so I went and checked the notes and checked the memory and notes of the guy who had been doing this before me, since the beginning, and we’d never seen such a thing. So we wrote up a paper about it, passed it by everyone at the agency, had it anonymously peer reviewed by three people, and it was published by Polar Biology.
Interviewers: OK, so what you’re saying is, you saw 7 polar bears. How can you say there were 30 dead polar bears out there?
Scientist: What?! I didn’t say there were 7 dead polar bears – I said there were three.
Interviewers: No, in your paper, you say 4 polar bears on one sweep, and 3 in the next. Four plus three equals seven.
Scientist: But …
Interviewers: Also, why didn’t you say 7 dead polar bears instead of 30, since that’s all you saw?
Scientist: Look, in the first place I only swept 10% of the area, so I multiplied the number I saw by 10 …
Interviewers: Why would you multiply by 10?
Scientist: Excuse me, but have you ever taken any fifth grade math?
Interviewer: And even if it was OK to multiply by 10, that would mean you were claiming you saw 70 dead polar bears.
Scientist: No, I’m claiming I saw 3 dead polar bears, and that the best guess for the total in my area was 30 dead polar bears.
Interviewers: Ah, so let me read back to you what you have said. ‘I am claiming in my paper that it is likely that there are 30 dead polar bears out there.’ Is that correct?
Scientist: No. Have you ever taken any statistics? Even just a little? It is not “likely” that there are 30 dead polar bears out there. It’s just the most likely number, and there is an almost 50% chance of a number less than that, and an almost 50% chance of a number grea - …
Interviewers: Well, I think we have all we need.
Scientist: In that case, on the record, let me tell you what’s really going on. First, the purpose of the paper was not to establish a final determination of what was going on, but to say that something odd was going on. Second, my hypothesis – that there are increased polar bear deaths because of ice withdrawal from land – has been amply proven since by scientific research, anonymously peer reviewed. Third, this department has persistently attempted to prevent me and others from publishing any research that might support global warming, even though this research is a clear part of my job as a scientist and a clear part of my task in this department. Fourth, I am supposed to be checking out anything that might affect the natives here, not just whales. They want to know about my research, whales and otherwise. The only people who don’t are the oil companies whose permits might be affected and the political appointees in this department who seem to be doing their bidding. Instead of investigating me, why don’t you investigate them for “scientific misconduct”? I can certainly document attempts to distort my research, to the point where I took my name off the product …
Interviewers: Goodbye.
Tuesday, July 26, 2011
Don't Cloud the Computing Carbon Emissions Reduction Issue
I recently read a post by Jon Koomey, Consulting Professor at Stanford, at www.climateprogress.org, called “4 reasons why cloud computing is efficient”. He argues (along with some other folks) that cloud computing – by which he apparently means almost entirely public clouds – is much more beneficial for reducing computing’s carbon emissions than the real-world alternatives. As a computer industry analyst greatly concerned by carbon emissions, I'd like to agree with Jon; I really would. However, I feel that his analysis omits several factors of great importance that lead to a different conclusion.
The study he cites compares the public cloud -- not a private or hybrid cloud -- to "the equivalent". It is clear from context that it is talking about a "scale-out" solution of hundreds and thousands of small servers, each with a few processors. This is, indeed, typical of most public clouds, and other studies have shown that in isolation, these servers do indeed have a utilization rate of perhaps 10-20%. However, the scale-up hundreds-of-processors servers that are a clear alternative, and which are typically not used in public clouds (but are often used in private clouds), have a far better record. The most recent mainframe implementations, which support up to a thousand "virtual machines", achieve utilization rates of better than 90% -- a three times better carbon efficiency than the public cloud, right up front.
The second factor Jon omits is the location of the public cloud. According to Carol Baroudi, author of "Green IT For Dummies", only one public cloud site that she studied is located in an area that has a strong record of electricity that is carbon-emission-light (Oregon). The others are in areas where the energy is "cheaper" because of fossil fuel use. That may change; but you don't move a public cloud data center easily, because the petabytes of data stored there to deliver high performance to nearby customers doesn't move easily, even over short distances. Corporate data centers are more movable, because the data storage sizes are smaller and they have extensive experience with "consolidation". While until recently most organizations were not conscious of the carbon-emission effects of their location, it appears that companies like IBM are indeed more conscious of this concern than most public cloud providers.
The third factor that Jon omits is what I call "flight to the dirty". High up-front costs of more efficient scale-up servers leads unconsciously to use of less energy-efficient scale-out servers. Controls over access to public and private clouds and data centers, and visibility of their costs, moves consumer and local computing onto PCs and smartphones. Apparent cheapness of labor and office space in developing nations leads companies to rapidly implement data centers and computing there using existing energy-inefficient and carbon-wasting electrical supplies. All of these "carbon inefficiencies" are not captured in typical analyses.
Personally, I come to three different conclusions:
1. The most carbon-efficient computing providers use scale-up computing and integrated energy management, and so far most if not all of those are private clouds.
2. The IT shops that are most effective at improving carbon efficiency in computing monitor energy efficiency and carbon emissions use not only inside but outside the data center, and those inevitably are not public clouds.
3. Public clouds, up to now, appear to be "throwing good money after bad" in investing in locations that will be slower to provide carbon-emission-light electricity -- so that public clouds may indeed slow the movement towards more carbon-efficient IT.
A better way of moving computing as a whole towards carbon-emission reductions is by embedding carbon monitoring and costing throughout the financials and computers of companies. Already, a few visionary companies are doing just that. Public cloud companies should get on this bandwagon, by making their share of carbon emissions transparent to these companies (and by doing such monitoring and costing themselves). This should lead both parties to the conclusion that they should either relocate their data centers or develop their own solar/wind energy sources, that they should move towards scale-up servers and integrated energy management, and that they should not move to less costly countries without achieving energy efficiency and carbon-emission reduction for their sites up front.
The study he cites compares the public cloud -- not a private or hybrid cloud -- to "the equivalent". It is clear from context that it is talking about a "scale-out" solution of hundreds and thousands of small servers, each with a few processors. This is, indeed, typical of most public clouds, and other studies have shown that in isolation, these servers do indeed have a utilization rate of perhaps 10-20%. However, the scale-up hundreds-of-processors servers that are a clear alternative, and which are typically not used in public clouds (but are often used in private clouds), have a far better record. The most recent mainframe implementations, which support up to a thousand "virtual machines", achieve utilization rates of better than 90% -- a three times better carbon efficiency than the public cloud, right up front.
The second factor Jon omits is the location of the public cloud. According to Carol Baroudi, author of "Green IT For Dummies", only one public cloud site that she studied is located in an area that has a strong record of electricity that is carbon-emission-light (Oregon). The others are in areas where the energy is "cheaper" because of fossil fuel use. That may change; but you don't move a public cloud data center easily, because the petabytes of data stored there to deliver high performance to nearby customers doesn't move easily, even over short distances. Corporate data centers are more movable, because the data storage sizes are smaller and they have extensive experience with "consolidation". While until recently most organizations were not conscious of the carbon-emission effects of their location, it appears that companies like IBM are indeed more conscious of this concern than most public cloud providers.
The third factor that Jon omits is what I call "flight to the dirty". High up-front costs of more efficient scale-up servers leads unconsciously to use of less energy-efficient scale-out servers. Controls over access to public and private clouds and data centers, and visibility of their costs, moves consumer and local computing onto PCs and smartphones. Apparent cheapness of labor and office space in developing nations leads companies to rapidly implement data centers and computing there using existing energy-inefficient and carbon-wasting electrical supplies. All of these "carbon inefficiencies" are not captured in typical analyses.
Personally, I come to three different conclusions:
1. The most carbon-efficient computing providers use scale-up computing and integrated energy management, and so far most if not all of those are private clouds.
2. The IT shops that are most effective at improving carbon efficiency in computing monitor energy efficiency and carbon emissions use not only inside but outside the data center, and those inevitably are not public clouds.
3. Public clouds, up to now, appear to be "throwing good money after bad" in investing in locations that will be slower to provide carbon-emission-light electricity -- so that public clouds may indeed slow the movement towards more carbon-efficient IT.
A better way of moving computing as a whole towards carbon-emission reductions is by embedding carbon monitoring and costing throughout the financials and computers of companies. Already, a few visionary companies are doing just that. Public cloud companies should get on this bandwagon, by making their share of carbon emissions transparent to these companies (and by doing such monitoring and costing themselves). This should lead both parties to the conclusion that they should either relocate their data centers or develop their own solar/wind energy sources, that they should move towards scale-up servers and integrated energy management, and that they should not move to less costly countries without achieving energy efficiency and carbon-emission reduction for their sites up front.
Subscribe to:
Posts (Atom)