Reading New Thoughts: O’Reilly’s What’s The Future, the Agile Entity, and Prediction

Disclaimer:  I am now retired, and am therefore no longer an expert on anything.  This blog post presents only my opinions, and anything in it should not be relied on.

Tim O’Reilly’s “WTF:  What’s the Future, and Why It’s Up to Us” is, alternately, an insightful memoir of many of the computer industry events that I lived with and to some extent participated in, a look at the oncoming technologies coming from the computer industry and its spinoffs, with particular over-emphasis on Uber and Lyft, and an attempt to draw general conclusions about how we all should anticipate what’s going to impact us in the future and how we should “ride the wave”.

First, I want to add a caveat that I think should become a Law:

The future happens faster than we think – and then it happens slower than we think.

By this I mean:  when new technological breakthroughs arrive, not all are obvious to a particular part of the economy that we attend to, even if (today) they are linked by software technology.  Then, even when they seem like the “new new thing” everywhere in our particular area, they typically take 10-30 years to spread to the world at large.  For example, smartphones and their apps (themselves over 10 years old) are by no means ubiquitous in the Third World, despite the hype.

I’d like to note here several instances of the future arriving “faster than we think”, some profiled in WTF.  Among the ones that I find amazing (and sometimes frightening):

·         We can now alter and replace 20-gene DNA and RNA segments, and hence genes in general, not only for the next generation but also in many cases over the course of a few months for our own.  The work to achieve that happened less than 10 years ago, practical implementation was achieved less than 5 years ago, and the Nobel Prize for that work (led by Jennifer Doudna and her team, described in her book) was awarded this month.

·         Pictures of anyone can be inserted seamlessly in a different scene, making it very hard to tell the truth of the news pictures that we see every day.

·         Understandable automated language translation (e.g., Google), automated voice recognition, and automated picture recognition have been achieved (although “good” speech recognition has still not been reached).

·         Semi-automated bots generate comments on articles and in blogs that are often indistinguishable from the ungrammatical and rambling comments of many humans.  Hence, the attempts at hacking political elections and the increasing difficulty of figuring out the truth of events in the public arena, “crowded out” as they now sometimes are by false “rumors.”

More subtly:

·         Uber and Lyft create “instant marketplaces” matching buyers and sellers of taxi services.

·         Supermarkets now dictate to growers production of specific items with detailed specification of quality and characteristics, based on narrow segments of the consumer market.

Now, let’s talk about what I think are the new thoughts generated by WTF.  In particular, I want to suggest that O’Reilly’s view of how to “predict” which oncoming technologies should be factored into one’s business, government, or personal strategy going forward, how to fit these into an overall picture, and how to use that picture to develop strategy, is really of most use in developing an “agile strategy” for an agile entity.

WTF Key Technologies and Strategies

Perhaps the best place to start is with WTF’s “Business Model of the Next Economy,” i.e. a model of the typical firm/organization in the future.  There are many subtleties in it, but it appears to break down into:

·         Central “Networked Marketplace Platforms”, i.e., distributed infrastructure software that provides the basis for one or many automated “marketplaces” in which buyers and sellers can interact.  In the supply chain, the firm would be the primary buyer; at the retail level, it would be the primary seller.

·         Feeding into these platforms, an approach that “replaces markets with information” – instead of hoarding information and using that hoarding to drive monopoly sales, the firm releases information openly and uses the commoditization of the product to drive dominance of new products.

·         Also feeding into the platforms, a new approach to user interfacing that seeks to create “magical experiences.”  This particularly enhances the firm’s and platform’s “reputation.”

·          Another “feeder” is “augmented” workers – workers enabled by rather than replaced by AI-type software.

·         A fourth “feeder” is “On-Demand” (applied flexibly, as needed) talent (workers given added value by their talents) and resources.  This includes an emphasis on actively helping workers to succeed, including over the long run.

·         A fifth feeder – somewhat complementary to the fourth – is “Alternatives to Full-Time Employment”, where the emphasis is on being flexible for the benefit of the worker, not the employer – the takeaway being that this actually benefits the employer more than WalMart-style “show up when we need you and never mind your personal life” approaches.  The key newness about this approach is that is “managed by algorithm” – the algorithm allows both the employer and employee to seek to manage their needs in a semi-automated fashion.

·         Returning to the business itself, the final feeder to the marketplace platform is “Services on Demand”, which offers to the consumer an interface that is providing an ongoing service rather than simply selling a product.  This is enhanced by “marketplace liquidity,” ways to make it easier for the consumer to buy the service.

At this point I revert to my caveat/Law in the beginning.  This “next economy” model is already operating in parts of the computer industry and related fields, e.g., Amazon, Google, Lyft – the future has already happened faster than we think.  At the same time, there will be a longer time than we think before it diffuses across the majority of organizations, if it does so at all.  Government and law are two obvious places considered in WTF where this model holds great potential, but will take a long, long time to effectively apply.

If the object of the game is to “ride the technology wave” by predicting which oncoming technologies should be factored into one’s business, then the technologies in this model are relatively safe bets.  They are already past the stage of “timing”, where the technology is attractive but it may not yet be time for the market to implement it.  As WTF points out, the trick is not to simply latch on to a strategy like this, but to constantly update the model and its details as new technologies arrive at their “timing” stage. 

Enter the agile strategy.

Prediction and the Agile Entity

The agile process is, on its face, reactive.  It does not attempt to get out ahead of the combined wisdom of developers/process-users and consumers/end-users.  Rather, it seeks to harvest that wisdom rapidly in order to get out in front of the market as a whole, and only for the purposes of each development/rollout process.

An agile strategy (which, up to this point, I haven’t examined closely) should be a different animal.  Precisely because any strategy bridges a firm/organization’s entire set of new-product-development efforts as well as aligning the rest of the organization with these, an agile strategy should be (a) long-term and (b) to a significant degree in advance of current markets.  

In the case of the strategy outlined in the previous section (i.e., implement the “new business economy model”), one very straightforward way of adding agility to the strategy would be to add agility to the software and analytics used to implement it.  One tried-and-true method for doing this is “refactoring” – adding a layer of abstraction to the software so that it is relatively easy to change.

Another method is simply to plan to revisit the strategy every 3-12 months.  The agile CEO I interviewed and reported on in a 5-years-old blog post did exactly that – a 5-year plan, revisited and informed with both his outside feedback and the information he gathered by attending scrum meetings.

WTF adds a third dimension:  attempt to discern upcoming technologies and approaches that are “important”, and then “time” the shift to a new strategy incorporating those technologies/approaches.  “Prediction,” in these terms, means anticipating which oncoming technologies/approaches are important and also the pace of their evolution into “timely” products and services.

I would argue, however, that this is precisely where an agile strategy adds value.  It does not assume that what seems important now stays important, or that an important technology/approach will arrive in the market in the next 5 years, but rather that whatever steps we take towards preparing the way for a new technology/approach must be flexible enough to switch to another technology/approach even midway in the process.  For example, we may move towards augmenting our workers with AI, but in such a way that we can instead fully automate one set of workers in order to augment a new type of worker whose responsibilities include that of the old.  We would be, in a sense, “refactoring” the worker-task definition.

So here’s my take from reading WTF:  It should be possible, using WTF’s method of anticipating change, to implement an agile strategy as described.  Moreover, an agile strategy should be clearly better than usual ones.  Usual strategies and agile processes do not anticipate the future; agile strategies such as this do.  WTF-type strategies anticipate the future but are not flexible enough to handle changes between identification of the future and the time for its implementation; an agile strategy should be able to do so.

The Transition From Agile Development to the Agile Organization Is Beginning to Happen

Disclaimer:  I am now retired, and am therefore no longer an expert on anything.  This blog post presents only my opinions, and anything in it should not be relied on.

Recently, new publications from CA Technologies via Techtarget arrived in my in-box.  The surveys mentioned in them confirmed to me that the agile organization or agile business is beginning to be a Real Thing, not just vendor hype. 

Five years ago, I wrote but did not publish a book on the evidence of agile development’s benefits and implications for creation of a truly agile business or other organization.  Now, it appears not only that the theoretical foundation has been laid for implementation at least of agile processes in every major department of the typical business, but also that a significant subset of businesses now think that they have at implemented agile according to that foundation across pretty much all of the enterprise – yes, apparently in a few cases including legal (what does legal-department agility mean?  I have no idea, yet).

So what are the details of this evidence?  And what benefits of an agile organization seem to be proving themselves?

The Solid Foundation of Agile-Development Benefits

It is now approaching a truism that agile development delivers benefits compared to traditional software-development approaches.  My own survey during my brief re-up at Aberdeen Group 9 years ago suggested improvements in the 20-30 % range for project cost and speed, product quality, and customer satisfaction (with the obvious implication that it also decreased product-development risk by around that amount, as Standish Group studies also showed).  One striking fact was that it was achieving comparable results when compared to traditional approaches focused on cost and/or quality.

One CA pub (“Discover the Benefits of Agile:  The Business Case for a New Way to Work) extends these findings to agile development plus agile project management.  It says that a “summary of research” finds that agile delivered 29 % improvements in cost, 50 % in quality, 97 % in “productivity” (something like my “speed”), 400 % (!) in customer satisfaction, 470 (!!) in ROI (a proxy for revenue and profit), compared to the “least effective” traditional approaches. 

While this may sound like cherry-picking, my research showed that the most effective traditional approaches were not that much better than the least effective ones.  So I view this CA-cited result as the “practice effect”:  experience with agile development has actually increased its advantage over all traditional approaches – in the case of customer satisfaction and profitability, by really large amounts.

The Theoretical Case For Business Agility

Note, as I did back when I did my survey, that agile development often delivers benefits that show up in the top and bottom line of the success-critical-software-developing business, even before the rest of the organization attempts to go agile.  So why would it be important to go the rest of the way and make most or all of the organization agile?

The CA pub “The State of Business Agility 2017” plus my own work suggest that potential benefits of “agile beyond software development” fall into three areas: 

1.      Hard-nosed top and bottom line benefits:  That is, effects on revenue, cost, and margin (profit).  For example, better “innovation management” via agile project management goes to the top line and eventually to the bottom line, and in some cases can be measured.

2.      “Fuzzy” corporate benefits, including competitive advantage, quality, customer satisfaction, speed to act and change strategies, and reduction in negative risks (e.g., project or IT-infrastructure failures) and “fire drills”.

3.      “Synergy” benefits stemming from most of the corporation being on the same “agile page” with coordinated and communicating agile processes, including better collaboration, better/faster employee strategy buy-in, better employee satisfaction through better corporate information, and better “alignment between strategy and execution.”

The results of the CA business-agility pub survey suggest that most respondents understand many but not all of these potential benefits before they take the first step towards the agile business.  I would guess, in particular, that they don’t realize the possible positive effects on combating “existential risks”, such as security breaches or physical IT-infrastructure destruction, as well as the effects on employee satisfaction and better strategy-execution alignment.

The Extent of Agile Organizations and Their Realized Benefits

Before I begin, I should note two caveats about the CA-reported results.  The first is that respondents are in effect self-selected to be farther along in agile-organization implementation and more positive about it.  These are, if you will, among the “best and the brightest.”  So actual agile-organization implementations “on the ground” are certainly far less than the survey suggests.

Second, CA’s definition of “agile” leaves out an important component.  CA’s project-management focus makes part of its definition of agility to be holding time and cost in a project constant while varying project scope.  What that really means is that CA de-emphasizes the ability to make major changes in project aims at any point in the project.  In the agile-organization survey, this means an entire lack of focus on the ability to incrementally (and bottom-up!) change a strategy rather than just roll out a whole new one every few years.  And yet, “more agile” strategy change is at the heart of agility’s meaning and is business agility’s largest long-term potential benefit.

How far are we towards the agile organization?  By some CA-cited estimates, 83% of all businesses have the first agile-development step at least in their plans, and a majority of IT projects are now “agile-centric.”  Bearing in mind caveat (1) above, I note that 22% of CA-survey respondents say they are just focused on extending “agile” to IT as a whole, 17% are also working on a plan for full business agility, 19% have gotten as far as organizational meetings to determine agility-implementation goals, and 39% are “well underway” with rollout.  Ignoring the question about “momentum” in partial departmental implementations for a moment, I also note that 47 % say IT is agile, 36% that Project Management is, and marketing, R&D, operations/manufacturing (are they counting lean methodologies as agile?), and sales (!) are a few percentage points lower. 

Getting back to partial implementation, service/support seems to be the “new frontier.”  Surprisingly, corporate communications/PR is among the laggards even in implementation, along with accounting/finance, HR, and legal.  What I find interesting about this list is that accounting and legal are even in the conversation, indicating that people really are planning for some degree of “agile” in them.  And, of course, the CEO’s agility isn’t even in the survey – as I said 9 years ago, the CEO is likely to be the last person in the organization to “go agile.”  Long discussion, not relevant here.

How about benefits?  In the hard-nosed category, agile organizations increase revenue 37% faster and deliver 30% greater profit (an outside survey).  For the rest of benefits, there is far less concrete evidence – the CA business-agility survey apparently did not ask what business benefits respondents had already seen from their efforts.  What we can deduce is that most of the 39% of respondents who said they were “well underway” believe that they are already achieving the benefits they already understand, including most of the “fuzzy” and “synergy” benefits cited above.

Implications:  The Cat Is In The Details

At this point, I would ordinarily say to you the reader that you should move towards implementing an agile business/organization, bearing in mind that “the devil is in the details.”  Specifically, the CA surveys note that the complexity of the business and cultural/political opposition are key (and the usual) problems in implementation.  And, indeed, this would be a useful thing to know.

However, I also want to emphasize that there is a “cat” in the details of implementation:  a kind of Schrodinger’s Cat.  In quantum physics, as I understand it, different states of basic particles (e.g., exists, doesn’t exist) are entangled until we disentangle them (e.g., by “opening the box” and measuring them).  Schrodinger imagined entangled states of “cat inside the box/no cat inside the box”, so that we wouldn’t know whether Schrodinger’s Cat existed until we opened the box.  In the same way, we not only don’t know what and how much in the way of benefits we get until we examine the implementation details, we won’t know just how really agile we are until we “open the box.”

Why does that matter?  Because, as I have noted above, the really big long-term potential benefit of business agility, is, well, agility:  The ability to turn strategically, instantly, on a dime and ensure the business’ long-term existence, as well as comparative success, much more effectively.  Just because some departments seem to be delivering greater benefits right now, that doesn’t mean you have built the basic infrastructure to turn on a dime.

And so, the cat is in the details.  Please open the box by testing whether your implementation details allow you to change overall business strategies fast, and then, if there is no cat there, what should you do?

Why, change your agile-implementation strategy, of course.  Preferably fast.

Tiempo Development and the Agile Business

I have heard a future of business organization, and it sounds as if it works.

After ten years of hearing companies only begin to appreciate agile development and then agile marketing, I certainly was not expecting to hear a company organize its management, its strategy, and its planning around a company-wide agile process. But when Tiempo Development, billing itself as yet another agile development outsourcer, gave me a telebriefing last week, I asked the obligatory question and was floored to hear that they had, indeed, done so. 

The Beginning Of The Epic

I am sure that, like many another agile-development startup, Tiempo asked themselves why they couldn’t apply agile principles to management – and yet, others I have talked to stopped there.  Tiempo, however, when faced with the need to begin doing better planning and budgeting as the business scales, seems to have had the guts to take a leap of faith and try to do it via a variant of Scrum, complete with Kanban-type lean scheduling, plus the “epics” and “stories” beloved of agile marketing.  In other words, there’s a five-year plan and budget (and, of course, in an agile organization that’s not set in stone) devolving into yearly, quarterly, monthly, and weekly epics and stories. 

Revisions to the plans are quarterly, flowing forward into a new rolling five-year plan. As in Scrum, daily task sessions and weekly reviews communicate widely, and change flows from below as well as above.  “Customer” feedback comes in laterally and frequently from project customers, as well as from above and below on the plans and their implementation.  To put it another way, the concentrated but broad communication of Scrum ensures the maximum of customer-driven evolution, while the five-year horizon ensures that long-term strategic considerations engage in a delicate dance with short-term customer –driven product specs.

We are long past the time when companies would ask whether agile development scales.  Still, the question should be asked:  how well does a Tiempo-style “Scrumban” management approach scale?  The apparent answer after about 1 1/2 years of practice is:  Quite well, so far.  The reason, I would suggest, lies in the initial establishment of five-year company goals in terms of “rocks”.  That is, the implementing CEO picked about five goals that formed what he saw as the fundamental things that the company needed to achieve over the next five years – the “rocks” in both the foundational and the risk sense. 

Again, as noted, the “rocks” are not made of concrete; they evolve over time, fairly frequently, as the company evolves.  Think of them as “virtual rocks”.  They therefore combine long-term perspective; frequent review based on customer and employee feedback; and a broad perspective, not only for the CEO but also in terms of employee understanding of (and buy-in to) strategy. Plug in more business, plug in new areas, plug in new employees.

To Understand Agile Management’s Effectiveness, Redefine Success

What does Tiempo report as evidence of success in agile management?  Interestingly, they start first with better communications between managers and “line” developers, followed by quick surfacing of bad news for rapid handling, and then “empowering” the employees.  It was only when I probed deeper that the follow-on effects emerged:  more rapid development without other sacrifices, leading to better profitability; greater customer satisfaction, leading to greater follow-on business and willingness of large enterprise customers to trust Tiempo in new areas; and a strong growth path that has the CEO more concerned with avoiding lack of focus via too many new areas than with short-term risks.

It was not a given that this would be so, but in this respect agile management is mirroring agile development.  My survey four years ago found a persistent pattern of comparative focus on change rather than cost, speed, profit, quality, or (downside) risk leading to greater cost, speed, profit, quality, and customer satisfaction as well as “upside” risk, while “downside” risk decreases – all compared to attitudes and processes that do focus on these things.  That Tiempo is focusing on the employee and customer satisfaction parts of the effect is what we should expect if Tiempo is “talking the agile talk and walking the agile walk.” And so, if agile management is indeed like agile development, out of this process we should indeed expect better top-line and bottom-line results that are likely to endure.

The problem is, of course, metrics that will establish these things – because the merits of agile management are not as easy to prove as those of agile development.   A fluke, a Board of Directors and the stock market will call the first agile management bottom-line results, caused by unique circumstances and remnants of past practices.  All these daily and weekly meetings, are they really doing any more than past business strategy fads?  I believe strongly that the answer is resoundingly yes. But I can’t do a survey like the agile development one to prove the point, because there are no obvious speed, cost, and risk-reduction metrics for the CEO’s actions, just customer and employee satisfaction, which have proven misleading in the past.

And so, we are back to CEO “culture”, just as we had to deal with IT and business management “cultures” that were highly resistant to the touchy-feely feel-good aspects of agile development.  Remember, CEOs are the byproduct of a selection process that emphasizes enormous, visible hard work and schmoozing as well as hard-nosed attention to the bottom line, and rewards these by surrounding the CEO with those who understand that their future depends on giving the CEO positive reinforcement, as well as with compensation that most fair-minded outside observers concede is well beyond the demonstrated relative effectiveness of the CEO.  In other words, power and money.  I once took a survey at B school that asked whether I wanted to make money, have power, do good for people, or accomplish something; it turned out that those who answered power or money got much more of both.

I mention this because (and this is only my impression) what really got the CEO excited, talking to me, was his own personal satisfaction – that is, the “accomplish something” part of the B school survey.  And yet, there was no reason he wouldn’t wind up with just as much power and money as my classmates who answered “power” or “money”.  In other words, in the agile world, you can make as much (or more, since the organization should be more successful) money and have as much power, but work less (remember, you’re not running around trying to single-handedly move an ocean liner of an organization or sell the world on your product) and get more real satisfaction out of it, because you can point to shared concrete projects finished successfully, not just praise from those who wish to share in your money.  And so, if the CEO and upper management can make the cultural shift, they may well find that they are better off in every way, including personal satisfaction – another agile paradox.

In fact, it turns out that the creators of agile processes may have been craftier than they knew in calling projects and project groupings “stories” and “epics”.  It speaks to something Tolkienesque in all of us.  A study recently reported in MIT Technology Review suggests that we don’t have fixed memories like snapshots of events, we have stories of the past, and each time we revisit them, we can change them, e.g., to fit new understandings of the past, a new narrative arc.  When I do TCO studies, it’s not all about yes/no, 1-5, and numbers; I ask the interviewee to tell his or her story, the story of him or her in the project.  Set the stage for me, what was your environment; tell me how the project unfolded; tell me what you found were the results; and only now tell me how this affected the TCO that you find justifies the project.  I have found the insights from such an approach far richer and better founded than those from standard surveys, even when the data is sparser.

In an agile business such as Tiempo’s, it sounds as if everyone becomes a hero or heroine in a shared epic that never ends, always new and unfolding.  Oh, and by the way, it should perform better than an old-style business.

Caveats and Not So Bottom Line

I have always been a bit dissatisfied with the analyst’s usual disclaimer:  Your mileage may vary.  Yes, and your hair may grow longer or get cut shorter.  In this case, my real disclaimer is not that agile management may or may not apply to your organization, but rather that I don’t have the figures or real-world use cases to support my belief that Tiempo’s agile-management experience applies well to most medium-sized and large-sized enterprises and organizations. 

Will Tiempo continue scaling?  Does the agile management model appropriate to a small-to-medium-sized software-development outsourcer apply to most other types of business – not the traditional “hardware” manufacturing firm that forms the “unconscious” of all businesses today, and which as far as I can tell is a small fraction of today’s businesses, but the rest of them?  What I see and hear says that it should; but until it happens, there are afaik no decent metrics to show it is working.  Your mileage will probably not vary, but you won’t know the mileage for sure until the end of your first long trip.

No now we can proceed to my not so bottom line – because, remember, agile is about focusing on change, and the bottom line will follow.  Agile development, by now, has thousands of successful use cases. New product development is getting there, and there is now a “hardware” (silicon foundry) use case, as cited in a blog post a while ago.  Agile marketing is approaching its thousandth use case – I note that in a little over a year, the Bay Area agile marketing group has gone from zero members to well over a thousand; and IT is doing its best to keep up.  And now, we have an agile management – and hence agile business – use case. It’s time to start shifting the burden of proof.  Instead of wondering why you should be an agile developer/marketer/business, you should be demanding good reasons why you shouldn’t be.

I have heard a future of business organization, and it sounds as if it works.   Please stop asking about the mileage and start installing the tires.