The Political Economics of the Singularity

"The singularity", in some sense at least, is already happening, and has been for the past couple of years.

Look, if you will, at the disconnect between the technology sector (booming; "talent" (labor) in exceedingly short supply, salaries rocketing) and the rest of the economy (tanking, many people out of work, surplus of labor, salaries plummeting).

Technology brings many new and unfamiliar nonlinearities into the economy. Access to the mass market does not (in all circumstances) require mass employment - For example, I am one individual, working from my home office, and I can easily make improvements to and deploy a product used (indirectly) by millions of people with a few keystrokes -- No expensive bureaucracy, no factory, no paperwork, and no infrastructure beyond a handful of laptops, an internet connection, and a few dozen rented Amazon EC2 machines.

The funny thing is this: We all expected the singularity to swing the balance of power firmly towards the side of capital, away from labor. After all, surely capital would simply buy robots instead of employing labor? However, it is not quite panning out as we expected. Developing a new technology product is now ridiculously cheap -- capital costs have all but disappeared. Technology startups now look to investors not so much for capital, but for advice, access to customers, and reputation. Just as the need for labor has (unevenly) diminished, so the need for capital has also (unevenly) diminished.

It is becoming clear that (in some circumstances at least) the old balance of power between labor and capital has been swept aside, with both, in some sense, having been made irrelevant.

What replaces it? The answer to that question is not easy to discern. One thing is clear though: this new world is far more complex, richly textured, baroque and interesting, and the old political battle-lines will need to be redrawn with greater subtlety and nuance than they ever were before.

Stupid is better than Smart (A call for humility)

Software development is a funny thing. It is full of nonlinearities and counterintuitive results.

Here is one of them: It is better to think of oneself as stupid (and be right) than it is to think of oneself as smart (and be wrong).

This sounds nonsensical, doesn't it? Surely it is better to be smart than it is to be stupid. Particularly since we spend so much of our time trying to demonstrate to other people just how smart we really are?

Well, if we were to start thinking of ourselves as smart people, relative to the rest of the population, then it is all too easy to start thinking of ourselves as being smart relative to the problems that we are trying to solve.

This is a problem, because *everybody* is pretty stupid in the grand scheme of things, and hubris is dangerous, particularly in the presence of complexity.

Complexity makes systems difficult to understand and manage, and difficult to fix when they go wrong. It is also difficult to gauge complexity, and humans have a consistent tendency to underestimate the complexity of unexplored functionality.

Let us consider a (mis) quote that illustrates the point:

"Debugging a system is harder than designing it in the first place, so if you are as clever as you possibly can be when you are designing the system, you are (by definition) too stupid to debug it."

The well known Dunning-Kruger effect applies here too: Because we tend to think that we are smarter than we really are, we tend to design systems that are too complex for us to debug and maintain.

In cases such as this, it is helpful to take a broader view. We may call somebody "Smart", but this term is defined relative to other humans, not relative to the problems that we need to solve.

We are frequently faced with problems that would be considered difficult by the very best of us. It is not a sign of weakness to acknowledge that; to treat the problems that we are trying to solve with the deference and respect that they deserve.

BAM! The Combinatorial Explosion and Planning

Solving problems requires understanding.

Understanding is built, in part, by measuring features.

One or two features might be sufficient to describe something simple, but describing something complex often takes many more.

With one or two features, a handful of measurements may suffice to capture the behavior of whatever-it-is that we are trying to understand. As the number of features that we need to measure grows, the number of measurements that we need to properly and comprehensively capture that behavior grows faster than fast: It grows stupendously, ridiculously, unreasonably, explosively fast. It grows so fast that it is akin to ... BAM! ... hitting a brick wall.

This is the Combinatorial Explosion:- Know it, and Fear it, because it is Important.

In other words, it stops being possible in any reasonable, practicable sense of the word, to understand any system or problem, the description of which involves more than a small handful of features.

This has some terribly important consequences. Consequences that, as our world becomes more complex, and contains more and more complex, interconnected systems, we particularly need to understand and internalize, for we fail to do so at our peril.

The past becomes less useful as a guide for predicting the future; Our intuition becomes less effective; Unintended consequences and unusual, exceptional events become more prevalent; our ability to predict what will happen next is weakened to the point where it disappears, and the utility of planning (in the way that we commonly understand it) diminishes dramatically.

At no point in history did plans ever survive first contact with the enemy, but as (consciously or unconsciously) we become more liberal with complexity, these traits and characteristics will become more and more prevalent.

We need to adapt, and learn to deal with them.

PS:

This line of thinking is particularly interesting when applied to organizations:
http://daltoncaldwell.com/thoughts-on-organizational-complexity

Which, of course, then go on to produce more complex products:
http://en.wikipedia.org/wiki/Conway's_law

Development Automation - The Revolution

In response to:

http://www.businessweek.com/articles/2012-12-17/google-s-gmail-outage-is-a-sign-of-things-to-come

It is really great to see an article on continuous deployment in the mainstream media, as it is an issue about which I am extremely enthusiastic.

When we delay the release of a piece of our work, the psychological importance that we place on the quality of that work increases, so we spend more time manually finessing and polishing the work (often resulting in more delay, and possibly also raising the psychological barriers to release still higher).

This is all well and good when all testing and quality control must, by necessity, be manual. However, this is less and less common today, as automated testing and deployment practices become more common (In the form of Test Driven Development & Continuous Integration).

This (extremely) high level of automation and the work practices that go with it, together offer a revolutionary step-change in the way that we engineer complex systems:- a revolution that companies like Google and Netflix have embraced; a revolution that the rest of us ignore at our peril.

Instead of simply engineering products, we must engineer organizations and systems that produce world-beating products time and time again.

That is the revolution of DevOps, of Test Engineering, and of Development Automation.

And that is the only sane way to go about making complex systems today.

Collaboration & Reuse

A spectrum of collaborative techniques exists between the two extremes presented below.

Loose Collaboration leads to Coarse Grained Reuse

* A collection of independent professionals rather than a tightly integrated team.

* Each developer brings different skills to the table.

* Each developer has a high level of expertise in his or her own domain.

* Each developer works independently and largely alone.

* Tasks are defined in broad terms: Developers have high levels of freedom and autonomy.

* Development Environments are varied, idiosyncratic and highly customized.

* Git / Hg - separate repositories for separate components.

* Few inter-dependencies.

* Quantum of re-use is large: A critical mass has to be achieved before something is placed into it's own repository and shared.

Tight Collaboration leads to Fine Grained Reuse

* A tightly integrated team rather than a collection of independent professionals.

* Work carried out by both junior and senior developers.

* Developers work in close collaboration with one another.

* Tasks are small and fine-grained.

* Working Practices & Development environments harmonised & centrally controlled.

* Subversion / CVS - single monolithic repository

* Many inter-dependencies.

* Techniques like continuous integration and test-driven development keep everything on track.

* Quantum of re-use is small: individual scripts and configuration settings are easily shared.

Which approach creates a better work environment? Well, it totally depends upon your personality. Personally, I enjoy working with other people, so I tend to find the tightly integrated team approach more fun, but I am happy to acknowledge that this is not everybody's cup of tea. I must also admit that my preferences continue to change as I grow and develop.

Which approach creates better software? I have no idea. System Architecture will definitely be impacted by the approach to collaboration. In some manner. So, I imagine that the development of some particular classes of software are better served by the first approach, others by the second approach. I cannot, for the moment, imagine what the mapping between class of software and most-appropriate approach would look like.

Guesses in comments, please.


PS:

There exist other spectra on which Software Development activities may be placed. One notable one is the Predictive/Reactive spectrum, which distinguishes between predictive, variance-minimizing techniques as espoused by SEI/CMMI (Chicago School), and reactive, "Agile" techniques. In contrast with the loose/tight collaboration spectrum, it is easier to judge the appropriateness of predictive vs reactive, as level of anticipated technical risk is pretty a good indicator of the degree to which a reactive approach is appropriate.

The Utility of Version Control Systems

In response to a post about the utility of version control systems.

The "backups" argument for version control is easily understood, but it is ultimately not that compelling. The real benefit that you get from version control is a dramatic cultural shift and a new way of working with collaborators.

First of all, a version control system provides a structured framework and vocabulary for collaboration and feedback. This means that it is possible to talk about "changes" and "branches'. This is not dissimilar from "track changes" functionality in MS Word. Secondly, it facilitates the implementation of automated services such as "continuous integration" and "continuous testing".

These two pieces of functionality enable a very significant cultural shift. In most domains, professionals are used to working independently for an extended period of time, only presenting results when they are finished, polished and (supposedly) error-free. This way of working leads to an intolerant attitude towards (rare) errors: The ensuing lack of humility creates and enforces a risk-averse culture. It quickly and inevitably becomes more important to be seen to be perfect than it is to innovate and solve problems.

The working culture and practices that version control systems and continuous integration support and encourage could not be more different. By publishing incomplete drafts and unfinished work, your mistakes and typos become part of the "public" record, along with a record of the way that you approach problems and do your work. This requires either a thick skin, or (more realistically) the realization that nobody really cares about your mistakes.

The benefit of this is strongest where the work being carried out by multiple professionals is highly interdependent. By sharing work earlier in the cycle, as well as more frequently, conflicts and misunderstandings are resolved up much sooner. In fact, it is highly likely that professionals that need to collaborate closely will already be using a form of version control (such as track changes) in their work already. Using tools like Subversion or Mercurial simply provides additional tools with finer grained control and a different approach to sharing and communication (changes pulled on demand from the repository rather than pushed out via email).

So, the prime benefit of version control is in the fact that it provides a structured framework for sharing work, as well as discussing and coordinating the collaborative development of documents that refer to, or otherwise depend upon, one another. Another, equally important benefit is that it encourages transparency and humility through the practice of regularly sharing unfinished work. Finally, as stated in the article, it acts as a record for understanding how work was carried out, and for recovering from past mistakes.

Bridging the Gap

In response to an article talking about using narrative in software specifications:

There certainly exists a reality gap between "the business" and the people who end up implementing the system, but I suspect that it is as much a matter of practicalities as it is a matter of cultural differences. The developer, by necessity, thinks in terms of the details, because sooner or later, he must confront them and overcome them. The customer typically thinks at a higher level of abstraction. He never needs to go into the details, so why would he waste the time? The act of bridging the gap is akin to weaving. You must go from a high level of abstraction, diving down to the details and then back out again, several times in succession before you can build the shared understanding, terminology & conceptual framework that is required for the business side and the technology side to fuse into a single, effective whole. This process is generally characterized by feelings of confusion and disorientation (on both sides) and sometimes accompanied by arguments, generally stemming from each side misunderstanding or failing to grasp the other's terminology and conceptual framework. All of this is natural, and part of the learning process. It is also exceedingly expensive and time consuming; a fact often under-appreciated at the start of the process.

You are probably familiar with the famous aphorism: "Computer science is no more about computers than astronomy is about telescopes". Well, if I may beg leave to corrupt and abuse the saying: "Software development is no more about computers than accountancy is about spreadsheets". Software developers may spend a lot of time using computers, to be sure, but then again, so do accountants. Software development is about understanding a problem, communicating it, and specifying it with such formality and precision that even a creature as simple and literal-minded as the typical computer can understand it. There may be a lot of technical jargon, to be sure, but ever since people ditched assembly language for C, Java, Python and the like, the jargon has had more to do with the precise communication and specification of philosophical concepts than anything to do with shuffling electrons across the surface of semiconductors. Software development is a misnomer. It is the search for the devil that lies in the details, and the communication of the elevated understanding that comes from that search. The software itself, the implementation, it is both the microscope that is constructed to aid in the search; a vehicle for sharing that understanding, and a tool that has utility in the problem domain.

It is worth reading the full article, which more fully lays out a very interesting vision of collaborative development not totally unlike that supported and encouraged by Agile tools like Scrum, albeit with the addition of some novel, interesting and potentially useful narrative concepts to structure the interaction.

Productivity Models.

The economic realities of software development are rather different from other forms of labor.

Firstly, productivity rates vary enormously, both for teams and for individuals. For example, one individual can be many orders of magnitude more productive than another. Equally, that same individual may experience variability in productivity of many orders of magnitude from one project to the next. People who talk about the 10Xer effect seem to (incorrectly) assume that people are constant - once a 10Xer, always a 10Xer. That is patently not the case. Personal productivity has a lot to do with emotional state, motivation, interpersonal relationships, office politics, the built environment, diet, length of commute and serendipitous fit-to-role, not to mention the ready availability of appropriate software for re-purposing and re-use. Great developers can do a lot to maximise the probability of exceptional performance, but they cannot guarantee it.

These apparent nonlinearities; these step-changes in productivity, make software development extraordinarily difficult to manage. The many factors involved and their extreme effects challenge our conventional approaches to modeling, prediction and control.

So, we have a challenge. I do love challenges.

Let us take our initial cue from the Machine Learning literature. When faced with a high-dimensional feature (or parameter) space; sparsely populated with data, such as we have here, we cannot fall back on pure data mining, nor can we deploy fancy or highly parameterized models. We must approach the problem with a strong, simple and well-motivated model with only a few free parameters to tune to the data. To make a Bayesian analogy: We require very strong priors.

So, we need to go back to first principles and use our knowledge of what the role entails to reason about the factors that affect productivity. We can then use that to guide our reasoning, and ultimately our model selection & control system (productivity optimization) architecture.

So, let us sketch out what we know about software engineering, as well as what we think will make it better, in the hopes that a model, or set of models will crystallize out of our thinking.

For a start, software development is a knowledge based role. If you already know how to solve a problem, then you will be infinitely faster than somebody who has to figure it out. The time taken to actually type out the logic in a source document is, by comparison with the time it typically takes to determine what to type and where to type it, fairly small. In some extreme cases, it can take millions of dollars worth of studies and simulations to implement a two or three line bug-fix to an algorithm. If the engineer implementing the system in the first place had already possessed the requisite knowledge, or even if he had instinctively architected around the potential problem area, the expenditure would not have been required.

20-20 hindsight is golden.

Similarly, knowledge of a problem doman and software reuse often go hand-in-hand. If you know that an existing library, API or piece of software solves your problem, then it is invariably cheaper to reuse the existing software than to develop your own software afresh. It is obvious that this reuse not only reduces the cost incurred by the new use-case, it also spreads/amortizes the original development cost. What is perhaps less obvious, is the degree to which an in-depth knowledge of the capabilities of a library or piece of software is required for the effective (re)use of that software. The capability that is represented by a piece of software is not only embedded in the source documents and the user manual, it is also embedded in the knowledge, skills and expertise of the practitioners who are familiar with that software. This is an uncomfortable and under-appreciated truth for those managers who would treat software developers as replaceable "jelly-bean" components in an organizational machine.

Both of these factors seem to indicate that good old specialization and division-of-labor are potentially highly significant factors in our model. Indeed, it appears on the face of it that these factors have the potential to be far more significant, in terms of effect on (software engineering) productivity, than Adam Smith could ever have imagined.

I do have to admit that the potential efficiency gain from specialization is likely to be confounded to a greater or lesser degree by the severe communication challenges that need to be overcome, but the potential is real, and, I believe, can be achieved with the appropriate technological assistance.

So, how do you make sure that the developer with the right knowledge is in the right place at the right time? How do you know which piece of reusable software is appropriate? Given that it is in the nature of software development to be dealing with unknown unknowns on a regular basis, I am pretty confident that there are no definitive and accurate answers to this question.

However, perhaps if we loosen our criteria we can still provide value. Maybe we only need to increase the probability that a developer with approximately the right knowledge is in approximately the right place at approximately the right time? (Taking a cue from Probably Approximately Correct Learning). On the other hand, perhaps we need the ability to quickly find the right developer with the approximately right knowledge at approximately the right time? Or maybe we need to give developers the ability to find the right problem (or opportunity) at the right time?

What does this mean in the real world?

Well, there are several approaches that we can take in different circumstances. If the pool of developers that you can call upon is constrained, then all we can do is ensure that the developers within that pool are as knowledgeable, flexible and multi-skilled as possible, are as aware of the business environment as possible, and have the freedom and flexibility that they need to innovate and provide value to the business as opportunities arise. Here particularly, psychological factors, such as motivation, leadership, vision, and apparent freedom-of-action are important. (Which is not to say that that they are unimportant elsewhere)

If, on the other hand, you have a large pool of developers available, (or are willing to outsource) then leveraging specialization and division-of-labour has the potential to bring disproportionate rewards. The problem then becomes an organizational/technical one:

How to find the right developer or library when the project needs them?

Or, flipping the situation on it's head, we can treat each developer to be an entrepreneur:

Given the current state of the business or organization, how can developers to use their skills, expertise and knowledge of existing libraries to exploit opportunities and create value for the business?

Looking initially at the first approach, it is a real pity that all of the software & skills inventory and reporting mechanisms that I have ever experienced have fallen pitifully short of the ease-of-use and power that is required. These systems need to interrogate the version control system to build the software components inventory, and parse it's log-file to build the skills inventory. These actions need to take place automatically and without human intervention. The reports that they produce need to be elegant and visually compelling. Innovation is desperately required in this area. I imagine that there are some startups somewhere that may be doing some work along these lines with GitHub:- if anybody knows of any specific examples, please mention them in the comments.

Another area where innovation is desperately needed is in the way that common accounting practices treat software inventory. Large businesses spend a lot of time and effort keeping accurate inventory checks on two hundred dollar Dell desktop machines, and twenty dollar keyboards and mice, but virtually no effort in keeping an accounting record on the software that they may have spent many tens of millions of dollars developing. For as long as the value of software gets accounted for under "intangibles" we will have this problem. We need to itemize every function and class in the organization's accounts, associate a cost with each, as well as the benefits that are realized by the organization. Again, this needs to happen automatically, and without any manual intervention from the individual developers. As before, if anybody knows of any organization who is doing this, please please let me know.

Moving our focus to the second approach, how do we match developers with the opportunities available at a particular organization; how do we let them know where they can provide value? How can we open organizations up so that outsiders can contribute to their success? This is a much harder problem which may invite more radical solutions. Certainly nothing reasonable springs to my mind at the moment.

Anyway, I have spent too long ruminating, and I need to get back to work. If anybody has any ideas on how this line of enquiry might be extended, or even if you think that I am wasting my (and everybody else's) time:- please, please say so in the comments.

This line of thought builds upon and extends some of my previous thinking on the subject.

Do Not Fear Conflict

The greatest creative partnerships that I have experienced have always involved an element of friction.

Not only are heated technical debates not to be feared, they are to be welcomed as they are one of the best ways (if not THE best way) of improving the design of a product.

...

(although there are limits, of course)

:-)

Why I like "working in trunk"

It forces you to:
* Commit Frequently.
* Communicate with your colleagues.
- Both of which are virtues in their own right.

Natural Semantics

Whilst doing some background reading on AMQP, I came across the following little tidbit. (from http://www.imatix.com/articles:necessary-changes-in-amqp)


"There is a concept I call "natural semantics". These are simple patterns that Just Work. Often they take a little while to appreciate. Natural semantics are like mathematical truths, they exist objectively, and independently of particular technologies or viewpoints. They are precious things. The AMQ exchange-binding-queue model is a natural semantic. A good designer should always search for these natural semantics, and then enshrine them in such ways as to make them inviolable and inevitable and trivial to use."

The notion of natural semantics resonates with my experiences as a developer:- which makes me happy, because something that I for so long felt to be true but could not articulate, now has a name.

Character Flaws.

My Modus Operandi is to identify my own serious character flaws, then exploit them for fun and profit!

I have mild obsessive-compulsive tendencies. I exploit this to help organize and categorize source documents & data, and to improve the discipline and quality of my work.

I also have mild bipolar tendencies. I use my mania to get lots of work done, then my depression to fix all of the bugs that I introduced during my manic coding sessions: My own personal (psychological) Carnot cycle.


We are all flawed in some way. But sometimes we can find real strength in apparent weakness.

Array languages rock

As a long-time MATLAB developer, I may be biased, but: Array languages rock!

Why?

They allow the developer to express simple numerical relationships in an incredibly declarative manner. In this, they go a further than typical functional languages, by allowing you to discard the use of "zip", "map", list comprehension constructs etc.. for a variety of simple cases.

procedural < functional < array

Indeed, when one gets to the point where much of the standard library of an array language is able to accept and return collections, the developer becomes able (in many cases) to purge flow control from great swathes of the program.

As a result, one ends up with a program where control flow proceeds linearly, from top to bottom, greatly reducing the mental gymnastics that are needed to understand what the algorithm is doing, greatly improving readability, and dramatically improving productivity.

Sure, the run-time performance of MATLAB often sucks, but there is no real reason why array languages cannot be highly performant. This is why I am feeling excited about the emergence of languages like Julia, that attempt to be the best of both worlds.

These new, emerging languages excite me:

• Go: For it's fast compile, opinionated toolset, close-to-the-metal feel, approach to object-orientation and its support for concurrency.
• Julia: For attempting to make array programming performant.
• Cobra: For bringing design-by-contract back into vogue.

My ideal software tool:

Designed primarily to allow academics, data scientists, mathematicians, financial engineers and other domain experts to prototype ideas quickly, it will feel a lot like MATLAB with a highly interactive development environment, excellent plotting and visualisation tools, and a large standard library filled with functions that accept and return arrays.

It will allow these prototypes to be quickly and easily brought up to production quality standards by supporting (out of the box) quality control mechanisms like strong typing, design by contract, assertions, built in static analysis, test coverage tools & continuous testing tools.

Like Go, it will compile very quickly to highly performant native binaries, support concurrency and multiprocessing out-of-the box, and be able to deploy to future production environments, which are likely to include highly heterogenous "cloud" environments as well as multicore CPU + GPU embedded systems.

Why declarative?


One of the challenges of software development is to try to identify those aspects of the system that are likely to remain constant, and those areas that are likely to change in future. These aspects are rarely specified, rarely understood, and rarely do we (as developers) get the architecture of the software right (in this respect).

Having done this, we try to make the transient, changeable aspects of the system as declarative as possible; perhaps going as far as to express them as statements declared in a configuration file.

Having done a bit of Prolog programming in the dim and distant past, my intuition is that trying to make everything declarative is a mistake; one ends up tying oneself into knots. The mental gymnastics simply are not worth it. However, splitting the program into declarative-and-non-declarative parts seems reasonable.

In fact, the idea that a a multi-paradigm approach can be used, with some parts of the system expressed in OO terms, some in functional terms, and some in declarative terms, seems to be gaining in acceptance and popularity.

Note: (See also Monads and some examples as another way of thinking about this)

On the importance of over-communication.

In a crisis, do not seek to control information flow. Instead, over-communicate. Have confidence in people's intelligence and tolerance. Confusion is a natural part of learning, and disagreement and debate are a natural part of team decision making.

You will either get through the crisis as a team, or you will end up postponing problems for later, and sewing seeds of distrust to trip you up in future.

I would rather work in an argumentative and critical but open and honest organization than in an organization that papers over disagreement and misunderstanding with dishonesty and spin.

Motivation trumps Method

We can either tackle the problem at hand by the most direct route possible, or we can try to build tools & abstractions to help future development efforts become faster and more efficient.

The first end of the spectrum corresponds largely to the notion of a "product-oriented" development attitude, and the second end to the notion of a "capability-oriented" development attitude.


The position on the spectrum that we select is driven both by immediate constraints and long term goals, and how we balance one against the other. Living exclusively at either extreme is generally harmful and counterproductive.

This dichotomy is very loosely related to that which contrasts the predictive/CMMI/waterfall development approach with the reactive/agile approach to development in that the reactive/agile approach is very well served by a toolset that has come into existence through the efforts of a capability-oriented team.

However, the predictive-vs-reactive dichotomy is really about managing risk, whereas the product-vs-capability dichotomy is really about amortizing costs.

There is no “right” answer, it all depends upon the situation that we are in and the culture of the organization(s) with which we are operating. One thing that is really important however, is motivation. Software is all about detail, and detail is all about attention, and attention is all about motivation, and motivation is all about the team. Maintaining motivation and enthusiasm in the team just about trumps everything else.

We should do what we enjoy, because we will be so much more productive that any other consideration will be secondary.

Development is Exploration.

People talk about software being grown, not built. Here is another perspective:

Software development is an act of exploration, uncovering a set of truths or relationships, whether perpetual or contingent, (and therefore transient), together with an abstract machine to turn that set of truths into a system that can interact with the problem and thereby do useful work.

Multi-project integration.

This is a response to the question: "How to maintain same code fragments on multiple projects":

The DRY principle states that there should be a single authoritative source of truth. This dictates the use of a single *source* repository for all program logic, with no duplication; files organized to promote sharing and reuse of documents.

The pragmatic requirements of communication in a distributed, multi-team environment suggests that there should be multiple independent repositories, one for each project or collaboration.

This presents a problem, as the requirements (at first glance) would seem to be contradictory. However, it is possible to use scripting & automation to smooth over this awkward fact, and to create two separate but consistent interfaces to the body of work.

The one unified repository acts as the single authoritative source of truth regarding all of the projects. The build process for each project copies all the files used by that project (and only those files) into an intermediate location, then build from that intermediate location. (Unison or some similar tool can be used to move deltas instead of whole files).

These intermediate locations can be used as local working copies for the set of secondary, derived, or downstream repositories. Post-commit hook scripts on the authoritative repository will update all of the intermediate locations, and, for each in turn, check if it has changed, and make the same commit to the corresponding secondary repository if a change is detected.

Likewise, post-commit hook scripts on the intermediate repositories can be implemented to update files in the central repository, enabling inbound changes from collaborators to propagate to the authoritative repository.

This way, the multiple secondary repositories are kept in sync with the single authoritative source repository, and the build process ensures that the secondary repositories contain all of the (possibly shared) documents and other files that are needed for the build to succeed. Finally, and most importantly, the development and build process ensures that files may be edited in one place and one place only, and that all copies are kept consistent and up to date.

This solution is tool agnostic, it should be possible to use Svn, Hg or Git as repositories, and bash or batch to implement the synchronizing logic. Something like Unison may be useful to synch the downstream repositories with the authoritative repository,

Capability vs Product Orientation

In response to: http://iamwilchung.wordpress.com/2012/09/17/make-tools-for-yourself-and-others-around-you/

I have always thought of the creation of internal tools and libraries in terms of the following dichotomy:

"capability-oriented" vs "product-oriented"

The "capability-oriented" approach seeks to maximise the capabilities of the organization over the long term, opening up opportunities for cost amortization over disparate products. By contrast, the "product-oriented" approach is "lean" in the sense that it seeks to minimize work not absolutely necessary for the current product.

The "product-oriented" approach seeks a local optimum, and is prevalent in industries with heavyweight cost controls and minimal opportunities for investment (Defense, for example), whereas the "capability-oriented" approach seeks a global optimum, and is only possible in situations where organizational leadership is capable and willing to get involved with detailed technical planning.

As with all dichotomies, organizations must seek a compromise position that varies from product to product and project to project.

A sweet-spot that I have frequently found: Third-party libraries are necessarily written in a very general manner, capable of being used in a wide variety of ways. A particular organization generally only uses a small subset of the features in a particular third-party library. As a result, an internal library that acts as a thin wrapper around an external/third-party library can help encourage and enforce a consistent approach within an organization, helping to create an institutional culture and improving interoperability between teams whilst reducing learning barriers for new team members.

Automated micro-outsourcing to reduce design costs by supporting hyper-specialization.

I have been re-reading Rodney Brooks "No Silver Bullet" paper. It is fantastic stuff, and I am glad that I came back to it (It has been several years since I last looked at it).

The paper is showing it age just a teensy tiny little bit (His enthusiasm for Ada and OOP seems slightly too strong in retrospect). With the benefit of experience, we now have a bit better understanding of how people react to complex tools, and as a result now tend to favor tools that encourage (or enforce) simplicity and consistency.

Anyway, that is not what I want to write about. The bit that particularly popped out at me this morning is the section on "Buy vs build". This chimes well with my belief that the only effective way to control software costs is through amortization. However, I had always limited my thinking to amortization within an organization, and so focussed my efforts entirely on how one could organize the source code repository to encourage reuse of components & libraries from within the organization, or engineer tools to help encourage reuse. (For example, an always-on background structural-similarity search to identify and bubble-up structurally similar source documents).

However, this does not address the concerns that one of my other closely-held beliefs implies. I believe that much of the value that is derived from development activities comes from the learning and resulting improvements in knowledge and understanding of the individual developers. This knowledge and expertise remains with the individual, and is a good part of the 10x performance differences that many people observe in software developers.

So, how about using the same sort of background structural-similarity search to identify potential collaborators / consultants whose expertise fits in well with the system that you are trying to create. In the same way that an advanced IDE would make the suggestion: "perhaps you want to buy this library to help you solve this problem here...", you could imagine the same IDE making the suggestion "perhaps you want to hire this developer to help you here..."

Or maybe it is just my recent involvement in the advertising industry that makes my mind turn so quickly to commercial transactions. :-)

Obsessive Perfectionism: Success and Failure.

Perfectionism is both a blessing and a curse - the obsessive pursuit of perfection is necessary for the creation of truly great works, but the obsessive pursuit of perfection is responsible for the most abject and catastropic failures