Wednesday 29 August 2012

Demographics and the Devil

I wonder how much specialization is driven by demographic shifts.

The number of educated, intellectually involved people is growing at a rapid rate. At the same time, the ability for people to communicate and share their knowledge is also increasing dramatically. All of this leads to a large quantitative shift in the amount of information and knowledge being generated and consumed.

Quantity being a quality all of its own, such large quantitative changes are bound to be accompanied by qualitative shift in the way that we use that information; the way that we contribute to the debate; and the way that we do our jobs.Increased specialization is an obvious consequence of this (ongoing) demographic change, but not the only one. What it means to be a specialist (or even generalist) has changed also.

I am convinced is that there is an increasing need for generalists and management professionals to be able to reason about issues at a level of fine detail that was not previously required or possible. Taking a broad-brush, 10,000 ft high approach, whilst still necessary, is no longer sufficient - we need to collectively sweat the details, because that, they say, is where the devil lies.

Pride and Culture

Just like the Toyota TPS, I have been influenced a lot by what W E Deming wrote, although he was concerned with the issues surrounding the management of industrial production rather than the management of design shops, so there is perhaps some scope for mismatch between our requirements and his philosophy. 

The thing that really resonates with me is the idea that pride in your work, and pride in your organization are both critically important motivational factors, and the idea that the pursuit of quality can be used to both improve productivity and the pleasure that work provides. 

This also chimes with this (world class) investment management advice: Invest in companies and organizations with a "soul" - with a strong, motivating culture that really makes people believe in, and belong to, an organization, over those that are hollow and impersonal. 

On a personal level, I want to take pride both in the organization for which I work, and the designs that I produce; I want to buy in to the culture and to believe in the mission. Moreover, I want to work for a company that has the homeostatic mechanisms in place to support that culture over the long term, and the profitability to sustain it. 

If this seems like a lot to ask, it is only because of the sad state of the world as it is right now. With a bit of organization, effort, and leadership, this sort of result is perfectly achievable, and with a bit of automation (yaaay technology!) it is perfectly reproducible.

Tuesday 28 August 2012

Breaking the liberal/conservative axis.

All models are wrong, but some of them are useful. 

The liberal/conservative dichotomy is a useful model, but the one thing that is more true of software than anything else is it's nonlinearity (and malleability).

If constructing an exception to a rule is possible, then we can automate it. If we can automate it, we can make it easy, and if we can make it easy, we can make it mainstream. In addition to describing the way things are, we also need to describe the way things should be.

Then make it happen.

Friday 24 August 2012

Is YAGNI always a good thing?

I am not sure that YAGNI really is good practice in all situations.

A minimalistic "only do what we need to right now" approach is appropriate under a wide range of situations, but really dramatic agility and performance comes about when you have developed a set of capabilities that are well-matched to emerging requirements.

I admit that this partly comes down to luck, but it is a powerful and uplifting experience when it happens.

So, typically, the business hands us a requirements spec, and we do some analysis to plan out how to tackle the project; a work breakdown is done, tasks are created, and we work away at those tasks to make a product that meets the spec. So far so conventional. All stakeholders are focussed on the product, and really do not think beyond the immediate project, because to do so would be wasteful, right? As software engineers, we have been told that modularity and reuse are good things, so we over-engineer components according to imagined modes of variation, but because everything is being done in the context of this one project, nothing ever stands any real chance of being reused anyway, because the organizational communication channels that are required to support that reuse simply do not exist. As a result, the software ends up being overcomplicated and expensive.

There are two (not necessarily exclusive) ways of tackling this problem. The first is pure YAGNI: "do not be precious". Your work is probably going to get thrown away anyway, so do the simplest thing that works for the immediate problem. This is, I think, a healthy way of thinking about problems, but it only gets you so far. The second is capability-building, which says: OK, while I am working away on this problem, let me make a bunch of libraries so that the next time I come to the same problem, I can move faster. Each of the libraries can start out simple, but as I move from project to project, product to product, my capabilities will improve and increase over time.

This is, in a sense, the opposite of YAGNI, but has helped me out on more than one occasion, when requirements come in that fit in very well with the capabilities at my disposal. On one particularly memorable occasion, I was able to turn around a small piece of functionality in about 45 minutes that one of our partner organizations had been working on for over a month. This simply because I had tools at my disposal that were a very good fit to the problem at hand.

My point is this; unless we build real capabilities (software libraries) that provide strategic differentiation and competitive advantage to the organizations for which we work, we will be forever stuck in a cottage industry, reinventing the wheel over and over and over and over again.

So:- whilst for many bike-shedding things it might be true that You Ain't Gonna Need It, if you choose carefully, maybe for some things You Are Gonna Need It, and when you do, you will be grateful that you have a capability to hand that enables you to spin on a sixpence and get product out the door FAST.

Agile: The 45 minute sprint

How long should a sprint be? 2 months? 2 weeks? Let us go right to one extreme. How about 45 minutes? Here is some homework for you: What tools would you need, what work would you have to do to make it possible to do an entire sprint in only 45 minutes?

Foundations of Agility

Being lightweight and low structure does not necessarily have anything to do with being Agile, although instances of the two characteristics do tend to be correlated in practice.

It is my strong belief that a humble approach to risk management is the distinguishing characteristic of agile processes.

Think about it like this: The term "Agile" deliberately evokes responsive, direction-changing imagery.  Indeed, all four Values in the agile manifesto either talk about combating impediments to change, or supporting mechanisms for change. The only thing that is not explicitly stated is the reason why responsiveness and changeability is important, although it is (I think) pretty obvious.

We do not know the future, so we have to be prepared for the unexpected. We cannot plan for everything, but that should not stop us from doing the best that we can. It is not "be prepared" in the boy-scout sense of being prepared for the worst, but rather a more sophisticated combination of anticipating change and removing current impediments to possible future maneuvers.

Agile is the technological equivalent of maneuver warfare, excepting that the objective is to merely keep up with our interlocutors and partners changing needs, rather than to overtake them and seize the initiative.

Processes and tools that support and facilitate change can (and sometimes do) have tremendous utility. It is just that (in the past) processes have tended to create harmful institutional inertia. Likewise, documentation traditionally has proven difficult to keep up-to-date in the face of changing systems and requirements, but this is not necessarily the case with documentation that is generated automatically from source comments. Similarly, contracts can be handled in an agile manner, providing that there is a well established, fast and easy way to handle change notices. Finally, as Ike Eisenhower said, "Plans are worthless, but planning is everything". If the presence of a plan imparts  inertia to a project, or if the process of planning takes anything other than an insignificant amount of time and resource away from execution, then it is clear that the activity is harmful; but to do away with planning altogether is folly.

Wednesday 22 August 2012

What is Agile?

Agile is humility. You do not have a crystal ball:- the unexpected can, (and frequently does) happen in this most uncertain of worlds.

Agile is old-fashioned, conservative decision making. Do not take risks that you can avoid.

Agile is the art of not committing to a course of action when you do not know what the future will bring. Postpone risky decisions, wait until you have more information before committing time and resources.

Agile is risk-aware prioritization. Schedule simple, low-risk activities that give you feedback and information before complex, high-risk activities that could go wrong.

Agile acknowledges the limits of prediction, and seeks to improve responsiveness and the capability to react quickly to external changes.

Agile is neither lightweight nor easy. You can be bad at being Agile just as you can be bad at any other activity.

Technology can help. The ability to respond quickly to external changes is an organizational capacity that must be built and nourished, just like any other capability that you want your organization to have.

You cannot plan change, but you can plan for change.

Monday 20 August 2012


We can increase transparency and prosperity through the disintermediation of informational and economic transactions:- I would rather trust a dumb algorithm than a conniving human.

Friday 17 August 2012

Sensor system (machine vision) development team organization and management.

A note on the organization of teams and processes for the development of automotive industry machine vision systems.

Sensor systems (particularly machine vision systems) operating in unconstrained environments pose significant design and engineering challenges. It is, however, possible to design these, and not necessary at exorbitant cost, either, but to do so means adopting some slightly unusual and highly, highly disciplined engineering practices.

First of all, the fact that the sensor system is to operate in an unconstrained environment with a very large number of free variables necessarily means that there will be lots of "black swan" / "long-tail" type problems that are not known in advance: the bird that flies in formation with the car for a short while, the van carrying a mirror on it's side; the torrential downpour limiting visibility; the train moving alongside the road; that sort of thing.

As a consequence, you need moderate-to-large volumes of pathological test data, together with the associated machinery and automation that will allow candidate system configurations to pit their wits against the worst that the test system can throw at them. 

The problem also requires that we get the basics right early on, so that the team can focus on the real problem without tripping themselves up with regressions and other problems (all to easy to do). Whilst solutions can be relatively simple (indeed they should be as simple as possible), the amount of work required to cover the basics is large, and no one person can make sufficient progress on their own. To succeed requires both teamwork and a level of focus, organization and automation that is not commonly found in an academic environment. It also requires that we master the tricky balance between discipline at the team level, and freedom to innovate at the individual level.

It is important that we pay attention to the organization of the system. We need to think about it for a little bit, and pick a good one, but move quickly and avoid bike-shedding. A wide range of organizations can be made to work. (Here is one I like). The organization of the system must be reflected in the organization of the team, and the organization of the filing system. The filing system would (ideally) be put in the version control system / repository, and the log used as a management tool to monitor progress. The important thing is that the organization is agreed-upon, and reflects the organization of the capabilities or systems being developed.

Having picked a good organizational structure, the next thing to focus on is total quality through test automation. 

Weekly meetings, daily standups, documentation & wikis etc... are all good ideas, but they are much less important than AUTOMATION, which IS ABSOLUTELY CRITICAL to success. This is why: humans are liars. We cannot help it. When we talk to other people, especially our superiors, we try to present ourselves in the best light. We spin the situation and use weasel words to make ourselves look better than we are. Honesty is crucially important for the team to make decisions. The only way to achieve honesty is to take reporting out of the hands of human beings.

You will need management buy-in for this. Use financial audit as an analogy. We audit the company books so investors can have some degree of faith in them, so why not also audit design progress?

First of all, you will need some hardware. Start with one server, running the repository and a simple post-commit hook script. (Ideally also stored in the repository) :-). Continuous Integration software like Hudson is not necessary (and perhaps not desirable). Better results can often be achieved with a Python, MATLAB or Go script (or some such) to run tests and measure performance.

The post commit hook script is the main way that we are going to feed-back performance to managers, so get it to save results to a file, and have a daily process summarize the contents and email the summary once a day to the management team. Do this even (especially) if the results file says "No progress measured today".

Initially, the management team might not want to read it, but get them to at least pretend that they do, so that there is at least the semblance of oversight. If they want more information, get them to ask for it to be included in the daily development status email. Try to encourage the view that "if it is not in the automated report, it is not done"


Bringing progress reporting and test automation together I think could be a powerful tool to help support and encourage transparent and professional development.

Manual reporting and communication attempts, such as documentation, manual progress reports, wikis and meetings have a smaller impact on success than tooling and automation.

It is about building a design production line for the organization.

Thursday 2 August 2012

Beware of false comforts

Ancient Sparta had no city walls:- I do not know why, but I like to think that it was a deliberate policy: A rejection of false comforts.

Static defensive positions can be useful, but only when coupled with a mobile army to disrupt the enemy's plans. On their own, all they can possibly do is postpone the inevitable. Of course, human nature being what it is, the appearance of safety that the city walls create soon leads to the withering of the real guarantor of safety: the army.

In software development too, we are surrounded by false comforts. We expend tremendous effort delaying integration and deployment, not touching legacy code "because it works" and so on, We should think hard about the procedures and tools that we use. Which ones give the appearance of safety, but in actuality provide only a false sense of comfort, and lead to a withering and neglect of the skills and tools that we really need to rely upon in a crisis?

Software development best practice for MATLAB users

Here is a response to a question on Stack Overflow: "Who organizes your MATLAB code?", that I posted back in 2011. It was nice to go back to this, and to realize that some of my recent rabid rants were, at one point in history, actually grounded in reality.


I have found myself responsible for software development best practice amongst groups of MATLAB users on more than one occasion.

MATLAB users are not normally software engineers, but rather technical specialists from some other discipline, be it finance, mathematics, science or engineering. These technical specialists are often extremely valuable to the organisation, and bring significant skill and experience within their own domain of expertise.

Since their focus is on solving problems in their own particular domain, they quite rightly neither have the time nor the natural inclination to concern themselves with software development best practices. Many may well consider "software engineer" to be a derogatory term. :-)

(In fact, even thinking of MATLAB as a programming language can be somewhat unhelpful; I consider it to be primarily a data analysis & prototyping environment, competing more against Excel+VBA rather than C and C++).

I believe that tact, diplomacy and stamina are required when introducing software engineering best practices to MATLAB users; I feel that you have to entice people into a more organised way of working rather than forcing them into it. Deploying plenty of enthusiasm and evangelism also helps, but I do not think that one can expect the level of buy-in that you would get from a professional programming team. Conflict within the team is definitely counterproductive, and can lead to people digging their heels in. I do not believe it advisable to create a "code quality police" enforcer unless the vast majority of the team buys-in to the idea.

In a team of typical MATLAB users, this is unlikely.

Perhaps the most important factor in promoting cultural change is to keep the level of engagement high over an extended time period: If you give up, people will quickly revert to follow the path of least resistance.

Here are some practical ideas:

If it does not already exist, set up the source file repository and organise it so that the intent to re-use software is manifest in it's structure. Try to keep folders for cross-cutting concerns at a shallower level in the source tree than folders for specific "products". Have a top-level libraries folder, and try to discourage per-user folders. The structure of the repository needs to have a rationale, and to be documented.

I have also found it helpful to keep the use of the repository as simple as possible and to discourage the use of branching and merging. I have generally used SVN+TortoiseSVN in the past, which most people get used to fairly quickly after a little bit of hand-holding.

I have found that sufficiently useful & easy-to-understand libraries can be very effective at enticing your colleagues into using the repository on a regular basis. In particular, data-file-reading libraries can be particularly effective at this, especially if there is no other easy way to import a dataset of interest into MATLAB. Visualisation libraries can also be effective, as the presence of pretty graphics can add a "buzz" that most APIs lack.

Coding Standards:
On more than one occasion I have worked with (otherwise highly intelligent and capable) engineers and mathematicians who appear to have inherited their programming style from studying "Numerical Recipes in C", and therefore believe that single-letter variables are de rigueur, and that comments and vertical whitespace are strictly optional. It can be hard to change old habits, but it can be done.

If people are modifying existing functions or classes, they will tend to copy the style that they find there. It is therefore important to make sure that source files that you commit to the repository are shining examples of neatness, full of helpful documentation, comments and meaningful variable names. This is particularly important if your colleagues will be extending or modifying your source files. Your colleagues will have a higher chance of picking up good habits from your source files if your make demo applications to illustrate how to use your libraries.

Development Methodologies:
It is harder to encourage people to follow a particular development methodology than it is to get them to use a repository and to improve their coding style; Methodologies like Scrum presuppose a highly social, highly interactive way of working. Teams of MATLAB users are often teams of experts, who are used to (and expect to continue) working alone for extended periods of time on difficult problems.

Apart from daily stand-up meetings, I have had little success in encouraging the use of "Agile" methodologies in teams of MATLAB users; most people just do not "get" the ideas behind test-driven development, development automation & continuous integration. In particular, the highly structured interaction with the "business" that Scrum espouses is a difficult concept to generate interest in, even though some of the more serious problems that I have experienced in various organisations could have been mitigated with a little bit of organisation in the lines of communcation.

Most of what constitutes "good programming practice" is simply a matter of good administration & organisation. It might be helpful to consider framing solutions as "administrative" and "managerial" in nature, rather than as "software engineering best practice".